• FilterEqualTypeInput - Input type is set to Yes/No, Select, or Multiple select. Your filter can contain the eq or in attribute. Use the eq attribute to exactly match the specified string. Use the in attribute to filter on a comma-separated list of values.
• FilterMatchTypeInput - Input type is set to Text Field or Text Area. Your filter must contain the match attribute, which will return all items that partially fuzzy match the specified string.
• FilterRangeTypeInput - Input type is set to Price or Date. Your filter can contain one or both of the to and from attributes, which serve to provide a range of values to filter on.

If, for any reason, you require the text attribute to support the "eq" filter, you can follow the steps below:

Text attributes are using match attributes OOTB. But Magento leaves open door to change this behavior by injecting the custom attribute using the below classes. This behaviour we can see for sku,url_key etc.

• Magento\CatalogGraphQl\Model\Config\FilterAttributeReader - inject style_code to private $exactMatchAttributes = ['sku']; allows to change graphQL queries to use eq attribute
• Magento\CatalogGraphQl\Plugin\Search\Request\ConfigReader - inject style_code to “exactMatchAttribute“ change request guery for Elasticsearch.

Changel Graphql query to use eq filter

Update the di.xml file to inject the custom attribute to the variable exactMatchAttributes to the class Magento\CatalogGraphQl\Model\Config\FilterAttributeReader

<type name="Magento\CatalogGraphQl\Model\Config\FilterAttributeReader">
    <arguments>
        <argument name="exactMatchAttributes" xsi:type="array">
            <item name="custom_attribute" xsi:type="string">style_code</item>
        </argument>
    </arguments>
</type>

Change request query for Elasticsearch

Update the di.xml file to inject the custom attribute to the variable exactMatchAttributes in the class Magento\CatalogGraphQl\Plugin\Search\Request\ConfigReader

<type name="Magento\CatalogGraphQl\Plugin\Search\Request\ConfigReader">
    <arguments>
         <argument name="exactMatchAttributes" xsi:type="array">
                <item name="style_code" xsi:type="string">style_code</item>
         </argument>
    </arguments>
</type>

Verify Elasticsearch Keyword mapping

If an attribute needs to be searched with eq filter the attribute should have the keyword index mapping in the ElasticSearch.

This can be vieified iwth the below curl comand

curl --location --request GET 'http://localhost:9200/*/_mapping/field/url_key'
{
  "mappings": {
    "url_key": {
      "full_name": "url_key",
      "mapping": {
        "url_key": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          },
          "copy_to": [
            "_search"
          ]
        }
      }
    }
  }
}

Generally for all the attributes where the backend_type is "text", The keword mapping would not be set by default.

Here the attribute "test_text" will not have a keword mapping in elasticsearch so we cannot use "eq" filter.

To know more preciesly on where keword mapping will not be added refer the below file.

\Magento\Elasticsearch\Model\Adapter\FieldMapper\Product\FieldProvider\StaticField::getField

But you can still make this type of attribute to support keyword mapping by updating the attribute index mapping. The index mapping for an attribute this set in the below method. So we can crete the afterPlugin for this method and have the mapping updated.

\Magento\Elasticsearch\Model\Adapter\FieldMapper\Product\FieldProvider\StaticField::getFields

The post How to Enable 'eq' Filter GraphQL Search for an Custom Attribute of Type Text in Magento 2 first appeared on Moses Dinakaran.

To get the Indices Details

List all indices

curl --location --request GET 'http://localhost:9200/_cat/indices?v'

Sample Output

health status index                   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   magento2_product_14_v18 WfscY4ghRRCJXCwQnXfqiA   1   1          7            0     17.3kb         17.3kb
yellow open   magento2_product_38_v18 zk6DJO9YRHaeGxk5XLKQOA   1   1         20            0     46.9kb         46.9kb

List all aliases

curl --location --request GET 'http://localhost:9200/_cat/aliases/?v'

Delete All Indices

curl -X DELETE "localhost:9200/*?pretty"

Sample Output

alias               index                   filter routing.index routing.search is_write_index
magento2_product_27 magento2_product_27_v18 -      -             -              -
commerce_product_7  commerce_product_7_v9   -      -             -              -

To Fetch the Data directly from Elasticsearch

Get Records Count

To fetch the total no of records in the indices, Replace * with index name to get the total no of records for a specific index

curl --location --request GET 'http://localhost:9200/*/_count'

Sample Output

{
    "count": 33,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    }
}

Get all the Records

To fetch all the records from all the indices, replace * with the index name to fetch all the records of a specific index

curl --location --request GET 'http://localhost:9200/*/_search' \
--header 'Content-Type: application/json' \
--data-raw '{
   "track_total_hits" : true
}'

Sample Output

{
    "took": 13,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 33,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "magento2_product_1_v26",
                "_type": "document",
                "_id": "340",
                "_score": 1.0,
                "_source": {
                    "store_id": "1",
                    "sku": "0986361001",
                    "updated_at": "2022-09-30T07:13:01+00:00",
                    "status": 1,
				}
            },
			{
                "_index": "magento2_product_1_v26",
                "_type": "document",
                "_id": "1714",
                "_score": 1.0,
                "_source": {
                    "store_id": "1",
                    "sku": "KKALSHAYA-001",
                    "updated_at": "2022-06-01T06:29:42+00:00",
                    "status": 1,
				}
			}
        ]
    }
}

Get all the Records with limit

To fetch from a specific index, replace * with the index name

curl --location --request GET 'http://localhost:9200/*/_search' \
--header 'Content-Type: application/json' \
--data-raw '{
  "query": {
    "match_all": {}
  },
  "_source": ["sku"],
  "from" : 1,
  "size" : 3
}'

Sample Output

{
    "took": 20,
    "timed_out": false,
    "_shards": {
        "total": 20,
        "successful": 20,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 207,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "commerce_product_1_v9",
                "_type": "document",
                "_id": "2",
                "_score": 1.0,
                "_source": {
                    "sku": "24-MB04"
                }
            },
            {
                "_index": "commerce_product_1_v9",
                "_type": "document",
                "_id": "3",
                "_score": 1.0,
                "_source": {
                    "sku": "24-MB03"
                }
            },
            {
                "_index": "commerce_product_1_v9",
                "_type": "document",
                "_id": "430",
                "_score": 1.0,
                "_source": {
                    "sku": "MJ12"
                }
            }
        ]
    }
}

To get a specific document

You should know the document ID. Here 1726 is the document id and magento2_product_1_v26 is the index name

curl --location --request GET 'http://localhost:9200/magento2_product_1_v26/_doc/1726'

Sample Output

{
    "_index": "magento2_product_1_v26",
    "_type": "_doc",
    "_id": "1726",
    "_version": 1,
    "_seq_no": 5,
    "_primary_term": 1,
    "found": true,
    "_source": {
        "store_id": "1",
        "sku": "giftcard_topup",
        "updated_at": "2022-05-06T11:23:10+00:00",
        "status": 1,
        "status_value": "Enabled",
        "visibility": 4,
        "ship_to_store": 1,
        "ship_to_store_value": "Enabled",
        "reserve_and_collect": 1,
        "reserve_and_collect_value": "Enabled",
        "is_buyable": 1,
        "tax_class_id": 0,
        "tax_class_id_value": "None",
        "is_sale": 0,
        "is_new": 1,
        "same_day_delivery": 0,
        "same_day_delivery_value": "No",
        "express_delivery": 0,
        "express_delivery_value": "No",
        "name": "topup",
        "url_key": "buy-topup",
        "product_activation_date": "2022-03-04T00:00:00+00:00",
        "category_ids": [
            2,
            3736
        ],
        "position_category_2": "10003",
        "name_category_2": "category1",
        "position_category_3736": "3",
        "name_category_3736": "HPS Gift Card",
        "price_0_1": "10.000000",
        "price_1_1": "10.000000"
    }
}

Fetch Data based on SKU

To fetch from a specific index, replace * with the index name

curl --location --request GET 'http://localhost:9200/*/_search' \
--header 'Content-Type: application/json' \
--data-raw '{
  "query": {
      "match": {"sku": "24-MB04"}
    }, "_source": ["name","sku","status","url_key","visibility","store_id","updated_at"]
}'

Sample Output

{
    "took": 7,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1.2039728,
        "hits": [
            {
                "_index": "commerce_product_1_v9",
                "_type": "document",
                "_id": "2",
                "_score": 1.2039728,
                "_source": {
                    "store_id": "1",
                    "visibility": 4,
                    "name": "strive shoulder pack",
                    "sku": "24-MB04",
                    "url_key": "strive-shoulder-pack",
                    "status": 1
                }
            }
        ]
    }
}

Get General Info on Elasticsearch Instance

To Get all the settings for all the indices.

Replace * with the index name to get the setting for a specific index.

curl --location --request GET 'http://localhost:9200/*/_settings'

Sample Output

{
    "magento2_product_41_v18": {
        "settings": {
            "index": {
                "routing": {
                    "allocation": {
                        "include": {
                            "_tier_preference": "data_content"
                        }
                    }
                },
                "mapping": {
                    "total_fields": {
                        "limit": "101938"
                    }
                },
                "number_of_shards": "1",
                "provided_name": "magento2_product_41_v18",
                "max_result_window": "100000",
                "creation_date": "1665213507317",
                "analysis": {
                    "filter": {
                        "default_stemmer": {
                            "type": "stemmer",
                            "language": "english"
                        },
                        "unique_stem": {
                            "type": "unique",
                            "only_on_same_position": "true"
                        }
                    },
                    "char_filter": {
                        "default_char_filter": {
                            "type": "html_strip"
                        }
                    },
                    "analyzer": {
                        "prefix_search": {
                            "filter": [
                                "lowercase",
                                "keyword_repeat"
                            ],
                            "char_filter": [
                                "default_char_filter"
                            ],
                            "type": "custom",
                            "tokenizer": "default_tokenizer"
                        },
                        "default": {
                            "filter": [
                                "lowercase",
                                "keyword_repeat",
                                "default_stemmer",
                                "unique_stem"
                            ],
                            "char_filter": [
                                "default_char_filter"
                            ],
                            "type": "custom",
                            "tokenizer": "default_tokenizer"
                        },
                        "sku_prefix_search": {
                            "filter": [
                                "lowercase",
                                "keyword_repeat"
                            ],
                            "type": "custom",
                            "tokenizer": "keyword"
                        },
                        "sku": {
                            "filter": [
                                "lowercase",
                                "keyword_repeat",
                                "default_stemmer",
                                "unique_stem"
                            ],
                            "type": "custom",
                            "tokenizer": "keyword"
                        }
                    },
                    "tokenizer": {
                        "default_tokenizer": {
                            "type": "standard"
                        }
                    }
                },
                "number_of_replicas": "1",
                "uuid": "3hoUumFITJi9cSnhwNDXog",
                "version": {
                    "created": "7170499"
                }
            }
        }
    }
}

Get All Index Mapping

Here commerce_product_1 is the index name, replace it with * to get the mapping for all the indices

curl --location --request GET 'http://localhost:9200/commerce_product_1/_mapping'

Get Mapping for a Specific Field

To get the mapping details of a specific field. In the below query "barcode" is the field name. Replace * with index name to get for specific index

curl --location --request GET 'http://localhost:9200/*/_mapping/field/barcode'

Sample Response

{
    "magento2_product_38_v28": {
        "mappings": {
            "barcode": {
                "full_name": "barcode",
                "mapping": {
                    "aims_barcode": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword"
                            }
                        },
                        "copy_to": [
                            "_search"
                        ]
                    }
                }
            }
        }
    },

Get Index Mapping Fields Count

To get the mapping fields count of all the indices. Replace * with index name to get for specific index

curl --location --request GET 'http://localhost:9200/*/_settings/index.mapping.total_fields*'

Sample Response

{
    "commerce_product_5_v9": {
        "settings": {
            "index": {
                "mapping": {
                    "total_fields": {
                        "limit": "1267"
                    }
                }
            }
        }
    },
    "commerce_product_1_v9": {
        "settings": {
            "index": {
                "mapping": {
                    "total_fields": {
                        "limit": "1267"
                    }
                }
            }
        }
    }
}

To get the Heap Size

curl --location --request GET 'http://localhost:9200/_cat/nodes?h=heap*&v'

Sample Response

heap.current heap.percent heap.max
       1.2gb           44    2.8gb

To get the cluster Health

curl --location --request GET 'http://localhost:9200/_cluster/health'

Sample Output

{
    "cluster_name": "elasticsearch",
    "status": "yellow",
    "timed_out": false,
    "number_of_nodes": 1,
    "number_of_data_nodes": 1,
    "active_primary_shards": 27,
    "active_shards": 27,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 20,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 57.446808510638306
}

To list the pending task of elastic search

curl --location --request GET 'http://localhost:9200/_cluster/pending_tasks'

To get the node stats

This output will contain details of documents, shards, indexing, search etc

curl --location --request GET 'http://localhost:9200/_nodes/stats'

Update Elasticsearch Settings

To update a setting to a specific index replace * with index name

curl --location --request PUT 'http://localhost:9200/*/_settings' \
--header 'Content-Type: application/json' \
--data-raw '{ "index" : { "max_result_window": 3 } }'

The post Magento 2 - Useful Elasticsearch Commands first appeared on Moses Dinakaran.

• How to Enable 'eq' Filter GraphQL Search for an Custom Attribute of Type Text in Magento 2
• Magento 2 - Useful Elasticsearch Commands
• Magento 2 - Elastic Search Concepts for a Magento Developer
• Magento 2 - How data is fetched from Elasticserach
• How Product Data is Pushed to Elasticsearch from Magento
• Elastic Search for Magento Developer

Elastic Search is a powerful search and analytics engine. It is known for its distributed nature, speed, and scalability. This document covers the important concepts of elastic search that would require for a magento developer

The Elasticsearch Instance

Once Elasticsearch is installed and running you have the instance of the Elasticsearch also known as node.

Elasticsearch Instance == Node

A node contains unique ID and Name and belongs to a single cluster. When you start elasticsearch instance (node) a cluster is automatically created. A cluster can have 1 to n nodes. Nodes can be distributed in several machines but still they can be part of single cluster.

Document

Elasticsearch stores data as JSON documents. Each document correlates a set of keys (names of fields or properties) with their corresponding values (strings, numbers, Booleans, dates, arrays of values, geolocations, or other types of data). Every document contains a unique ID

Indices

Document that shares similarities are grouped in to indices. Multiple Index can be stored in a node. Index is just a virtual thing that knows where the data is stored, Data is actually stored in the shard.

• MySQL => Databases => Tables => Columns/Rows
• Elasticsearch => Indices => Types => Documents with Properties

Shard

Shard is where data is stored, this is where you run your search query. When you create an index, a shard is created by default. No of documents that a shard can store depends on the size of the document and the instance size. Shards are horizontally scalable. A Single index can be stored across multiple nodes.

The total no of index you have in the node, the total no of shards will be created. This no can be changed based on the index.number_of_shards configuration.

Sharding

The practice of creating an index with multiple shards is called sharding. Its explained in the below image. Here we have elasticsearch with three instance and contains two indexes. These indexes are shared in all three instances. Each index contains 2 shards.

There are two main reasons why sharding is important,

• With the first one being that it allows you to split and thereby scale volumes of data.
• The other reason is that operations can be distributed across multiple nodes and thereby parallelized. This results in increased performance because multiple machines can potentially work on the same query.

To Avoid data loss, use replica of shards.

A cluster with too many indices or shards is said to suffer from oversharding. An oversharded cluster will be less efficient. The best way to prevent oversharding and other shard-related issues is to create a sharding strategy.

Index Mapping

Every index in an elastic search contains a mapping.

Mapping determines how a document, and its field are stored and indexed. It will define which fields needs to be indexed and which not. Mapping plays a important role in how ES stores and searches the data. Depending upon the use cases the mapping can be customized to make storage and indexing more efficient.

If a mapping is not defined, Elasticsearch defined its own mapping based on the document data this is called dynamic mapping.

A sample mapping is shown below.

The data is Elasticsearch  is stored as TEXT or KEYWORD. By Default, every string get mapped by both text and keyword. This can be changed based on our requirement.

Only One mapping will be created for a index, After the mapping is created we cannot edit the existing field but we can add new fields to mapping. If we need to edit the existing mapping, we need to delete the indices and create a new once.

Elasticsearch Indexing

When a document is stored, it is indexed and fully searchable in near real-time--within 1 second. Elasticsearch uses a data structure called an inverted index that supports very fast full-text searches. An inverted index lists every unique word that appears in any document and identifies all the documents each word occurs in.

Any field in Elasticsearch can be of type Text or Keyword.

• Text --> Full Text Search
• Keyword --> Keyword is defined for Exact Search, Aggregation and Sorting.

Text Field Indexing

Lets see how the text field indexing works, For our ex we will take the product attribute description which contains the below text mapping

"description": {
        "type": "text"
}

and the data that we are inserting is “These pineapples are sourced from New Zealand.”

By default, strings are analyzed when it is indexed. The string is broken up into individual words also known as tokens. The analyzer further lowercases each token and removes punctuations.

Once the text are analyzed its stored in the inverted Index

Now that we have the data indexed if the client is searching for the word pineapples, The ES just need to look for the term pineapples in the inverted index and be able to find the doc id.

In short inverted index is 

• words --> to --> documents.

Only text fields support the analyzer mapping parameter.

Keyword Field Indexing

When a keyword field is created, the content of the field is not analyzed and not stored in the inverted index instead Elasticsearch uses a data structure call the DOC value

So for each document, the document id along with the field value the actual string is stored as is. This data structure is optimal for exact search aggregation and sorting.

Search Relevance & Full text Search

A search is made up of 2 parts Relevance and Ranking

Relevance:

All Relevant documents that match a certain query. Precision / Quality(AND) and Recall /Quantity ( OR) determines which document are relevant for the search query but it will not determine which is the most relevant compared to the other.

i.e Precision and Recall will include all the available "Phones" but it will not tell which phone should be listed first.

Ranking:

This determines which record should come first. Ranking of a document is determined by score using a scoring algorithm. A score is a value that represents how relevant a document for a specific query. A score for a document is determined by multiple factors such as

Term Frequency

How Often the search term appears in this document. More frequency == More often

Inverse Document Frequency

How often does this term appear in all the documents. The more often the lower the weight. For ex Buy Mobile for Rs 10000

Here Buy and for will appear more in all the document which will reduce its weight

Vector Space Model

This provides a way for comparing a multi term query against a document.

Field Length normalization

How long is the field, The shorter the higher the weight ex title vs body

Aggregation

There are 2 main ways to fetch data from Elasticsearch.

Query

• Retrieve document that matches the specific search criteria. Ex Search All Products with Name “Mobile Phones”

Aggregation

• Summarizes your data as metrics, statistics and other analytics Ex What is the total no of Mobiles that belongs to Apple Brand

Elasticsearch organizes aggregations into three categories:

Metric Aggregation

Aggregations that calculate metrics, such as a sum or average, from field values.

Bucket aggregations

When you want to aggregate document on several subset of documents based on the field value, range or other criteria bucket aggregation will be handy. Bucket aggregation groups documents into several set of documents called Bucket all document within a bucket share a same criteria.

Pipeline aggregation

Aggregations that take input from other aggregations instead of documents or fields are called pipeline aggregation.

The below is a sample aggregation query.

The post Magento 2 - Elastic Search Concepts for a Magento Developer first appeared on Moses Dinakaran.

• How to Enable 'eq' Filter GraphQL Search for an Custom Attribute of Type Text in Magento 2
• Magento 2 - Useful Elasticsearch Commands
• Magento 2 - Elastic Search Concepts for a Magento Developer
• Magento 2 - How data is fetched from Elasticserach
• How Product Data is Pushed to Elasticsearch from Magento
• Elastic Search for Magento Developer

Overview

Once we have the product data pushed to Elasticsearch we can now use ES API's to fetch data.

The data from ES helps to

1. Identify the products ids that are relevant for the specific search query.
2. Filter Navigation / Aggregation

Once we have the products ids, The relevant product data such as pricing, inventory, product attributes are fetched from the mysql directly.

A search request to elastic search contains 3 parts.

1. The Search String
2. Sorting
3. Aggregation (Filter Navigation)

These requests are built based on configuration in search_request.xml file. By default, Magento uses the following query types:

• quick_search_container – Quick Search
• advanced_search_container – Advanced Search
• catalog_view_container – Category page with layered navigation
• graphql_product_search_with_aggregation – Used by GraphQL API when you pass aggregation query
• graphql_product_search – Used by GraphQL API

Code Reference :

• vendor/magento/module-catalog-graph-ql/etc/search_request.xml

The Search Query

There are three types are search query defined in search_request.xml

• boolQuery
• matchQuery
• filteredQuery

With the bool query, you can combine multiple queries into one request and further specify boolean clauses to narrow down your search results.
There are four clauses to choose from:

• Must
  • One or more queries can be specified here. A document MUST match all of these queries to be considered as a hit.
• must_not
  • A document must NOT match any of the queries specified here. It it does, it is excluded from the search results.
• Should
  • A document does not have to match any queries specified here. However, it if it does match, this document is given a higher score.
• Filter
  • These filters(queries) place documents in either yes or no category. Ones that fall into the yes category are included in the hits.
    You can build combinations of one or more of these clauses. Each clause can contain one or multiple queries that specify the criteria of each clause.
    These clauses are optional and can be mixed and matched to cater to your use case. The order in which they appear does not matter either!

Syntax:

GET name_of_index/_search
{
  "query": {
    "bool": {
      "must": [
	Filter by attributes, Category Ids, Visibility
      ],
      "should": [
        { The Search String }
      ],
    }
  }
}

How Search Query is Generated

The search queries are defined in the below 4 files, Based on the type of search request the respective container is selected.

• vendor/magento/module-elasticsearch-catalog-permissions/etc/search_request.xml
• vendor/magento/module-elasticsearch-catalog-permissions-graph-ql/etc/search_request.xml
• vendor/magento/module-catalog-graph-ql/etc/search_request.xml
• vendor/magento/module-catalog-search/etc/search_request.xml

When search request is processed these xml files are processed and the relevant search query is generated and stored in the cache. The process of generating the search aggregation query is explained below

File Reference	Comments
\Magento\Framework\Config\Data::initData	XML file content and parsed and merged
\Magento\Framework\Search\Request\Config\Converter::convert	Search Query is formed
\Magento\CatalogSearch\Model\Search\Request\SearchModifier::modify	At this point we will have the category and price bucket generated but filterable attribute bucket are not generated yet
\Magento\CatalogSearch\Model\Search\RequestGenerator::generate \Magento\CatalogSearch\Model\Search\RequestGenerator::generateRequest	At this point all the filterable bucket attributes are generated. $request['aggregations'][$bucketName] = $generator->getAggregationData($attribute, $bucketName);

The ouput of these parsed xml files is stored in the $data array

Container	Sample Data
Quick Search Container
Catalog View Container
GraphQl Product Search With Aggregation
GraphQl Product Search

Lets see some actual elastic search requests that gets triggered in real time.

Category Page Query

To List all the Products from a specific Category

Frontend URL	Elastic Search Request
Category Page http://commerce.development/gear/bags.html	URL: http://localhost:9200/commerce_product_1/document/_search?track_total_hits=true Body:
Category Page with Filter Selected http://commerce.development/gear/bags.html?material=47	Api Request: http://localhost:9200/commerce_product_1/document/_search?track_total_hits=true Request Body:

Search Page Query

If you are searching for a string “pack” the below query is issues to elastic search

Frontend URL : http://commerce.development/catalogsearch/result/?q=pack

In Short difference between PLP and Search Page query is the query clause

• PLP -> must
• Search Page -> must & should

Sorting

By default magneto supports position, product name and price sorting. Additional attributes can be added for sorting. In that case the respective mapping will get updated note that a full reindexing is required in that case.

Below are the few sort query examples.

Sort By Position
Sort By Name

Filter Navigation / Aggregation

The data for the filter navigation is generated using the aggregation buckets in the search query

Search Query	Frontend Display

There are primarily three types of aggregation buckets

• Category Bucket
• Price Bucket
• Filterable Product Attributes Bucket

The Response from the aggregation bucket from the elastic search will be like the below

These data from the response is parsed and displayed in the Frontend Filtered Navigation

The post Magento 2 - How data is fetched from Elasticserach first appeared on Moses Dinakaran.

To serve data from elastic search we first need to push data to elastic search, This is done through the catalog search fulltext indexing.

Lets explore more in to this through the below sections.

Identity the Relevant Products to Push to Elasticsearch

The products that are enabled, visibility = catalog search will be pushed to elastic search. Each product data will be stored in the elastic search as a document.

A sample product data would be like the following

In the case of configurable product, The data from the child product is also merged and then pushed to elastic search. For ex if we have the below configurable product with 3 children where each child has a separate barcode.

Reference :

• \Magento\CatalogSearch\Model\Indexer\Fulltext\Action\Full::rebuildStoreIndex
• \Magento\CatalogSearch\Model\Indexer\Fulltext\Action\DataProvider::getSearchableProducts
  • Fetches all the products that needs to be send to elastic search

Generate Elasticsearch Index Name

The first step in creating the elastic search index is to generate an index name. Magento creates a separate elastic search Indices for each store view.  For ex if we have 4 store views then 4 indices will be created as show below.

At times we will see there are multiple index available for a store view, In the below table for Kuwait English Store we could see two elastic search indices are created which we will discuss later.

The Elasticsearch Index pattern “commerce_product_1_v15” contains the below parts

• commerce à Index Prefix which is configurable
• product àRepresent Product Index
• [1|2|3..] à Store ID
• [v15 | v11] à A random version no given for each index name.

The below steps are involved to identify the index name for a store view.

For ex lets consider we are reindexing Kuwait English Store

• Identify the primary index of Kuwait English Store, The primary index of a store can be identified using the elastic search alias which we will see in the next section.
• Remove all the unwanted or orphaned index of Kuwait English Store apart from the primary index.
  • In our case, lets assume we have two indices commerce_product_1_v15 and commerce_product_1_v16, where commerce_product_1_v16 is the primary index so commerce_product_1_v15 will get deleted.
• Now do +1 to the current primary index version so our new index name will be commerce_product_1_v17

Code Reference

• \Magento\CatalogSearch\Model\Indexer\Fulltext::executeByDimensions
  • Indexing Process Starts Here
• \Magento\Elasticsearch\Model\Indexer\IndexerHandler::cleanIndex
  • Delete the old unwanted Index and Create the New Index
• \Magento\Elasticsearch7\Model\Client\Elasticsearch::existsAlias
  • Check if alias exists

Create the Index

Once the index name is identified, we can not create the new index.

Code Reference :

• \Magento\Elasticsearch\Model\Indexer\IndexerHandler::saveIndex

Index Setting

While creating the index, Index level settings is also set such as filter setting, analyser, mapping fields count etc.

Request : PUT http://localhost:9200/commerce_product_1_v17

Body :

Response : 
{
"acknowledged":true,
"shards_acknowledged":true,
"index":"commerce_product_1_v17"
}

The response indicates that we have created the new index commerce_product_1_v17 along with the settings.

Code Reference:

• \Magento\Elasticsearch\Model\Adapter\Elasticsearch::prepareIndex
  • Create new index with mapping.

Analyzer

The analyzer parameter specifies the analyzer used for text analysis when indexing or searching a text field.

An analyzer  — whether built-in or custom — is just a package which contains three lower-level building blocks: character filters, tokenizers, and token filters.

Tokenizer

A tokenizer receives a stream of characters, breaks it up into individual tokens (usually individual words), and outputs a stream of tokens. For instance, a whitespace tokenizer breaks text into tokens whenever it sees any whitespace. It would convert the text "Quick brown fox!" into the terms [Quick, brown, fox!].

Character filters

A character filter receives the original text as a stream of characters and can transform the stream by adding, removing, or changing characters

Token filter

A token filter receives the token stream and may add, remove, or change tokens. For example, a lowercase token filter converts all tokens to lowercase, a stop token filter removes common words (stop words) like the from the token stream, and a synonym token filter introduces synonyms into the token stream.

Code Reference

• \Magento\Elasticsearch\Model\Adapter\Index\Builder::build
  • Set the analyzer setting

Create Index Mapping

Before we start pushing the data to the index, we need to create the index mapping.

The mapping will be created for each and every product attribute that is available in the store.

Depending on the product attribute type the mapping type is set. Below are the few sample mapping

Searchable Product Attributes.

All the product attributes that is set to use in Search will be pushed to elastic search.

And based on the attribute input type the relevant mappings are set.

For ex, Lets have a look at the below product data.

Name	Sku	Url Key	Description

The copy_to parameter allows you to copy the values of multiple fields into a group field, which can then be queried as a single field.

So while searching for a record either in listing page or graphql etc, magento appends the below match query automatically.

So basically instead of searching in each field we search in _search field as all the values are copied here.

For the name attribute we see fields.sort_[attrbutecode], This is because name attribute has the sorting setting enabled.

In the case of description, we dont have the keyword mapping which means we cannot use the eq operator

But we can use the match operator

i.e For all the attributes which doesn't contains the keyword mapping we cant use eq operator to search.

Code Reference:

• \Magento\Elasticsearch\Model\Adapter\FieldMapper\Product\CompositeFieldProvider::getFields

• \Magento\Elasticsearch\Model\Adapter\FieldMapper\Product\FieldProvider\StaticField::getField

• \Magento\Elasticsearch7\Model\Client\Elasticsearch::addFieldsMapping
  • Set the mapping for each attribute
• \Magento\Elasticsearch7\Model\Client\Elasticsearch::applyFieldsMappingPreprocessors

In case if we need to customize mapping we can do here,

<type name="Magento\Elasticsearch7\Model\Client\Elasticsearch">
    <arguments>
        <argument name="fieldsMappingPreprocessors" xsi:type="array">
            <item name="elasticsearch7_nested_type_field_mapping" xsi:type="object">YourModule\NestedFieldMapping</item>
        </argument>
    </arguments>
</type>

•  \Elasticsearch\Namespaces\IndicesNamespace::putMapping
  • Save the Mapping

Filterable Product Attributes

All the filterable attributes will have both text and keyword mapping represented in 2 fields attrcode & attrcode_value

Mapping	Product Data

Non-Indexed Attributes

Mapping	Product Data
	We don’t push these kind of data to elastic search, But while creating the mapping still set these field and made is as index false

Sorting Fields

There are certain fields that is used for sorting such as Position, Product Name and Price.

For the purpose of position sorting position_category_[category_id] mapping is created. This mapping field is created for all the available store categories.

Mapping	Product Data

While fetching the products from the es we issue the sorting query as the follows

Elastic search Alias

An alias is a secondary name for a single index or a multiple indices. Magento create a alias for each store view and index (the one with version no) is assigned to the alias.

 In our case we always assign one index (the primary index) to the alias as shown below.

The data from the elastic search is always fetched using the alias. ex

http://localhost:9200/commerce_product_1/document/_search?

While full reindexing, Magento will not touch the current primary index that is assigned to the alias instead it will create a new index and push the data to the new index. Once all the data is pushed to the new index, The alias is switched to the new index and the old index is deleted.

Index Processing Summary

The Process of creating the elasticsearch indices for a store involves the below steps, For ex lets consider we are reindexing Kuwait English Store

• First remove all the unwanted index apart from the primary index of the respective store
  • In our case, lets assume we have two indices commerce_product_1_v15 and commerce_product_1_v16, where commerce_product_1_v16 is the primary index so commerce_product_1_v15 will get deleted.
• Now create a new elastic search indices (commerce_product_1_v17) along with the index settings and mapping.
• Push the data to the new index.
• Once all the data is pushed to this new indices, This new indices will be set as the primary indices and the old one commerce_product_1_v16 will get removed.
• So while full reindexing, The data to the frontend will be served without any interruption from the  current primary index. On the completing of the  reindex process the new indices will be set as the new primary index while the old one get deleted.

The post How Product Data is Pushed to Elasticsearch from Magento first appeared on Moses Dinakaran.

The below are the two primary actions involved with magento integration with elastic search which we will see in detail.

• Push Data
  • The product data from Magento is pushed to elastic search through catalogsearch_fulltext indexing.
• Fetch Data
  • Through query and aggregation search, Ex graqlql query, plp etc

We can use any REST API client for interacting with Elasticsearch. Following are some sample http methods we will be using to interact

• GET : Used to retrieve records based on query
  • GET 'http://localhost:9200/magento2_product_1/_search'
  • GET 'http://localhost:9200/*/_settings
• POST : Used to update the existing record in Indices
  • POST 'http://localhost:9200/indexname/_doc'
• PUT : Used to create and update the records
• DELETE : To remove the existing record from indices

Magento_Elasticsearch is the magento core module that handles the elastic search functionality. This module internally uses the third party php client https://github.com/elastic/elasticsearch)to communicate with the elastic search instance.

Code Reference

• \Elasticsearch\Namespaces\BooleanRequestWrapper::performRequest
• \Elasticsearch\Transport::performRequest
  • The place where actual request that is triggered to elastic search

In the below articles we will look in to the elastic search concepts and how its works with Magento.

• How to Enable 'eq' Filter GraphQL Search for an Custom Attribute of Type Text in Magento 2
• Magento 2 - Useful Elasticsearch Commands
• Magento 2 - Elastic Search Concepts for a Magento Developer
• Magento 2 - How data is fetched from Elasticserach
• How Product Data is Pushed to Elasticsearch from Magento

The post Elastic Search for Magento Developer first appeared on Moses Dinakaran.