Elastic Search

Infra Setup (Open Search):

• Deployment Option: Domain With standby is recommended (leaving the AZ for standby to AWS itself)

• Set Fielddata cache allocation a little bigger

  • this will give us Improved performance of searches that involve lots with large numbers of values.

  • Need to consider memory consumption here.

  • max clause count - TBD

• Monitor the usage metrics and consider scaling up/down, minimizing the costs, and maximizing the performance.

  • OpenSearch error logs

  • Search slow logs

  • Indexing slow logs

  • Audit logs

Things that impact performance:

Sample Config:

{
	"index.requests.cache.enable": True,
	"number_of_shards": 3,
	"number_of_replicas": 1,
}

Replicas:

AWS recommends configuring at least two replicas for each index in a three-zone setup to avoid disruption in performance and availability.

Shards

Number of nodes --> Number of Shards

Number of shards --> Performance (Also results in performance degradation if we use more shards which might cause a Mapping Explosion)

AWS Suggested Formulae: Number of shards ==> 25 x (1/2 x Instance Memory)

For r6g.large.search instance with a Memory of 16 GB, we can utilize 200 shards which provides us with better performance.

Things to do:

• Benchmarking the Instances with different queries.

Queries

• What about isolating the data of different clients in ES?

Exploration Pending

• Best-Practices-for-Optimizing-Elasticsearch-Performance
• ES Serverless

• Asynchronous Search
• ES Ingestion

• ES Ultra Warm

References:

• AWS: manage-open-search-domains-multi-availability-zone
• AWS: configure-amazon-OpenSearchopensearch-service-for-high-availability
• AWS: amazon-OpenSearch-service-now-supports-99.99%-availability-using-multi-az-with-standby/
• Elastic Search: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html