Controlling CPU Utilisation during indexing in elastic search

At INDmoney we use elastic search extensively to power mutual funds data. Our entire mutual fund catalogue depends on elastic search hence it’s a very critical component . We use airflow that schedules jobs which upserts data into our elastic search cluster.

Current design

Recently we realised that one such job that runs every 3 hours to update mutual funds data causes CPU spike to over 95% which in turn degrades our user facing Go service performance.
So we decided to dig deep and solve this issue.

Cloudwatch metric for CPU Utilisation

Updating ES and Spark Config

1. Index config — Elastic search is near realtime. ES config refresh_interval helps tune this. We ended up setting refresh interval to -1 (completely disabling it). Our spark job during batch inserts explicitly call the refresh api (default behaviour), that makes the new data available for search. We were able to set refresh interval to -1 because in our case only scheduled jobs updates the data in the index.
   PUT fund-details/_settings
{
“index” : {
“refresh_interval” : “-1”
}
}
2. Reduce indexed fields — We store a lot of data per document, but search happens only on limited fields. So we ended up explicitly disabling indexing for fields that we never search on.
   “mappings” : {
“properties” : {
“Active” : {
“type” : “long”,
“index” : false
}
}}
3. Reduce parallelism — Spark DataFrame coalesce() is used only to decrease the number of partitions. We set it to 1 in our spark job to reduce parallelism, thus the load on our elastic search cluster also reduces.

Tradeoff — Slower indexing rate which means increased time to complete the job

After the changes the CPU remains mostly around 60% during indexing job which is something we can totally work with.

CPU usage after changes

That’s about it! Thank you for reading, and I hope you enjoyed the article.