When OpenSearch becomes overwhelmed with requests, it can start rejecting them. As a result, HTTP 429 error messages like Too Many Requests are returned to svi-indexer service. However, despite the error message, the indexer finishes the job successfully.
This issue leaves the data in a bad state where metrics state that all records were written to OpenSearch. In reality, though, there are records missing from the indexed data.
To work around this issue, tune the OpenSearch cluster shows signs of stress, which involves additional resource allocation and index management.