When the Indexer sends documents to OpenSearch using the /bulk API, OpenSearch might return errors for some of the document operations.

HTTP Errors

Today, the Indexer handles HTTP 429 (Too Many Requests) errors intelligently. When an HTTP 429 error occurs, the Indexer does the following:

• waits before retrying (backoff)
• reduces the amount of data sent in each request
• retries in smaller batches.

However, HTTP 413 (Content Too Large) errors are not handled the same way. When an HTTP 413 error occurs, the Indexer currently treats every document in the batch as potentially causing the problem and retries them one document at a time. This behavior comes from existing Spring Batch retry logic.

Retrying documents individually is very inefficient and can cause indexing jobs to take hours instead of minutes. The HTTP 413 error should be handled similarly to an HTTP 429 error by reducing the batch size and retrying smaller chunks, rather than sending every document individually.

Impact

Large indexing batches can exceed OpenSearch's http.max_content_length limit. As a result, indexing slows down dramatically.

Here is an example:

• Expected: ~200,000 records indexed in less than five minutes.
• Actual: The same job can take about 5.5 hours because documents are retried one-by-one.

Desired Behavior

When an HTTP 413 error occurs, the following behavior is desired:

• reduce the batch size
• retry with smaller chunks
• continue reducing the chunk size until the request succeeds or reaches the configured tolerance limits

This behavior would dramatically improve performance and avoid unnecessary one-document-at-a-time processing.

Resolution

The hot fix for this issue resolves the HTTP 413 error by changing the behavior so that documents are not processed one-by-one.