When Elasticsearch re-indexes jobs, a failure message similar to the following might occur in the Search and Discovery (SAND) log:
2019-12-09 05:38:05.560 ERROR 29897 --- [0-auto-1-exec-1] c.s.s.w.e.GlobalExceptionHandler : sas.svi-indexer@@compliance [c10dab7289496cdc] [REQUEST_HANDLING_ERROR_CAUSED_BY] [458cacde-ba7b-4662-a69d-2bd5d517f824] Caused by: exception: ElasticsearchException message: Elasticsearch engine response: Code: 403. Error: cluster_block_exception:blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];.
By design, when disk space runs out during an indexing job, Elasticsearch modifies the indexes to be Read-only.
Note: System administrators should check system disk space usage, the size of the indices (see SAS KB0039693, "Scripts that analyze Elasticsearch configurations and svi-indexer logging can enhance configuration and performance in SAS® Visual Investigator" for instructions), remove redundant indexes, and allocate more space if necessary before attempting to index again.
There are two known workarounds to circumvent this issue.
- Run a full re-indexing, which results in all indexes being re-created.
- Run one of the following commands depending on your SAS Visual Investigator version, Elasticsearch key, and cert files that you contain. Note that you need to replace host-name and port-number with the correct Elasticsearch host name and port information for your system. Then, restart your indexing job.
- If you use SAS® Visual Investigator 10.6 or SAS® Visual Investigator 10.7 and your system contains /opt/sas/viya/config/etc/elasticsearch/default/keys/searchguard/sgadminkey.pem and /opt/sas/viya/config/etc/elasticsearch/default/certs/searchguard/sgadmincert.pem files, run the following command:
sudo curl --request PUT --cacert /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem --key /opt/sas/viya/config/etc/elasticsearch/default/keys/searchguard/sgadminkey.pem --cert /opt/sas/viya/config/etc/elasticsearch/default/certs/searchguard/sgadmincert.pem https://host-name:port-number/_all/_settings --data '{"index.blocks.read_only_allow_delete": null}' -H "Content-Type:application/json"
- If you use SAS Visual Investigator 10.6 or SAS Visual Investigator 10.7 and your system contains /opt/sas/viya/config/etc/elasticsearch/default/keys/searchguard/sghealthcheck-key.pem and /opt/sas/viya/config/etc/elasticsearch/default/certs/searchguard/sghealthcheck-cert.pem files, run the following command:
sudo curl --request PUT --cacert /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem -–key /opt/sas/viya/config/etc/elasticsearch/default/keys/searchguard/sghealthcheck-key.pem --cert /opt/sas/viya/config/etc/elasticsearch/default/certs/searchguard/sghealthcheck-cert.pem https://host-name:port-number/_all/_settings --data '{"index.blocks.read_only_allow_delete": null}' -H "Content-Type:application/json"
- If you use SAS® Visual Investigator 10.8, run the following command:
sudo curl --request PUT --cacert /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem --key /opt/sas/viya/config/etc/elasticsearch/default/keys/opendistro/opendistrohealthcheck-key.pem --cert /opt/sas/viya/config/etc/elasticsearch/default/certs/opendistro/opendistrohealthcheck-cert.pem https://host-name:port-number/_all/_settings --data '{"index.blocks.read_only_allow_delete": null}' -H "Content-Type:application/json"