When you run a full index on a data object, the first step in openSearch is to create a new index and then write data from the source database table to the index until all data rows are exhausted. At the end of the indexing process, the new index is registered to SAS Visual Investigator, and the old index is deleted, releasing disk space. If a failure occurs during the indexing process, it is possible that the old index might not be deleted and remains intact on openSearch, which consumes disk space.

As mentioned above, the old copy of the data exists while the new index is being populated. This process doubles the disk space that is needed in openSearch during indexing, even if for a brief time. As a result, disk usage is a very important metric to monitor closely to prevent the disk usage from reaching full capacity and affecting data accessibility in SAS Visual Investigator.

Workaround

1) Determine the current disk usage for each opendistro pods.

Complete the following steps to view the disk space for each opendistro pod in openSearch:

1. Get namespaces:
   kubectl get namespaces
   
   
2. Get the name of the pods for opendistro. Note that your namespace might differ, and you might have multiple opendistro pods.
   kubectl get pods -n viya | grep sas-opendistro
   
   
3. Get secrets for opendistro. Note that your namespace might differ.
   kubectl get secrets -n viya | grep -i opendistro
   
   
4. Decode the user secret user name. Note that your namespace and pod name might differ.
   kubectl get secret sas-opendistro-sasadmin-secret -n viya -o jsonpath="{.data.username}" | base64 -d
   
   
5. Decode the password. (Your namespace and pod name might differ.) By setting the password to a variable, you can use $PASSWORD in all the following commands:
   PASSWORD=$(kubectl get secret sas-opendistro-sasadmin-secret -n viya -o jsonpath="{.data.password}" | base64 -d)
   
   
6. Go into the opendistro data pods to check their disk space. (Your namespace and pod name might differ, and you might have multiple opendistro pods.) Check the data pods first. Then, check the master pods at the end. This order ensures that you are in one of the master pods if you need to move on to step 2: Determine whether there are duplicate indices.
   1. kubectl exec -it sas-opendistro-default-0 -n viya -- /bin/bash
      Note that the example below is in the sas-opendistro-custom-data-2 pod, and the /usr/share/elasticsearch/data disk usage is at 100%.
      
   2. Repeat steps I and II for each opendistro pod.
      
      Note:Do not exit the last master opendistro pod if you need to proceed to Step 2 below. Otherwise, exit the pod.

2) Determine whether there are duplicate indices in openSearch that cause high disk usage

Compete the following steps to view the indices on openSearch:

1. Run the /_cat/indices curl command from within one of the master pods. Note that your user name might differ, and you might have to use your host name instead of localhost.
   curl -u sasadmin:$PASSWORD http://localhost:9200/_cat/indices?v | sort 
   
   If the above command does not work, try the following:
   curl -k -u sasadmin:$PASSWORD https://localhost:9200/_cat/indices?v | sort
   
   
2. Assess the results, and look for duplicate indices. Here are some examples of duplicate indices:
   • sand__document_link__1773560365767
   • sand__document_link__1774767185777
3. Do not exit from the master opendistro pod until you determine that you do not need to delete the indices due to duplicates. If you do need to delete the indices, proceed with step 3. Otherwise, exit the pod.

Step 3: Delete all data indices in openSearch.

Note: The steps in this section require you to fully re-index all of the data.

Complete the following steps to delete the indices on openSearch. These instructions take care to avoid deleting anything besides the data indices.

1.  Delete all data indices for disk space reclamation. (Note: Delete only indices that begin with sand__*.)
   curl -X DELETE -u sasadmin:$PASSWORD http://localhost:9200/sand__*
   
   
2. Run the /_cat/indices command again to verify that the indices were deleted:
   curl -u sasadmin:$PASSWORD http://localhost:9200/_cat/indices?v | sort 
   
   If the above command does not work, try the following:
   curl -k -u sasadmin:$PASSWORD https://localhost:9200/_cat/indices?v | sort
   1. Verify that all of the sand* indices are deleted. 
   2. Repeat step 1: Determine current disk space usage for all opendistro pods to verify whether you reclaimed the space. 
   3. Exit the pod.

At this point, the indices should be deleted, and you should have reclaimed the disk space. If one or more of the openSearch data pods still contains data, then you need to remove the orphaned data (dangling indices). Complete the additional steps below to reclaim the disk space.

4) Delete the orphaned data to reclaim disk space.

Execute the following instructions from outside of the opendistro pod.

1.  Verify that the variable to your password is still set, or reset variable from step 1.
   echo $PASSWORD 
   
   
2.  Generate a list of dangling indices:
   kubectl exec -n viya sas-opendistro-default-0 -- curl -s "http://localhost:9200/_dangling" -u sasadmin:$PASSWORD
   
   Here is the response when no dangling indices exist: 
   Defaulted container "sas-opendistro" out of: sas-opendistro, sas-certframe (init), sas-opendistro-sysctl (init) {"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"sas-opendistro","dangling_indices":[]}
   
   
3. If a list of dangling_indices exists, execute the following command to delete each one. Replace the {index_uuid} with the values from list generated above.
   kubectl exec -n viya sas-opendistro-default-0 -- curl -s "http://localhost:9200/_dangling/{index_uuid}" -u sasadmin:$PASSWORD
   
   Here is an example of an index_uuid. The following results are from the /_cat/indices command.
   
   health status index                                                           uuid                   
   yellow open   sand__party_interest_list__1775499654842   p7gOHO0fQGeQJnWwFLW2rw

If you are unable to delete the orphaned data (dangling indices), you can also delete the Persistent Volume Claims (PCVs) to reclaim the disk space. 

5) Delete PVCs from openSearch if disk space is not reclaimed when you delete all of the data pods.

Note: The steps in this section require you to fully re-index all of the data. In addition, your pod names might differ. Lastly, these instructions are for stateful pods.  

All of the following commands below use a variable for namespace: NS=<namespace>.

1. Shut down OpenDistroCluster.
   1. Get a list of opendistro pods. (The examples that accompany these instructions are on a system with three masters pods and three data pods.)
      kubectl get pods -n $NS | grep sas-opendistro
      
      
   2. Check the number of replicas (for startup later):
      data_replicas=$(kubectl get statefulset sas-opendistro-custom-data -n "$NS" -o jsonpath='{.spec.replicas}')
      master_replicas=$(kubectl get statefulset sas-opendistro-custom-master -n "$NS" -o jsonpath='{.spec.replicas}')
      echo "data_replicas: $data_replicas"; echo "master_replicas: $master_replicas"
      
      
   3. Shut down all opendistro pods:
      kubectl -n $NS patch opendistrocluster sas-opendistro --type=json -p='[{"op": "replace", "path": "/spec/shutdown", "value":true}]'
      
      
   4. Stop the pods. Note that you use the word statefulset for stateful pods to ensure that the pods will not restart.
      kubectl scale statefulset sas-opendistro-custom-data -n $NS --replicas=0
      kubectl scale statefulset sas-opendistro-custom-master -n $NS --replicas=0
      
      
2. Verify that all opendistro* pods are stopped. Delete any orphaned pods. Note: Do not delete the sas-opendistro-operator pod
   1. Get list of opendistro pods to check their status:
      kubectl -n $NS get pods | grep opendistro
      
      
   2. Wait a few minutes. If there are leftover opendistro pods running, you must delete them:
      kubectl -n $NS delete pods <pod-name>
      
      
3. Remove the PVCs associated with the OpenDistro pods:
   --delete PVCs
   kubectl -n $NS delete pvc --selector=type=opendistro-nodeset,app=sas-opendistro
   
   An output similar to the following should occur for each PVC:
   persistentvolumeclaim "data-sas-opendistro-custom-data-0" deleted 
   
   
4. Start up the OpenDistroCluster:
   kubectl -n $NS patch opendistrocluster sas-opendistro --type=json -p='[{"op": "replace", "path": "/spec/shutdown", "value":false}]'
   
   IF NEEDED: If you do not see the number of replicas from before shutdown, execute the following:
   kubectl scale statefulset sas-opendistro-custom-data -n $NS --replicas=$data_replicas
   kubectl scale statefulset sas-opendistro-custom-master -n $NS --replicas=$master_replicas
   
   
5. Wait a few minutes for the pods to come back up.
   
   
6. Verify the health and the composition of the cluster:
   kubectl exec -n viya sas-opendistro-default-0 -- curl -s "http://localhost:9200/_cat/nodes" -u sasadmin:$PASSWORD
   kubectl exec -n viya sas-opendistro-default-0 -- curl -s "http://localhost:9200/_cluster/health?pretty" -u sasadmin:$PASSWORD
   
   The output of /_cat/indices and /_cat/nodes should show all pods as part of the cluster. The output of /_cluster/health should ideally display a "green" status.
   
   
7. Restart the sand pods:
   sand_index=$(kubectl -n "$NS" get pods -o name | grep -E 'sand' | cut -d'/' -f2)
   echo "$sand_index"
   kubectl -n $NS delete pod $sand_index
   
   
8. Re-index the data in SAS Visual Investigator. If the output from step VI shows a healthy cluster with all the nodes running, then you can proceed with the re-indexing.