You might encounter several performance issues in SAS Visual Investigator if your environment contains many entities and relationships. 

Performance Issue Symptoms

This section describes two different types of symptoms you might encounter.

Fails to Load with an HTTP 502 Error

When you export a configuration or load a list of relationships via SAS Visual Investigator, it loads for only a few minutes and issues an HTTP 502 error. In this scenario, an error similar to the following might also occur in the sas-svi-datahub log:

Loads Indefinitely

When you log on to SAS Visual Investigator, one of the following occurs:

• It loads indefinitely. 
• You are able to log on successfully. But, when you click the Alerts or Relationship tabs, alerts and relationships load indefinitely.

In these scenarios, the user interface performance slows down significantly when you move between the other tabs, enter text into text boxes, and perform similar actions. In your sas-svi-visual-investigator log or the sas-visual-investigator-app pod log, errors similar to the following about SAS Visual Investigator struggling to connect to the Data Hub service with “ResourceAccessException” and “Read timed out” errors and reporting it as being down might occur:

{"level":"error","message":"ResourceAccessException, datahub is probably down. org.springframework.web.client.ResourceAccessException: I/O error on POST request for \"/svi-
datahub/documents/labels\": Read timed out; nested exception is java.net.SocketTimeoutException: Read
timed out","properties":{"logger":"com.sas.fcs.workspace.services.RecentlyViewedService","thread":
"https-jsse-nio-8080-exec-
2","traceId":"5671dc145a98a7f2","spanId":"5671dc145a98a7f2","username":
"sukcag","__session":"e9f0cd23-f1a7-488d-add1-79e6488e90d5"},"source":
"sas-visual-investigator-app","timeStamp":"2024-01-23T10:15:41.1130000Z",
"version":1,"messageParameters":{"0":"org.springframework.web.client.ResourceAccessException:
I/O error on POST request for \"/svi-datahub/documents/labels\": Read timed out;
nested exception is java.net.SocketTimeoutException:Read timed out"}}{"version":1,
"timeStamp":"2024-01-23T11:32:13.0240000Z","level":"error","source":
"sas-visual-investigator-app","message":"Error in
restTemplateCachedTokenInterceptor.\njava.net.SocketTimeoutException: Read timed out",
"properties":{"logger":"com.sas.fcs.commons.security.SpringSecurityContextHelper","thread":
"https-jsse-nio-8080-exec-8","traceId":"7f93f5607245b312","spanId":"1149095e00467c86","__session":
"a594d319-64ac-4233-acbf-294102e91148","username":"sukcag"}}

Performance Issues

There are two different situations that can often occur at the same time and lead to issues such as the following.

Issue 1: Excessive Cache Misses

Excessive cache misses can cause the following issue: 

The default cache size settings of the Visual Investigator Data Hub service are exceeded due to large numbers of entities and relationships, which result in excessive cache misses and a need to increase the default cache size. 

Determine Whether this Issue Is Occurring

To confirm that you are encountering the excessive cache misses issue, enable TRACE level logging for the com.sas.fcs.datahub.core.metadata.cache.local logger against the service. An excessive number of messages similar to the following appear in the sas-svi-datahub log with the TRACE level logging enabled:

LOG.trace("Cache MISS for key: {}", key);

Note: For information about how to specify loggers and logging levels for services, refer to the SAS^® Viya^® 3.5 Administration Guide if you are on SAS^® Viya^® 3.5. If you are on the SAS^® Viya^® platform, refer to the version of the SAS^® Viya^® Platform Administration Guide that matches the version of your platform.

Resolution

To resolve Issue 1, which results from excessive cache misses, increase the metadata cache size and increase the refreshAfterWriteInMinutes/ refreshAfterWriteInSeconds property as follows: 

1. Log on to SAS^® Environment Manager as a SAS administrator and navigate to the Configuration page.

2. Select Definitions from the View drop-down list.

3. Search for the word datahub to filter for definitions related to the Data Hub service.

4. (SAS Viya platform users only) Skip this step if you use SAS Viya 3.5. Select sas.svi-datahub.cache.metadata-types from the list of definitions and click the New Configuration button on the right. In the resulting window, change the value of the maximumSize property from 3000 to 6000 as follows and click Save:

5. Select the sas.svi-datahub.cache.metadata.client definition from the list of definitions and click the New Configuration button on the right. In the resulting window, change the value of the maximumSize property from 3000 to 6000. Also perform the following adjustment based on your platform:

(SAS Viya platform) Change the refreshAfterWriteInMinutes property from 1 to 20 and click Save. 

(SAS Viya 3.5) Change the refreshAfterWriteInSeconds property from 60 to 1200 and click Save:

6. Restart the sas-svi-datahub service/pod and wait for at least six minutes for the changes to be picked up and the cache to fully reinitialize.

Issue 2: Cache Refreshes Occur Too Frequently

Cache refreshes occurring too frequently can cause the following issue:

If there are a large number of entities in the system, refreshing the cache might take more than one minute to complete. If the next cache refresh is triggered before the current refresh has completed, the system's performance can suffer.

Determine Whether this Issue Is Occurring

In this scenario, high CPU usage might occur in your sas-svi-datahub pod(s) if you are on the SAS Viya platform. If you are on Viya 3.5, the process associated with your sas-svi-datahub service might use significantly more CPU than usual.

Resolution

To resolve Issue 2, which results from cache refreshes occurring too frequently, increase the time between metadata cache refreshes as follows: 

By default, the metadata caches are refreshed once per minute. Increasing this rate to once per ten minutes (600 seconds) should increase the refresh time between caches and reduce the high CPU usage by the sas-svi-datahub pod/process. 

To do this, implement a sas.svi-datahub.cache.bulk-refresher setting and set it to 600 in consul and restart the sas-svi-datahub pod/service. Note that it is not possible to do this via SAS^® Environment Manager. You will have to use kv Write commands to write to consul as follows based on your platform: 

SAS Viya Platform

Use one of the following commands, depending on your TLS configuration:

If your environment has just front-door TLS is enabled or no TLS is enabled, issue the following:

kubectl -n <Viya namespace> exec sas-consul-server-0 -- bash –c '/opt/sas/viya/home/bin/sas-bootstrap-config kv write --force config/svi-datahub/sas.svi-datahub.cache.bulk-refresher "600"'

If your environment has full-stack TLS enabled, do the following:

kubectl -n <Viya namespace> exec sas-consul-server-0 -- bash -c 'export CONSUL_HTTP_ADDR=https://localhost:8500; /opt/sas/viya/home/bin/sas-bootstrap-config kv write --force config/svi-datahub/sas.svi-datahub.cache.bulk-refresher "600"'

SAS Viya 3.5

Use these commands:

source /opt/sas/viya/config/consul.conf

export CONSUL_HTTP_TOKEN=$(sudo cat
/opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/consul/default/client.token)

/opt/sas/viya/home/bin/sas-bootstrap-config kv write --force config/svi-datahub/sas.svi-datahub.cache.bulk-refresher "600"

Both Platforms

Make sure you restart your sas-svi-datahub pod/service in order for the change to be picked up. There is no need to wait after the restart. The caches should populate automatically in response to user requests. After a short period of normal usage, the Data Hub caches should be populated.