This KB article documents an issue encountered when enabling or using SAS Viya Copilot where the activation fails due to reaching the maximum number of secrets for Consumption Units (CUs). The issue manifests as failures in secret creation within the GenAI Gateway and prevents SAS Viya Copilot from obtaining access tokens.

Symptoms

The following error occurs during SAS Viya Copilot activation or runtime and prevents the GenAI Gateway from creating or retrieving the required credentials:

Error: Maximum number of secrets reached for CUs

One or more of the following symptoms might occur:

• Copilot activation fails with an internal server error.
• SAS Model Studio displays an error indicating that Copilot cannot be contacted.
• GenAI Gateway reports missing credentials.
• HTTP 400 or 500 errors are returned from GenAI-related endpoints.

Example Errors

Here are logs from gen-ai-gateway micro service:

Here is the SAS Viya Copilot activation error:

 

Here is the SAS Model Studio error:

Root Cause

The GenAI Gateway has a limited number of retries when generating secrets, and this limit can be exceeded when multiple environments are deployed concurrently under the same order.

Here is what happens when the limit is reached:

• New secrets cannot be created.
• Existing environments might lose access to credentials.
• Copilot becomes unavailable.

SAS has updated the internal limit, but environments that were created before the fix or with high-parallel activity might still be affected.

Enable DEBUG Logging for GenAI Gateway

To gather additional diagnostics, enable DEBUG logging on the GenAI Gateway deployment:

kubectl set env deploy/sas-gen-ai-gateway SAS_LOG_LEVEL=DEBUG -n viya-namespace 

After enabling DEBUG, do the following:

• Reproduce the issue.
• Collect logs from the sas-gen-ai-gateway pod. 
• Look for errors related to secret creation or Secrets Management API calls. 

Workaround

A viable workaround is to deploy environments using more than one order, which reduces contention for secret creation under the same CU pool. This approach avoids encountering the secret generation limits when multiple environments are created or activated simultaneously.

Resolution

1.  Regenerate API Credentials

As an initial remediation step, generate a new API Key and Secret and add them to the affected environment. If the credentials are correctly configured, SAS Viya Copilot should become enabled.

2.   Cross-Environment Validation

If you have multiple environments deployed using the same order's assets (such as Prod, QA, Dev, Test), do the following:

• Verify whether other environments continue to work after regenerating the credentials.
• If enabling Copilot in one environment causes another to fail, this strongly indicates a shared secret limit issue.