When you initially deploy SAS Viya, an internally generated TLS certificate is applied, which functions as the Root CA (Certificate Authority) for SAS Viya services. This certificate might be used for generating the ingress certificate, signing requests to generate additional security certificates, and other SSL/TLS certificate management tasks.

• The default expiration of this Root CA certificate is five years from when it was created.
• This Root CA certificate is persistent between SAS Viya version updates.

If your SAS Viya environment has been in use for around five years, it is very likely that the SAS Viya Root CA certificate will soon expire and needs to be renewed. If your SAS Viya environment has been in use for more than five years and is unexpectedly down, it is possible that the SAS Viya Root CA certificate has already expired.

SAS Viya administrators should proactively plan for the internal SAS Viya Root CA expiration and renewal in their SAS Viya environments that approach five years of time since initial deployment.

SAS Technical Support recommends that SAS Viya administrators perform a renewal of the SAS Viya Root CA certificate before it is fully expired in order to prevent an unexpected service interruption.

---

Verify Conditions

You must have performed the initial deployment five or more years ago for a problem to occur from this condition, as the SAS Viya Root CA certificate created at initial deployment is valid for five years. You should perform proactive maintenance near the five-year anniversary of your initial SAS Viya deployment but before the five-year time frame is exceeded to prevent a service outage.

If your SAS Viya Root CA certificate nears expiration, note the following:

• No information is provided to warn of the upcoming Root CA certificate expiration.
• You can manually perform administrative checks to determine the status and expiration date of the SAS Viya Root CA certificate.

If your SAS Viya Root CA certificate has fully expired, the following occurs:

• Various SAS Viya services fail to start.
  • Consul
  • sas-readiness
• Various running SAS Viya services do not communicate properly or report communication errors.
  • Consul
  • SAS^® Cloud Analytic Services Server
• Warnings or errors occur in various SAS Viya service logs related to certificate validity messages, such as the following.
  • Consul Server log: 
    failed to get conn: x509: certificate has expired or is not yet valid
    
    
  • CAS Server log:
    {"level":"error","message":"Secure communications error status 807ff008 description \"192.168.0.1: The certificate sent from the remote host cannot be validated by any of the public keys in the root certificate store specified by SSLCALISTLOC or SSLCACERTDIR.
    
    
  • sas-readiness log:
    error retrieving application config from consul (results from Consul Server communication failures)

---

Cause

SAS Viya services fail to communicate if the Root CA certificate is expired. As a result, a service interruption occurs when the Root CA certificate expires.

Depending on the certificate generation and management method used in your SAS Viya deployment, the behavior that occurs when the Root CA certificate nears expiration varies slightly.

• If cert-manager is NOT in use (the default SAS Viya configuration), a renewed Root CA certificate is NOT automatically generated. In this scenario, manual actions are required to both generate the new Root CA certificate and apply it.
• If cert-manager is in use, a renewed Root CA certificate is automatically generated one hour prior to the expiration of the original Root CA certificate. Note that cert-manager cannot apply the renewed certificate to running services; manual actions are still required.

Regardless of the certificate generation and management method used in your SAS Viya deployment, a maintenance outage is required when the SAS Viya Root CA certificate is renewed. You must restart SAS Viya services with a newly generated Root CA secret in order to apply a renewed Root CA certificate to the SAS Viya services.

Note that, in most production deployments, the internal SAS Viya Root CA certificate is NOT used as the ingress certificate's root certificate authority. Most organizations require a trusted certificate authority to sign the certificates presented by web applications in order to avoid browser security warnings.

• An organization's IT department typically provides the Root CA for the ingress certificate, which is configured as described in IT-Provided Ingress Certificates.
• The internal SAS Viya Root CA certificate might be used as the certificate authority for the ingress certificate (instead of an IT-Provided Root CA certificate) if such a trust chain is not required, as described in Provisional Ingress Certificates.

Although you should be cautious not to confuse these two Root CA certificate functions and the fact that they might be different Root CA certificates, the configuration used in your SAS Viya deployment is not relevant for the maintenance task outlined in this SAS KB article.

• All SAS Viya deployments still use an internal SAS Viya Root CA certificate, which is automatically generated by a certificate generator, for various internal functions.
• All SAS Viya deployments periodically require the maintenance task outlined in the Resolution section of this article.

---

Resolution

Prerequisite: Determine the Certificate Generator in Use

Note: The default configuration of SAS Viya uses openssl to generate certificates and does NOT use cert-manager.

You can configure SAS Viya to use a proprietary openssl-based solution (default) or the open-source cert-manager utility for certificate generation. The commands and processes used when performing maintenance on the SAS Viya Root CA certificate vary depending on the certificate generator being used.

If you are not sure which method was set up to handle certificate generation in your SAS Viya environment, you should check the deployment's configMaps or the deployment assets to determine the configured certificate generator. The following methods, as outlined in Determine Which Certificate Generator Is Being Used, explain how to check these items:

Method 1: Examine a Running Deployment

Review the contents sas-certframe-user-config configMap by running the following command:

kubectl -n name-of-namespace describe configmaps $(kubectl -n name-of-namespace get configmaps | grep sas-certframe-user-config | awk -F" " '{print $1}') 

An environment variable called SAS_CERTIFICATE_GENERATOR might be defined and present in the data returned by the command.

• If its value is set to openssl, then openssl is in use.
• If its value is set to cert-manager, then cert-manager is in use.
• If its value is set to something else, then a nonstandard certificate generator might be in use. In this scenario, contact your Kubernetes administrators for guidance.
• If the SAS_CERTIFICATE_GENERATOR environment variable does not exist, then openssl is in use.

Method 2: Examine the Deployment Assets

Only use this method if the SAS Viya deployment is fully stopped. Begin by reviewing the $deploy/kustomization.yaml file that your SAS Viya deployment uses. In the generators section, look for a generator that modifies the sas-certframe-user-config configMap. 

• The generators are user-defined and typically contain a name that references their function.
• If you are not sure which generator might make this modification, you might need to review all the generators listed in the kustomization.yaml file.

Once you identify the correct generator file that makes modifications to the sas-certframe-user-config configMap, check the value of the SAS_CERTIFICATE_GENERATOR literal defined within that generator's referenced yaml file.

• If its value is set to openssl, then openssl is in use.
• If its value is set to cert-manager, then cert-manager is in use.
• If its value is set to something else, then a nonstandard certificate generator might be in use. In this scenario, contact your Kubernetes administrators for guidance.
• If the SAS_CERTIFICATE_GENERATOR environment variable does not exist, then openssl is in use.
  
  

Instructions for When the openssl Certificate Generator is in Use

If the openssl certificate generator is in use, complete the steps in this section to verify Root CA certificate details, renew the Root CA certificate, or apply a newly generated SAS Viya Root CA certificate.

Verify Certificate Details and Expiration (managed by openssl)

When openssl is in use, you can check the details of the SAS Viya Root CA certificate by querying the certificate location from its reference in the related secret.

kubectl -n name-of-namespace get secret sas-viya-ca-certificate-secret -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout

Locate the Validity section of the certificate details printed by the previous command. Inspect the properties "Not After" and "Not Before".

• The Not After property is the date and time that the certificate will expire.
  
• The Not Before property is the date and time that the certificate first becomes valid. Typically, this is the date and time the certificate was issued.

The date and time noted by the Not After and Not Before properties provide valuable information about the current status of the SAS Viya Root CA certificate.

• If the Not After property is a date and time value that has already passed, the certificate is fully expired and needs to be renewed in order for the SAS^® Viya^® platform to operate.
• If the Not After date value is in the near future, proactive renewal is advised.
• If the Not Before date value is approximately five years before the current date, proactive renewal is advised.

If the SAS Viya Root CA certificate is fully expired, messages related to an expired x509 certificate condition occur within various SAS Viya service logs. For example, messages similar to the following might occur in the Consul log: 

x509: certificate has expired or is not yet valid .

If these messages occur, you should complete the steps to renew the SAS Viya Root CA certificate.

Contact SAS Technical Support if you require assistance analyzing potential warnings or errors in your SAS Viya service logs.

Renew, Generate, or Apply a New SAS Viya Root CA Certificate (Managed by openssl)

If you determine that the SAS Viya Root CA certificate is expired, needs to be re-applied, or want to proactively renew it to extend its expiration date, complete the instructions in this section. Kubernetes administrator permissions might be required. The actions to perform are the same for all conditions. (If the current SAS Viya Root CA certificate is fully expired, or if you are doing a proactive renewal, you should complete the same steps).

1) Stop the SAS Viya deployment:

kubectl create job sas-stop-all-`date +%s` --from cronjobs/sas-stop-all -n name-of-namespace

2) Ensure that SAS Viya services have stopped. There are multiple ways to check the services. For example, most of the pods typically observed in a running SAS Viya deployment should not be seen in the output of the following command. (The previous command should have scaled them to zero replicas.)

kubectl -n name-of-namespace get pods

3) Delete the existing secret linked to the SAS Viya Root CA certificate.

kubectl -n name-of-namespace delete secret sas-viya-ca-certificate-secret

4) Start the SAS Viya deployment.

kubectl create job sas-start-all-`date +%s` --from cronjobs/sas-start-all -n name-of-namespace

After you delete the secret associated with the SAS Viya Root CA and restart your SAS Viya platform, the first SAS Viya service pod restarts and generates a new SAS Viya platform Root CA certificate. This new Root CA certificate is then used to issue certificates to all services based on the updated CA.

Once SAS Viya services finish starting, your SAS Viya platform should be operational and using the renewed Root CA certificate.

You can verify the updated SAS Viya Root CA certificate and its new expiration date by re-running the command that you used to list the certificate details. The updated dates should be visible in the NotAfter and NotBefore fields.

kubectl -n name-of-namespace get secret sas-viya-ca-certificate-secret -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout

Instructions for When the cert-manager certificate generator is in use

If the cert-manager certificate generator is in use, complete the steps in this section to verify Root CA certificate details, renew the Root CA certificate, or apply a newly generated SAS Viya Root CA certificate.

Verify Certificate Details and Expiration (Managed by cert-manager)

When cert-manager is in use, you can check the age of the SAS Viya Root CA certificate (and its related secret) by running the following command:

kubectl -n name-of-namespace get certificates

If the AGE column for sas-viya-ca-certificate shows 5y (five years) or longer, you should inspect further and check the certificate's expiration date.

You can inspect the existing SAS Viya Root CA certificate's expiration date by running kubectl describe on the certificate. Here is an example command:

kubectl -n name-of-namespace describe certificate sas-viya-ca-certificate

Locate the "Not After", "Not Before", and "Renewal Time" properties in the describe output.

• The Not After property is the date and time that the certificate will expire.
  
• The Not Before property is the date and time that the certificate first becomes valid. Typically, this is the date and time that the certificate was issued.
  
• The Renewal Time property is the date and time that the certificate should be automatically renewed by cert-manager.

The date and time noted by the Not After property provides valuable information about the current status of the Root CA certificate.

• If the Not After property is a date and time value that has already passed, the certificate is fully expired and needs to be renewed in order for the SAS Viya platform to operate.
• If the Not After date is almost exactly five years in the future, then cert-manager has likely renewed the Root CA certificate, and it needs to be re-applied. In this condition, SAS Viya services would report errors due to the expired certificate.
• If the Not After date and time value is in the near future, proactive renewal is advised.

If the SAS Viya Root CA certificate is fully expired or was renewed but needs to be re-applied to the running services (such as if cert-manager has just renewed it), messages related to an expired x509 certificate condition occur within various SAS Viya service logs. For example, messages similar to the following might occur in the Consul log: 

x509: certificate has expired or is not yet valid .

If these messages occur, you should complete the steps to renew the SAS Viya Root CA certificate.

Contact SAS Technical Support if you require assistance analyzing potential warnings or errors in your SAS Viya service logs.

Renew, Generate, or Apply a New SAS Viya Root CA Certificate (Managed by cert-manager)

If you determine that the SAS Viya Root CA certificate is expired, needs to be re-applied, or want to proactively renew it to extend its expiration date, complete the instructions in this section. Kubernetes administrator permissions might be required. The actions to perform are the same for all conditions. (If the current SAS Viya Root CA certificate is fully expired, or if you are doing a proactive renewal, you should complete the same steps.)

1) Stop the SAS Viya deployment.

kubectl create job sas-stop-all-`date +%s` --from cronjobs/sas-stop-all -n name-of-namespace

2) Ensure that the SAS Viya services have stopped. There are multiple ways to check the services. For example, most of the pods typically observed in a running SAS Viya deployment should not be seen in the output of the following command. (The previous command should have scaled them to zero replicas.)

kubectl -n name-of-namespace get pods

3) Delete the existing secret linked to the SAS Viya Root CA certificate.

kubectl -n name-of-namespace delete secret sas-viya-ca-certificate-secret

4) Start the SAS Viya deployment.

kubectl create job sas-start-all-`date +%s` --from cronjobs/sas-start-all -n name-of-namespace

After you delete the secret associated with the SAS Viya Root CA and restart your SAS Viya platform, the first SAS Viya service pod restarts and generates a new SAS Viya platform Root CA certificate. This new Root CA certificate is then used to issue certificates to all services based on the updated CA.

Once SAS Viya services finish starting, your SAS Viya platform should be operational and using the renewed Root CA certificate.

You can verify the updated SAS Viya Root CA certificate and its new expiration date by re-running the command that you used to list the certificate details. The updated dates should be visible in the NotAfter and NotBefore fields.

kubectl -n name-of-namespace describe certificate sas-viya-ca-certificate

You can verify the new SAS Viya Root CA secret (which references the new SAS Viya Root CA certificate) by re-running the command that you used to list secrets in the Kubernetes namespace for SAS Viya. The AGE column should show a new, lower value. The AGE should approximately match the time that you restarted the SAS Viya services after completing steps 3 and 4 above.

kubectl -n name-of-namespace get certificate