When working with data from multiple regions that was produced in multiple languages or otherwise contains non-Latin characters, data-conversion errors might occur with the default configuration of SAS 9.4. These errors might cause unexpected behavior in processing or inaccurate output or results. Using Unicode resolves this issue but might introduce challenges with legacy data.

There are various approaches to handling a mix of Unicode-dependent and Unicode-incompatible code, depending on your data and usage requirements. Not all of those methods are outlined in this SAS KB article.

This article outlines common options and details an approach using multiple SASApp Application Server instances, which is often a preferred method for maintaining support of multiple encodings.

When a data set contains characters from an encoding that differs from that which the current SAS session runs, Cross-Environment Data Access (CEDA) is invoked. Messages similar to the following might occur, which are informative and do not indicate a problem:

NOTE: Data file xxxxx.xxxxx.DATA is in a format that is native to another host, or the file encoding does not match the session encoding. Cross Environment Data Access will be used, which might require additional CPU resources and might reduce performance.

However, if CEDA is not possible because the current session encoding is too restricted to support the characters, character data loss might occur, and output or results might not be displayed as expected. As a result, problems with the output data might occur. When this issue occurs, the following message is also printed to the log:

WARNING: Some character data was lost during transcoding in the dataset xxxxx.xxxxx. Either the data contains characters that are not representable in the new encoding or truncation occurred during transcoding.

Note: SAS 9.4 uses LATIN1 (WLATIN1 on Microsoft Windows) encoding by default. SAS^® Viya^® uses Unicode (UTF-8) encoding by default, so the problems outlined in this article do not apply to SAS Viya deployments.

Resolution

Method 1: Globally Switch to Unicode Default Encoding

For modernization, this is the preferred approach. However, if you work with legacy data sets (and do not want to convert the encoding of those data sets), or if your workflows interact with systems that do not support Unicode, this approach is not recommended. Note that, in the SAS^® Viya^® platform, the default session encoding is already UTF-8 (Unicode).

If you are configuring a new SAS 9.4 platform deployment, or you do not access any legacy data sets in your existing SAS 9.4 deployment, it is recommended to simply make a global switch to using SAS 9.4 in UTF-8 mode. This switch allows support for a broader character set in SAS 9.4 and also enables easier movement of data sets that you create between SAS 9.4 and SAS Viya deployments.

See SAS KB0036261 for instructions about how to switch SAS 9.4 to run fully in Unicode UTF-8 mode.

If you are performing a new deployment of SAS 9.4, you will be prompted during the deployment process if you want to configure the server for Unicode support. If you select this option, SAS^® Deployment Wizard automatically completes the steps outlined in SAS KB0036261.

Method 2: Per-session Encoding

If you do not regularly work with non-Latin character data, you might prefer to adjust encoding for the specific data-read task or for the session.

You can adjust the encoding for specific tasks or sessions using the ENCODING= system option. For details, see Setting the Encoding of a SAS Session.

To use UTF-8 Unicode encoding, set ENCODING=utf8

You can find a complete list of available session encoding values at SBCS, DBCS, and Unicode Encoding Values for Transcoding Data.

Method 3: Configure Separate Application Server (SASApp) Instances for (W)LATIN1 and UTF-8 Modes

By configuring a separate SASApp Application Server instance, you can selectively run SAS in various encoding modes without requiring a total switch or per-session options. This is an effective method for maintaining support of legacy applications, datasets, or integrations that cannot support Unicode while also providing Unicode session availability for processes where it is required.

Note: A SAS Administrator must complete this method as it requires access and modifications to the Compute Server where SAS 9.4 is installed. Users who access SASApp application server instances must be running SAS through a client application like SAS^® Enterprise Guide^® or SAS^® Studio.

1) Deploy a new SAS Application Server.

The SAS administrator should first deploy and define an additional SASApp Application Server Context to use for Unicode support by running SAS Deployment Wizard on the Compute Server machine.

The recommended selections to complete this task are outlined in Defining Multiple Application Servers: Create a SAS Application Server for a SAS Solution.

Note: Be sure to note the name of the new Application Server context, which is used in the server configuration paths that you must modify in the following steps. In the examples below, the Application Server context was named SASAppU8. Replace this with the name that you defined in this step.

2) Locate the SAS Application Server context configuration Usermods file.

The new SAS Application Server context should have created configuration files on the Compute Server. Locate these files and identify the correct Usermods file, which will be edited to call the Unicode-mode SAS binaries.

a) On the Compute Server machine, navigate to <SAS-Configuration-directory>/LevN/SASAppU8/.

b) Open the usermods file for the Application Server Context that matches your Operating System and Configuration in a text editor.

• • Linux non-GRID: <SAS-Configuration-directory>/LevN/SASAppU8/appservercontext_env_usermods.sh
  • Linux GRID: <SAS-Configuration-directory>/LevN/SASAppU8/GridServer/grid_usermods.cfg
  • Microsoft Windows: <SAS-Configuration-directory>\Levn\SASAppU8\sasv9_usermods.cfg

3) Modify the SAS Application Server Context Configuration

Using the file that you found in step 2, apply the changes required to call SAS 9.4 in Unicode mode. The changes required vary based on Operating System and Configuration.

• Linux non-GRID: Modify or add the following line to set the SAS_COMMAND environment variable definition:

SAS_COMMAND=$SASROOT/bin/sas_u8

• Linux GRID: Modify or add the following lines to set the SASEXE environment variable definition and ensure it is exported:

SASEXEFILE=$SASROOT/bin/sas_u8
export SASEXEFILE

• Microsoft Windows: Locate and comment out the existing global usermods file reference, then add a reference to the Unicode binary version:

i. Comment out the following line: -config <SAS-Installation-directory>\SASFoundation\9.4\sasv9_usermods.cfg
ii. Then, add the following line: -config <SAS-Installation-directory>\SASFoundation\9.4\nls\u8\sasv9_usermods.cfg

4) Restart the SAS Compute-tier services.

To ensure the usermods file is processed and the changes are made available, you should restart the SAS services.

Typically, restarting the SAS^® Object Spawner on the Compute-tier machine forces the usermods file to be re-read.

If this step does not produce the desired effect, or if problems occur, fully restart your SAS machines and SAS services to ensure a clean boot.

5) Validate SAS Session Encoding for the new Application Server context.

After making the above changes and restarting SAS services, the new application server context should be available in your SAS client application. Select the Application Server context and then run the following SAS code:

proc options option=encoding value;
run;

If successfully configured, the SAS log output should display a session encoding reported as UTF-8. The configuration file location might vary based on your Operating System and is dependent on the file that you modified in step 3. The following example is from a non-GRID Linux environment: