Part 2: A long–term strategy for addressing unresponsive SAS^® Stored Process Servers with SAS^® 9.1.3

SAS Technical Support has received reports of previously working SAS Stored Process Servers becoming unresponsive over time for unknown reasons. By unresponsive, we mean that the SAS Stored Process Servers are up and running but no requests from client applications are getting through to the server. These servers might also be referred to as "hung" or "orphaned" SAS^® processes.

What to do?

1. Initially, addressing this issues with all customers starts as a "put out the fire" situation in which Technical Support offers suggestions to
   
   
   1. Confirm that the servers are down or unresponsive.
   2. Clean up and recover from the problem immediately. (See SAS Note 30952, "Tips for addressing unresponsive SAS 9.1.3 Stored Process Servers, Part 1" for instructions on how to recover from the immediate problem of unresponsive servers.)
2. Then, a long–term strategy is needed to gather information and determine why the problem occurs.

This document provides a suggested long–term strategy to facilitate this evaluation.

 

Modify the Load Balancing properties and logging for the SAS Stored Process Server and enable verbose logging for the Object Spawner.

 

To examine this problem over time, we must modify the load balancing parameters for the SAS Stored Process Server to optimize server performance and add logging information to the Object Spawner and Stored Process Server logs. We’re not only looking for obvious errors in the logs but also clues as to a pattern to the problem, for example, what was the last step/program that successfully executed? What is the last entry written in the log?

From within SAS^® Management Console:

1. Expand the Server Manager plug-in.
   1. Highlight the SASMain - Logical Stored Process Server definition and right-click to select Properties.
   2. Select the Load Balancing tab. The default settings are as follows:

Balancing Algorithm: Cost
Response Refresh Rate: 0
Cost Per Client: 100
Logical Server Credentials: (None)

3. Change the Cost Per Client to 1 (one) and click OK to save the change and close the Properties window.

2. Expand the SASMain - Logical Stored Process Server definition.

1. 1. Highlight the SASMain - Stored Process Server definition and right-click to select Properties.
   2. Select the Options tab.
      • In the Object Server Parameters field, add applevel=3
      • Click the Advanced Options... button.

3. Select the Load Balancing Properties tab. The default settings are as follows:

Maximum Clients: 5
Maximum Cost: 1000
Startup Cost: 10
Availability Timeout (msec): 60000
Start Size: 0
Recycle Activation Limit: 0
Inactivity Timeout: not selected, and blank

4. Change the Maximum Cost to 500 and click OK.

5. Click OK to save the change and close the Properties window.

From outside of SAS Management Console:

Windows Operating Systems:

1. Locate your ObjectSpawner.bat file that is located in your SAS Configuration directory. For example:

<configdir>\Lev1\SASMain\ObjectSpawner\ObjectSpawner.bat

2. Add the saslogfile (-slf) and sasverbose (-sv) options to the ":install" section of the ObjectSpawner.bat file:

1. 1. Edit the install section of the ObjectSpawner.bat file similar to the following:

   :install 

      start /b "Object Spawner" "%sasdir%objspawn" –sasSpawnerCn 

      "%SPWNNAME%" –OMRconfigfile 

           "C:\SAS\EntBIServer\Lev1\SASMain\ObjectSpawner\OMRConfig.xml" 

           –slf "C:\SAS\EntBIServer\Lev1\SASMain\ObjectSpawner\logs\objspawn.log" 

       –sv –install –name "%SASServiceNAME%" %DEPENDS% 

    goto end2  

NOTE: The second line in the example above that begins with "start /b..." and the four lines which follow it must be specified on the SAME LINE in your ObjectSpawner.bat file. The long line above was split into multiple lines for readability. So, the line begins with "start /b" and ends with "%DEPENDS".

2. Stop the Object Spawner service through the Windows Services window.

3. Remove the Object Spawner service by executing "ObjectSpawner.bat -remove".

4. Reinstall the Object Spawner service by executing "ObjectSpawner.bat -install".

5. Start the Object Spawner service through the Windows Services window.

NOTE: The Object Spawner must be stopped and restarted to enable changes made to Object Spawner or Stored Process Server options in SAS Management Console or changes made in the Object Spawner or Stored Process Server script files.

UNIX Operating Systems:

1. Find your ObjectSpawner.sh file which is located in your SAS Configuration directory. For example:

!SASROOT/BIArch/Lev1/SASMain/ObjectSpawner/ObjectSpawner.sh

2. Add the –saslogfile (–slf) and –sasverbose (–sv) options to the ObjectSpawner.sh file.

1. 1. Edit the ObjectSpawner.sh command similar to the following:

      nohup $CMD –sasSpawnerCn "$SPWNNAME" –sv –slf $DIR/logs/objspawn.log –omrconfigfile $XMLCFG > $DIR/logs/objspawn_console.log 2>&1 &

   2. Stop the Object Spawner process by executing "ObjectSpawner.sh stop".
   3. Restart the object spawner process by executing "ObjectSpawner.sh start".

NOTE: Object Spawner must be stopped and restarted to enable changes made to Object Spawner or Stored Process Server options in SMC or changes made in the Object Spawner or Stored Process Server script files.

Review SAS Notes relating to unresponsive SAS Stored Process Servers:

Are you using custom style sheets in your stored process(es)?

Are you using the Sessions feature of stored processes or are you producing graphics output in your stored process(es)?

Are you submitting your stored processes from a Web client?

Are your Object Spawner and SAS Stored Process Servers running on a UNIX platform?

Are you using an older SAS image without service packs installed?

General debugging information for unresponsive SAS Stored Process Servers

After modifying your Object Spawner and SAS Stored Process Server configurations to enable logging and adjust load balancing, and after reviewing the above SAS Notes and applying workarounds or hot fixes as appropriate, wait for the problem to occur again.

Once you believe there's a problem, save a copy of the objspawn.log file (grab it before you restart Object Spawner or it will be overwritten) and all of the SAS Stored Process Server logs that correspond to the same time frame when the problem occurred.

Here are some things you will need to think about:

• How often does the problem occur? Hourly? Daily? Weekly? Sporadically?
• When does the problem occur?
  • Does it occur after running a particular stored process or sequence of stored processes, or does it seem to occur randomly?
  • Does it occur at the same time of day?
  • Does it occur when graphics output is being generated in the stored process?
  • Does it occur when stored processes are executed from one particular client (such as SAS^® Enterprise Guide^®, SAS^® Add-In for Microsoft Office, STP Web Application, SAS^® Web Report Studio)?
  • Does it occur with a high user load?

If the problem can be isolated to a particular stored process or sequence of stored processes, use the macro code described in SAS Note 19573,"Macro that helps when debugging "hanging" Stored Process Server problems" to help identify the problematic step in the program.

 

If the cause of the unresponsive SAS Stored Process Servers cannot be determined using the above tips, please compile the information obtained from these steps and contact SAS Technical Support for further assistance.

 

If you are experiencing unresponsive stored process servers using SAS 9.2, please refer to the following notes:

SAS Note 43160, "Tips for addressing unresponsive SAS^® 9.2 Stored Process Servers, Part 1"

SAS Note 43163, "Tips for addressing unresponsive SAS^® 9.2 Stored Process Servers, Part 2"