When you schedule a large number of SAS jobs at the same time (for example, through Jobs and Flows or the SAS Scheduler) in a SAS Viya deployment running on Kubernetes or OpenShift, not all of the jobs run. Above a certain level of concurrency, some jobs never start. These jobs are not displayed on the Jobs and Flows monitoring tab, and no SAS error occurs for the missing jobs.

Note: Reducing the number of jobs launched at the same time avoids the problem.

This issue occurs when SASWORK (the temporary work location of the SAS Compute and Batch servers) is backed by a dynamically provisioned storage class, LVMS (which uses TopoLVM), rather than by the default emptyDir volume. In that configuration, Kubernetes creates one PersistentVolumeClaim (PVC) for each job.

Cause

The behavior comes from how these per-job volumes are created. When SASWORK uses a storage class, the storage provisioner must create and bind a dedicated volume for every job pod, and that creation is serialized: LVM serializes metadata operations on a volume group, and the LVMS or TopoLVM node component creates one volume at a time per node, giving a ceiling of approximately one volume per second per node. When many jobs start at once, the volumes cannot be created quickly enough. Pods whose volume is not bound in time are never scheduled, and the corresponding jobs are dropped without any SAS error.

This issue is a provisioning throughput limit, not a disk, capacity, or compute problem. During such a run, disk activity is low, the volume group is healthy with free space, and there is no CPU or memory pressure.

How to Confirm That the Cause Is Volume Provisioning and Not SAS

The following approach distinguishes this storage provisioning limit from a compute, capacity, or SAS application problem.

1. Reproduce the issue and collect get-k8s-info, covering the exact time window of the run. Note the launch time so that you can correlate the events and logs.
2. Look at the scheduling events for the compute and batch pods. Affected pods stay unschedulable with messages such as waiting for ephemeral volume controller to create the persistentvolumeclaim and could not find PersistentVolumeClaim. The claim name matches the pod name followed by a suffix, which shows that each job maps to one pod and one dedicated SASWORK volume.
3. Rule out compute. There are no Insufficient CPU or Insufficient memory events, and the compute nodes show ample CPU and memory headroom during the run.
4. Rule out storage capacity and performance. The volume group is healthy with free space (for example, from the per-node volume group status), and disk activity on the nodes stays low throughout the run.
5. Inspect the storage provisioner logs for the same window (on OpenShift LVM Storage, the lvms-operator, and the per-node vg-manager). Volume creation is processed one at a time per node. Under a large burst, repeated object has been modified; please apply your changes and try again conflicts occur, which add latency.
6. Apply the decisive test. Rerun the same total number of jobs, but stagger the jobs in smaller waves instead of all at once. If the losses disappear when the launches are spread out, the limit is provisioning throughput and not capacity or compute. The same total succeeds simply because the arrival rate is lower.

Address the Storage Provisioning Limit

There are three ways to address this issue, and you can combine the three methods.

(Recommended for High Currency) Use the Default emptyDir Volume for SASWORK

emptyDir is node-local and does not create a PVC for each job, which removes the bottleneck at its source. emptyDir is set through the SAS deployment overlay for the SAS Programming Run-Time Environment (sas-bases/examples/sas-programming-environment/storage) without changing the LVM operator.

Confirm first that the compute nodes contain enough local storage for the peak SASWORK usage. Note that emptyDir supports only a soft sizeLimit, which evicts the pod when exceeded, and not a hard maximum WORK size per session.

Stagger the Job Submissions

Submit the jobs in smaller waves instead of all at once so that the submission rate matches the provisioning rate. The workable wave size and interval depend on the environment and sizing—in particular, the number of nodes that run compute pods. Because provisioning runs at about one volume per second per node, larger clusters sustain larger waves.

Start conservatively and increase the batch size until you find the level that your environment handles without losing jobs. Staggering reaches a high total number of jobs without changing the storage configuration, but it is an operational practice rather than a configuration change.

Add Worker Nodes

Because provisioning is serialized at about one volume per second per node, the aggregate rate scales with the number of nodes that run compute pods. Adding nodes shortens the time needed to bring up a large batch. There is no supported user-facing setting to raise the per-node provisioning concurrency and is not changed by a version upgrade. The OpenShift or storage team handles this rate.

By default, SASWORK is backed by an emptyDir volume that uses storage on the node. A storage class is configured only when larger or dedicated work volumes are required. That choice reintroduces per-job dynamic provisioning and the behavior described above. For more information, see the SAS documentation topic "Configure Storage for the SAS Programming Run-Time Environment."