sas-logon-app fails to start with a flyway migration error


During a SAS® Viya® upgrade, the sas-logon-app might fail to start, which in turn prevents other pods from initializing.

As a result, the following error messages might occur in the sas-logon-app logs:

Migration of schema "logon" to version "4.110 - Add ExternalKey To Identity Provider" failed! Changes successfully rolled back.
...
ERROR: column "external_key" of relation "identity_provider" already exists
...
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.

Cause

The Flyway migration script, V4_110__Add_ExternalKey_To_Identity_Provider.sql, attempts to add an external_key column to the identity_provider table. If this column already exists, possibly due to a previous partial upgrade or manual schema modification, the migration fails, which causes the sas-logon-app to crash.

Workaround

To circumvent this issue, manually remove the external_key column from the identity_provider table in the SharedServices PostgreSQL database by completing the following steps:

  1. Retrieve the dbmsowner password:

    kubectl get secret sas-crunchy-platform-postgres-pguser-dbmsowner -o jsonpath="{.data.password}" | base64 -d

  2. Set the environment variables:

    export NAMESPACE=<your_viya_namespace>
    export PG_CLUSTER=sas-crunchy-platform-postgres

  3. Identify the primary PostgreSQL pod:

    kubectl get pod -n $NAMESPACE -o name -l postgres-operator.crunchydata.com/cluster=$PG_CLUSTER,postgres-operator.crunchydata.com/role=master

  4. Access the pod shell:

    kubectl exec -it -n $NAMESPACE <primary_pod_name> -c database -- sh

  5. Connect to PostgreSQL and drop the column:

    psql -h localhost -d SharedServices -U dbmsowner -p 5432
    ALTER TABLE logon.identity_provider DROP COLUMN external_key;
    \q

  6. Restart the sas-logon-app pod:

    kubectl rollout restart deployment sas-logon-app -n $NAMESPACE