Problem

The admin services stops working while generating Internal CA in ST with external MSSQL database. The generation of new Internal CA takes over a minute after which the admin service stops working. In the catalina.out log we can see a message like this:

Coherence <Error>: Halting JVM due to unrecoverable service failure

The issue is not know to occur on ST with embedded MySQL database or external Oracle database.

Resolution

This error condition occurs when the MSSQL database does not have the READ_COMMITED transaction encapsulation level with READ_COMMITED_SNAPSHOT option enabled. Because of this during the generation of the Internal CA one transaction is locking database resources read in another transaction. When the READ_COMMITED_SNAPSHOT is not enabled, the second transaction is forced to wait a very long time. The Coherence Service Guardian process treats this as a dead lock condition and halts the Admin JVM process, effectively stopping the admin service.

The READ_COMMITED_SNAPSHOT option is a required prerequisite during the installation of ST. To check if it is enabled you can execute the following query:

SELECT is_read_committed_snapshot_on FROM sys.databases WHERE name = yourdatabase

To turn on the READ_COMMITTED_SNAPSHOT option, use the following statement:

ALTER DATABASE yourdatabase SET READ_COMMITTED_SNAPSHOT ON