Problem

-- When using systemd as init system to start cassandra when API Gateway system boots, sometimes API Gateway starts too quickly after cassandra, and cassandra is not available yet. When it happen, the following error appears in instance trace:

ERROR 02/Dec/2016:15:51:24.882 [0f27:000000000000000000000000] Could not start connection pool for host 127.0.0.1(127.0.0.1):9160
ERROR 02/Dec/2016:15:51:24.903 [0f27:000000000000000000000000] Error initializing schema:
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:401)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:232)
at me.prettyprint.cassandra.service.AbstractCluster.describeKeyspace(AbstractCluster.java:199)
at com.vordel.kps.storeImpl.cassandra.CassandraSchema.initSchema(CassandraSchema.java:72)
at com.vordel.kps.Model.initialiseCassandraSchema(Model.java:115)
at com.vordel.kps.Model.start(Model.java:95)
at com.vordel.kps.impl.KPS.load(KPS.java:169)
at com.vordel.kps.impl.KPS.configure(KPS.java:135)
at com.vordel.precipitate.SolutionPack$ConfigModule.configure(SolutionPack.java:296)
at com.vordel.precipitate.SolutionPack.loadModules(SolutionPack.java:414)
at com.vordel.dwe.Service.refresh(Service.java:504)
at com.vordel.dwe.Service.<init>(Service.java:456)

and API Manager won't be available.

-- This happens because by default, systemd considers cassandra is started as soon as the start command is launched (and not when it is really available)

Resolution

- In order to check cassandra availability, it is possible to use the nodetool status command:

nodetool -h hostname status

- The return code of this command will be 0 if cassandra is available, higher than 0 otherwise.

- That's possible to wait for cassandra availability in the cassandra systemd unit file by using the following command:

/bin/bash -c "status=1; while ((status!=0)); do /path/to/cassandra/bin/nodetool -h localhost status; status=$(echo $?); done"

in the ExecStartPost directive of the service.