Problem

When updating API Manager to 7.7.20230530 (May 23) or later there is an issue for systems where the Cassandra tables were originally created in version 7.5.2 or earlier. The problem is a failure to add a new column to a table when the API Manager instance is restarted after running the update script...

INFO 23/Nov/2023:17:11:40.721 [3bba:000000000000000000000000] Loading KPS configuration.
INFO 23/Nov/2023:17:11:41.716 [3bba:000000000000000000000000] Using existing Cassandra keyspace: x109978f7_5230_4046_95ea_a02d697eef1a_group_2
ERROR 23/Nov/2023:17:11:41.780 [3bba:000000000000000000000000] Error initializing Cassandra schemas:
com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on apim-cassdb1-dev/172.16.200.168:9042: java.lang.AssertionError
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:62)
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:26)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
at com.datastax.driver.core.ChainedResultSetFuture.getUninterruptibly(ChainedResultSetFuture.java:59)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:58)
at com.axway.apigw.cassandra.impl.datastax.kps.BasicDdlOperationsImpl.addColumn(BasicDdlOperationsImpl.java:213)
at com.vordel.kps.storeImpl.cassandra.CassandraSchema.addColumn(CassandraSchema.java:151)
at com.vordel.kps.storeImpl.cassandra.CassandraSchema.updateColumnFamily(CassandraSchema.java:131)
at com.vordel.kps.storeImpl.cassandra.CassandraSchema.initSchemaForStore(CassandraSchema.java:109)
at java.base/java.util.ArrayList.forEach(Unknown Source)
at com.vordel.kps.storeImpl.cassandra.CassandraSchema.initSchemas(CassandraSchema.java:48)
at com.vordel.kps.impl.KPS.initialiseCassandraSchema(KPS.java:245)
at com.vordel.kps.impl.KPS.load(KPS.java:219)
at com.vordel.kps.impl.KPS.configure(KPS.java:179)
at com.vordel.precipitate.SolutionPack$ConfigModule.configure(SolutionPack.java:317)
at com.vordel.precipitate.SolutionPack.loadModules(SolutionPack.java:437)
at com.vordel.dwe.Service.refresh(Service.java:582)
at com.vordel.dwe.Service.<init>(Service.java:483)

and this turns out to be because 7.5.2 creates tables with COMPACT STORAGE. We used an older Hector driver with Thrift support back in 7.5.2 and successive API Gateway upgrades/updates doesn't remove COMPACT STORAGE. However it's only a problem at 7.7.20230530 or after with Cassandra 3 or 4.

To check if the API Manager tables have COMPACT STORAGE, search the output of "cqlsh desc keyspace <keyspace-name>" and look for COMPACT STORAGE as below

CREATE TABLE x109978f7_5230_4046_95ea_a02d697eef1a_group_2.api_portal_portaluserstoreldap (
    key text PRIMARY KEY,
    cn text,
    "createTimestamp" text,
    description text,
    id text,
    "jpegPhoto" text,
    "loginName" text,
    mail text,
    mobile text,
    o text,
    sn text,
    "telephoneNumber" text,
    "userDn" text,
    "userPassword" text,
    "userPasswordType" text
) WITH COMPACT STORAGE 

Resolution with dsbulk and the recommended approach for zero downtime

Follow the steps in the attached document "How to Upgrade Cassandra Schema with DSBULK"

Resolution with kpsadmin backup and restore

1) In kpsadmin, backup the API Manager group (option 26); note the UUID used in filenames and only continue if no errors

2) Verify backup - restore the keyspace in a test environment; verify everything API Manager is running ok

3). Take a cassandra backup and test the restore just in case (Cassandra backup and restore documentation)

4). Stop the API Manager instance using the keyspace

5. a) in cqlsh> drop keyspace KEYSPACE-NAME

(note that an auto snapshot is taken this can be removed with nodetool clearsnapshot)

OR

b) In API Gateway configuration (normally with Policy Studio) change the keyspace name from x${DOMAINID}_${GROUPID} to some unique name

6. Start API Gateway instances (check logs - tables should be recreated with trace log as below)

INFO    23/Jan/2024:13:33:58.498 [7710:000000000000000000000000]     Loading KPS configuration.
INFO    23/Jan/2024:13:34:01.863 [7710:000000000000000000000000]         Created Cassandra keyspace: apim_new_keyspace_name. Replication factor: 1
INFO    23/Jan/2024:13:34:01.864 [7710:000000000000000000000000]         Creating table: api_portal_portalconfigstore
INFO    23/Jan/2024:13:34:02.527 [7710:000000000000000000000000]         Creating table: api_portal_portaluserstore

5. Clear KPS group in kpsadmin (option 25)

6. Restore KPS group in kpsadmin with UUID from 1. (option 27)

Create a new schema without COMPACT STORAGE

This is an alternative to having API Manager startup create the new 7.7 schema. We get a copy of the old schema, modify it appropriately removing COMPACT STORAGE and apply any other 7.7 changes. Also other recommendations from Axway Support or Datastax could be applied at this time.

1. Copy the schema from existing keyspace. The old keyspace name can be found in kpsadmin (option 30)

e.g.

cqlsh -e "DESCRIBE KEYSPACE x109978f7_5230_4046_95ea_a02d697eef1a_group_2" > keyspace_source.txt

2. Process this schema with sed to
- remove compact storage
- change to a new keyspace name
- adjust other table props to align with 7.7
- add any other recommendations here to commands.txt

sed -f commands.txt keyspace_source.txt > keyspace_source_77_without_compact_storage.txt
cat commands.txt
s/) WITH COMPACT STORAGE/)/g
s/AND bloom_filter_fp_chance/WITH bloom_filter_fp_chance/g
s/x109978f7_5230_4046_95ea_a02d697eef1a_group_2/apim_new_keyspace_name/g
s/read_repair_chance = 1.0/read_repair_chance = 0.0/g
s/speculative_retry = 'NONE'/speculative_retry = '99PERCENTILE'/g
/CLUSTERING ORDER BY/d

3. apply new schema to cassandra DB

./cqlsh -f keyspace_source_77_without_compact_storage.txt

4. Check new schema with cqlsh desc keyspace

5. Make sure the API Manager instance keyspace name is set to apim_new_keyspace_name in Policy Studio.

6. Copy data with either dsbulk or kpsadmin restore (see above)

7. Deploy and reload with the new keyspace - test API manager on the new keyspace