Problem

The system being upgrade contains an ActiveMQ broker in one group with another group being a client to this service.

During the "sysupgrade apply" step of the ActiveMQ client node a generic sysupgrade error occurs :-

Node Manager started.
Starting gateway HOST for group GROUPNAME

Data apply (create topology, import data) failed
------------------------------------------------
ERROR: com.vordel.api.nm.NodeManagerAPIException: Node Manager error: error occurred on call from Node Manager on 'HOST' to https://HOST:8085' for URL '/api/configuration' and 'PUT'
.
HTTP status: 500.
Error code: 102 

Looking at the last trace file for the new instance being created we see that it's blocked trying to connect to JMS because the last line doesn't change after a long wait.

INFO    22/Nov/2020:13:05:56.073 [75a1:e653ba5f0200aa2d85d76ebd]         JMS service: max number threads: 20
INFO    22/Nov/2020:13:05:56.077 [75a1:e653ba5f0200aa2d85d76ebd]         JMS service: com.vordel.jms.apache.activemq.InitialContextFactory => failover://(ssl://amq-server1:61616,ssl://amq-server2:61616)
DEBUG   22/Nov/2020:13:05:56.079 [75a1:e653ba5f0200aa2d85d76ebd]         Executing reconnect to JMS Service GrpName
INFO    22/Nov/2020:13:05:56.142 [75a1:e653ba5f0200aa2d85d76ebd]         Connecting to JMS service GrpName with no credentials
INFO    22/Nov/2020:13:05:56.283 [75a1:e653ba5f0200aa2d85d76ebd]         Starting JMS connection to GrpName

Resolution

This was solved by adding a timeout parameter to the ActiveMQ JMS URL:-

Edit the project via Policy Studio and enter the connection string as below

failover://(ssl://amq-server1:61616,ssl://amq-server2:61616)?initialReconnectDelay=2000&maxReconnectAttempts=2

This new fed will need to be placed in the upgrade location for the group that failed to deploy

/<New Install dir>/upgrade/bin/out/upgrade/esgroups/groups/group-N/

Re-run the sysupgrade apply with --force which should now succeed because of the connection timeout.

For further information about the Active MQ failover timeouts see https://activemq.apache.org/failover-transport-reference.html

also for TCP transport http://activemq.apache.org/tcp-transport-reference