Problem

The deployment of the configuration is no longer possible, nor from Policy Studio to ANM and instance, neither using deployment scripts (Jenkins, SVN)

Errors thrown in Policy Studio:

HTTP status: 500.

Error code: 102.

Errors thrown in NM trace are:

ERROR 01/Jan/2020:00:00:00.000 [0000:000000000000000000000000] java exception:

com.vordel.api.nm.NodeManagerAPIException: Node Manager error: error occurred on call from Node Manager on 'server1' to https://server2:8090' for URL '/api/router/service/instance-4/api/configuration' and 'PUT'

Errors thrown in Jenkins log

00:00:00 TASK [Fed-Deployment] **********************************************************

00:00:00 fatal: [server1]: FAILED! => {"changed": true, "cmd": "/apigateway/posix/bin/managedomain --deploy -g GROUP -n Node01 --archive_filename /tmp/fed_file.fed --username user --password pass", "delta": "0:02:09.677678", "end": "2020-01-00 00:00:00.000000", "msg": "non-zero return code", "rc": 1, "start": "2020-01-00 00:00:00.000000", "stderr": "Failed to deploy to API Gateway 'Node01'.\nNode Manager error: error occurred on call from Node Manager on 'server1' to https://server1:8085' for URL '/api/configuration' and 'PUT'\n. \nHTTP status: 500. \nError code: 102.", "stderr_lines": ["Failed to deploy to API Gateway 'Node01'.", "Node Manager error: error occurred on call from Node Manager on 'server1' to https://server1:8085' for URL '/api/configuration' and 'PUT'", ". ", "HTTP status: 500. ", "Error code: 102."], "stdout": "Loading archive 'f65caa44-f3cb-48b0-b33d-63c8a2e55644' to hosts running selected API Gateways...\nLoaded archive 'ffffffff-ffff-0000-0000-000000000000' to hosts running group.\nDeploying to API Gateway 'Node01'...", "stdout_lines": ["Loading archive 'ffffffff-ffff-0000-0000-000000000000' to hosts running selected API Gateways...", "Loaded archive 'ffffffff-ffff-0000-0000-000000000000' to hosts running group.", "Deploying to API Gateway 'Node01'..."]}

See that in complex topologies the errors may occur from ANM to NM or from NM to its instance on the same host or remote.

CAUSE

During the normal development of a configuration instance the size of the *.fed gets bigger and bigger than initial size with the time, and it may take +500Kb or +1Mb. Thus, the time to deploy from one entity to another increases and it clashes with

- Active Timeout (default 60000 ms = 1 min)

- Transaction Timeout (default 240000 ms = 4 min)

already set for API Gateway NM and instances.

Resolution

For all ANM, NM and instances the Active Timeout and Transaction Timeout must be increased to something longer than the deployment time.

As the deployment of the changed configuration is no longer possible, you could not use deployment from Policy Studio to address this. You have to change directly the configuration files.

The usual way is:

- backup the configuration files you want to change;

- stop all NM and instances on the node you change;

- change the values;

- start the NM and instances on the node;

- continue will all the nodes;

- retest by deploying.

Files and values to change:

== for each NM

- check if PrimaryStore-[long hash].xml (when using NM registered in ANM) is so or simply PrimaryStore.xml exist, then search for:

maxTransTimeout

activetimeout

[user@server ~]$ vi /apigateway/conf/fed/PrimaryStore.xml

[user@server ~]$ vi /apigateway/conf/fed/PrimaryStore-[hash].xml

<fval name="activetimeout"><value>240000</value></fval>

<fval name="dateFormat"><value>MM.dd.yyyy HH:mm:ss,SSS</value></fval>

<fval name="exportCorrelationId"><value>true</value></fval>

<fval name="externalDataCacheRefreshIntervalSecs"><value>5</value></fval>

<fval name="idletimeout"><value>15000</value></fval>

<fval name="inputEncodings"><value>.inherit</value></fval>

<fval name="ldapServiceProvider"><value>com.sun.jndi.ldap.LdapCtxFactory</value></fval>

<fval name="maxInputLen"><value>20971520</value></fval>

<fval name="maxOutputLen"><value>20971520</value></fval>

<fval name="maxTransTimeout"><value>2400000</value></fval>

== for each instance

activetimeout

grep -i timeout app/753/api/apigateway/groups/group-9/instance-16/conf/mgmt.xml

<SSLInterface activetimeout="60000" address="localhost" auditConn="1" backlog="64" ciphers="FIPS:!SSLv3:!aNULL" clientAuth="required" depth="1" dhParams="MIICCAKCA

....

QI=" enabled="1" idletimeout="60000" inputEncodings=".inherit" maxRequestMemory="16777216" name="Internal Management HTTPS Interface" opsSettingsUsage="PORT" outputEncodings=".inherit" port="${env.PORT.MANAGEMENT}" protocol="any" recordCircuitPath="0" recordInboundTxns="0" recordOutboundTxns="0" recordTrace="0" resolveSubjectCNtoNetAddr="0" reuseAddress="false" tracelevel="INHERIT" transparentProxy="false">

maxTransTimeout

grep -R "240000" app/762main/a/apigateway/groups/group-2/

/apigateway/groups/group-2/conf/9405fa86-b30a-45aa-83e6-63f612c54985/PrimaryStore.xml: <fval name="maxTransTimeout"><value>2400000</value></fval>

The ESID is found as follows:

The Entity Store ID (esid) is found by looking at esid.xml:

cat app/753/api/apigateway/groups/group-2/instance-12/conf/esid.xml

<?xml version="1.0" encoding="UTF-8"?>

<ConfigurationFragment>

<set property="esid" value="9405fa86-b30a-45aa-83e6-63f612c54985"/>

</ConfigurationFragment>