Changes

⚠ Acknowledging a fluid situation (3 vulnerabilities reported in the first 8 days and various changes with regards to mitigation options) we are adding this new section with important Changes made to this KB to facilitate the monitoring and decisions.

2022-03-10 Noted that log4j 2.17.1 is now part of the February 2022 release.

2022-01-25 Added links to log4j 2.17.1 patches, removed the 2.17 ones

2022-01-19 Adjusted statement and recommendation with regards to Client Tools (Policy Studio, Configuration Studio)

2022-01-05 Added details about CVE-2021-44832 and position regarding newer log4j versions in the "Context" section

2021-12-23 Added procedure for Mitigation Option 3 in EMT environments

2021-12-22 Added "Changes" section, details about CVE-2021-45105 and patch procedure in EMT environments

2021-12-22 Adjusted the title and "Context" section

2021-12-21 Added links to log4j 2.17 patches, removed the 2.16 ones

2021-12-16 Added links to log4j 2.16 patches, removed the 2.15 ones

2021-12-15 Marked Mitigation option #1 as insufficient

2021-12-13 Marked Mitigation option #2 as not effective and removed its content

2021-12-12 Added links to log4j 2.15 patches

2021-12-11 Published this KB referring to initial CVE-2021-44228

Context

The current article intends to provide recommendations and technical clarifications with regards to the impact of log4j vulnerabilities in API Gateway, Manager, Portal. As new conclusions and recommendations are available we will be updating this KB for API related products and https://support.axway.com/news/1331/lang/en for all products.

CVE-2021-44228 (base CVSS Score = 10)

A 0-day vulnerability in the popular Java logging library, log4j, was published on Dec 10th along with a POC available on GitHub that shows the possibility of Remote Code Execution (RCE) if log4j logs an attacker-controlled string value.

This vulnerability is solved in log4j 2.15 which is included in the patches released by Axway on Dec 12th.

Alternatively, this vulnerability can best be mitigated by applying the Mitigation Option #3 described below.

CVE-2021-45046 (base CVSS Score = 9)

This vulnerability was published on Dec 14th showing that the fix provided in Log4j 2.15 was incomplete in certain non-default configurations.

This vulnerability is solved in log4j 2.16 which is included in the patches released by Axway on Dec 16th.

Alternatively, this vulnerability can best be mitigated by applying the Mitigation Option #3 described below.

CVE-2021-45105 (base CVSS Score = 5.9)

This third vulnerability was published on Dec 17th showing that a DOS (Denial of Service) attack was possible even in log4j 2.16 when the logging configuration uses a non-default Pattern Layout with a Context Lookup - for example, $${ctx:loginId}. Its initial CVSS score of 7.5 was later adjusted (on Dec 30th) to 5.9.

This vulnerability is solved in log4j 2.17 which is included in the patches released by Axway on Dec 21st.

After the analysis of this CVE Axway concluded that recent API releases (7.7, 7.6 SP 1 or newer) do not have config files containing the "ctx" context lookup pattern and, therefore, default configurations are not vulnerable. Older releases do contain one config file with the "ctx" pattern however it does not originate from user input.

The conclusion is that a default API Gateway installation could be vulnerable to this CVE only if the attacker has access to the topology case in which they could do even worse damage if that was the case. Axway considers that the same mitigations the customer takes to secure their topology can be used to to ensure they're not vulnerable to this DoS attack.

As a general approach, mitigation options for this vulnerability are different compared with the other log4j ones and consist in:

1. Make sure there is no use of Context Lookups (the ctx: syntax)
2. If this is used then:

• remove it or
• replace it with safe syntax e.g. %X, %mdc or %MDC or
• ensure that the input for the lookup e.g. ${ctx:username} is not user-controlled

CVE-2021-44832 (base CVSS Score = 6.6)

This fourth vulnerability against log4j was published on Dec 28th showing that log4j versions up to 2.17 are vulnerable to a remote code execution attack if an attacker has permissions to modify the log4j configuration file to construct a malicious configuration.

When identifying the log4j configuration file one should consider:

• "log4j.xml" for API Gateway versions up to and including 7.7.20200330
• "log4j2.yaml", "topologyLog.yaml", "eventLog.yaml" and "openTrafficLog.yaml" for API Gateway versions 7.7.20200530 or higher.

Axway analyzed this vulnerability and considers that a default configuration of API Gateway is not vulnerable and that the file system / environment would need to be compromised for any risk to be exposed.

One mitigation option is to review configuration and secure the access to the log4j configuration files while also making sure the system is not pulling a remote log4j configuration file.

This vulnerability is fixed in log4j 2.17.1 and Axway will release patches with log4j 2.17.1 (or an eventual newer version, if this will be the case) for the supported versions of API Gateway (7.7.20201130 - 7.7.20211130) by January 28th (or sooner if a higher score CVE is identified meanwhile).

Additionally, Axway is actively monitoring the log4j news and vulnerabilities and plans to continue patching the supported versions of API Gateway if other relevant log4j vulnerabilities are identified. The CVSS score of vulnerabilities will be taken into account when deciding the patch and delivery date.

The points below are relevant for the first 2 vulnerabilities (CVE-2021-44228 & CVE-2021-45046) and do not apply to the other ones (CVE-2021-45105 & CVE-2021-44832)

Impacted products

The impact derives from the use of Apache log4j within the products and all log4j versions between and including 2.0 and 2.14.1 are impacted. Some variations in impact exist based on the exact log4j and JRE version.

• API Portal (all versions): API Portal is not affected by these vulnerabilities as log4j is not being used.
• Cassandra (versions 2.2.8, 2.2.12, 2.2.19, or 3.11.11): Cassandra versions delivered with API products are not impacted by these vulnerabilities and are safe. Cassandra does not use log4j.
• Policy Studio: While Client Tools do not run as a service and, therefore, not directly exposed to this type of vulnerabilities Axway is currently reviewing a theoretical impact of log4j vulnerabilities on Policy Studio and Configuration Studio. Recommended actions and extra details are presented in Article ID #182001.
• API Gateway/Manager: impact confirmed for some versions. See below the results of our investigation and recommendations:
• API Gateway Analytics: API Gateway Analytics has been investigated and the recommendation is to apply the same solution as for API Gateway.

3rd party products such as AppDynamics used in the context of API Gateway may be vulnerable and need to be upgraded or mitigated as advised by their vendors. If issue is not fixed or mitigated in such 3rd party product, it may cause API Gateway to be vulnerable even if issue has been patched or mitigated in API Gateway.

Resolution

Permanent solution: use log4j version 2.17.1 or higher

On January 25th Axway has released patches with log4j 2.17.1 for APIM versions between 7.7.20201130 (Nov 20) and 7.7.20211130 (Nov 21). Users using these versions are advised to update and use the new patches in which the 4 CVEs (CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 and CVE-2021-44832) are fixed. Please find below direct links to new patches:

• Patch for 7.7.20211130 (November 21): API Gateway 7.7.20211130 Patch 26337 (allOS)
• Patch for 7.7.20210830 (August 21): API Gateway 7.7.20210830 Patch 26338 (allOS)
• Patch for 7.7.20210530 (May 21): API Gateway 7.7.20210530 Patch 26339 (allOS)
• Patch for 7.7.20210330 (March 21): API Gateway 7.7.20210330 Patch 26340 (allOS)
• Patch for 7.7.20201130 (November 20): API Gateway 7.7.20201130 Patch 26341 (allOS)

For API Gateway Analytics installations the same patches provided for APIM above should be used and installed in the "analytics" directory. For example:

tar xvzf patch_name.tgz -C INSTALL_DIR/analytics

The instruction to remove the jar from "apigateway" should also be applied to the "analytics" installation: remove INSTALL_DIR/analytics/system/lib/modules/log4j-slf4j-impl-2.13.2.jar

Procedure for EMT environments:

Axway tested and proposes the following procedure for EMT environments.

Note: This change must be removed when upgrading to February release or later

EMT Patch Instructions:

1. Create a "patch" directory and extract the patch into it.
2. Navigate to ./apigw-emt-scripts/Dockerfiles/gateway-base/
3. Edit the Dockerfile. Below the line

RUN chmod a+x /opt/runInstall.sh && opt/runInstall.sh /opt/APIGateway_Install.run
    

add the following

RUN rm -rf /opt/Axway/apigateway/system/lib/modules/log4j-slf4j-impl-2.13.2.jar \
           /opt/Axway/apigateway/system/lib/modules/log4j-1.2-api-2.13.2.jar \
           /opt/Axway/apigateway/system/lib/modules/log4j-api-2.13.2.jar \
           /opt/Axway/apigateway/system/lib/modules/log4j-core-2.13.2.jar \
           /opt/Axway/apigateway/system/lib/modules/disruptor-3.4.2.jar
COPY {patch_dir_path} /opt/Axway/apigateway/
    

4. From the ./apigw-emt-scripts directory run ./build_base_image.py with your usual parameters

For analytics:

1. Navigate to ./apigw-emt-scripts/Dockerfiles/emt-analytics/
2. Edit the Dockerfile. Below the line:

RUN chmod a+x /opt/runInstall.sh && opt/runInstall.sh /opt/APIGateway_Install.run

add the following

RUN rm -rf /opt/Axway/analytics/system/lib/modules/log4j-slf4j-impl-2.13.2.jar \
           /opt/Axway/analytics/system/lib/modules/log4j-1.2-api-2.13.2.jar \
           /opt/Axway/analytics/system/lib/modules/log4j-api-2.13.2.jar \
           /opt/Axway/analytics/system/lib/modules/log4j-core-2.13.2.jar \
           /opt/Axway/analytics/system/lib/modules/disruptor-3.4.2.jar
COPY {patch_dir_path} /opt/Axway/analytics/

From the ./apigw-emt-scripts directory run ./build_aga_image.py with your usual parameters.

For versions older than 7.7.20201130 (Nov 20) one can rely on Mitigation Option 3 that completely addresses both vulnerabilities (CVE-2021-44228 and CVE-2021-45046).

Mitigation Option 1: log4j.formatMsgNoLookups=true

Recent articles showed that this option may be insufficient in some contexts where message lookups could still occur. This is no longer a recommended option.

If using log4j v2.10 or higher one can set the property log4j2.formatMsgNoLookups=true in jvm.xml file. Axway tested this option and confirms it is effective against currently known attacks. The steps below apply to both APIM and Analytics installations and should be applied in each such installation:

1. Identify the jvm.xml location. Possible locations of jvm.xml:
   For APIM installations:

apigateway/conf/jvm.xml (recommended) (jvm.xml file to be created). Effective at instance and NM level, inherited by new instances and preserved during update. A jvm.xml sample is attached to this KB.

apigateway/groups/[group X]/[instance Y]/conf/jvm.xml - effective at instance level but new instances and NM will not inherit these parameters.

apigateway/system/conf/jvm.xml - effective at instance level and NM level, used also for new instances but it is replaced when applying an update.

For API Gateway Analytics the location of the jvm.xml is: analytics/system/conf/jvm.xml

2. Edit jvm.xml and adjust or add <VMArg name="-Dlog4j2.formatMsgNoLookups=true"/> so that jvm.xml contains a structure like:

<ConfigurationFragment>
<VMArg name="-Dlog4j2.formatMsgNoLookups=true"/>
</ConfigurationFragment>
    

3. Restart the product after adjusting the jvm.xml.

To confirm that the modification has been taken into account after the restart one can use jcmd and the following approach to verify:

Retrieve the process id (pid) using a command like:

ps -eaf | grep -i vshell | grep -v grep

Navigate to jvm.xml file location and check that log4j2.formatMsgNoLookups=true is present in the file:

grep -Ri format jvm.xml

should display:
<VMArg name="-Dlog4j2.formatMsgNoLookups=true"/>

Download openjdk8 and within it use the following:

cd /path/openjdk8/jdk8.0.265-linux_x64/bin
./jcmd pid  VM.system_properties | grep -i format

Output of this command should display:
log4j2.formatMsgNoLookups=true

This means that the property is taken into account.

Mitigation Option 2: %m{nolookups}

Mitigation Option #2 based on pattern layout has been removed from the list of possible solutions following test results showing the solution is not effective.

Mitigation Option 3: removal of JndiLookup and JndiManager classes from classpath

This is the safest mitigation option available as it removes completely the jndi class providing protection against both CVE-2021-44228 and CVE-2021-45046. This is the recommended option to mitigate these 2 log4j related vulnerabilities.

This mitigation option is based on removing below classes from log4j-core jar.

This has been tested by Axway in the base configuration without encountering issues and Axway confirms it is effective against currently known attacks. However, removal of the JndiManager will cause the JndiContextSelector and JMSAppender to no longer function and impact on service availability may be encountered due to custom code or configuration therefore this option should be thoroughly tested on customer side.

This option can be implemented on any Service Pack level but needs to be re-applied after installing a new Service Pack/Update/Patch as the adjusted log4j-core-*.jar can be replaced during the update.

The steps below apply to both APIM and Analytics installations and should be applied in each such installation:

1. Identify the log4j-core-*jar file within your installation using the following command:
   

   For APIM: cd ../apigateway/ 
   For API Gateway Analytics: cd ../analytics/
   find ./ -name log4j-core-*jar
       

2. Backup log4j-core-*jar by creating a copy in a folder outside APIM or Analytics classpath.
3. Remove the JndiLookup and JndiManager classes from the log4j-core-*jar. For example, you can run a command like:

   zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
   zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/net/JndiManager.class
       

4. Restart Node Manager, API Gateway instance and or API Gateway Analytics after the change is performed.

EMT environments

This mitigation option can be implemented in an EMT environment using the approach below. Note that these commands need zip installed in the Docker image:

1. find ./ -name log4j-core-*jar

Note the path that is returned, e.g. /opt/Axway/apigateway/system/lib/modules/log4j-core-2.13.2.jar.

2. Modify the Dockerfile as follows: after

RUN chmod a+x /opt/runInstall.sh && opt/runInstall.sh /opt/APIGateway_Install.run
    

add:

RUN zip -q -d <PATH> org/apache/logging/log4j/core/lookup/JndiLookup.class
RUN zip -q -d <PATH> org/apache/logging/log4j/core/net/JndiManager.class