Topology SSL Error After EMT Upgrade

Problem

After upgrading API Gateway / API Manager in EMT mode, the following SSL-related errors appear in the logs:

ERROR 31/Jul/2025:11:32:45.393 [01b3:000000000000000000000000] [SSL alert read 0x22e, 0x2e]: certificate unknown [fatal] { subject: /DC=host-1/OU=emt_anm_group/CN=nodemanager-1, thumbprint: 8A:79:ED:72:61:A5:C6:B5:70:D2:0A:66:F7:D6:74:2B:BC:D6:07:AF }.
ERROR 31/Jul/2025:11:32:45.393 [01b3:000000000000000000000000] [SSL_accept, 0x2e]: error - error.
ERROR 31/Jul/2025:11:32:45.393 [01b3:000000000000000000000000] error handling connection: SSL protocol error
error:0A000416:SSL routines::sslv3 alert certificate unknown, source location: ssl/record/rec_layer_s3.c:1605 ssl3_read_bytes, additional info: SSL alert number 46

These errors typically indicate that the node manager's certificate is not trusted or mismatched during inter-node communication.

Resolution

This issue is commonly caused when a custom FED is used for the Admin Node Manager (ANM), and its certificate differs from the domain certificate bundled with the container image.

The recommended resolution is to use your own domain Certificate Authority (CA). EMT-mode API Gateway supports injecting custom domain certificates for containerized deployments.

Prerequisites

1. PVC and Mount Setup

• The following Persistent Volume Claims (PVCs) should be present:
  • PVC for the Admin node manager deployment (apigateway-anm)
  • PVC for the traffic and apimgr deployments (apigateway-apitraffic, apigateway-apimgr)
• Mount each PVC under the /merge directory.
• Within each mounted directory, create the following structure:

apigateway/
└── groups/
    └── certs/
        ├── domaincert.pem
        └── private/
            └── domainkey.pem

Documentation: Mount a Component Configuration | Update Domain Certificates at Runtime

2. Domain CA Generation

Use the following script to generate a self-signed domain CA certificate and encrypted private key:

#!/bin/bash
# Output files
KEY_FILE="domainkey.pem"
CERT_FILE="domaincert.pem"
# Generate encrypted private key
openssl genrsa -aes256 -out "$KEY_FILE" 2048
# Create a temporary config file with extensions
cat > ca_openssl.cnf <<EOF
[ req ]
distinguished_name = req_distinguished_name
x509_extensions = v3_ca
prompt = no
[ req_distinguished_name ]
CN = DefaultDomain
[ v3_ca ]
basicConstraints = critical, CA:TRUE, pathlen:0
keyUsage = critical, Digital Signature, Key Encipherment, Data Encipherment, Key Agreement, Certificate Sign
EOF
# Generate the self-signed certificate with the config
openssl req -new -x509 -days 3650 -key "$KEY_FILE" \
  -out "$CERT_FILE" -config ca_openssl.cnf
# Cleanup
rm ca_openssl.cnf
echo "Done. Files generated:"
echo "  Private Key: $KEY_FILE"
echo "  Certificate: $CERT_FILE"

Note: You will be prompted for the passphrase three times — use the same password each time.

3. Helm Configuration

In your values.yaml for anm, apimgr, and apitraffic, set the following environment variable:

extraEnvVars:
  - name: DOMAIN_KEY_PASSPHRASE
    value: "SECRET123"

Replace SECRET123 with the passphrase you used when generating domainkey.pem.