Problem

Whenever a transfer goes into VVTIMO, it will continue to get a VVTIMO and all transfers start to pile up

Error: CFTH23E NPART=[partner] rejected EVENT=VVTIMO

What is the meaning of VVTIMO

Issuing a CFTUTIL command just stores the command in the COM and it will never get executed

By recreating the COM and CAT files it seemed to solved the issue but it started again short after.

Resolution

The message "CFTH23E NPART=[partner] rejected EVENT=VVTIMO" is added in log by CFTTPRO when CFTMAIN process does not respond (VVTIMO meaning timeout).

CFTMAIN does not handle the command in CFTCOM either.

In fact even before the first CFTH23E message, we can see the following messages:

CFTW09I CFTSEND My_IDF <IDTU=E0000ZS0 PART=My_Part IDF=My_iDF IDT=K2206320
CFTW09I+NIDF=My_IDF>

CFTT82E transfer aborted <IDTU=E0000ZS0 PART=My_Part 240 RTO>

CFTW09I is displayed by CFTMAIN and CFTT82E is displayed 4 minutes and 20s later by CFTTFIL.

From there, we cannot tell exactly what happened nor why.

When the issue occurred, it looks like CFTMAIN process was blocked but there is not enough information in the cft_support to analyze the root cause.

It can be due to a file access on a NFS disk, but it does not seem to be used.

Most likely, a deadlock occurred between CFTMAIN and another process.

When the issue occurs, when the message CFTH23E is seen in CFTLOG, create a cft_support archive before stopping CFT.

Then to understand where CFTMAIN is blocked, use this command:

kill -s SIGSEGV $(pidof CFTMAIN)

This will generate a core that will have to be provided for the Root Cause Analysis.