Problem

This article will provide insight about the purpose and functions of the .m_inproc files in ST. Mainly, the following points will be covered:

• What is the purpose of the .m_inproc files?
• What is the behavior of SecureTransport in the presence of these files?
• How do SSH/FTP clients handle these files?
• What are the issues the .m_inproc files could cause?

Resolution

The .m_inproc files are the way for ST logic to assure that any events related to file processing post-upload are kicked in, only after the file is completely uploaded. In essence, the .m_inproc files can be considered "lock files" - they lock the original file that is being uploaded and prohibit any other action with it. The several use cases below should clarify this short explanation:

The .m_inproc files are used to lock any incomplete files from being accessed by anyone else but the user(s) that are still performing any actions on them. In short, as long as there is an .m_inproc file associated to a file on the storage, the latter will be not visible for anyone, but for admins and the user uploaded it.

The .m_inproc files prevent any Post Transfer Actions (PTA) from being performed.

The .m_inproc files are also used in a cluster environment with shared storage, to assure that only the node that is used for uploading the original file, has access to it for any operations.

Consider the following real use case example:

1. Two ST users - Alice and Bob are subscribed to a Shared Folder application which allows them to share files with each other.
2. Alice begins uploading a file to the "Shared" directory.
3. During the upload an .m_inproc file is created in the upload directory and the original file is invisible to Bob. Alice, however, is still able to see the file and perform actions to it, including delete.
4. The .m_inproc file remains in the directory until the transfer is completed. Once the transfer finishes, the .m_inproc file is deleted from the folder, which makes the original file visible to Bob.

The .m_inproc files are supposed to be removed by ST automatically when they had fulfilled their purpose. In case the files are not removed by ST for some reason, the following effects will be observed:

All uploaded files that have a leftover .m_inproc file associated, will be reported as "In Progress" in the File Tracking page of Admin UI. Whether the actual uploads were successful or not will not change this - it's the sole presence of the .m_inproc file that will trigger the "In Progress" status in the UI.

No PTAs will be executed on successfully uploaded files that hasleftover .m_inproc files associated with them. Effectively this will prevent any post-upload processing from being executed.

The above two effects can be mitigated by manually removing the leftover .m_inproc files from the disk AFTER confirming that the files were properly uploaded. The ways to do this is to either verify their MD5 checksum with the checksum of the source file (in case you have access to it) or by monitoring their size on the file system for a short amount of time and verifying that it doesn't increase (which would mean that the file is no longer being uploaded).

Removing the lefover .m_inproc files will NOT trigger the PTAs or post-upload processing. In most cases you can use the Resubmit functionality, but the safest way to retry the entire flow is to have the original files re-uploaded to ST.

The possible reasons for .m_inproc files being left after a successful upload are generally related to file system locks or file system cache inconsistencies. Such issues are very often observed with NFS file systems. Disabling caching/locking on the file system will help with the .m_inproc leftover files, but this will sacrifice storage performance. Practical tests with ST uploads and consultation with the storage administrator can help finding the best configuration values for the storage.

Another very important consequence of the usage of the .m_inproc files in ST is the inability to rename uploaded files within the actual upload session. In other words, a user will not be able to upload a file and immediately rename it, the .m_inproc file will prevent the action. What the user should do is to send the rename request several seconds after the upload had completed, thus giving time to ST to remove the lock.

This inability to rename uploaded files can run into conflict with some clients and servers. Legacy expectations from "traditional FTP" servers forced renames in FTP scripts from partners to ensure partial files were not received by internal polling applications. Such "traditional" servers supported only client actions with the files – a client uploads and an internal client polls for files. There was no automation and no event driven actions, and the renames were the way to ensure that one side did not pull a file that the other side had not completed writing.

As SecureTransport is an event-driven application and possesses different methods of assuring data integrity, and since renames (even renames in the same session) could trigger consequent actions or even lead to problems with automated actions set on application level, renames were disabled as of ST 4.9.2 SP2 and later.

In the real world, the lack of renaming affects mostly automations which use customized upload scripts that depend on renaming, or file transfer clients that use renaming as core function. One such client is WinSCP, which by default uploads the file with a temporary name and sets the original name once the upload is completed successfully. If WinSCP is configured to disable this default behavior, uploads to ST should succeed. Axway KB 67579 describes how to disable client-side renaming for WinSCP.