This article discusses the maintenance and export procedures of the internal MySQL database of SecureTransport, for the Core(Backend) and Edge Servers. This includes how to setup and tune the export and purge of old data from the database, monitoring and configuring the purge of old exported logs, and the considerations regarding database size, export logs storage for avoiding out-of-space storage issues.

Log management in the SecureTransport Core Server

In a default SecureTransport installation, the logs are kept in the internal MySQL database and <FILEDRIVEHOME>/var/logs folder. During the database rotation and data export process, the exported data is written as flat text file logs, in the default directory <FILEDRIVEHOME>/var/db/hist.

In the SecureTransport Core(Backend) Server this is achieved by several Applications managed by the Transaction Manager:

• Log entry Maintenance Application, which exports the Server logs from the database to the <FILEDRIVEHOME>/var/db/hist/log-entry folder
• Audit log Maintenance Application, which exports the Audit logs from the database to the <FILEDRIVEHOME>/var/db/hist/audit-log folder
• Transfer log Maintenance Application, which exports the Transfer logs from the database to the <FILEDRIVEHOME>/var/db/hist/transfer-log folder

These applications should be managed carefully and fine-tuned to prevent database bloat or export logs directories from consuming too much storage space. The critical values for fine-tuning are "Delete transfer log when*: X days old" and "Delete exported files when data is: X days old". The default values in a fresh ST installation are 30 days and 180 days. However, for an ST environment that faces a considerable amount of traffic and thus generates a lot of Transfer and Server logs, these values could be reduced, to keep the database at a reasonable size and to prevent the exported logs from accumulating.

For example, a reduction of "Delete transfer log when*: X days old" from the default 30 days to 10 days or 5 days, means that any log data older than 10 days would be exported to flat logs and then deleted from the database - thus keeping it trim. Another example is changing the "Delete exported files when data is: X days old" from the default 180 days (6 months) to 60 days (2 months). This means that instead of keeping 6 months of Transfer logs or Server logs, the system will keep only 2 months' worth of archive and delete anything older. These can be fine-tuned individually to comply with data/log retention policies if needed.

Should the database grow past storage constraints a database shrink procedure may be implemented. Please see the relevant articles regarding Database shrink procedure, linked below.

Log management in the SecureTransport Edge Server

Unlike SecureTransport Backend Server, the SecureTransport Edge Server does not have an 'Applications' menu where you could set maintenance applications to manage the export of log files from the database and their retention. The SecureTransport Edge Server does not have a 'Transaction Manager' service so Maintenance applications would not work as they would on the Backend server.

For that reason, the 'rotate' and 'rotate_db' shell scripts are used. They are located in the <FILEDRIVEHOME>/bin folder. The logs exported by 'rotate_db' are the Server Logs, while the logs managed by the 'rotate' script are the ones in <FILEDRIVEHOME>/var/logs. Transfer logs do not exist in the SecureTransport Edge Server as it does not manage the actual transfers. By default, these scripts are set to run daily in the cron scheduler of the 'root' user.

Checking on the configuration for logs rotation

1. For a ST Edge server, running on a Linux/Unix OS, login as root and user the following command on the console:

crontab -e

This will display all scheduled jobs for the root user. If the ST is not installed as root, you would need to add -u "STuser" where "STuser" is the actual user with which SecureTransport operates in the environment.

2. For an ST Edge server, running on Windows open the file:

<CYGWINHOME>\var\cron\tabs\SYSTEM Where <CYGWINHOME> is the folder named 'cygwin' above your <FILEDRIVEHOME> folder. Cygwin essentially is used by ST to run Linux-like commands, and this includes the needed scripts.

Managing other logs, including customized loggers and logs redirected to flat files

Logs in SecureTransport could be customized and redirected directly to flat files. Commonly these files are stored in <FILEDRIVEHOME>/var/db/hist/logs, although this location may differ. Any such logs, which are not managed by the database and the Maintenance applications, including the <FILEDRIVEHOME>/var/logs folder, will not be administered and deleted automatically. Those logs are the responsibility of the system administrator.

To automate the removal or archival of such logs, a cronjob may be utilized on a Linux/Unix OS. This can be very useful for space management on SecureTransport Appliances, including VM Appliances. For Windows, this could be accomplished manually or by a custom script.

Linux/Unix cronjob directions and examples

1. Login as root and user the following command on the console:

crontab -e

2. Add the new cronjob command at the end of the document, on the next new line.

2.1 For deleting logs older than X amount of days:

59 23 1 * * find /[FullPathToDirectoryHere] -type f -mtime +15 | egrep "(*.)" | xargs rm -rf

Where the [FullPathToDirectoryHere] should be replaced by the full path to the exact directory in which the logs are located. This can be obtained by entering the directory in question and using the pwd command. The -mtime +15 specifies all files older than 14 days. If other periods of time are needed, simply use -mtime +X where X is the amount of days +1. The egrep component can filter the file names using regular expressions - simply add the regex needed in the space "( )" - this can be leverage if only specific logs should be deleted by the command.

59 23 1 * * is the calendar on which the command will be executed - on the 59th minute of the 23rd hour of the 1st of every month of every weekday. For more details, see crontab and cronjob instruction guides for Linux/Unix.

To finish and save the changes, hit Esc, then enter insert mode for the text editor by hitting the : symbol, the type the wq symbols and hit Enter.

2.2 For moving/archiving logs older than X amount of days, to a shared storage:

First, ensure there is a shared directory that is accessible and can be written to. Then follow the steps above with the following cronjob command:

59 23 1 * * find /[FullPathToDirectoryHere] -type f -mtime +15 | egrep "(*.)" |  xargs -I '{}' mv {} ../[YourSharedStorage]

Note that ../[YourSharedStorage] is the full directory path to the shared storage directory.

To finish and save the new cronjob, hit Esc, then enter insert mode for the text editor by hitting the : symbol, the type the wq symbols and hit Enter.

It is recommended to frequently review the state of the SecureTransport logs rotation, logs directories and storage consumption, as to prevent out-of-space issues.

• SecureTransport MySQL database maintenance and size shrink for ST 5.3.1 and below
• SecureTransport MySQL database maintenance and size shrink for ST 5.3.3 and above