This article is intended to be a guidance and provide an example basic setup for an Hadoop Transfer Site in SecureTransport.

Table of contents

1. Introduction
2. Installation
3. Limitations
4. Setup
5. Test the setup
   1. Server Initiated Pull by ST from the Hadoop host
   2. Server Initiated Push from ST to the Hadoop host
6. Troubleshooting Steps

Introduction

The Hadoop Connector is available as a plugin and is not included in the core SecureTransport product. The plugin can be downloaded from the Axway Marketplace free of charge. Once the plugin is deployed, the Hadoop Transfer Site will appear in the SecureTransport Admin UI along with the rest of the Transfer Sites.

The Hadoop transfer site is based on WebHDFS REST API.

Limitations

EDGE Socks proxy: The Backend machine is used for DNS resolution instead of the Edge proxy one. The DNS resolution will be performed by the ST server, if it has access to the DNS service, regardless of the "Use the Edge DNS configuration" option.

Hadoop Secure mode and proxying: Kerberos authentication requests go directly from the backend ST server to the KDC and are not proxied. All other connections are proxied via the defined proxy(s).

Installation

Download

The Hadoop Connector for SecureTransport can be downloaded from the Axway Marketplace from this link.

Prerequisites

Before proceeding further, check the Prerequisites section of the README.md file, which is available in the Connector zip archive.

Remove previous deployments (if any)

Make sure that there are no files/folders from previous versions of the Hadoop Connector. For that purpose, check the <FILEDRIVEHOME>/plugins/transferSites/ folder, and remove everything related the previous version of this plugin (if they exist). Refer to the README.md for the current complete list of files and/or folders that has to be removed.

Unzip the package

Extract the zip archive with the Hadoop Connector securetransport-plugins-site-hadoop-<version>.zip into <FILEDRIVEHOME>/plugins/transferSites/.

Add a logger

You need to add a dedicated Transaction Manager (TM) logger for the newly deployed Hadoop Connector, so that events related to the Connector are available in the Server Log. Follow the steps below.

Take a backup copy of the <FILEDRIVEHOME>/conf/tm-log4j.xml file;

Open the <FILEDRIVEHOME>/conf/tm-log4j.xml file for editing;

Add the following logger in the beginning of the loggers section of the file, with the desired logging level:

<logger name="com.axway.st.plugins.site.ts" additivity="false">
    <level value="info" />
    <appender-ref ref="ServerLog" />
</logger>

To enable debug logging, set the level value to debug. However, change the value with caution. It is recommended to switch to levels different than INFO only temporarily, e.g. during troubleshooting sessions.

Save the file

Restart ST's services

<FILEDRIVEHOME>/bin/stop_all

<FILEDRIVEHOME>/bin/start_all

Verify the installation

The installation can be verified by opening a ST user account and creating a new Transfer Site under Admin UI → Accounts → [user] → Transfer Sites → Add New. The Transfer Protocol drop-down list should contain "Hadoop".

The Hadoop protocol in a Transfer Site

Setup

The screenshots below illustrate the basic configuration required for a working Hadoop Transfer Site:

Basic configuration of a Hadoop Transfer Site

You can use the contextual help (i.e. the [?] buttons) for information about each of the fields in the Transfer Site.

Test the setup

Server Initiated Pull by ST from the Hadoop host

Create a Subscription to a Basic Application (BA).

in the For Files Received from this Account or its Partners section, select "Automatically retrieve files from" and select the Hadoop Transfer Site from the drop-down.

Set a schedule or alternatively, use the Retrieve Files Now button after the Subscription was saved (i.e. save the Subscription, re-open it and then use the button).

[optional] Under For Files Sent to this Account or its Partners, select "Send Files Directly To" and select a different Site where the test file should be sent to (local or remote location) after it's pulled from the Hadoop host.

Save the Subscription.

Place a test file on the Hadoop host in the folder that was defined under "Download Settings" of the Hadoop Site in ST.

Wait for the scheduler to trigger or use the "Retrieve Files Now" button in the Subscription.

Go to Operations → File Tracking → Show Advanced Search, then filter by "Protocol: hadoop" and make sure each of the transfers succeeded with green check Transfer Status icon.

Server Initiated Push from ST to the Hadoop host

Create another Subscription to a Basic Application (BA).

Under For Files Sent to this Account or its Partners, select "Send Files Directly To" and select the Hadoop Transfer Site.

Login to ST with the user account and upload a test file in the Subscription Folder. Wait for a few seconds.

Go to Operations → File Tracking → Show Advanced Search, then filter by "Protocol: hadoop" and make sure each of the transfers succeeded with green check Transfer Status icon.

If all have succeeded, go to the Hadoop host and check the folder defined under "Upload Settings" of the Hadoop Site in ST.

The file should be there.

The latest patch for SecureTransport 5.4 allows you to see the transfers' security parameters when you click on a transfer on the File Tracking page.

Troubleshooting

When a transfer from/to the Hadoop host has failed, the steps below can be used as guidelines how to troubleshoot the issue.

Go to ST's Admin UI → Operations → File Tracking, expand the "Show Advanced Search" and filter by File, or by Protocol: hadoop, etc.

Click on the red X Transfer Status icon related to the failed transfer, expand the window and take a screenshot of the entire content. The screenshot will be requested by Axway Support in case it is needed to open a Support ticket.

Click on the SessionID link, which redirects to the Server Log with filtered results related to the transfer session.

Add an additional filter for ERROR log level and hit Go.

The detailed error/root cause is likely to be presented in these event messages. Click on the timestamp link next to the error message(s) and expand the window. Detailed error message/stack trace is presented which may contain further details for the root cause.

In case the error does not provide an explanation what had happened, take a screenshot of the window mentioned in the last bullet point and/or export the log results using the "Export Log" button and open a ticket with Axway Support for further investigation. In case the Transaction Manager (TM) log is exported in flat file, send an archived copy of the TM log to Axway Support, along with the timeframe/timestamp details of the failed transfer.