Reliable File Transfer Service [Final]

Overview

The Reliable Transfer Service (RFT) is an OGSA based service that provides interfaces for controlling and monitoring 3rd party file transfers using GridFTP servers. The client controlling the transfer is hosted inside of a grid service so it can be managed using the soft state model and queried using the ServiceData interfaces available to all grid services. It is essentially a reliable and recoverable version of the GT2 globus-url-copy tool and more.

Prerequisites and Dependencies

The Prerequisites to RFT are:

PostgreSQL is used to store the state of the transfer to allow for restart after failures. The interface to PostgreSQL is JDBC so any DBMS that supports JDBC can be used, although no other has been tested.

GridFTP perfoms the actual file transfer.

GridFTP server can only be run on Unix or Linux. There are 2 ways to get GridFTP:

If you don't already have GridFTP from GT2 installed, use the GT3 bundled version. In either case, configuration instructions are documented below.

PostgreSQL Setup

PostgreSQL can be installed both on Unix and on Windows. For setup instructions, see:

Here is a summary of what you need to do after installation:

After completing either the Unix or Windows specific setup above, do the following from your user shell (postgres or Administrator).

$ export PGDATA=/usr/local/pgsql/data
$ initdb
$ createdb ogsa
$ psql -d ogsa -f rft_schema_ogsa.sql
(rft_schema_ogsa.sql is provided in the ogsa distribution root, under the etc directory)

To start the db server run:

$ pgctl -o  "-i" -l logfile start

Configure and Run a GridFTP Server

Typically two GridFTP servers on separate machines are used to perform a file transfer. However to perform a simple test of RFT, one GridFTP server is sufficient. Another alternative is to start two GridFTP servers on one machine using different ports.

A properly installed host certificate is required to run GridFTP. See the Admin Guide, Security Configuration for details.

RFT Grid Service Setup

The Core Final distributions includes a deployable GAR for RFT named multirft.gar.

First, set the following parameters in multirft-server-config.wsdd:

Build the GAR from Source Distribution

This step is optional.

If you downloaded the RFT source code then you can make a deployable gar file as follows:

  1. Change the ogsacore.javadir property in build.xml to point to ogsa/impl/java directory in your ogsa installation (or you can override the default property by doing ant -Dogsacore.dir=<your ogsa installation> compile)
    Run:

  2. $ cd data_management/multirft
    $ ant deployGar

    This builds a gar that can be deployed in a service container. Now update the multi-rft-server-config.wsdd and deploy the GAR as described above.

Testing the RFT Service using the Command Line Client

  1. Deploy multirft.gar
    $ source setenv.sh
  2. You can provide the list of transfers in the form of a file. The format of the file is as follows :

    (the "# comments" are provided for explanation of the parameters and should not be included in the transfers list )
    true        # type of transfer true=binary false=ascii
    16000       # block size in bytes
    16000       # TCP buffer size in bytes
    false       # Notpt
    1           # Number of parallel streams
    true        # DCAU
    2           # Concurrency
    gsiftp://dg0n1.mcs.anl.gov/sandbox/madduri/smalltest    # source URL1
    gsiftp://dg0n1.mcs.anl.gov/sandbox/madduri/test1        # destination URL1
    gsiftp://dg0n1.mcs.anl.gov/sandbox/madduri/smalltest    # source URL2
    gsiftp://dg0n1.mcs.anl.gov/sandbox/madduri/test2        # destination URL2
  3. Submit the transfer to the deployed RFT service:
    $ java org.globus.ogsa.gui.RFTClient \
        <handle to RFT factory> \
        <path to transfers list> 

Service Data Elements for RFT

Version :version of RFT.

FileTransferProgress:
SDE that denotes the percentage of file that is transferred

FileTransferRestartMarker:
SDE for the last restart marker for a particular transfer

FileTransferJobStatusElement:
SDE for status of a particular transfer

FileTransferStatusElement:
SDE that denotes the status of all the transfers in the request

GridFTPRestartMarkerElement:
SDE of Restart marker of the transfer

GridFTPPerfMarkerElement:
SDE of Performance Marker of the transfer