GridFTP Plugin Structure

The Globus Toolkit code can be found at

• https://github.com/globus/globus-toolkit

Examples of plugins can be found at

• https://github.com/stfc/gridFTPCephPlugin
• https://github.com/globus/globus-toolkit/blob/globus_6_branch/gridftp/server/acl/example/globus_gfs_acl_example.c
• https://github.com/EUDAT-B2STAGE/B2STAGE-GridFTP
  • blog post at https://irods.org/2017/02/setting-up-irods-4.2-with-gridftp-using-b2stage-gridftp/
• Posix DSI https://github.com/wyang007/gridftp_dsi_posix
• Open Science Grid DSI https://github.com/opensciencegrid/globus-gridftp-osg-extensions/blob/master/src/osg_extension_dsi.c

Data Storage Interface (DSI)

Information about what a DSI is can be found at http://toolkit.globus.org/toolkit/docs/6.0/gridftp/developer/index.html#gridftp-developer-dsi

DSI modules are loaded by adding the following to the gridftp.conf file

# Load modules
load_dsi_module {plugin name}
allowed_modules {plugin name}

Similarly you can use the -dsi {plugin name} command line option

DSI structure

The set of interface functions that define the DSI can be found in globus_gridftp_server.h.

All type definitions starting with globus_gfs_storage_*() are part of the DSI interface.

Bare bones DSI can be found at https://github.com/globus/globus-toolkit/blob/4fe78a3686cd37bd66cdb73650a1b568c7871525/gridftp/server/src/dsi_bones/globus_gridftp_server_dsi.c.in

Custom commands in a DSI

This file seems to use CUSTOM CMDs: https://github.com/globus/globus-toolkit/blob/4fe78a3686cd37bd66cdb73650a1b568c7871525/gridftp/server/src/globus_i_gfs_control.c

This Open Science Grid DSI https://github.com/opensciencegrid/globus-gridftp-osg-extensions/blob/master/src/osg_extension_dsi.c seems to implement a custom command for SITE USAGE. This is a good model for what we want to do.

Our plan should be to implement a SITE GETPERMISSIONS command to read permissions and then look at SITE SETPERMISSIONS to write permissions.

GridFTP alternatives

• IBM Aspera FASP https://asperasoft.com/technology/transport/fasp/

  • open protocol but patented implementation, used by most video editing companies

• Google QUIC https://www.chromium.org/quic

  • multiplexing UDP used to stream YouTube, etc, currently only works in Chrome and Chromium but the technology is browser independent.
  • not quite mature enough for production yet.

• Just use native python sockets (socket.sendfile method is already a zero-buffer implementation), this would then need a file segmentation to occur.