GridFTP

This is a commonly used and efficient transfer protocol which makes use of grid authentication. GridFTP servers are already in production and are being used to efficiently move large amounts of data. As the data fabric becomes available storage that it provides will be available via the GridFTP protocol so that other services, such as Globus job submission, can continue to access storage. Accessing SRB data through the GridFTP is now a proven technology which ARCS has experience with.

dCache

The ILDG community uses dCache for storing, transferring and replicating its data. An instance of this is maintained in production by ARCS. This has some useful features for caching and replicating data and can be provided as an alternative technology for research groups if it is more suitable than the default SRB-based data fabric. It is unknown, at the moment, how dCache may interact with storage provided by SRB and this will be investigated.

LDR

LDR (LIGO data replicator) is used by the LIGO community to manage and replicate its data. An instance of this is maintained in production by ARCS. As well as this LDR is being implemented as the data replication solution for the SkyMapper astronomy project as well as other astronomy projects. Using LDR for replication may be suitable for other research groups and can be provided by ARCS on request. LDR can utilise storage in the data fabric since it too uses GridFTP as a transport protocol.

MDS

The data fabric and other data services provided by ARCS will be published in to MDS (meta data service) so that developers can write software to discover these data services and their capabilities without them having to be hard wired.

Testing and monitoring

The ARCS Data Fabric and other data collaboration services will be tested and monitored to ensure their reliability and availability. Certain performance information will be available to users to help them decide if the services provided are suitable to their needs.

Managed file transfers

A managed file transfer service will be provided between sites to improve the efficiency and reliability of data transfers. It will allow administrators to manage the use of their network resources and to provide a means of indicating when data services are not available. This is in a prototype stage and some work will be required to integrate it with the emerging data fabric.

TPAC Oceans and Climate Digital Library Portal

TPAC Digital Library Portal catalogues just under 60 ocean and climate datasets from within Australia and abroad. These datasets are served by OPeNDAP servers over the internet. You can use the TPAC Digital Library to search for data files based on published metadata. The portal allows you to select files and download resource URLs. It is possible to download OPeNDAP, GridFTP, WMS and WCS URLs depending on individual site's configuration.

-- StephenMcMahon - 22 Apr 2008