A Lightweight, Scriptable, Web-based Frontend to the SRB
Toby O. H. White
1
, Rik P. Tyer
2
, Richard P. Bruin
1
, Martin T. Dove
1
, Katrina F. Austen
1
1
Department of Earth Sciences, Downing Street, University of Cambridge. CB2 3EQ
2
CCLRC Daresbury Laboratory, Daresbury, Warrington. WA4 4AD
Abstract
The San Diego Supercomputing Centre’s Storage Resource Broker (SRB) is in wide use in our
project. We found that none of the supplied interfaces fulfilled our needs, so we have developed a
new interface, which we call TobysSRB. It is a web-based application with a radically simplified
user interface, making it easy yet powerful to operate for novices and experts alike. The web
interface is easily extensible, and external viewers may be incorporated. In addition, it has been
designed such that interactions with TobysSRB are easily automatable, with a stable, well-defined
and well-documented HTTP API, conforming to the REST (Representational State Transfer)
philosophy. The focus has been on lightweightness, usability and scriptability.
Introduction
Introduction to the SRB
The Storage Resource Broker (SRB) is a distributed
data storage product written and maintained by the
San Diego Supercomputing Centre (SDSC)[1]. It is
in wide use throughout the UK eScience community
[2].
It is intended to allow users to access files, and
collections of files seamlessly within a distributed
environment. It provides an abstraction layer
between the storage of data, which may be in
multiple locations, on multiple filesystems, and the
access of data, which is presented through a unified
interface, transparent to the details of storage type or
location.
An SRB system consists of four components:
•
The MCAT (Metadata Catalogue) database,
which stores internal SRB information - most
importantly, mappings between a file’s SRB
address, and its storage location.
•
The MCAT SRB server, which contains much
of the logic for manipulating files and internal
SRB state.
•
The SRB server, which exports an interface
across the network, accepting requests from
clients, and translating these requests into
interactions with the MCAT database and MCAT
server.
•
The SRB client, which is what the user or
developer sees, and interacts with, and which
exports the transparent, seamless, abstraction layer
which is the point of the SRB.
Within the eMinerals project, we have been using
the SRB for several years now as our primary data
repository. It enables us to share data across the
project in a very simple fashion. We have also built
workflow tools using the SRB as a universally-
accessible storage layer[3]. These tools are widely
used across the project, and several large-scale
studies have been performed using them.
SRB interfaces
From the user’s perspective, regardless of the
interface used, the SRB appears much like a
distributed filesystem, with data in files, and files in
collections that may be nested like directories.
Interfaces for users
SDSC provide three user-accessible front-ends to the
SRB, through which end-users may interact with
files, and navigate through the collections and
datasets of the SRB:
•
The Scommands, which are a series of command-
line tools for Unix-like operating systems, written
in C, which very roughly mimic native POSIX
commands; thus to list the contents of a collection,
Sls is used, where ls lists directory contents on a
traditional Unix file system.
•
MySRB, which is a web-based graphical interface.
•
InQ, which is a native graphical MS Windows
application.
In addition, there are a number of third-party user
interfaces available, which all support subsets of the
SRB’s functionality[4].
Interfaces for developers
In addition, there are a number of officially
supported SRB APIs available:
•
There is a fully-featured C API, which is the
primary developers’ interface.
•
Jargon is a Java API which exports the full
range of functionality.
•
Matrix exports a WS (Web Services) SRB
interface, which is built on top of Jargon.

Again, there are also a number of third-party
interfaces. Mostly these consist of language-specific
wrappers around the C API.
Use of the SRB
The SRB’s primary selling point is as a way to
abstract access to files stored in multiple logical
locations, through a single interface, which bears a
resemblance to a hierarchical filesystem. In addition,
it offers a number of additional features - limited
user-editable metadata, and replication of files.
However, within our project, we have found that
the only aspect of the SRB that we are interested in is
the common view of a familiar filesystem-like
interface.
This manner of usage is encouraged by the
analogies that can be drawn between the Scommands
and native unix filesystem tools.
Indeed, for both SRB versions 2 and 3, there are
filesystem plugins, which enable the use of the SRB
transparently, and allow an SRB repository to be
mounted as a native filesystem within Linux, and a
similar plugin to Windows Explorer, which enables
an analogous interface for Windows.
Given that we use SRB only as a network
filesystem, many of the features offered by the
existing interfaces are beyond our needs.
Deficiencies in current methods of SRB
interaction
User interfaces
The Scommands are the SRB developers'
recommended interface. However, navigating an
SRB repository through the Scommands has many of
the same strengths and drawbacks as navigating a
normal filesystem from the command line.
In its disfavour is the fact that visualizing a
directory structure, and navigating through it when
you are not sure of the destination, can be clumsy
from the command line; and for users unused to
command-line interaction, it is a daunting prospect.
Indeed it is for this reason that graphical file system
browsers were invented in the 1970s.
Furthermore, a network filesystem can never
deliver the performance of a local filesystem, due to
network latency. This is a familiar problem, from
which all network filesystems (NFS, AFS) suffer.
Thus Scommand-ing one's way through a repository
is always slower and less efficient even than
navigating through a filesystem using the analogous
native unix tools.
A further deficiency is the necessity to install the
Scommands on any system where access is required.
It is impossible to connect to the SRB using this
method from an arbitrary computer which knows
nothing about the SRB. Furthermore, even when the
Scommands are installed, it is necessary for each
user individually to be set up correctly to use the
SRB - thus it is not possible to, for example, lean
over someone's shoulder, while they are logged on,
and quickly retrieve a file from your own SRB
collection.
In its favour is the fact that, since the interface is
expressed and used through a unix shell, SRB
interaction can be very easily incorporated in a script,
whether written in shell script, or in a higher-level
language such as Perl or Python.
However, again, network effects conspire to make
repeated Sls's a great deal slower than lots of ls's.
In addition, our experience has been that the SRB is
not sufficiently robust to allow scripting of many
SRB interactions (on the order of a few hundred in
less than a minute). This is due to an architectural
flaw in the SRB server, which will report success for
a transaction, even before the database has been
updated, thus breaking the atomicity of SRB
operations. Clearly this introduces an enormous
number of potential race conditions. It is therefore
impossible to set up the SRB infrastructure to allow
high volumes of requests of the sort which are
essentially trivial for a real filesystem. Repeated
failures of various sorts will be seen.
MySRB and InQ both try to solve some of the
problems associated with the Scommands, in
different ways.
InQ primarily tries to solve the problems
associated with a textual interface. As a native
Windows application, it must be installed locally, on
a Windows computer. Thus, it does not solve the
problems associated with needing to be at a
particular computer. It does at least not require
setting up for each user - it can be used to retrieve
anyone's files from a single installation.
MySRB is a web-based interface to the SRB. It is
a CGI script, implemented in C, which provides a
stateful session of limited duration, and most SRB
actions are possible through it. Since it is a web
application, it is accessible from anywhere with a
working web-browser and internet connection.
However, it suffers from a number of deficiencies
in its user interface (UI). Firstly, due to the necessity
to make available through MySRB almost all of the
functionality of the SRB, the interface suffers from
complexity and overcrowding.
Secondly, when using it as a simple method for
retrieving files, there are two particular irritations
which render it frustrating for the expert, and
confusing for the novice.
•
It second-guesses the browser’s ability to show
files - if a request is made to view a file of a type
MySRB is not aware of, or indeed that MySRB
thinks cannot be displayed by the browser, then
MySRB will escape all HTML-sensitive
characters, and insert the resultant translated file
into an HTML frame between <PRE> tags. This
naturally renders it impossible for the browser to
render the output sensibly when retrieving an SVG
file.
•
When downloading a file, MySRB
communicates with the browser such that every
time, the browser’s save dialogue tries to name the
file MySRB.cgi.