As videotape gradually disappears from the broadcast plant,
storage in a digital format becomes more important. In most cases, this is a combination of spinning disk and removable
media, either data tape or DVD. With
that comes the management of material as it moves from video server to nearline
to archive and back again. The class of
software products which perform this function are generally called hierarchical
storage management or HSM systems. This
is actually in misnomer in that HSM refers to a specific type of storage
management rather than a product category.
Storage management really consists of four broad categories
of products, each of which has advantages and disadvantages as it applies to
on-air, production, and news environments.
Specifically, the categories are: shared filesystem, HSM systems, disk
extenders, and data movers.
Shared filesystems came about in response to SAN systems
being introduced into a heterogeneous computer environment. In the IT world, they allow disparate
operating systems to share files seamlessly.
This allows common storage to be used for multiple application and
computer types. This does not
necessarily translate to a broadcast environment where video formats and real
time processing require a more closed environment. Shared filesystems are more reasonable to implement in a
production environment where there is less of a real time requirement.
HSM systems are designed to migrate material between
different levels of storage based on rules defined by the user. These are often based on the fill level of
the primary storage which initiates a move to secondary storage based on least
used algorithms. That is, the disk
fills up to a certain level and when that threshold is crossed, the files which
have not been accessed in a while are sent off to the next level of
storage. This is advantageous in that
these rules execute automatically but they don’t necessarily have the rules
needed for a broadcast operation. For
example, a broadcaster would not want the movie they ingested but have not
played yet, moved off the on-air server ten minutes before air.
Disk extenders operate under the premise of making removable
media look like a part of the spinning disk system. The robotic system looks like just another drive letter. This is implemented with a disk subsystem is
the front end that provides the first level cache. The disk extender keeps a “stub” of the file on disk and moves
the majority of the file off to cheaper removable media. The file looks like it is still on disk and
when the application requests a file, the data blocks on removable media are
restored to the cache and then transferred to the application. This makes integration with existing
applications very easy but it does have unintended consequences. It is a two step process for moving for
removable media to cache to application which can introduce a lot of
non-determinism in when material will arrive at the on-air server, for example.
Finally, data movers operate as explicit migration tools
rather than implicit migration tools as described above. This means that data movers to move any
files from one level to another until they are told to do so by a controlling
application. Controlling applications
include automation, editing, and asset management. This introduces a high level of determinism into the movement of
files but also means that the process is less automated.
This paper will discuss in detail the differences as well as
the advantages and disadvantages of each approach. There is no single approach which will work across the board for
all broadcast applications and the implementation needs to be balanced based on
a number of factors.
Steve Atkinson has been involved in high performance storage
systems for the last 20 years with the last 15 years applying those storage
systems to broadcast video. Company experience includes Ampex, StorageTek
and Avalon with his current position being VP of Sales, Americas for Software
Generation, Ltd. Steve has degrees in Journalism from Kansas State
University and Computer Science from University of Texas - Dallas.