High-performance metadata indexing and search in petascale data storage systems

This paper presents Spyglass, a fast file metadata search system that achieves high scalability by exploiting storage system properties.

Large-scale storage systems used for scientific applications can store petabytes of data and billions of files, making the organization and management of data in these systems a difficult, time-consuming task. The ability to search file metadata in a storage system can address this problem by allowing scientists to quickly navigate experiment data and code while allowing storage administrators to gather the information they need to properly manage the system. In this paper, we present Spyglass, a file metadata search system that achieves scalability by exploiting storage system properties, providing the scalability that existing file metadata search tools lack. In doing so, Spyglass can achieve search performance up to several thousand times faster than existing database solutions. We show that Spyglass enables important functionality that can aid data management for scientists and storage administrators.

In Proceedings of the Conference of Scientific Discovery through Advanced Computing 2008 (SciDAC ’08)

Resources

A copy of the paper is attached to this posting. indexing-scidac08.pdf