Dynamic Data Partitioning in the WAFL File System
Date
September 30, 2023
Author
Jian Hu, NetApp, Inc.; Matthew Curtis-Maury, NetApp, Inc.; Vinay Devadas, NetApp, Inc.
IEEE High Performance Extreme Computing Conference
The WAFL file system leverages data partitioning to manage parallel processing of client and internal operations. Such operations are mapped to data partitions based on the data they touch. Changes in load over time can result in imbalances between data partitions, which can limit system parallelism. Ideally, the system would respond by periodically re-optimizing the partition mappings. However, there are a variety of challenges in making such changes on-the-fly without disrupting client accesses. In this paper, we present an approach that dynamically repartitions file system objects to balance partition load and increase parallelism. Our evaluation shows substantial performance gains across a variety of realistic workloads and configurations.
Resources
The paper can be found at: https://ieeexplore.ieee.org/document/10363474
