Sign in to my dashboard Create an account

Dynamic Data Partitioning in the WAFL File System


September 30, 2023


Jian Hu, NetApp, Inc.; Matthew Curtis-Maury, NetApp, Inc.; Vinay Devadas, NetApp, Inc.

IEEE High Performance Extreme Computing Conference

The WAFL file system leverages data partitioning to manage parallel processing of client and internal operations. Such operations are mapped to data partitions based on the data they touch. Changes in load over time can result in imbalances between data partitions, which can limit system parallelism. Ideally, the system would respond by periodically re-optimizing the partition mappings. However, there are a variety of challenges in making such changes on-the-fly without disrupting client accesses. In this paper, we present an approach that dynamically repartitions file system objects to balance partition load and increase parallelism. Our evaluation shows substantial performance gains across a variety of realistic workloads and configurations.


The paper can be found at:

Drift chat loading