Agni: An Efficient Dual-Access File System over Object Storage

Abstract

Object storage is a low-cost, scalable component of cloud ecosystems. However, interface incompatibilities and performance limitations inhibit its adoption for emerging cloud-based workloads. Users are compelled to either run their applications over expensive block storage-based file systems or use inefficient file connectors over object stores. Dual access, the ability to read and write the same data through file systems interfaces and object storage APIs, has promise to improve performance and eliminate storage sprawl. We design and implement Agni1, an efficient, distributed, dual-Access object storage file system (OSFS), that uses standard object storage APIs and cloud microservices. Our system overcomes the performance shortcomings of existing approaches by implementing a multi-Tier write aggregating data structure and by integrating with existing cloud-native services. Moreover, Agni provides distributed access and a coherent namespace. Our experiments demonstrate that for representative workloads Agni improves performance by 20%-60% when compared with existing approaches.