Efficient File Distribution Using Adaptive Chunking and Gossip-Based Node Discovery in DisktroByte

Abstract

The exponential growth of digital data and rising concerns over privacy have highlighted the limitations of centralized storage systems, which suffer from scalability bottlenecks, single points of failure, and insufficient built-in security. To address these challenges, we present a peer-to-peer (P2P) distributed file storage system that integrates lightweight compression, strong encryption, chunk-based distribution, and replication with automated self-healing. Files are segmented, compressed using LZ4, and encrypted with authenticated encryption schemes (AES-256-GCM or ChaCha20-Poly1305). Consistent hashing distributes chunks evenly across peers, while replication ensures durability. Metadata is managed using BadgerDB to maintain file–chunk mappings, cryptographic keys, and operational logs. The system provides a command-line interface (CLI), uses HTTP endpoints for peer discovery, and leverages gRPC for efficient data transfer. Preliminary evaluation indicates reduced storage overhead, improved retrieval speed, and resilience under node churn. This work lays the foundation for future enhancements such as access control, duplication, versioning, and dashboard-based observability.