Distributed File Storage Service

About

A fault-tolerant, scalable, distributed object storage system built with a Master-Worker microservices architecture. DFS Core automatically partitions files into 1MB chunks, generates SHA-256 checksums for data integrity, and seamlessly replicates data across multiple storage nodes. Built with resilience in mind, the system dynamically monitors cluster health via heartbeats and guarantees zero data loss by automatically falling back to replica nodes during server crashes.

---

Screenshots

System Dashboard

---

Features

Distributed Storage Engine

• Data Partitioning: Large files are automatically split into manageable 1MB chunks.
• Data Integrity: SHA-256 checksum generation for every individual chunk.
• High Availability: Enforced Replication Factor of 2 (every chunk is duplicated).
• Fault-Tolerant Retrieval: Smart download routing automatically skips dead nodes and fetches backup replicas.

Cluster Management

• Master-Worker Architecture: 1 Coordinator Node and 2 Storage Replicas.
• Health Monitoring: 5-second heartbeat intervals to track node latency and active/dead status.
• Dynamic Routing: Automated IP and Port resolution within Docker network.

Frontend Features

• Minimalist YC-Style UI: Clean, high-contrast, enterprise-grade dashboard.
• Drag-and-Drop Ingestion: Seamless file upload interface.
• Real-time Status: Boot sequence animations and active transfer tracking.
• Offline Simulation: UI gracefully falls back to visual simulations if the backend is unreachable.

Backend Features

• Microservices Architecture: Independently scalable Master and Storage nodes.
• Containerization: Fully orchestrated multi-container deployment.
• Centralized Metadata: PostgreSQL database tracking files, chunks, and node health.
• Cross-Origin Support: Fully configured CORS mapping for local development.

---

Tech Stack

Backend & Infrastructure

• Java 21
• Spring Boot 3.x
• Spring Data JPA
• PostgreSQL
• Docker & Docker Compose
• RESTful APIs (RestTemplate)
• Maven

Frontend

• React 18
• Vite
• Tailwind CSS 3
• Lucide React (Icons)
• Axios / Fetch API

---

Project Structure

distributed-file-system/

├── master-node/           # Central Coordinator Service
│   ├── controller/
│   ├── service/
│   ├── repository/
│   ├── model/
│   └── Dockerfile
│
├── storage-node/          # Worker Node Service
│   ├── controller/
│   ├── service/
│   └── Dockerfile
│
├── frontend/              # React UI
│   ├── src/
│   │   ├── App.jsx
│   │   ├── index.css
│   │   └── main.jsx
│   └── tailwind.config.js
│
└── docker-compose.yml     # Orchestration File

---

API Endpoints

Master Node (Coordinator)

Method	Endpoint	Description
POST	/api/files/upload	Chunks, replicates, and routes file to storage nodes
GET	/api/files/download/{fileId}	Fault-tolerant file retrieval and stitching
POST	/api/system/heartbeat	Receives health status from storage nodes

Storage Nodes (Workers)

Method	Endpoint	Description
POST	/api/storage/chunks/{chunkId}	Saves raw chunk data to local disk
GET	/api/storage/chunks/{chunkId}	Retrieves raw chunk data for download

---

System Architecture Flow

1. Upload: Client sends file to Master Node.
2. Process: Master generates DB metadata, splits file, and hashes chunks.
3. Replicate: Master routes each chunk to 2 active Storage Nodes based on heartbeats.
4. Download: Client requests file ID. Master groups chunks by index.
5. Recover: If a Storage Node is dead during download, Master instantly switches to the secondary replica.
6. Stitch: Master rebuilds the binary file and streams it to the client.

---

Installation & Setup

Prerequisites

• Docker & Docker Desktop
• Node.js & npm

Backend Setup (Dockerized)

1. Clone Repository

git clone https://github.com/saarv3sh7/Distributed-File-Storage-Service.git
cd distributed-file-system

2. Spin up the Cluster (Database, Master Node, 2 Storage Nodes)

docker compose up --build -d

3. Monitor Cluster Health

docker compose logs -f

Frontend Setup

1. Navigate to the frontend directory

cd frontend

2. Install Dependencies

npm install

3. Run the Vite Development Server

npm run dev

Frontend URL:

http://localhost:5173

---

Future Enhancements

• Dynamic Node Scaling: Allow adding N storage nodes on the fly without restarting.
• Data Rebalancing: Automatically move chunks if a node exceeds storage limits.
• File Encryption: AES-256 encryption at rest on the storage nodes.
• Cloud Deployment: Migrate Docker orchestration to Kubernetes (K8s).

---

Author

Sarvesh Yeutkar

M.Tech CSE (Information Security)
College of Engineering Pune (COEP)