cloudDB

cloudDB is a distributed, DynamoDB-compatible database system. It features a high-performance Zig-based LSM-tree storage engine with Bloom filters, Multi-Paxos consensus, Gossip cluster membership, Percolator transactions, and a Spring Boot Java control plane.

---

Prerequisites

Ensure you have the following installed:

• Zig 0.16+ (for the storage engine)
• C++17 Compiler (Clang or GCC)
• CMake (version >= 3.14)
• JDK 17 & Maven (for the Java Control Plane)

---

How to Build and Run

1. Build and Run the Zig Storage Engine

The storage engine handles local LSM storage, Paxos consensus, and network requests.

# Build the Zig storage server
cd storage-engine
zig build

# Start the storage node
./zig-out/bin/cloudDBServer --port 9099 --gossip-port 9098 --rf 1 --w 1 --r 1

2. Build and Run the Java Control Plane

The control plane exposes a DynamoDB-compatible REST API that translates HTTP requests to TCP requests for the Zig storage engine.

# Start the Spring Boot REST application (exposes port 8080)
cd control-plane
mvn clean install
mvn spring-boot:run

---

Supported DynamoDB Operations

cloudDB supports the following DynamoDB operations:

Operation	Type	Description
PutItem	Write	Write an item to a table
GetItem	Read	Read a single item by primary key
UpdateItem	Write	Update an existing item (SET, REMOVE, ADD)
DeleteItem	Write	Delete an item (tombstone)
BatchGetItem	Read	Read up to 25 items in a single request
BatchWriteItem	Write	Write or delete up to 25 items
Query	Read	Query a table by partition key and sort key conditions
Scan	Read	Scan the entire table with optional filters
TransactWriteItems	Write	Write multiple items atomically
TransactGetItems	Read	Read multiple items transactionally
CreateTable	DDL	Create a table with optional LSIs and GSIs
DeleteTable	DDL	Delete a table
DescribeTable	DDL	Get table metadata and schema
ListTables	DDL	List all tables

Expression Support

Filter Expressions:

• Comparison operators: =, <>, >, <, >=, <=
• Logical operators: AND, OR, NOT
• Functions: begins_with(), contains(), attribute_exists(), attribute_not_exists()
• BETWEEN operator (e.g., age BETWEEN 18 AND 65)
• IN operator (e.g., status IN ('active', 'pending'))

Update Expressions:

• SET — set attribute values
• REMOVE — remove attributes
• ADD — add to numbers or sets

Attribute Types

• S (String)
• N (Number)
• SS (String Set)
• NS (Number Set)

---

How to Run Tests

Zig Tests

From the storage-engine directory:

zig build test

Java Tests

From the control-plane directory:

mvn test

---

Project Structure

• control-plane/: Java Spring Boot application (REST API, DynamoDB-compatible endpoints, TCP client to storage engine)
• storage-engine/: Zig storage engine
  • src/storage/ — LSM storage engine, memtable, SSTable, Bloom filters
  • src/distributed/ — Multi-Paxos, gossip, Percolator transactions, vector clocks
  • src/network/ — RPC server/client
  • src/recovery/ — Write-Ahead Log (WAL)
• docs/ — Architecture Decision Records (ADRs)
• README.md — This documentation file

---

Architecture

                    ┌─────────────────┐
                    │   AWS CLI /     │
                    │   HTTP Client   │
                    └────────┬────────┘
                             │ HTTP
                             ▼
                   ┌─────────────────────┐
                   │  Java Control Plane │
                   │  (Spring Boot)     │
                   │  Port 8080          │
                   └────────┬───────────┘
                            │ TCP
                            ▼
              ┌─────────────────────────────┐
              │     Zig Storage Engine       │
              │  (LSM tree, Paxos, gossip)  │
              │  Port 9099                   │
              └─────────────────────────────┘

Key Components

1. Java Control Plane — DynamoDB-compatible REST API; translates HTTP to TCP RPC
2. Zig Storage Engine — High-performance LSM-tree with memtable, SSTable, Bloom filters
3. Multi-Paxos — Leader-based consensus for strong consistency
4. Percolator Transactions — Distributed ACID semantics with snapshot isolation
5. Gossip Protocol — Dynamic cluster membership and failure detection