From 2b11b9ebf590e2e972e4237ca7c6f1122578c0a6 Mon Sep 17 00:00:00 2001 From: iteye Date: Mon, 13 Jul 2026 11:01:50 +0800 Subject: [PATCH 1/4] Add "QuarkChain Slave Migration Design" --- L1/qkc-slave-migration-design.md | 146 +++++++++++++++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 L1/qkc-slave-migration-design.md diff --git a/L1/qkc-slave-migration-design.md b/L1/qkc-slave-migration-design.md new file mode 100644 index 0000000..f58bb94 --- /dev/null +++ b/L1/qkc-slave-migration-design.md @@ -0,0 +1,146 @@ +# QuarkChain Slave Migration Design Document + +## 1. Migration Goal + +Migrate the Slave component in QuarkChain cluster communication from Python to Go while maintaining byte-level protocol +compatibility with the Python Master. + +The Python implementation is the only protocol specification. + +## 2. Boundary + +This phase implements: + +- Frame encoding and decoding +- Metadata / Opcode / Message definitions +- RpcConn (basic connection framework) +- MasterConn (Master-Slave communication) +- XshardConn (Slave-Slave communication) +- PeerConn (Peer virtual connection) +- Dispatcher (message routing) +- Slave Runtime (lifecycle management) +- Handler Registration (registration mechanism) +- Handler Dispatch (dispatch scheduling) + +This phase does not implement business logic, such as: + +- Shard Runtime +- StateDB +- TxPool +- Miner +- State Transition + +## 3. Current Status + +| PR | Content | Status | +|-----|--------------------------------------------|-------------| +| PR1 | Wire Frame encoding/decoding | ✅ Completed | +| PR2 | Metadata and Opcode definitions | ✅ Completed | +| PR3 | Message definitions and protocol constants | Review | +| PR4 | RpcConn + XshardConn foundation | Review | + +## 4. Migration Plan + +``` +PR1-PR3 (Protocol foundation layer) +↓ +PR4 (RpcConn / XshardConn / Xshard Handler registration) +↓ +PR5 (MasterConn / Master Handler registration and dispatch) +↓ +PR6 (PeerConn / Dispatcher / Peer Handler registration and dispatch) +↓ +PR7 (SlaveServer Runtime) +``` + +### PR5: MasterConn + +Corresponds to Python: `MasterConnection` + +Main contents: + +- Master-Slave TCP connection +- ClusterMetadata handling (12B) +- MasterConn lifecycle +- Master Handler registration and dispatch framework +- Stub implementations for Handlers depending on business components + +### PR6: PeerConn + Dispatcher + +Corresponds to Python: `PeerShardConnection`, `VirtualConnection` + +Main contents: + +- Peer virtual connection + forwarding mechanism +- Dispatcher message routing +- Peer Handler registration and dispatch framework +- Stub implementations for Handlers depending on business components + +### PR7: SlaveServer Runtime + +Corresponds to Python: `SlaveServer` + +Main contents: + +- Slave lifecycle management + TCP listener +- Unified connection management (MasterConn / PeerConn / XshardConn) +- Dispatcher orchestration +- Wiring of all components + +## 5. Handler Implementation Strategy + +Handlers are categorized by connection type: + +- Xshard Handler → XshardConn +- Master Handler → MasterConn +- Peer Handler → PeerConn + +This phase completes: + +- Handler registration (Registration) +- Handler dispatch (Dispatch) +- RPC request/response flow + +For Handlers depending on other components, such as: + +- Shard Runtime +- StateDB +- TxPool +- Miner + +Only Stub implementations are provided, without implementing actual business logic. + +The goal of the Stubs is to ensure: + +- Opcode can be recognized +- Messages can be parsed +- Handlers can be correctly dispatched +- RPC request/response flow is complete +- Protocol-compatible placeholder responses are returned + +## 6. Testing + +The Python implementation is the protocol authority. + +Compatibility coverage: + +- Frame encoding/decoding +- Metadata encoding +- Opcode mapping +- Message serialization +- RPC request/response flow +- Connection lifecycle + +Test vectors generated by the Python implementation should be used whenever possible. Do not rely solely on Go self +encode/decode tests. + +## 7. Expected Result + +After completing PR1-PR7: + +- Complete the migration of the Slave communication framework +- Master-Slave, Slave-Slave, and Peer virtual connections are all ready +- All RPC Opcodes have completed registration and dispatch +- Business Handlers provide compatible Stub implementations + +This phase only completes the communication layer infrastructure. Business logic will be implemented in later phases. \ No newline at end of file From 273baa41eafd95091b80e8833461cc63eb70d95e Mon Sep 17 00:00:00 2001 From: iteye Date: Mon, 13 Jul 2026 19:16:39 +0800 Subject: [PATCH 2/4] fix comment --- L1/qkc-slave-migration-design.md | 113 +++++++++++++++++++++++++------ 1 file changed, 93 insertions(+), 20 deletions(-) diff --git a/L1/qkc-slave-migration-design.md b/L1/qkc-slave-migration-design.md index f58bb94..79286df 100644 --- a/L1/qkc-slave-migration-design.md +++ b/L1/qkc-slave-migration-design.md @@ -5,9 +5,12 @@ Migrate the Slave component in QuarkChain cluster communication from Python to Go while maintaining byte-level protocol compatibility with the Python Master. -The Python implementation is the only protocol specification. +The Python implementation remains the sole protocol specification. -## 2. Boundary +The goal of this phase is to complete the migration of the communication infrastructure while keeping the existing +Python Master unchanged, providing a foundation for future business logic migration. + +## 2. Scope This phase implements: @@ -38,21 +41,87 @@ This phase does not implement business logic, such as: | PR2 | Metadata and Opcode definitions | ✅ Completed | | PR3 | Message definitions and protocol constants | Review | | PR4 | RpcConn + XshardConn foundation | Review | +| PR5 | MasterConn, handler registration | Review | +| PR6 | PeerConn, Dispatcher, and peer routing | Planned | +| PR7 | Slave runtime and lifecycle management | Planned | + +## 4. Communication Flow + +QuarkChain cluster communication consists of three communication paths: + +- Master ↔ Slave +- Peer ↔ Slave (forwarded through Master) +- Slave ↔ Slave + +### 4.1 Master → Slave RPC + +``` + Master Slave (MasterConn) + │ │ + │──[TCP Frame]──────────────────►│ + │ 12B ClusterMetadata + │ + │ op + rpc_id + payload │ + │ │── cluster_peer_id == 0 + │ │── lookup MASTER_OP_RPC_MAP + │ │── dispatch handler + │◄──[TCP Frame]──────────────────│── return response + │ 12B ClusterMetadata + │ + │ op + rpc_id + payload │ +``` + +### 4.2 Peer Virtual Connection Forwarding + +``` + External P2P Peer Master Slave (MasterConn) + │ │ │ + │──[P2P Frame]─────►│ │ + │ │── convert to ClusterMetadata + │ │──[Cluster Frame]─────►│ + │ │ │── cluster_peer_id ≠ 0 + │ │ │── route to PeerConn + │ │◄──[Cluster Frame]─────│── return response + │◄──[P2P Frame]─────│ │ +``` -## 4. Migration Plan +### 4.3 Slave → Slave RPC + +``` + Slave A (XshardConn) Slave B (XshardConn) + │ │ + │──[TCP Frame]──────────────────►│ + │ 0B Metadata + │ + │ op + rpc_id + payload │ + │ │── lookup XSHARD_OP_RPC_MAP + │ │── dispatch handler + │◄──[TCP Frame]──────────────────│── return response + │ 0B Metadata + │ + │ op + rpc_id + payload │ +``` + +## 5. Migration Plan ``` PR1-PR3 (Protocol foundation layer) ↓ -PR4 (RpcConn / XshardConn / Xshard Handler registration) +PR4 (RpcConn / XshardConn) ↓ -PR5 (MasterConn / Master Handler registration and dispatch) +PR5 (MasterConn) ↓ -PR6 (PeerConn / Dispatcher / Peer Handler registration and dispatch) +PR6 (PeerConn / Dispatcher) ↓ -PR7 (SlaveServer Runtime) +PR7 (Slave Runtime) ``` +### PR4: RpcConn + XshardConn + +Corresponding Python components: `Connection`, `SlaveConnection` + +Main contents: + +- RpcConn: connection framework (TCP read/write loops, RPC ID management, request/response matching) +- XshardConn: Slave-to-Slave TCP connections (0B Metadata) +- Xshard handler registration and Ping/Pong handshake + ### PR5: MasterConn Corresponds to Python: `MasterConnection` @@ -76,7 +145,7 @@ Main contents: - Peer Handler registration and dispatch framework - Stub implementations for Handlers depending on business components -### PR7: SlaveServer Runtime +### PR7: Slave Runtime Corresponds to Python: `SlaveServer` @@ -87,7 +156,7 @@ Main contents: - Dispatcher orchestration - Wiring of all components -## 5. Handler Implementation Strategy +## 6. Handler Implementation Strategy Handlers are categorized by connection type: @@ -118,23 +187,27 @@ The goal of the Stubs is to ensure: - RPC request/response flow is complete - Protocol-compatible placeholder responses are returned -## 6. Testing +## 7. Testing The Python implementation is the protocol authority. -Compatibility coverage: +Testing is performed in two stages: -- Frame encoding/decoding -- Metadata encoding -- Opcode mapping -- Message serialization -- RPC request/response flow -- Connection lifecycle +- PR1–PR6: Use Python-generated test vectors and lightweight Python protocol peers to validate serialization, RPC flows, + and connection lifecycles. These protocol peers are used solely for protocol compatibility testing and do not require + a full Python Master or business components. +- PR7: Run end-to-end interoperability tests using a real Python Master and the Go Slave implementation. + +Testing focuses on: -Test vectors generated by the Python implementation should be used whenever possible. Do not rely solely on Go self -encode/decode tests. +- Frame and Metadata compatibility +- Message serialization compatibility +- RPC request/response flow +- Master-Slave communication +- Slave-Slave communication +- Peer communication -## 7. Expected Result +## 8. Expected Result After completing PR1-PR7: From 727b37f43227f7421a78a011fdbb9f605ba9e8b7 Mon Sep 17 00:00:00 2001 From: iteye Date: Tue, 14 Jul 2026 11:47:28 +0800 Subject: [PATCH 3/4] fix comment --- L1/qkc-slave-migration-design.md | 43 +++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 6 deletions(-) diff --git a/L1/qkc-slave-migration-design.md b/L1/qkc-slave-migration-design.md index 79286df..fea606a 100644 --- a/L1/qkc-slave-migration-design.md +++ b/L1/qkc-slave-migration-design.md @@ -98,6 +98,34 @@ QuarkChain cluster communication consists of three communication paths: │ op + rpc_id + payload │ ``` +### 4.4 Connection Model Rules + +**Connection Identity** + +| Connection Type | Description | +|-----------------|-----------------------------------------------------------------------------------------| +| MasterConn | Connection used for Master-Slave communication | +| XshardConn | Connection used for Slave-Slave communication | +| PeerConn | Virtual connection representing forwarded peer traffic, identified by `cluster_peer_id` | + +**RPC Isolation** + +- Each logical connection is an independent RPC channel. +- RPC IDs are scoped per connection and are not required to be globally unique. +- Multiple PeerConn instances sharing the same MasterConn transport may use overlapping RPC IDs. + +**Lifecycle** + +- PeerConn lifecycle is driven by Master commands, never autonomously by the Slave +- Closing MasterConn closes all associated PeerConn instances. + +**Message Routing** + +MasterConn routes frames based on `cluster_peer_id`: + +- `cluster_peer_id == 0`: Master RPC, handled locally by MasterConn +- `cluster_peer_id != 0`: Peer RPC, forwarded by the Dispatcher to the corresponding PeerConn + ## 5. Migration Plan ``` @@ -119,7 +147,7 @@ Corresponding Python components: `Connection`, `SlaveConnection` Main contents: - RpcConn: connection framework (TCP read/write loops, RPC ID management, request/response matching) -- XshardConn: Slave-to-Slave TCP connections (0B Metadata) +- XshardConn: Slave-to-Slave communication - Xshard handler registration and Ping/Pong handshake ### PR5: MasterConn @@ -128,9 +156,8 @@ Corresponds to Python: `MasterConnection` Main contents: -- Master-Slave TCP connection -- ClusterMetadata handling (12B) -- MasterConn lifecycle +- Master-Slave communication +- MasterConn serves as the transport channel for forwarded peer traffic - Master Handler registration and dispatch framework - Stub implementations for Handlers depending on business components @@ -140,8 +167,12 @@ Corresponds to Python: `PeerShardConnection`, `VirtualConnection` Main contents: -- Peer virtual connection + forwarding mechanism -- Dispatcher message routing +- PeerConn: virtual connection for forwarded external peer traffic. +- PeerConn does not own a TCP socket and uses MasterConn as its transport. +- PeerConn is an independent RPC channel with its own RPC ID namespace. +- PeerConn lifecycle is controlled by Master commands. +- Dispatcher: routes frames to the correct PeerConn based on `cluster_peer_id` (`cluster_peer_id == 0` routes to MasterConn + itself) - Peer Handler registration and dispatch framework - Stub implementations for Handlers depending on business components From fd9a1e8b92807003dcc366902d26042301cca172 Mon Sep 17 00:00:00 2001 From: iteye Date: Tue, 14 Jul 2026 12:03:40 +0800 Subject: [PATCH 4/4] fixing the obfuscated runtime --- L1/qkc-slave-migration-design.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/L1/qkc-slave-migration-design.md b/L1/qkc-slave-migration-design.md index fea606a..73c98b3 100644 --- a/L1/qkc-slave-migration-design.md +++ b/L1/qkc-slave-migration-design.md @@ -35,15 +35,15 @@ This phase does not implement business logic, such as: ## 3. Current Status -| PR | Content | Status | -|-----|--------------------------------------------|-------------| -| PR1 | Wire Frame encoding/decoding | ✅ Completed | -| PR2 | Metadata and Opcode definitions | ✅ Completed | -| PR3 | Message definitions and protocol constants | Review | -| PR4 | RpcConn + XshardConn foundation | Review | -| PR5 | MasterConn, handler registration | Review | -| PR6 | PeerConn, Dispatcher, and peer routing | Planned | -| PR7 | Slave runtime and lifecycle management | Planned | +| PR | Content | Status | +|-----|------------------------------------------------------|-------------| +| PR1 | Wire Frame encoding/decoding | ✅ Completed | +| PR2 | Metadata and Opcode definitions | ✅ Completed | +| PR3 | Message definitions and protocol constants | Review | +| PR4 | RpcConn + XshardConn foundation | Review | +| PR5 | MasterConn, handler registration | Review | +| PR6 | PeerConn, Dispatcher, and peer routing | Planned | +| PR7 | Slave communication runtime and lifecycle management | Planned | ## 4. Communication Flow @@ -171,21 +171,22 @@ Main contents: - PeerConn does not own a TCP socket and uses MasterConn as its transport. - PeerConn is an independent RPC channel with its own RPC ID namespace. - PeerConn lifecycle is controlled by Master commands. -- Dispatcher: routes frames to the correct PeerConn based on `cluster_peer_id` (`cluster_peer_id == 0` routes to MasterConn - itself) +- Dispatcher: routes frames to the correct PeerConn based on `cluster_peer_id` (`cluster_peer_id == 0` routes to + MasterConn itself) - Peer Handler registration and dispatch framework - Stub implementations for Handlers depending on business components -### PR7: Slave Runtime +### PR7: Slave Communication Runtime Corresponds to Python: `SlaveServer` Main contents: -- Slave lifecycle management + TCP listener +- Slave communication service lifecycle management +- TCP listener for cluster connections - Unified connection management (MasterConn / PeerConn / XshardConn) - Dispatcher orchestration -- Wiring of all components +- Wiring of communication components ## 6. Handler Implementation Strategy