Skip to content

qkc/slave 03: define message structs and protocol factory#31

Open
iteyelmp wants to merge 4 commits into
goshard/basefrom
slave-03
Open

qkc/slave 03: define message structs and protocol factory#31
iteyelmp wants to merge 4 commits into
goshard/basefrom
slave-03

Conversation

@iteyelmp

@iteyelmp iteyelmp commented Jul 6, 2026

Copy link
Copy Markdown

Define the wire protocol message layer for slave↔master and peer↔slave
communication, byte-for-byte aligned with pyquarkchain:

  • quarkchain/cluster/rpc.py
  • quarkchain/cluster/p2p_commands.py

This PR establishes a complete opcode → message struct mapping for all
ClusterOp and CommandOp messages.


Changes

  • types.go
    Custom wire types:

    • PrependedSizeBytes4
    • PrependedSizeHashList4

    Implements Python-style 4-byte length-prefixed serialization for
    nested slices and hash lists, with basic safety constraints.


  • rawbytes_placeholder.go
    RawBytes is a temporary schema placeholder.

    Current goal:

    • preserve wire schema compatibility
    • preserve field ordering
    • preserve optional semantics

    RawBytes is not intended to provide semantic deserialization of the underlying Python types.

    Review focus should be on:

    • schema compatibility
    • field ordering
    • optionality
    • opcode mapping

    not on placeholder limitations.


  • messages.go
    94 message structs covering all ClusterOp and CommandOp messages.

    • Field order strictly matches Python FIELDS definitions
    • All opcodes defined upfront for exhaustive factory coverage

  • protocol.go
    Opcode dispatch layer:

    • NewClusterMessage(opcode)
    • NewCommandMessage(opcode)
    • Direction enum

    Provides strict opcode → struct mapping (fail-fast on unknown opcodes)


  • messages_test.go
    Test coverage includes:
    • custom type round-trips
    • message serialization/deserialization
    • ser:"nil" presence marker behavior
    • field order verification vs Python
    • factory completeness checks
    • hand-computed wire compatibility vectors

Notes

  • RawBytes is a migration-only placeholder and will be removed once
    all Python complex types are ported.
  • Wire compatibility is strictly byte-level and must remain aligned
    with Python Serializable behavior.

Comment thread qkc/cluster/wire/messages.go Outdated
Comment thread qkc/cluster/wire/messages.go Outdated
Comment thread qkc/cluster/wire/rawbytes_placeholder.go Outdated
@qzhodl

qzhodl commented Jul 8, 2026

Copy link
Copy Markdown

Overall, this PR is not ready yet. The three inline P1s are all deterministic byte-level incompatibilities with pyquarkchain: QKC Address is 24 bytes, uint256 is fixed 32-byte big-endian, and embedded Python Serializable fields must not get an extra RawBytes length prefix.

These issues were found by Codex 5.5 Extra High without any task-specific prompt or hints. Before submitting wire/serialization code, please run a Codex scan against the Python source and include Python-compatible golden vectors so this class of mismatch is caught before review.

@qzhodl

qzhodl commented Jul 8, 2026

Copy link
Copy Markdown

Additional testing note: the current tests did not catch these issues because they mostly prove Go-side self-consistency, not Python wire compatibility. A Go encode/decode/encode round trip will still pass when both sides share the same wrong assumptions, such as Address being 20 bytes or uint256 using variable-length encoding.

The hand-computed vectors have the same weakness: they can encode the reviewer/author assumption instead of the pyquarkchain contract. RawBytes is an especially good example: the test currently expects the extra 4-byte prefix, so it locks in the incompatibility rather than detecting it.

For this layer, please add Python-generated golden vectors for representative messages and primitives, including Address, uint256, biguint, Optional(Address), and embedded Serializable fields such as RootBlock, MinorBlockHeader, TypedTransaction, and CrossShardTransactionList. The target invariant should be pyquarkchain bytes == goshard bytes, not only goshard round-trip success.

@blockchaindevsh

blockchaindevsh commented Jul 20, 2026

Copy link
Copy Markdown

Each of the PR series should link to the design doc, and the PR description should be succinct, ideally within 1 screen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants