Skip to content

ntx-builder: move to sqlite framework#2353

Open
SantiagoPittella wants to merge 9 commits into
nextfrom
migrate-ntx-builder-off-diesel
Open

ntx-builder: move to sqlite framework#2353
SantiagoPittella wants to merge 9 commits into
nextfrom
migrate-ntx-builder-off-diesel

Conversation

@SantiagoPittella

@SantiagoPittella SantiagoPittella commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

closes #2250

Summary

Migrates the ntx-builder database layer off Diesel and onto the shared miden-node-db SQLite framework.

Why:

  • Removes the last Diesel dependency and its schema/codegen tooling from the ntx-builder.
  • Unifies persistence on one connection-pool + transaction model across the node.

How:

  • Replaces the Diesel query modules with plain query functions over ReadTx/WriteTx and hand-written SQL in db/sql/.
  • Drops the pinned "loop connection"; writes now serialize through the framework's single writer connection, reads use the shared reader pool.
  • On-disk layout is unchanged (same to_bytes()/read_from_bytes() codec, same 001_initial.sql schema), so existing databases stay compatible.

Changelog

changelog = "none"
reason    = "Internal change only."

@Mirko-von-Leipzig Mirko-von-Leipzig changed the title chore: mvoe ntx-builder to sqlite framework ntx-builder: move to sqlite framework Jul 17, 2026
Comment thread bin/ntx-builder/src/db/queries/note_scripts.rs Outdated
Comment thread bin/ntx-builder/src/db/queries/mod.rs Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some thoughts, leaving inline so we can discuss more easily.

First some organizational ones:

  • I think we should wrap the db crate's read and write tx again here, and then expose the queries / writes as methods instead of free functions.
  • I would prefer separate folder per sqlite query i.e. not bundling multiple per file, and no sql folder.

More important though:

  • Everywhere still has access to the Database struct.
    • This means everywhere can still write if they so choose.
  • I think the SQLite framework should not give back the Database struct on construction.
    • Instead return: (WriteTx, Pool<ReadTx>).
    • That way we cannot mix and match.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented the wrapper struct (NtxDb) and replace the read/writes with methods, and did that folder organization that you mention. It looks cleaner now.

Regarding the other comments:

Everywhere still has access to the Database struct.

With the wrapper this was fixed, since all read/writes happens through the methods.

I think the SQLite framework should not give back the Database struct on construction.

I don't think I fully understand, so I want to clarify before changing things. Essentially in the ntx-builder we have the block subscription, the server, and the actor context. And only the block stream one (main loop) needs write access, so what I should do is to have 2 wrapper structs, one that only implements reading methods, and another with write capacity?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm getting at, is that it should not be possible to create more than one write connection. And in addition, we want to enforce that writing is unavailable to read-only paths.

For example, in the node, the RPC module should have zero access to database writing. At present it would own a Database and can invoke write at any time. And in the ntx-builder, the actors should have zero access to writing.

What I'm suggesting is that once a database is configured, you not longer need Database, and you only care about the connection pool for reads and the single write connection.

One way of modelling this is:

// A single `WriteConnection`, and a connection pool returning only `ReadOnlyConnection`.
let (db_writer, db_read_pool) = Database::new();

// In ntxb, the writer is owned by the central coordinator, and
// the actors get only the read pool.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I addressed that in df523c1 . Though, and regarding previous comments, when I introduced the framework and migrated the validator component to it I used free functions for the queries instead of the wrapper struct + methods that I used in this PR. I will open a another PR migrating the validator to that design instead of doing that here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate ntx-builder off diesel

2 participants