diff --git a/CLAUDE.md b/CLAUDE.md index cb3e415..480bc39 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,6 +8,8 @@ InfiniteZero / DIN Protocol DevNet: a federated-learning network coordinated by The Solidity tree at `hardhat/contracts/` (Hardhat workflow) is used for deployment/verification tooling. +Docs convention: `Documentation/` describes what exists in the code on `develop` (not the live Sepolia deployment) — `Documentation/public/` for network participants, `Documentation/technical/` for code readers. `Developer/` holds forward-looking material: `design/` (planned mechanism designs), `issues/` (backlog), `proposals/` (tooling proposals), `tasks/` (contributor task specs), `discussion/`, `rejected-ideas/`, plus ROADMAP.md and process docs. Placement rules: `Developer/README.md`. + ## Commands ### Python / dincli @@ -44,8 +46,8 @@ dincli system connect-wallet --account ### Two layers: protocol contracts vs. per-model task contracts -- **Platform-level contracts** (deployed once by the DIN-Representative): `DinCoordinator` (ETH<->DIN exchange, slasher registry admin), `DinToken` (ERC20, minted on ETH deposit), `DinValidatorStake` (validator staking/slashing), `DinModelRegistry` (model registration, open-source vs proprietary fee). See `Documentation/DIN-workflow.md`. -- **Task-level contracts** (deployed per model by the model owner): `DINTaskCoordinator` and `DINTaskAuditor`. These must be authorized as "slashers" on `DinValidatorStake` (via `DinCoordinator`, only callable by the DIN-Representative) *before* the model owner can register the model in `DinModelRegistry`. See `Documentation/Model-workflow.md` for the full step-by-step (deploy → slasher auth request → genesis model → register → global iterations). +- **Platform-level contracts** (deployed once by the DIN-Representative): `DinCoordinator` (ETH<->DIN exchange, slasher registry admin), `DinToken` (ERC20, minted on ETH deposit), `DinValidatorStake` (validator staking/slashing), `DinModelRegistry` (model registration, open-source vs proprietary fee). See `Documentation/public/workflows/din-workflow.md`. +- **Task-level contracts** (deployed per model by the model owner): `DINTaskCoordinator` and `DINTaskAuditor`. These must be authorized as "slashers" on `DinValidatorStake` (via `DinCoordinator`, only callable by the DIN-Representative) *before* the model owner can register the model in `DinModelRegistry`. See `Documentation/public/workflows/model-workflow.md` for the full step-by-step (deploy → slasher auth request → genesis model → register → global iterations). A model's lifecycle runs in **Global Iterations (GI)**: aggregator/auditor registration → Local Model Submission (LMS) by clients → auditor evaluation/scoring → two-tier aggregation (T1 sub-batches, T2 combines T1 into the new global model) → slashing of misbehaving validators → GI end. Each phase is opened/closed explicitly by the model owner via `dincli model-owner ...` subcommands, and other roles act only within an open phase. @@ -64,7 +66,7 @@ Differential privacy is opt-in per model via a nested `dp` block in the manifest ### IPFS abstraction -`dincli/services/ipfs.py` supports three interchangeable upload/retrieve backends (env-var-configured IPFS node, Filebase, or a fully custom Python provider) selected via `resolve_ipfs_config()`/`ipfs_provider` config — see `Documentation/guides/ipfs.md`. All CID-bearing artifacts (services, manifests, model weights, ABIs) flow through this layer rather than direct HTTP calls scattered through the CLI. +`dincli/services/ipfs.py` supports three interchangeable upload/retrieve backends (env-var-configured IPFS node, Filebase, or a fully custom Python provider) selected via `resolve_ipfs_config()`/`ipfs_provider` config — see `Documentation/public/guides/ipfs.md`. All CID-bearing artifacts (services, manifests, model weights, ABIs) flow through this layer rather than direct HTTP calls scattered through the CLI. ### Networks diff --git a/Developer/CONTRIBUTING.md b/Developer/CONTRIBUTING.md index 792990a..ff929f6 100644 --- a/Developer/CONTRIBUTING.md +++ b/Developer/CONTRIBUTING.md @@ -21,6 +21,8 @@ Create a feature branch, commit your changes, and open a pull request to `develo The materials in `/Developer` are working design input, not fixed specifications. They represent current thinking, not final decisions. +**Where documents live:** `Documentation/` describes what exists in the code on `develop` (`public/` for network participants, `technical/` for people modifying the code); `Developer/` holds plans, designs, proposals, tasks, and process docs. Before adding a document, read "Where does a new document go?" in [README.md](README.md). + Good contributors challenge assumptions, question tradeoffs, and propose alternatives. If you see a better path, say so. The goal is the right architecture, not defending the existing one. ## Ways To Contribute diff --git a/Developer/DEVELOPMENT_SETUP.md b/Developer/DEVELOPMENT_SETUP.md index 9cb91b1..3b66ff9 100644 --- a/Developer/DEVELOPMENT_SETUP.md +++ b/Developer/DEVELOPMENT_SETUP.md @@ -2,7 +2,7 @@ This file is the minimal setup entrypoint for contributors working on DevNet. -**Prerequisite:** [Wallet Setup](../Documentation/guides/wallet-setup.md) — for development, use demo mode or `ETH_PRIVATE_KEY_` in `.env`. Never commit real keys. +**Prerequisite:** [Wallet Setup](../Documentation/public/guides/wallet-setup.md) — for development, use demo mode or `ETH_PRIVATE_KEY_` in `.env`. Never commit real keys. ## Local Python Setup diff --git a/Developer/GOOD_FIRST_ISSUES.md b/Developer/GOOD_FIRST_ISSUES.md index a0f38ed..1753286 100644 --- a/Developer/GOOD_FIRST_ISSUES.md +++ b/Developer/GOOD_FIRST_ISSUES.md @@ -134,7 +134,7 @@ Relevant code paths: **Difficulty:** Intermediate → Advanced **Area:** Client Services & Tooling **Detailed issue:** [issues/client-data-labeling.md](issues/client-data-labeling.md) -**Detailed tooling:** [tooling/client-labeling.md](tooling/client-labeling.md) +**Detailed tooling:** [tooling/client-labeling.md](proposals/client-labeling.md) Address the challenge of client-side data labeling where raw edge data is unlabeled but federated learning requires high-quality labeled datasets. @@ -145,7 +145,7 @@ Current focus areas: #### Curated Review Packet - [Developer/issues/client-data-labeling.md](/home/azureuser/projects/devnet/Developer/issues/client-data-labeling.md) -- [Developer/tooling/client-labeling.md](/home/azureuser/projects/devnet/Developer/tooling/client-labeling.md) +- [Developer/proposals/client-labeling.md](/home/azureuser/projects/devnet/Developer/proposals/client-labeling.md) - [cache_model_0/services/client.py](/home/azureuser/projects/devnet/cache_model_0/services/client.py) - [dincli/services/client.py](/home/azureuser/projects/devnet/dincli/services/client.py) @@ -172,5 +172,5 @@ Open a discussion or issue if you need onboarding help, architecture clarificati We welcome contributors from AI/ML, cryptography, distributed systems, blockchain, privacy-preserving computing, and open-source communities. [Contribution Guide →](https://github.com/InfiniteZeroFoundation/DevNet/blob/develop/Developer/CONTRIBUTING.md) -[Getting Started →](https://github.com/InfiniteZeroFoundation/DevNet/blob/main/Documentation/GettingStarted.md) +[Getting Started →](https://github.com/InfiniteZeroFoundation/DevNet/blob/develop/Documentation/public/getting-started.md) [Say hello →](mailto:abrahamnash@protonmail.com) diff --git a/Developer/README.md b/Developer/README.md index e69de29..a5e4f21 100644 --- a/Developer/README.md +++ b/Developer/README.md @@ -0,0 +1,31 @@ +# Developer + +Forward-looking material for people building the DIN Protocol: plans, designs, proposals, contributor process, and work tracking. + +> **Scope rule:** documents here describe **what we plan, propose, decide, or how we work** — they stay valid even as the code changes. Documentation of what currently exists on `develop` lives in [`Documentation/`](../Documentation/README.md) instead (`public/` for participants, `technical/` for people modifying the code). + +## Layout + +| Location | Contents | +|---|---| +| [ROADMAP.md](ROADMAP.md) | Phase plans (P2–P4) and the full work-package table with owners, dependencies, and status | +| [`design/`](design/) | Target designs not yet (fully) implemented: [DevNet 2.0 mechanism design](design/MECHANISM_DESIGN.md), [production staking/slashing spec](design/suggested-staking-mechanism.md), [feasibility report](design/feasibility-report.md) | +| [`issues/`](issues/) | Backlog of design/implementation write-ups per mechanism or feature (some with `design.md` / `implementation.md` / `simulation.md` subdocs) | +| [`proposals/`](proposals/) | Tooling proposals — tools that don't exist yet (client labeling, model-owner contract/service builders) | +| [`tasks/`](tasks/) | Contributor task specs (`task_DDMMYY_n.md`) | +| [`discussion/`](discussion/) | Open discussions (Filecoin support, Foundry migration) | +| [`rejected-ideas/`](rejected-ideas/) | Ideas evaluated and rejected, with rationale (e.g. TKNN-Shapley) | +| [CONTRIBUTING.md](CONTRIBUTING.md) | Contribution process and code standards pointers | +| [CODE_STANDARDS.md](CODE_STANDARDS.md) | Code style and quality expectations | +| [DEVELOPMENT_SETUP.md](DEVELOPMENT_SETUP.md) | Setting up a development environment | +| [GOOD_FIRST_ISSUES.md](GOOD_FIRST_ISSUES.md) | Curated starter tasks for new contributors | + +## Where does a new document go? + +1. **Describes code that exists on `develop`?** → `Documentation/` (never here). Participant-facing → `Documentation/public/`; for code readers/auditors → `Documentation/technical/`. +2. **A design or spec for something not built yet?** → `design/` (protocol mechanisms) or `proposals/` (tooling). +3. **A backlog item someone could pick up?** → `issues/`. +4. **A scoped task for a specific contributor?** → `tasks/`. +5. **An open question or debate?** → `discussion/`. A decision *not* to do something → `rejected-ideas/`. + +**Graduation rule:** when a design from `design/` or `issues/` ships, don't move the document — write (or update) the current-state description in `Documentation/technical/` and mark the design here as shipped with a pointer. The design doc remains as the record of intent; `Documentation/` records reality. (Example: `issues/staking-mechanism.md` → `Documentation/technical/mechanisms/staking-mechanism.md`.) diff --git a/Developer/design/MECHANISM_DESIGN.md b/Developer/design/MECHANISM_DESIGN.md new file mode 100644 index 0000000..d0d8aef --- /dev/null +++ b/Developer/design/MECHANISM_DESIGN.md @@ -0,0 +1,243 @@ +# DIN DevNet 2.0 — Cryptoeconomic Mechanism Design + +**Status:** Working design document — immediate requirement for DevNet 2.0 +**Owner:** Umer +**Scope:** Consolidated design for staking, slashing, rewards, scoring/auditing, tokenomics, and fees, organized by network role. +**Roadmap anchors:** P3-4.1/4.2/4.3 (slashing), P3-SCR (scoring), P3-5.1/5.2/5.3 (token utility, emission, fees), P3-DOC2/3/5 (public docs). + +--- + +## 1. Role × Mechanism Matrix + +This is the one-page view. Every mechanism below is designed per role — a mechanism that isn't tied to a role and a concrete misbehaviour/contribution is out of scope. + +| Role | Stakes? | Slashable? | Earns rewards? | Pays fees? | Scored/audited? | +|---|---|---|---|---|---| +| **Client** | No (data contributor, low barrier) | No | ✅ Yes — proportional to BlockFLow contribution score | No | ✅ Yes — local model updates scored by auditors | +| **Auditor** | ✅ Yes (validator stake) | ✅ Yes — liveness + correctness | ✅ Yes — per-GI evaluation service fee share | No | Indirectly — cross-auditor median exposes outliers | +| **Aggregator** | ✅ Yes (validator stake) | ✅ Yes — liveness + correctness | ✅ Yes — per-GI aggregation service fee share | No | Indirectly — T1/T2 output verifiable against inputs | +| **Model owner** | No (pays instead) | No (loses fees/deposit on abandonment) | Gets the trained model | ✅ Yes — registration fee + per-GI service fee + reward deposit | No | +| **DIN-Representative / DAO** | No | No | Treasury cut of all fees | No | Governance-accountable, not economically | +| **Treasury** | — | — | Receives network fee split + (decision pending) slashed stake | — | — | + +Design principle: **clients face zero economic barrier** (staking clients would kill data-contribution supply; Sybil resistance for clients comes from scoring, not stake). **Validators (auditors + aggregators) face full economic accountability** (stake → slash → reward is their loop). **Model owners are the paying customers** whose fees fund validator rewards and the treasury. + +--- + +## 2. Current On-Chain State (what DevNet 1.0 already has) + +Grounding: everything below references `foundry/src/` as of `develop`. + +| Mechanism | Implemented today | Gap for 2.0 | +|---|---|---| +| **Staking** | `DinValidatorStake`: flat `MIN_STAKE = 10 DIN`, 7-day `UNBONDING_PERIOD`, pending withdrawals remain slashable, blacklist, slasher registry (task contracts authorized via `DinCoordinator`) | No role-specific stake sizing, no stake-weighted selection, single pending withdrawal only, no delegation | +| **Slashing** | Liveness-only, flat `minStake()` amount: `DINTaskAuditor.slashAuditors` (missed vote), `DINTaskCoordinator.slashAggregators` (missed T1/T2 submission). Slashed stake is decremented in place — it stays locked in the stake contract (de facto burn), no redistribution | No correctness-based slashing, no partial/full tiers, no dispute resolution, no explicit destination for slashed stake | +| **Rewards** | `DINTaskAuditor.totalDepositedRewards` + `RewardDeposited` event — deposits are recorded but **there is no distribution or claim logic at all** | Entire reward engine: funding, per-GI settlement, claim flow | +| **Scoring** | On-chain: auditor score submission (`setAuditScorenEligibility`), pass-score eligibility voting, `finalizeEvaluation`, `setTier2Score`. Off-chain: BlockFLow-inspired scoring implemented in Python (WP 2.2 ✅, validation pending P3-SCR) | Median aggregation of auditor scores on-chain, score → reward wiring, score-deviation detection for auditor slashing | +| **Tokenomics** | `DinToken` minted only via `DinCoordinator.depositAndMint` (ETH → DIN at `dinPerEth = 1,000,000`), owner can withdraw deposited ETH. No cap, no burn, no emission | Supply policy, emission schedule, burn sinks, treasury | +| **Fees** | `DINModelRegistry`: ETH registration + manifest-update fees (open-source vs proprietary tiers), `withdrawFees` to owner | Per-GI service fees, treasury routing, fee splits, dynamic fees, storage cost line item | + +--- + +## 3. Staking — auditors & aggregators + +**Goal:** every validator action is backed by capital at risk that exceeds the profit from cheating on one GI. + +### Design + +1. **Who stakes:** auditors and aggregators only. One stake pool (`DinValidatorStake`) covers both roles — a validator's stake backs whichever role(s) it registers for in a GI. Keep the unified pool for 2.0 (simpler accounting, slashers already wired); revisit per-role pools only if role risk profiles diverge materially. +2. **Stake sizing:** + - Keep a network-wide `minStake` floor (DAO-settable, replacing the `constant`). + - Add a **per-model stake requirement** settable by the model owner at task deploy (bounded by DAO min/max) so high-value models can demand more skin in the game. + - Registration for a GI should check `getStake(validator) >= max(networkMin, modelMin)` — today it only checks activity. +3. **Stake-at-risk accounting:** a validator registered in N concurrent GIs has the same stake backing all N. For 2.0, cap concurrent registrations per validator (e.g., stake must cover `k × slashable-amount` across active registrations) to prevent one 10-DIN stake backing 20 tasks. +4. **Unbonding:** keep 7 days, keep unbonding stake slashable (already correct). Fix the single-pending-withdrawal limitation (queue of withdrawals) as a quality-of-life item, not a blocker. +5. **Selection weighting:** for 2.0, registration remains permissionless above min stake; batch assignment stays random/rotational. Stake-weighted selection is deliberately **not** adopted (it concentrates work in whales and weakens the cross-validator median). Document as a rejected idea. +6. **Sybil bound:** minimum stake × validator-to-participant ratio is the Sybil barrier (Roadmap §5). The DAO parameter set must include both. + +### Parameters (DAO-settable via P3-5.1 governance hooks) + +`minStake`, `unbondingPeriod`, `maxConcurrentRegistrationsPerStakeUnit`, `modelMinStakeBounds[min,max]`. + +### Open decisions + +- Per-role min stake (auditor vs aggregator) — recommend **no** for 2.0, single floor. +- Delegation — out of scope until P5+. + +--- + +## 4. Slashing — auditors & aggregators (P3-4.1 / 4.2 / 4.3) + +**Goal:** misbehaviour is cheaper to punish than to commit; honest-but-offline is punished less than malicious. + +### Slashing conditions taxonomy + +| # | Condition | Role | Detection | Tier | +|---|---|---|---|---| +| S1 | Missed audit vote in assigned batch | Auditor | On-chain (already implemented) | Partial | +| S2 | Missed T1/T2 aggregation submission | Aggregator | On-chain (already implemented) | Partial | +| S3 | Score deviation beyond threshold from cross-auditor median (per model, per GI) | Auditor | On-chain comparison at `finalizeEvaluation` — requires median computed on-chain | Partial → Full on repeat | +| S4 | Invalid aggregation: submitted T1/T2 CID whose recomputation from inputs fails verification | Aggregator | Off-chain challenge + on-chain dispute (S4 is the main dispute-resolution consumer) | Full | +| S5 | Repeated liveness failures (≥ `r` partial slashes within `w` GIs) | Both | On-chain counter | Full + jail/blacklist | +| S6 | Registration without capacity (registers, never participates, across models) | Both | On-chain counter | Partial, escalating | + +### Penalty tiers + +- **Partial slash:** fixed fraction of `minStake` (e.g., 25–50%), not the flat `minStake()` used today — today's flat amount fully unstakes a floor-staked validator for one missed vote, which over-punishes liveness faults. Make the fraction a DAO parameter. +- **Full slash:** entire slashable stake (active + unbonding) + blacklist. Reserved for provable malice (S4) and recidivism (S5). +- **Jailing:** `_syncValidatorStatus` already produces a Jailed state below min stake; keep — jailed validators cannot register until they top up. + +### Slashed-stake destination (decision required in P3-4.2) + +Options: (a) burn, (b) treasury, (c) redistribute to honest validators of the same GI, (d) split. +**Recommendation: split — 50% burn / 50% treasury.** Burning creates a deflationary sink and avoids perverse incentives (validators profiting from peers being slashed encourages false disputes); the treasury half funds dispute adjudication costs. Do **not** redistribute to the reporting validator beyond a fixed dispute bounty (see below). Today's behaviour (tokens stranded in the stake contract) is an accidental burn — make it explicit either way. + +### Dispute resolution (P3-4.3) + +- Any staked validator can open a dispute against an S3/S4 outcome within a **dispute window** (e.g., 1 GI or fixed time) by posting a **dispute bond** (forfeited if frivolous, returned + fixed bounty from treasury if upheld). +- Upheld dispute → re-evaluation by a **fresh validator subgroup** (reassignment selector excludes accused + original batch). +- Failure modes to document (P3-DOC3): validator offline mid-evaluation, dispute window expiry, all-validator collusion (out of threat scope > ~50%, per Roadmap §5). + +### Invariants (feed P3-6.3b fuzz tests) + +- Total staked supply never increases after a slash. +- Slashed amount always reaches its declared destination (burn address / treasury), never a third party. +- A slash can never make `activeStake + pendingWithdrawals` negative or leave a validator Active below min stake. + +--- + +## 5. Rewards — clients, auditors, aggregators + +**Goal:** every honest contribution in a GI is paid, proportionally to value, from a funded per-GI pool. Today there is **no** payout path — this is the largest 2.0 gap. + +### Funding: the per-GI reward pool + +Each GI's pool is funded from (in priority order): + +1. **Model-owner reward deposit** — the existing `totalDepositedRewards` deposit in `DINTaskAuditor`, made a *precondition of `startGI`* (a GI cannot open unless the pool covers the configured per-GI minimum). Market-set: model owners compete for validator/client attention via pool size (P3-5.1 "market-set pool"). +2. **Protocol emission top-up** (bootstrap phase only) — per-GI minted subsidy from the emission contract (P3-5.2), decaying on schedule, so early networks with few paying model owners still reward participation. Steady state: fees fund everything, emission → 0. + +### Split across roles (DAO-settable fractions, initial proposal) + +| Share | Recipient class | Basis | +|---|---|---| +| **60%** | Clients | Proportional to BlockFLow marginal-gain score among *accepted* models (score ≤ 0 or rejected ⇒ 0) | +| **20%** | Auditors | Equal per completed audit assignment in the GI (correctness enforced by slashing, not reward weighting — mirrors BlockFLow's dropped honesty-score, Roadmap §2) | +| **15%** | Aggregators | Per finalized T1/T2 batch completed | +| **5%** | Treasury | Network cut (see Fees §7) | + +### Settlement & claims + +- **Settlement cadence:** per-GI, at `endGI`. `endGI` computes each participant's entitlement and credits an on-chain `claimable[address]` balance — **pull-payment claim pattern** (`claimRewards()`), never push transfers in loops (gas + reentrancy). +- **What's on-chain vs off-chain:** scores land on-chain (auditor submissions, already implemented); the median + proportional split must be computed on-chain at settlement so payouts are verifiable. Heavy scoring math stays off-chain in the Python services; the chain only sees submitted scores. +- **Unclaimed rewards:** claimable indefinitely; no expiry for 2.0. +- **Anti-gaming:** duplicate-update discounting is inherent in BlockFLow sequential fold-in (P3-SCR verifies); score inflation by a single auditor is bounded by the median; a client scoring ≤ pass-score threshold earns nothing, so spam costs compute for zero return. + +### Open decisions + +- Exact split percentages (simulate in P3-5.2 at 10–50 validators / 100–500 clients per model). +- Whether aggregator reward should scale with batch size (recommend yes, linear in models aggregated). +- Denominating pool minimums in DIN vs fiat-oracle terms — recommend plain DIN for 2.0, no oracle dependency. + +--- + +## 6. Scoring & auditing — client local models (WP 2.2 ✅, P3-SCR) + +**Goal:** turn each client's submitted update into (a) an accept/reject decision and (b) a contribution score driving reward share — without auditors ever seeing raw data. + +### Pipeline (per model, per GI) + +1. **Pre-filters (cheap, off-chain, per auditor):** norm-bound check (rejects scaling/garbage), basic sanity (shape, dtype, NaN). +2. **BlockFLow-inspired scoring (off-chain, per auditor):** marginal-gain of the update vs current global model on the auditor's held-out audit test set (assigned via `assignAuditTestDataset`); cosine-to-consensus; sequential fold-in with permutation averaging (discounts duplicates). +3. **On-chain submission:** each auditor in the batch submits score + eligibility vote (`setAuditScorenEligibility`) — already implemented. +4. **Consensus:** **median across the auditor batch** is the canonical score (tolerates < 50% dishonest auditors). 2.0 change: compute the median on-chain at `finalizeEvaluation` so it is the slashing reference (S3) and the reward basis (§5) — today eligibility voting exists but the canonical per-model score isn't derived on-chain. +5. **Eligibility:** median ≥ `passScore` (`updatePassScore`, exists) ⇒ model enters T1 aggregation; else rejected, zero reward. +6. **Auditor accountability:** |auditor score − median| > deviation threshold ⇒ S3 slash candidate. Threshold must be validated empirically in P3-SCR before slashing on it (avoid punishing honest variance from heterogeneous audit test sets — start with a wide threshold + warning-only "shadow mode" for the first weeks of DevNet 2.0, then tighten). + +### Threat coverage (per Roadmap §5) + +In scope: crude poisoning (label-flip, sign-flip, scaling, garbage — caught by norm bound + marginal-gain gate), Sybil clients (spam costs compute, earns nothing below pass score), score inflation (median-bounded). Out of scope for 2.0: backdoors (RES-2), >50% auditor collusion. + +### Aggregator auditing + +T1/T2 outputs are deterministic functions of accepted inputs ⇒ verifiable by recomputation. 2.0 mechanism: any validator may recompute and open an S4 dispute; no proactive re-audit of every batch (cost). Proactive spot-checking can come later. + +--- + +## 7. Tokenomics — DIN token (P3-5.1 / 5.2) + +**Goal:** DIN is the unit of stake, reward, and fee — with a supply policy that doesn't hyperinflate during bootstrap or starve rewards at steady state. + +### Utility (demand sinks) + +1. **Staking** — validators must hold and lock DIN (the primary sink). +2. **Fees** — model-owner fees payable in DIN (see migration note in §8; today they're ETH). +3. **Reward-pool deposits** — model owners pre-fund GI pools in DIN. +4. **Governance** — future DAO weight (P5+, design hook only). + +### Supply policy + +| Flow | Mechanism | 2.0 design | +|---|---|---| +| **Mint — deposit** | `depositAndMint` ETH→DIN at `dinPerEth` (exists) | Keep for devnet as the faucet/on-ramp; the deposited ETH backing should route to treasury, not `owner()` withdrawal. Flag: this is effectively an unlimited mint at a fixed admin-set rate — acceptable for devnet, must be capped or replaced before any real-value network. | +| **Mint — emission** | None today | New emission contract (P3-5.2): per-GI reward subsidy, **diminishing schedule** (recommend geometric decay per epoch), hard `MAX_SUPPLY` cap, inflation guard (per-GI mint ceiling), mint triggers tied to GI lifecycle events only. | +| **Burn — slashing** | Accidental (stranded in stake contract) | Explicit: slashed-stake burn share (§4). | +| **Burn — fees** | None | Optional: burn a fraction of protocol fees (decide in P3-5.3; start at 0%, keep the hook). | +| **Treasury** | None (owner-withdraw patterns) | Dedicated treasury address/contract receiving: network fee split, slashed-stake share, deposit backing. DAO-controlled spend (multisig path per Roadmap §1). | + +### Modelling requirement (P3-5.2, before contracts) + +Simulate: does the pool + emission produce stable validator income at target / 10× / 0.1× participation? Wrong answers here are expensive post-audit — design time is scheduled ahead of contract code, keep it that way. + +--- + +## 8. Fee mechanism — model owners & network treasury (P3-5.3) + +**Goal:** model owners pay for the network services they consume; a slice of every flow sustains the treasury. + +### Fee lines + +| Fee | Payer | When | Exists today? | Routing | +|---|---|---|---|---| +| **Model registration fee** (open-source vs proprietary tiers) | Model owner | `requestModelRegistration` | ✅ (ETH, in `DINModelRegistry`, owner-withdrawable) | → treasury (change from `withdrawFees(owner)`) | +| **Manifest update fee** | Model owner | `requestManifestUpdate` | ✅ (ETH) | → treasury | +| **Per-GI service fee** | Model owner | Precondition of `startGI` | ❌ — this **is** the reward-pool deposit (§5); fee and pool are one flow | → validator pool (95%) + treasury network fee (5%) | +| **Storage cost line item** | Model owner | Per-GI, alongside service fee | ❌ | Design the routing slot now, keep at 0 until Filecoin migration (RES-1) | +| **Dispute bond** | Disputing validator | Dispute open | ❌ | Returned + bounty if upheld; → treasury if frivolous | + +### Routing contract (P3-5.3) + +Single fee-router with DAO-settable split fractions: `modelOwner → {validatorPool, treasury, burn, storage}` with fractions summing to 100%. All splits are basis-point parameters behind governance hooks (P3-5.1); nothing hard-coded except sane bounds (e.g., treasury cut ≤ 20%). + +### Denomination decision + +Registration fees are ETH today; staking/rewards are DIN. **Recommend converging on DIN for all protocol fees in 2.0** (single-asset accounting, reinforces token utility; the `depositAndMint` on-ramp makes acquisition trivial on devnet). Migration: add DIN-denominated fee params alongside, deprecate ETH fees after one release. If ETH fees are kept, the router must handle both assets — added complexity for no devnet benefit. + +### Dynamic fees + +Admin/DAO-settable base rate + per-role multipliers (P3-5.3). No algorithmic (demand-based) fee logic in 2.0 — parameterize now, automate later. + +--- + +## 9. Consolidated open-decision list + +Decisions that must be made (with owner + roadmap slot) before DevNet 2.0 contracts freeze: + +1. **Slashed-stake destination** — burn/treasury/split (rec: 50/50). Owner: Umer, P3-4.2. +2. **Partial-slash fraction & S3 deviation threshold** — needs P3-SCR empirical data; shadow-mode first. Owner: Umer, P3-4.1 + P3-SCR. +3. **Reward split percentages across roles** — simulate in P3-5.2. Owner: Umer. +4. **Fee denomination** — DIN-only vs dual-asset (rec: DIN-only). Owner: Umer, P3-5.3. +5. **Emission schedule shape + MAX_SUPPLY** — P3-5.2 simulation. Owner: Umer (design), Robbert (contract). +6. **Per-model stake requirement bounds** — P3-5.1. Owner: Umer. +7. **Dispute bond size & window length** — P3-4.3. Owner: Umer (design), Robbert (contract). +8. **`depositAndMint` cap/retirement plan for testnet** — flag before audit. Owner: Umer. + +--- + +## 10. Cross-references + +- `Developer/ROADMAP.md` — WP scheduling (P3-4.x, P3-5.x, P3-SCR, P3-6.x, P3-DOC2/3/5). +- `Documentation/public/workflows/din-workflow.md`, `Documentation/public/workflows/model-workflow.md` — current GI lifecycle these mechanisms attach to. +- `foundry/src/DinValidatorStake.sol`, `DINTaskCoordinator.sol`, `DINTaskAuditor.sol`, `DINModelRegistry.sol`, `DinCoordinator.sol`, `DinToken.sol` — current implementations referenced in §2. +- Roadmap Discussion §2 (BlockFLow over Shapley), §3 (Filebase→Filecoin fee hook), §5 (threat model scope). +- PR #13 upgradeable proxy layer — all new mechanism contracts should follow its Transparent Proxy + `_disableInitializers()` + `__gap` conventions from day one. diff --git a/Developer/feasibility-report.md b/Developer/design/feasibility-report.md similarity index 100% rename from Developer/feasibility-report.md rename to Developer/design/feasibility-report.md diff --git a/Developer/design/sdk-interface-proposal.md b/Developer/design/sdk-interface-proposal.md new file mode 100644 index 0000000..ff09048 --- /dev/null +++ b/Developer/design/sdk-interface-proposal.md @@ -0,0 +1,402 @@ +# DIN-SDK — Public Interface & JSON Output Conventions (proposal v0.3) + +**Issue:** #20 · **Branch:** `feat/din-sdk` · **Status:** proposal for review (candidate seed for P3-DOC7) + +Proposes the public interface and output conventions for `dincli/sdk/`, extracted from the existing +`dincli` code. Concrete before/after from the current code so it can be reviewed against DOC7 and, if +accepted, become its seed. Nothing here is built yet. + +> **v0.2** incorporated a code-audit of v0.1: plan/apply as the *default* for irreversible ops; +> wallet/session non-interactive *by contract*; typed amounts stringified only at the JSON boundary; +> `meta` gains `schema_version`/`chain_id`/`correlation_id`; `error.details` sanitization; finer tx/rpc +> subcodes; summary/detail request objects; golden-CLI + import-boundary tests; tx retry-state. +> +> **v0.3** incorporates a second code-audit against real call sites. Changes: `TxReceiptInfo` now +> carries `contract_address` + raw receipt/decoded events (deploy flows read `tx_receipt.contractAddress`; +> registry flows call `process_receipt`); plan/apply for CID-producing ops is a **three-phase** +> prepare→upload→on-chain split (register uploads the manifest mid-flow today, so a "pure" plan can't +> yield a CID); `SignerProvider` given an explicit Protocol; `sanitize_details()` is allowlist-oriented +> (per-code schema); import-boundary test runs in a fresh subprocess; fixed an invalid illustrative +> snippet. + +--- + +## 1. Design principles + +1. **Return data, don't print.** SDK functions return dataclasses. Never `console.print`, `rich` + tables, or stdout. +2. **Raise, don't exit.** SDK functions raise typed domain exceptions (`DinError` subclasses). Never + `typer.Exit` / `sys.exit`. +3. **Non-interactive by contract.** SDK functions never perform interactive I/O — not `typer.confirm`, + **and not `getpass()` or any hidden prompt**. A missing password/signer/decision raises (§2, §6) + rather than blocking on input. This is what makes the SDK safe for `dind`. +4. **JSON-serializable by construction.** Every returned dataclass serializes to the envelope in §3 via + the shared encoder only (bytes→hex, wei→string, enums→name). +5. **Importable standalone.** `dincli/sdk/` imports without `typer`, `rich`, or any `dincli.cli.*` + module (enforced by a test — §7). The CLI depends on the SDK, never the reverse. + +The CLI keeps its exact current behavior; it becomes a thin layer that calls the SDK, renders the +result, and maps exceptions to exit codes (§7). + +--- + +## 2. The session object + +Replaces the runtime-resolution parts of `DinContext`, minus console/exit — and minus prompting. + +```python +# dincli/sdk/session.py +class DinSession: + """Lazily resolves network, web3, account, config. No printing, no exit, no prompts.""" + def __init__(self, network: str | None = None, wallet: str | None = None, + signer: SignerProvider | None = None): ... + + @property + def network(self) -> str: ... + @property + def w3(self) -> Web3: ... # raises NetworkError if RPC unreachable + + @property + def account(self) -> LocalAccount: + """Resolve the signing account via the configured SignerProvider. + + NEVER calls getpass or any interactive prompt. Raises SignerUnavailable + if no non-interactive credential is available, WalletError if the + keystore is malformed or missing. + """ + + @property + def config(self) -> dict: ... +``` + +Today `load_account` can fall back to an interactive `getpass` prompt; the SDK must not. The consumer +supplies the `SignerProvider`: the CLI's adapter *may* prompt; the daemon's is backed by a configured +secret/agent. The SDK core only ever sees the provider result — or a raised error. + +```python +# dincli/sdk/session.py — the signing contract (no prompts inside the SDK) +@runtime_checkable +class SignerProvider(Protocol): + def address(self) -> str: ... + # checksummed address for this signer; raises SignerUnavailable if none resolvable + def can_decrypt(self) -> bool: ... + # True if this provider holds/points to a usable key without further input + def sign_transaction(self, tx: dict) -> SignedTransaction: ... + # raises SignerUnavailable (no credential), WalletError (bad keystore/decrypt failure) +``` + +- **CLI adapter** wraps the existing keystore flow; it *is* allowed to prompt (that's the interactive + boundary), then hands a resolved signer to the session. +- **Daemon adapter** is backed by a configured password source / signing agent and **must never** + prompt — on any missing credential it raises `SignerUnavailable`, which the daemon turns into a job + error rather than a hang. + +Contract getters and tx-param building move to `sdk/contracts.py` and `sdk/tx.py`, taking a `DinSession`. + +--- + +## 3. JSON output convention (the contract) + +SDK functions **return dataclasses with typed fields** (`int` for wei, `bool`, enums, etc.). The JSON +*envelope* — with amounts stringified — is produced only at the consumer boundary (daemon always; CLI +on `--json`). Python consumers keep numeric ergonomics; JS consumers get precision-safe strings. + +```jsonc +{ + "status": "ok", // "ok" | "error" + "data": { /* dataclass, serialized */ },// null on error + "error": null, // null on success; object below on error + "meta": { + "schema_version": "din-sdk-envelope/v1", // bump on breaking envelope changes + "sdk_version": "0.1.0", + "network": "sepolia_op_devnet", + "chain_id": 11155420, + "correlation_id": "…" // optional for CLI, first-class for daemon jobs/logs + } +} +``` + +Error shape: + +```jsonc +{ + "status": "error", + "data": null, + "error": { + "code": "tx_reverted", // stable machine string (§4) + "message": "T1 aggregation submission reverted", // human-readable + "details": { "tx_hash": "0x…", "gi": 3, "nonce": 42 } // structured, sanitized (§4a) + }, + "meta": { "schema_version": "din-sdk-envelope/v1", "network": "sepolia_op_devnet", "chain_id": 11155420 } +} +``` + +**Serialization rules** (shared encoder, `sdk/serialize.py`): +- **Amounts are typed `int` inside SDK dataclasses**; the encoder emits wei/token/`uint256` values as + decimal **strings** (JS `Number` precision). Fields opt in via metadata, e.g. + `fee_wei: int = field(metadata={"json": "uint256_string"})`. Small counts (GI number, indices, enum + ints) stay JSON numbers. +- `bytes` / `HexBytes` → `0x` hex string. Addresses → checksummed string. `Decimal` → string. +- Enums (e.g. GI state) → name string; include the int in `details` where useful. + +--- + +## 4. Exception hierarchy + +```python +# dincli/sdk/errors.py +class DinError(Exception): + code: str # stable, machine-readable (goes into error.code) + def __init__(self, message: str, *, details: dict | None = None): ... + +class ConfigError(DinError): code = "config_error" +class NetworkError(DinError): code = "network_unreachable" # rpc subcodes below +class WalletError(DinError): code = "wallet_error" +class SignerUnavailable(DinError): code = "signer_unavailable" # no non-interactive credential (§2) +class ManifestError(DinError): code = "manifest_error" +class IpfsError(DinError): code = "ipfs_error" +class ContractError(DinError): code = "contract_error" +class NotFoundError(DinError): code = "not_found" +class ValidationError(DinError): code = "validation_failed" # e.g. wrong GI/state +class TransactionError(DinError): code = "tx_failed" # stable subcodes below +class ConfirmationRequired(DinError): code = "confirmation_required" # see §6 +``` + +**Reserved stable subcodes** (set as `code` on the raised error; daemon retry policy keys off these): +- Transaction: `tx_estimation_failed`, `tx_reverted`, `tx_timeout`, `tx_nonce_conflict`, + `tx_replacement_underpriced`, `receipt_missing`. +- RPC/network: `rpc_unreachable`. +Contract revert reasons stay optional inside `details`; the top-level `code` must be stable enough to +drive retry decisions on its own. + +### 4a. `error.details` sanitization rules + +`details` is structured context, never a secret sink — critical once daemon logs are JSON and +long-lived. Sanitization is **allowlist-oriented**, not a generic scrubber (a generic scrubber misses +secrets nested in arbitrary values). Each error `code` declares a **schema of allowed detail keys**, +each with a type and bound; `sanitize_details(code, raw)` drops anything not in that code's allowlist: + +- Free-form / unlisted keys are **discarded**, not passed through. +- String values are length-bounded (e.g. revert reasons capped; `stdout`/`stderr` truncated to a + bounded tail). +- URLs are explicitly sanitized to host + path, stripping any embedded API keys / credentials. +- Never allowlisted anywhere: private keys, wallet passwords, mnemonic/seed material, raw env secrets. + +Example: `tx_reverted` allows `{tx_hash: hex, nonce: int, gi: int, reason: str<=256}`; a stray +`raw_signed_tx` or `password` key simply never survives `sanitize_details`. + +--- + +## 5. Function interface patterns — real before/after + +### 5a. Reads: compute-and-print → return data (summary vs detail) + +**Before** (`dindao.py::list_pending_requests`): loops contracts, `console.print`s each row, returns +nothing; fields by raw tuple index (`req[6]`). + +**After** — list endpoints return lightweight *summaries*; single-item lookups return *detail* objects: + +```python +# dincli/sdk/registry.py +@dataclass +class ModelRequestSummary: + request_id: int + requester: str # checksummed address + open_source: bool + fee_wei: int = field(metadata={"json": "uint256_string"}) + processed: bool + +@dataclass +class ModelRequestDetail(ModelRequestSummary): + manifest_cid: str | None + task_coordinator: str | None + task_auditor: str | None + approved: bool + approved_model_id: int | None + created_at: int | None # block timestamp if available + +@dataclass +class PendingRequests: + model_requests: list[ModelRequestSummary] + manifest_requests: list[ManifestUpdateRequestSummary] + +def list_pending_requests(session: DinSession, req_type: str | None = None) -> PendingRequests: ... + # raises ValidationError if req_type not in {None,'model','manifest'} + +def get_model_request(session: DinSession, request_id: int) -> ModelRequestDetail: ... + # raises NotFoundError if absent (replaces explore_request's broad except→Exit) +``` + +CLI renders the tables it prints today; daemon reads `.model_requests` for its job queue. This is also +the call site P4-IDX2 later swaps for an indexed query — extracting it now gives Robbert a clean seam. + +### 5b. Transactions: the keystone `build_and_send_tx` + +**Before** (`utils.py::build_and_send_tx`): takes `ctx` + three human strings, prints progress + tx +hash, and `raise typer.Exit(1)` on estimation/revert. + +**After** — returns retry-informative state; raises with stable subcodes: + +```python +# dincli/sdk/tx.py +@dataclass +class TxReceiptInfo: + tx_hash: str # 0x hex + status: int # 1 success, 0 revert + block_number: int + gas_used: int + nonce: int + contract_address: str | None # set for deployment txs (deploy flows read this) + logs: list[dict] # raw, JSON-serializable log entries + _raw: "TxReceipt" = field(repr=False, metadata={"json": "omit"}) + # web3 AttributeDict receipt, kept for in-SDK event decoding; NOT serialized + +def send(session: DinSession, contract_function, *, tx_params: dict | None = None, + on_event: Callable[[str, dict], None] | None = None) -> TxReceiptInfo: + """Estimate → build → sign → send → wait. Returns receipt info. + Raises TransactionError with subcode tx_estimation_failed | tx_reverted | tx_timeout | + tx_nonce_conflict | tx_replacement_underpriced | receipt_missing (§4).""" + +def decode_events(receipt: TxReceiptInfo, contract_event) -> list[dict]: + """Decode a specific event from a receipt, e.g. registry.events.ModelRegistrationRequested(). + Wraps web3's process_receipt so the operations layer can build typed results + (e.g. ModelRegistrationResult.request_id) without exposing raw web3 objects to consumers.""" +``` + +Why the richer receipt: deploy flows read `tx_receipt.contractAddress` (`dindao.py`, +`modelownerd/deploy.py`) → `contract_address`; registry flows decode events via +`contract.events.X().process_receipt(tx_receipt)` (`task.py`) → `decode_events()` over the retained +`_raw`. The serialized surface (`to_envelope`) omits `_raw`; operations decode events into their own +typed result dataclasses. + +Retry safety (§10): on failure the raised `TransactionError.details` carries enough to decide a retry — +`tx_hash` **if broadcast**, `nonce` if known, and a `broadcast: bool` flag (did we hit the network +before failing?). No message strings, no printing: the CLI supplies its own text and prints the +explorer URL; the daemon logs structured `on_event("submitted", {"tx_hash": …})`. + +### 5c. IPFS: already SDK-shaped + +`services/ipfs.py` already returns values and raises real exceptions. Extraction = move to +`sdk/ipfs.py`, replace the ~3 `console.print("Uploading via …")` lines with `logger`/`on_event`, wrap +raw `RuntimeError` into `IpfsError`. + +--- + +## 6. Confirmations without a console — plan/apply is the default + +Several pipelines gate on `_confirm_or_exit` / `typer.confirm` *mid-logic* (`task.register_request`, +`client.train_lms`, `model.*`, `gi.start`). **Plan/apply is the standard interface for all irreversible +operations** — on-chain writes, file mutations, uploads that feed on-chain state, and task execution. + +**`plan` is strictly pure — it performs no side effects, including no IPFS upload.** This matters +because IPFS upload is itself an irreversible, CID-producing side effect: `register_request` today +uploads the manifest mid-flow (`task.py`) precisely to obtain the CID it then submits on-chain. A plan +therefore cannot both be pure *and* contain a freshly-produced CID. For CID-producing operations the +split is **three-phase** — prepare → upload → on-chain-apply: + +```python +def register_request_prepare(session, manifest_path) -> RegisterPlan + # PURE. Reads/validates the manifest, resolves the fee, and describes intent: + # RegisterPlan(manifest_path=…, fee_wei=…, would_upload=True, manifest_cid=None, …) + # If a CID is already known, would_upload=False and manifest_cid is populated. + +def register_request_upload(session, plan) -> RegisterPlan + # SIDE EFFECT: uploads the manifest to IPFS, returns the plan with manifest_cid filled in. + # Skipped when plan.would_upload is False. + +def register_request_apply(session, plan) -> RegisterResult + # ON-CHAIN: requires plan.manifest_cid set; submits the tx. Decision already made upstream. +``` + +- **CLI:** `prepare` → render (incl. "will upload manifest at ") → `_confirm_or_exit` → + `upload` → `apply`. +- **Daemon:** `prepare` → evaluate the inspectable plan against policy/preferences → `upload` → + `apply` (or stop after `prepare` if policy rejects — nothing was uploaded or spent). + +Operations with **no** CID-producing step collapse to the simple two-phase `prepare`/`apply`. +`assume_yes: bool` / an injected `confirm` callback exist **only as a CLI-compatibility adapter**, not +as the SDK's primary design. Where no decision is supplied, the default policy **raises +`ConfirmationRequired`** rather than silently proceeding. + +--- + +## 7. Backward-compatibility guarantees + test strategy + +The refactor is internal. Guaranteed unchanged: **command names/arguments/options**, **human-readable +output**, **exit codes** (CLI maps `DinError` → today's non-zero exits; success `0`), and +**config/keystore/manifest on-disk formats and locations**. + +Test strategy: +- **Keep the current suite green** — baseline on `develop` @ `8c4735d`: **70 passed, 1 skipped** across + `tests/test_*.py` (the earlier connect-wallet / dintoken failures were fixed by PR #16 follow-ups in + `c12b9e1`). The `tests/dincli/` integration suite is tracked separately — it hardcodes an + environment path and needs a live Hardhat/IPFS/Docker stack, so it's out of scope for this refactor's + green-bar. +- **Golden CLI-compatibility tests** for the first extracted commands: assert command names/options, + exit codes, and key output lines — "same human-readable output" needs enforcing, not just "tests + pass". +- **Import-boundary test:** assert `import dincli.sdk` pulls in no `typer`, `rich`, or `dincli.cli.*` + — directly enforces principle 5. **Must run in a fresh Python subprocess** (else another test having + already imported those modules gives a false pass/fail), e.g. + `subprocess.run([sys.executable, "-c", "import dincli.sdk, sys; assert not {'typer','rich'} & set(sys.modules); assert not any(m.startswith('dincli.cli') for m in sys.modules)"])`. +- **New SDK-level unit tests** where extraction creates a directly-callable function — logic that today + can only be exercised through `CliRunner` (because it's welded to the Typer command signature) + becomes unit-testable in isolation once it lives in the SDK. + +--- + +## 8. Proposed package layout + +``` +dincli/sdk/ + __init__.py # curated public exports + __version__ + session.py # DinSession (was DinContext, minus console/exit/prompts) + SignerProvider + errors.py # DinError hierarchy + reserved subcodes (§4) + sanitize_details (§4a) + serialize.py # to_envelope() + shared JSON encoder (§3) + config.py # config/env/network/IPFS-config resolution (from utils.py) + wallet.py # keystore/account/demo-key, non-interactive signing (from utils.py + system.connect_wallet) + web3.py # get_w3 (from utils.py) + contracts.py # ABIs + contract getters + artifact resolution (contract_utils + DinContext) + cid.py # from services/cid_utils.py (verbatim) + ipfs.py # from services/ipfs.py (§5c) + manifest.py # manifest/CID-cache/din_info (from utils.py + DinContext) + runtime.py # from services/runtime.py (verbatim) + worker.py # from cli/worker.py (console param → on_event/logger) + state.py # GI-state enums/converters + validate_* predicates (return/raise) + tx.py # send() + tx-param building (§5b) + operations/ # role operations returning dataclasses (plan/apply where irreversible) + registry.py gi.py aggregation.py auditor.py client.py model.py dao.py +``` + +Consumers: +- **CLI:** `command → build DinSession (interactive SignerProvider) → call sdk fn → render / map errors + to exit codes` (+ optional `--json` emits `to_envelope(...)`). +- **Daemon:** `job → build DinSession (non-interactive SignerProvider) → call sdk fn → + to_envelope(...) → update state/job queue`, keying retries off the stable error subcodes. + +--- + +## 9. Resolved decisions (from review) + +1. **Envelope shape** — `{status, data, error, meta}`, with `schema_version`, `sdk_version`, + `network`, `chain_id`, and optional `correlation_id` in `meta`. ✔ +2. **Amounts** — typed `int` in SDK dataclasses; decimal **strings** for all wei/token/`uint256` at the + JSON boundary. ✔ +3. **Confirmations** — **plan/apply is the standard**; `assume_yes`/injected confirm is a CLI adapter + only. ✔ +4. **Session naming** — **`DinSession`** (signals the no-console / no-exit / no-prompt contract). ✔ +5. **Error taxonomy** — base hierarchy in §4 plus explicit, reserved **transaction and RPC subcodes** + now. ✔ +6. **Wallet** — **non-interactive by contract**: no `getpass`; raise `SignerUnavailable` / + `WalletError` instead. ✔ +7. **Doc home** — accepted version lives in **`Developer/design/`** as the DOC7 seed; summarize/link + from `Documentation/technical/ARCHITECTURE.md` once implemented. ✔ + +## 10. Idempotency / retry expectations (daemon-facing writes) + +The SDK need not implement full retry policy yet, but write results/errors must expose enough for a +consumer to retry safely: on success, `TxReceiptInfo` carries `tx_hash`, `nonce`, `status`; on failure, +`TransactionError.details` carries `tx_hash` **if broadcast**, `nonce` if known, and a `broadcast: bool` +flag distinguishing pre-broadcast failures (safe to rebuild) from post-broadcast ones (must confirm the +existing tx, not resend). Stable subcodes (§4) let the daemon pick the retry strategy. +``` \ No newline at end of file diff --git a/Documentation/technical/mechanisms/suggested-staking-mechanism.md b/Developer/design/suggested-staking-mechanism.md similarity index 100% rename from Documentation/technical/mechanisms/suggested-staking-mechanism.md rename to Developer/design/suggested-staking-mechanism.md diff --git a/Developer/discussion/add-filecoin-support.md b/Developer/discussion/add-filecoin-support.md index 46641e7..8639f9b 100644 --- a/Developer/discussion/add-filecoin-support.md +++ b/Developer/discussion/add-filecoin-support.md @@ -5,11 +5,14 @@ status: open participants: - Umer Majeed (Principal Engineer) - Abraham Nash (Protocol Founder) + - Similoluwa Abidoye (provider research, Discussion #18) related-issue: Developer/issues/filecoin-integration.md decision-target: P3 fee design (before August 2026) --- +> **Update (2026-07):** Hands-on evaluation in [Discussion #18](https://github.com/InfiniteZeroFoundation/DevNet/discussions/18) ruled out both providers recommended below. Lighthouse retrieval turned out to be payment-gated even for the uploader's own files (violates the "readers pay nothing" requirement). Storacha's infrastructure (console, API host, both retrieval gateways) is down/deprecated. The current leading candidate is **Filecoin Onchain Cloud** via the `filecoin-pin` CLI — see [`Developer/proposals/filecoin-onchain-cloud.md`](../proposals/filecoin-onchain-cloud.md) for the full writeup (Web/CLI/auth, Session Key delegation model, and dincli integration notes). The provider comparison and recommendation below predate that research and are kept for the record; treat the proposal doc as current. + ## Background DIN currently uses Filebase as its default IPFS pinning provider — a centralised S3-compatible service that wraps IPFS. All model artifacts (client updates, aggregated models, manifests, service files) flow through `dincli/services/ipfs.py` and are referenced on-chain by CID. The contract layer is already provider-agnostic; the coupling is entirely operational. @@ -91,20 +94,22 @@ The sponsored upload architecture (see below) requires on-chain storage budget a ## Best Filecoin-backed providers +*Superseded by hands-on testing — see the 2026-07 update at the top of this doc. Left in place for the record; do not use Lighthouse or Storacha based on this section.* + Three services offer Filecoin-backed storage with IPFS-compatible APIs that would slot into DIN's existing `custom` provider path with minimal friction: -### Lighthouse +### Lighthouse — ruled out - Filecoin storage with fast IPFS retrieval via gateway - Supports end-to-end encryption for stored files - Simple SDK and REST API - Per-account billing; no raw FIL management required -- **Strong candidate** for DIN's first Filecoin integration +- **Ruled out:** Discussion #18 found retrieval is payment-gated — even the uploader's own files return `402 Payment Required`. Fails the "readers pay nothing" requirement outright. -### Web3.Storage / Storacha +### Web3.Storage / Storacha — ruled out - W3C UCAN-based delegated upload capabilities natively supported - UCAN delegation is precisely the scoped credential model DIN needs for sponsored uploads (see below) - Multi-provider Filecoin storage -- **Strongest fit** for the sponsored upload architecture because delegated upload permissions are a first-class protocol primitive, not a custom workaround +- **Ruled out:** Discussion #18 found the console, API host, and both retrieval gateways (storacha.link, w3s.link) down/deprecated as of 2026-07. Filecoin Onchain Cloud's Session Key model is the closest available substitute for the UCAN delegation property (see [`Developer/proposals/filecoin-onchain-cloud.md`](../proposals/filecoin-onchain-cloud.md)). ### Filebase (Filecoin bucket option) - Filebase offers a Filecoin-backed storage option alongside its IPFS pin option @@ -112,7 +117,7 @@ Three services offer Filecoin-backed storage with IPFS-compatible APIs that woul - Less decentralised than Lighthouse or Storacha (Filebase is still the intermediary) - **Lowest-risk migration path** if the goal is just switching the backing layer without changing operational model -**Recommendation:** Lighthouse for a clean provider swap; Storacha/Web3.Storage if delegated upload capabilities are a P3 priority. +**Recommendation (superseded, see 2026-07 update above):** Lighthouse for a clean provider swap; Storacha/Web3.Storage if delegated upload capabilities are a P3 priority. --- @@ -213,7 +218,7 @@ If third parties can upload against a model owner's storage budget, explicit con ## Open questions -1. Which provider first — Lighthouse (simpler) or Storacha (native UCAN delegation)? +1. Which provider first — settled: neither Lighthouse (retrieval is payment-gated, even for the uploader's own files) nor Storacha (infrastructure is obsolete/deprecated). Current candidate is Filecoin Onchain Cloud — open question is whether its Session Key model covers the sponsored-upload delegation needs below as well as UCAN would have. 2. Should the storage broker be an off-chain DIN-operated service initially, with on-chain accounting added in P4? 3. What is the minimum per-task storage budget, and how is it priced relative to expected model size and round count? 4. Should audit datasets require Filecoin storage, or is short-lived IPFS pinning sufficient for the auditor path? @@ -228,7 +233,7 @@ If third parties can upload against a model owner's storage budget, explicit con | Should DIN add Filecoin support? | Yes — it is the correct long-term storage layer | | When? | Adapter now; sponsored upload in P3 fee design | | Primary blocker? | Sponsored upload credential architecture, not technical integration | -| Best provider? | Storacha for UCAN delegation; Lighthouse for simplicity | +| Best provider? | Filecoin Onchain Cloud (`filecoin-pin`) — Storacha is obsolete, Lighthouse retrieval is payment-gated (both ruled out by Discussion #18) | | Can model owner pay per model? | Yes — per-task storage budget funded at task creation | | Can abuse be prevented? | Yes — through scoped credentials, quotas, acceptance gating, and economic controls | | Contract changes required? | Not for provider swap; yes for per-task budget accounting (P3/P4) | diff --git a/Developer/discussion/ows-delegation-feasibility.md b/Developer/discussion/ows-delegation-feasibility.md new file mode 100644 index 0000000..e2f1716 --- /dev/null +++ b/Developer/discussion/ows-delegation-feasibility.md @@ -0,0 +1,188 @@ +# OWS (Open Wallet Standard) delegation — feasibility spike + +**Author:** Santiago Rodriguez Cetran +**Date:** 2026-07-07 +**Context:** `task_300626_3` §1c ("OWS integration exploration") · PR #16 review follow-up +**Status:** Hands-on spike complete. Supersedes the earlier desk-research write-up. +**Versions exercised:** `ows` CLI **1.4.2** (`76c8c67`), Python SDK `open-wallet-standard` **1.4.2** +(importable module name: **`ows`**), on Linux x86_64. Note: the bundled agent skill doc is +**v1.3.2** and is stale relative to the 1.4.2 package — this explains two discrepancies below +(import name and vault layout). + +--- + +## Why this exists + +The task asked whether `dincli` can use OWS as an account backend — enumerate accounts and +request a **signed transaction without ever handling the raw key** — and what the failure +modes are. The first pass answered from documentation only; the PR #16 review flagged that as +a gap. This document records an actual spike: OWS installed, wallets created and inspected on +disk, a real Optimism Sepolia transaction signed and cryptographically verified. + +**Method:** local-verify (no on-chain broadcast, no funds). A signature is proven correct by +recovering the signer address from it — no funds and no private key required. + +## Verdict — scoped precisely + +- **CONFIRMED:** OWS can sign an EVM (Optimism Sepolia, chainId 11155420) transaction and + return a signature that recovers to its own account, **without exposing the private key** to + the caller. `dincli` can enumerate accounts and drive this via CLI subprocess or the + in-process Python SDK. No daemon required. +- **NOT yet proven (do not ship on this basis):** the stronger production claim — *"`dincli` + can safely support OWS delegation with a mandatory, scoped signing policy."* The verified + path is **unscoped local vault signing by wallet name**, which appears to bypass the + policy/API-key layer entirely, and (see below) is **not even passphrase-gated**. Policy- + enforced signing (the `key`/`policy` "agent access layer", and any REST/MCP transport) was + **not exercised**. Until that is tested end-to-end — plus passphrase behavior and adapter + compatibility — OWS delegation is a **follow-up**, not a production recommendation. + +## What OWS is (verified from the running binary + on-disk vault) + +`ows` 1.4.2 — a local, multi-chain wallet manager (Rust core). Keys live in an encrypted vault +under `~/.ows/wallets/.json` (flat file per wallet; `ows_version: 2`. The v1.3.2 skill +doc's `~/.ows/wallets//wallet.json` layout is stale). **Crypto confirmed by reading the +keystore file directly** (not just docs): `crypto.cipher = "aes-256-gcm"`, `crypto.kdf = +"scrypt"` (cost params in `crypto.kdfparams`; `auth_tag`, `cipherparams`, `ciphertext` +present). One mnemonic derives addresses for 12+ chains via BIP-44. + +Access surfaces: **CLI**, **Node SDK** (`@open-wallet-standard/core`), **Python SDK** +(`open-wallet-standard`, PyO3 — Rust core in-process; import as `ows`). Install: +`curl -fsSL https://docs.openwallet.sh/install.sh | bash` (global `npm install -g +@open-wallet-standard/core` also provides the CLI). Command surface: +`wallet {create,import,export,delete,rename,list,info}`, `sign {message,tx,send-tx}`, +`key {create,list,revoke}`, `policy {create,list,show,delete}`, `mnemonic`, `fund`, `pay` +(x402), `config show`. + +## The four task questions, answered with evidence + +### 1. What interface does OWS expose? +CLI subprocess **and** an in-process Python SDK (`import ows`). Both produced a byte-identical +signature for the same input; no daemon/socket. A `key`/`policy` "agent access layer" with +scoped tokens exists in the command surface but its enforcement path (and any REST/MCP +transport) was **not exercised in this spike**. + +### 2. Can `dincli` enumerate accounts? +Yes. `ows wallet list` (CLI) and `ows.list_wallets()` (SDK) return structured data; each wallet +lists per-chain accounts, e.g. `{"chain_id":"eip155:1","address":"0x…","derivation_path": +"m/44'/60'/0'/0/0"}`. + +### 3. Can `dincli` request a signed tx without holding the raw key? +Yes (for EVM, unscoped local signing). `ows sign tx --wallet --chain 11155420 --tx +` and `ows.sign_transaction(,"11155420",)` both return a 65-byte +compact signature (`r‖s‖recovery_id`) — **not** an assembled tx, so the caller reassembles it +(or uses `sign send-tx --rpc-url …` to broadcast). Verified two independent ways (recover from +the signing hash via `eth_keys`, and `Account.recover_transaction()` on the reassembled raw +tx): both recover exactly the wallet's own address. See the transcript under **Reproduction**. + +### 4. Chain / account mapping (verified) +For EVM, OWS derives **one** secp256k1 account (`m/44'/60'/0'/0/0`) shared across all `eip155:*` +chains, so the enumerated `eip155:1` address **is** the signer for `--chain 11155420`. Proven: +the address recovered from a chainId-11155420 signature equalled the enumerated `eip155:1` +address byte-for-byte. The chainId only affects the signed payload (it is embedded in the tx +bytes). `dincli` would map its network → OWS `--chain ` (e.g. `sepolia_op_devnet` +→ `11155420`). Note: `ows config show` ships **no RPC for `eip155:11155420`** — irrelevant for +offline `sign tx` (which `dincli` broadcasts itself via its own `w3`), but `sign send-tx` would +need an explicit `--rpc-url`. + +### 5. Failure modes (all exercised) + +| Scenario | Observed | +|---|---| +| OWS not installed | exit **127**, `No such file or directory` — clean `FileNotFoundError` for fallback | +| Unknown wallet name | exit **1**, `error: wallet not found: ''` | +| Wrong chain (EVM bytes, `--chain solana`) | exit **1**, cryptic: `invalid transaction: transaction too short for declared signature slots` | +| Malformed tx bytes (`0xdeadbeef`) | ⚠️ exit **0** — **signed blindly** (see caveat 1) | +| Wrong passphrase (via SDK) | `RuntimeError: decryption failed: aead::Error` (clean failure) | +| "OWS not running" | N/A — no daemon; CLI/SDK are one-shot/in-process | + +## Passphrase / authentication of local signing — material finding + +In v1.4.2 there is **no CLI or env way to create a passphrase-protected wallet**: `ows wallet +create`/`import` expose no passphrase option, and `OWS_PASSPHRASE` at creation was **ignored** +(the resulting wallet signed with no secret and rejected `passphrase="secret"` with +`aead::Error`). Consequences for the verified flow: + +- A CLI-created OWS wallet is encrypted at rest but effectively **empty-passphrase**: `ows sign + tx`/`sign_transaction()` decrypt and sign with **no secret from the caller**. Protection + reduces to **filesystem permissions on `~/.ows/wallets/`**. +- This is **weaker** than `dincli`'s own encrypted keystore, which *requires* a passphrase to + decrypt. So the "highest security / dincli never sees the key" framing is only half-true: the + key isn't exposed to `dincli`, but neither is it passphrase-gated in the path we verified. +- The passphrase param *is* the KDF input (a wrong one fails with `aead::Error`), so a + passphrase-protected mode presumably exists via some path we did not find — that, plus + non-interactive credential passing without leaking through args/env/shell history, is + **untested and part of the follow-up**. + +## Security caveats + +1. **Blind signing.** `ows sign tx` (EVM) signs arbitrary bytes without validating they decode + as a well-formed tx (`0xdeadbeef` → exit 0 + signature). The caller owns payload correctness. +2. **Unscoped API keys, and policy enforcement unverified.** `ows key create --name X --wallet + Y` succeeds with **no `--policy`**, minting a token that can sign anything for that wallet. + Whether attaching a policy actually blocks a disallowed tx was **not tested** — the verified + local-signing path does not go through the key/policy layer at all. +3. **Unauthenticated local signing** (see the passphrase section) — filesystem-permission-only + protection in the verified flow. + +## Integration seam (for a future opt-in — larger than a drop-in) + +Every signature funnels through `DinContext.account`, used at `dincli/cli/utils.py:881` +(`signed_tx = account.sign_transaction(tx)`) and `dincli/cli/system.py:589`, followed by +`w3.eth.send_raw_transaction(signed_tx.raw_transaction)`. So an `OwsAccount` adapter must be +**eth-account-compatible**, not a thin shim — it has to: + +1. expose `.address` (from `list_wallets`, the `eip155:*` account); +2. accept a **web3 transaction dict** (as produced by `contract_function.build_transaction(...)` + / the ETH-transfer dict), including EIP-1559 fee fields; +3. **serialize** it to unsigned typed (EIP-1559) transaction bytes; +4. call `ows.sign_transaction(...)` to get `r/s/recovery_id`; +5. **reassemble** the signed raw tx (`0x02 || rlp([...,y_parity,r,s])`); +6. return a result object exposing **`.raw_transaction`** (and ideally `.hash`, `.r`, `.s`, `.v`) + so the existing `send_raw_transaction(signed_tx.raw_transaction)` call sites are unchanged. + +Gate on `ows`/`open-wallet-standard` being importable; fall back to the keystore path. + +## Recommendation + +- **Ship named multi-keystore as the default production path** (merged, self-contained, no + external dependency, passphrase-gated). +- **Treat OWS delegation as a follow-up, not production-ready as described.** EVM signing + without key exposure is genuinely feasible and low-integration-cost via the Python SDK — but + before recommending it for production the follow-up must verify end-to-end: (a) **policy- + enforced** signing (create a policy, scope an API key to it, confirm a disallowed tx is + rejected); (b) **passphrase-protected** wallets and non-interactive credential passing without + leaking secrets; (c) the eth-account-compatible **adapter** above. OWS remains an operator + tool we should **not vendor** (per the task). +- The `wallet-setup.md` corrections in this PR are independent of that follow-up and land now. + +## Reproduction + +``` +# versions +$ ows --version +ows 1.4.2 (76c8c67) +$ python -m pip install open-wallet-standard # 1.4.2; import module is `ows` + +# create + enumerate (no passphrase prompt) +$ ows wallet create --name ev + eip155:1 → 0xaDdA6AEe49109DbB1e87Fe4948Cf075df4C68c5B +$ ows wallet list # -> eip155:1 address above + +# unsigned EIP-1559 OP-Sepolia tx (chainId 11155420, nonce 0, to …dEaD, value 1e15, empty data/accessList): +UNSIGNED=0x02f083aa37dc80830f4240843b9aca0082520894000000000000000000000000000000000000dead87038d7ea4c6800080c0 + +$ ows sign tx --wallet ev --chain 11155420 --tx $UNSIGNED +87839cda2ce3434678d591fc7c8dfbc279109013567c9ef153093bc36a5707db685878656e2ba3879874b6a0128f88f3f2744fee3f827b22727d05ab138a1dbd01 +# (r‖s‖recovery_id; recovery_id=1) + +# verify (no key needed): recover signer from the signing hash and assert == enumerated address +# eth_keys.Signature(vrs=(1,r,s)).recover_public_key_from_msg_hash(TypedTransaction.from_dict(tx).hash()) +# observed -> 0xaDdA6AEe49109DbB1e87Fe4948Cf075df4C68c5B (== enumerated eip155:1 address) ✓ +``` + +The unsigned-tx construction uses `eth_account.typed_transactions.TypedTransaction` + +`_unsigned_transaction_serializer` (EIP-1559 fields RLP-encoded, prefixed with `0x02`). Address +values above are from one spike run; the mnemonic is random per `wallet create`, so a fresh run +yields a different address/signature but the same verification (recovered signer == enumerated +`eip155:1` address). Spike wallets and the API key created during testing were deleted +afterward (`ows wallet delete`, `ows key revoke`). diff --git a/Developer/issues/client-reward-mechanism.md b/Developer/issues/client-reward-mechanism.md index 627fd04..d8f0cfd 100644 --- a/Developer/issues/client-reward-mechanism.md +++ b/Developer/issues/client-reward-mechanism.md @@ -58,7 +58,7 @@ Required scoring outputs: - `eligible`: whether the client model/update passed admission checks; - `utility_score_bps`: normalized utility score used for model admission; - `contribution_score_bps`: normalized reward contribution score; -- `contribution_mode`: for example `leave_one_out` or `marginal_global_delta` (note: `tknn_shapley` and `dp_tknn_shapley` are rejected - see [Rejected Ideas: TKNN-Shapley](rejected-ideas/tknn-shapley.md)); +- `contribution_mode`: for example `leave_one_out` or `marginal_global_delta` (note: `tknn_shapley` and `dp_tknn_shapley` are rejected - see [Rejected Ideas: TKNN-Shapley](../rejected-ideas/tknn-shapley.md)); - `contributionReportCID`: off-chain report containing detailed contribution evidence; - optional `anomaly_score_bps` or disagreement fields for reward suppression. diff --git a/Developer/issues/manifest-acess-in-services.md b/Developer/issues/manifest-acess-in-services.md index d76bd74..9b776c1 100644 --- a/Developer/issues/manifest-acess-in-services.md +++ b/Developer/issues/manifest-acess-in-services.md @@ -31,7 +31,7 @@ There is already a helper in `dincli.cli.utils`: - `get_manifest_key(network, key, model_id=None, task_coordinator_address=None)` -And the documentation in `Documentation/manifest.md` says custom manifest fields can be accessed in services through `get_manifest_key`. +And the documentation in `Documentation/public/manifest.md` says custom manifest fields can be accessed in services through `get_manifest_key`. But there is a practical issue: @@ -56,7 +56,7 @@ Relevant code paths today: - `load_custom_fn()` dynamically imports a function from a service file - `dincli/cli/utils.py` - `get_manifest_key()` reads manifest data from cache or task directories -- `Documentation/manifest.md` +- `Documentation/public/manifest.md` - states that custom manifest fields can be accessed in services The current loading mechanism returns a callable, but does not inject: diff --git a/Developer/issues/scoring-mechanism/implementation.md b/Developer/issues/scoring-mechanism/implementation.md index 109860b..26c51be 100644 --- a/Developer/issues/scoring-mechanism/implementation.md +++ b/Developer/issues/scoring-mechanism/implementation.md @@ -23,7 +23,7 @@ Make the task’s scoring scenario explicit and machine-readable. - `Documentation/technical/manifest.md` - `cache_model_0/manifest.json` -- optionally `Documentation/auditors.md`, `Documentation/model-owner.md`, and `Developer/tooling/model-owner-services.md` +- optionally `Documentation/public/roles/auditors.md`, `Documentation/public/roles/model-owner.md`, and `Developer/proposals/model-owner-services.md` ### Changes @@ -280,12 +280,12 @@ Make the new scenario-driven model understandable for contributors and model own ### Files To Update -- `Documentation/technical/DINTaskAuditor.md` +- `Documentation/technical/contracts/DINTaskAuditor.md` - `Documentation/technical/manifest.md` -- `Documentation/auditors.md` -- `Documentation/model-owner.md` -- `Developer/tooling/model-owner-services.md` -- optionally `Developer/tooling/model-owner-contracts.md` +- `Documentation/public/roles/auditors.md` +- `Documentation/public/roles/model-owner.md` +- `Developer/proposals/model-owner-services.md` +- optionally `Developer/proposals/model-owner-contracts.md` ### Changes diff --git a/Developer/issues/validator_selection/implementation.md b/Developer/issues/validator_selection/implementation.md index 7dbaecb..6340183 100644 --- a/Developer/issues/validator_selection/implementation.md +++ b/Developer/issues/validator_selection/implementation.md @@ -23,7 +23,7 @@ Make workload requirements machine-readable and available to every role through - `Documentation/technical/manifest.md` - `cache_model_0/manifest.json` -- optionally `Documentation/model-owner.md`, `Documentation/aggregators.md`, and `Documentation/auditors.md` +- optionally `Documentation/public/roles/model-owner.md`, `Documentation/public/roles/aggregators.md`, and `Documentation/public/roles/auditors.md` ### Changes @@ -255,12 +255,12 @@ Make the new model understandable and reproducible for contributors and model ow ### Files To Update -- `Documentation/technical/DINTaskCoordinator.md` -- `Documentation/technical/DINTaskAuditor.md` +- `Documentation/technical/contracts/DINTaskCoordinator.md` +- `Documentation/technical/contracts/DINTaskAuditor.md` - `Documentation/technical/manifest.md` -- `Documentation/aggregators.md` -- `Documentation/auditors.md` -- `Documentation/model-owner.md` +- `Documentation/public/roles/aggregators.md` +- `Documentation/public/roles/auditors.md` +- `Documentation/public/roles/model-owner.md` ### Changes diff --git a/Developer/tooling/client-labeling.md b/Developer/proposals/client-labeling.md similarity index 100% rename from Developer/tooling/client-labeling.md rename to Developer/proposals/client-labeling.md diff --git a/Developer/proposals/filecoin-onchain-cloud.md b/Developer/proposals/filecoin-onchain-cloud.md new file mode 100644 index 0000000..4bf640f --- /dev/null +++ b/Developer/proposals/filecoin-onchain-cloud.md @@ -0,0 +1,159 @@ +# Filecoin Onchain Cloud (FOC) Integration Guide + +FOC is the Filecoin ecosystem's native cloud layer — verifiable storage, retrieval, and payments running through smart contracts (the Synapse SDK + Filecoin Warm Storage Service) rather than a company's backend. It launched on testnet November 18, 2025, and on mainnet January 31, 2026. + +This guide leads with **Filecoin Pin** — the `filecoin-pin` CLI/library, an IPFS-persistence layer built on top of FOC's core Synapse SDK — since it's the closer match to how `dincli` already thinks about storage (CID-referenced artifacts, CI/CD-friendly), rather than the lower-level Synapse SDK directly. Same three-part structure as the Storacha guide, since that's what's being replaced: Web UI, CLI, Authentication. + +## 1. Web — setting up access and managing data + +There's no account/email/console model the way Storacha has one. "Setting up an account" here means: get a wallet, fund it, and everything else happens through the CLI or SDK talking directly to the contracts. The web surfaces that do exist are narrower and split by purpose: + +- **A demo upload site** (`pin.filecoin.cloud`) exists for trying Filecoin Pin in the browser, but per Filecoin Pin's own glossary it runs on "a hardcoded wallet and session key on the Calibration network" — a shared demo credential, not your own. For kicking the tires only, not real usage. +- **`filecoin.cloud/service-providers`** — directory of storage providers with live performance stats, if you want to pick one manually instead of letting the SDK auto-select (auto-select is the recommended default). +- **PDP Explorer** (`https://pdp.vxb.ai/calibration/dataset/{datasetID}` on testnet) — proof-status viewer for a given Data Set: confirms your storage provider hasn't faulted on its onchain proofs. Closest thing to a "view my files" page, except it shows cryptographic proof state, not a file browser. `filecoin-pin data-set ` (or its alias `filecoin-pin dataset show `) is the CLI equivalent, querying the chain directly rather than a cache. +- **No general management console exists yet.** Filecoin Pin's own README lists this as "Planned" (tracked in [issue #74](https://github.com/filecoin-project/filecoin-pin/issues/74)). Right now it's CLI/Explorer only — worth knowing if anyone on the team expects a GUI. + +### Setting up access (the actual "account setup" step) + +```bash +# Node.js 24+ required +node --version + +npm install -g filecoin-pin +filecoin-pin --version +``` + +Then get a wallet and fund it: + +- **Calibration testnet** (for testing — data isn't permanent, infra resets regularly): fund with [test FIL](https://faucet.calibnet.chainsafe-fil.io/funds.html) (gas) and [test USDFC](https://forest-explorer.chainsafe.dev/faucet/calibnet_usdfc) (storage payments, a stablecoin). +- **Mainnet** (real usage): fund the wallet with real FIL and USDFC. + +**Important, and the opposite of what an earlier pass at this research assumed: the CLI defaults to Mainnet.** `filecoin-pin add myfile.txt` with no flags spends real money. Pass `--network calibration` explicitly to test first. This is worth being deliberate about — there's no "safe by default" here the way Storacha's free tier was. + +## 2. CLI — upload/retrieve workflow + +```bash +# 1. One-time payment setup — approves the Warm Storage contract and seeds a +# USDFC deposit sized from current on-chain pricing. +filecoin-pin payments setup --auto --network calibration + +# 2. Upload (defaults to 2 storage-provider copies for redundancy; --copies to change) +filecoin-pin add ./local-model.bin --network calibration +``` + +Output: +``` +Root CID: bafybeibh422kjvgfmymx6nr7jandwngrown6ywomk4vplayl4de2x553t4 +Piece CID: bafkzcibcfab4grpgq6e6rva4kfuxfcvibdzx3kn2jdw6q3zqgwt5cou7j6k4wfq +Transaction: 0xc85e49d2ed745cc8c5d7115e7c45a1243ec25da7e73e224a744887783afea42b +``` + +Two CIDs, and the distinction matters: +- **Root CID** — the actual IPFS content identifier. What you'd reference on-chain and what any IPFS gateway/tool retrieves by. +- **Piece CID** — Filecoin's cryptographic commitment for the whole stored CAR file. Not for retrieval; it's what storage providers get proof-challenged on. `/piece` on a provider's endpoint returns the exact stored bytes; `/ipfs` (the same trustless-gateway protocol public gateways use) resolves by Root CID. + +Directories work the same way (`filecoin-pin add ./my-data/`). Multiple uploads under one payment configuration group into a **Data Set**, each file a numbered **Piece** inside it. + +There's also `filecoin-pin import ./archive.car` for uploading a pre-built CAR directly instead of packing a path. + +### Retrieve + +```bash +curl https://.ipfs.dweb.link/ +# or, in a browser: https://inbrowser.link/ipfs/ +# (dweb.link/ipfs.io now redirect browser navigations to inbrowser.link, +# a verifiable Service Worker gateway that checks blocks against the CID +# client-side before rendering; curl and other non-browser clients are +# served directly by dweb.link) +``` + +No credentials needed to read — matches the retrieval requirement that ruled out Lighthouse. One real difference from "instant": **retrieval only works once IPNI (their content-routing indexer) has propagated the advertisement.** The CLI itself waits for this before it prints the retrieval URL, so you won't get a CID back that isn't yet fetchable — but it means "upload returns" and "third party can fetch it" aren't the exact same instant, worth knowing for round-timing assumptions. `filecoin-pin`'s own content-routing FAQ describes indexer propagation in the seconds-to-low-minutes range for index updates. + +By default, retrieval is CDN-accelerated through **FilBeam** (`--egress-provider beam`, the CLI default — `none` is the other option), with egress cost drawn from the *uploader's* lockup, not charged to the reader. That's the "model owner pays, readers don't" property directly, out of the box, without needing a separate paid tier the way Storacha needed Forge for it. + +## 3. Authentication — wallet keys and Session Keys, not accounts or UCANs + +No account system: identity is your wallet (private key). But there's a real delegation mechanism here too, closer to Storacha's UCAN model than it might first look — **Session Keys**, registered on-chain in a Session Key Registry and used by the Filecoin Warm Storage Service as an alternative to signing every operation with the owner's raw key. + +A session key: +- has **scoped permissions** (e.g. `CREATE_DATA_SET`, `ADD_PIECES` — explicitly *not* fund transfer) +- has an **expiration** (`--validity-days`, default 10, max 365) +- is **revocable** by the wallet owner before expiry + +That's a genuinely close parallel to a Storacha UCAN delegation: a narrow, time-boxed, revocable capability grant to a different keypair, rather than handing out the account's full authority. + +### The pattern for a backend/CI environment + +Two-party flow, mirroring Storacha's "generate a key locally, get it delegated" shape: + +```bash +# On the CI/consumer side — generate a keypair locally, no chain interaction: +filecoin-pin session generate +# → outputs a session address + private key + +# Send the session address to whoever owns the wallet/funds. They run, on their machine: +filecoin-pin session authorize --validity-days 30 + +# Back in CI, use the session key instead of the owner's raw private key: +export WALLET_ADDRESS=0x... # the owner's wallet +export SESSION_KEY=0x... # the session key generated above +filecoin-pin add ./artifact.bin --network calibration +# (or --wallet-address / --session-key flags instead of env vars) + +# Revoke when done / if it leaks: +filecoin-pin session revoke +``` + +There's also a single-party shortcut (`filecoin-pin session create`) if the same party both generates and authorizes the key — less relevant for a backend/CI split where the CI environment shouldn't hold owner-level authority in the first place. + +**This materially changes the risk picture from a raw private key.** Using `PRIVATE_KEY` directly in CI means a leak drains whatever's in that wallet, full stop. Using a session key means a leak is bounded: scoped to storage operations, time-limited, and revocable — much closer to the blast radius of a leaked Storacha delegation than "leaked wallet key," as long as session keys are actually used instead of the raw key. + +### Official CI/CD support: GitHub Action + +```yaml +- name: Upload to Filecoin + # Pin to a release tag or commit SHA for production — floating @v1 tags + # are not maintained, and @master tracks whatever's newest. + uses: filecoin-project/filecoin-pin/upload-action@v1.1.0 + with: + path: dist + walletPrivateKey: ${{ secrets.FILECOIN_WALLET_KEY }} + network: calibration + minRunwayDays: 30 # security checklist: always hardcode this + maxBalance: "100" # and this, in trusted workflows +``` + +The action's own security checklist (worth taking as seriously as it's presented — it's specific, not boilerplate): +- Pin to a version tag or commit SHA, not `@master`, for production. +- Always hardcode `minRunwayDays` and `maxBalance` in trusted workflows. +- Never use `pull_request_target`; use the documented two-workflow pattern instead. +- Consider GitHub Environments with required manual approval before a workflow can make deposits. +- Fork PRs are blocked entirely by design (rejected before wallet input validation) specifically to prevent non-maintainer PR actors from draining funds. + +Note the Action's `walletPrivateKey` input is exactly that — a raw private key, not a session key (the Action doesn't currently expose the session-key flow). So the Action's own checklist about a dedicated, minimally-funded CI wallet matters more here than it would if session keys were an option at that layer; the session-key delegation pattern above is CLI-only for now. + +Telemetry is on by default (`FILECOIN_PIN_TELEMETRY_DISABLED=true` or the standard `DO_NOT_TRACK=1` to opt out) — no PII collected per their docs, but per-upload outcome metrics are sent to a third-party ingestion endpoint by default, worth a policy check before this runs in DIN's CI. + +## What this means for `dincli` + +The genuinely good find here: **`filecoin-pin` implements the standard [IPFS Pinning Service API specification](https://ipfs.github.io/pinning-services-api-spec/)** directly — confirmed from the repo's own description ("IPFS Pinning Service API implementation that pins to Filecoin's PDP service"), not just inferred. That's the same API shape Filebase-style adapters are typically built against. + +```bash +filecoin-pin server +# runs a localhost IPFS Pinning Service API server backed by Filecoin PDP +``` + +If `dincli`'s existing Filebase adapter in `ipfs.py` really does speak that standard API, pointing it at a locally-run `filecoin-pin server` could be closer to a config change than a rewrite — worth checking the adapter code against the spec before assuming a full rewrite. That said, don't lean on this for production yet: the server mode is explicitly labeled **Beta, not intended for production use** in Filecoin Pin's own README, with specifics that go beyond "less tested" — +- **state is held in memory and lost on restart** (no persistence across a crash/redeploy) +- no enforced rate limits, per-user quotas, max DAG size, or concurrency caps — a single caller can exhaust disk/CPU/network with one oversized CID +- requires a bearer token (`ACCESS_TOKEN`) on every request; running with `ALLOW_NO_AUTH=true` outside local dev isn't recommended +- `delegates` in pin responses is always empty (each pin spins up a short-lived Helia node, stopped when the pin finishes — no long-lived node to advertise) + +The CLI, GitHub Action, and JS library are all separately labeled **production-ready** in the same README — it's specifically the server/daemon affordance that isn't there yet. For a first integration, driving the CLI directly (or the JS library, which is what powers the CLI) is the safer path; treat `filecoin-pin server` as a promising shortcut to revisit once it's out of beta, not something to point at DIN's real round artifacts today. + +## Sources + +- https://github.com/filecoin-project/filecoin-pin — canonical repo (README, `src/` command source, `documentation/` — retrieval.md, content-routing-faq.md, glossary.md) +- https://github.com/filecoin-project/filecoin-pin/tree/master/upload-action — GitHub Action README (security checklist, versioning, egress-provider default) +- https://registry.npmjs.org/filecoin-pin — package metadata (confirmed active: v1.1.1, published 2026-06-26, 41 versions) +- https://filecoin.io/blog/posts/introducing-filecoin-onchain-cloud-verifiable-developer-owned-infrastructure/ — FOC launch (testnet, Nov 18 2025) diff --git a/Developer/tooling/model-owner-contracts.md b/Developer/proposals/model-owner-contracts.md similarity index 99% rename from Developer/tooling/model-owner-contracts.md rename to Developer/proposals/model-owner-contracts.md index efd0e1a..ed05803 100644 --- a/Developer/tooling/model-owner-contracts.md +++ b/Developer/proposals/model-owner-contracts.md @@ -38,7 +38,7 @@ The tool should let a model owner: 5. connect a wallet and deploy them 6. surface the deployed addresses as manifest-ready outputs -The generated workflow should stay compatible with the current deployment path documented in `Documentation/model-owner.md`. +The generated workflow should stay compatible with the current deployment path documented in `Documentation/public/roles/model-owner.md`. ## Current Contract Model diff --git a/Developer/tooling/model-owner-services.md b/Developer/proposals/model-owner-services.md similarity index 98% rename from Developer/tooling/model-owner-services.md rename to Developer/proposals/model-owner-services.md index d694008..a7e45af 100644 --- a/Developer/tooling/model-owner-services.md +++ b/Developer/proposals/model-owner-services.md @@ -38,7 +38,7 @@ The tool should let a model owner: 4. generate the final Python artifacts 5. surface the generated artifact paths and manifest-ready entries -The output should remain compatible with the existing service and manifest model documented in `Documentation/services.md` and `Documentation/manifest.md`. +The output should remain compatible with the existing service and manifest model documented in `Documentation/public/services.md` and `Documentation/public/manifest.md`. ## Expected Inputs diff --git a/Developer/tasks/task_060726_4.md b/Developer/tasks/task_060726_4.md index 723e7d8..019bafc 100644 --- a/Developer/tasks/task_060726_4.md +++ b/Developer/tasks/task_060726_4.md @@ -43,8 +43,8 @@ cd foundry && forge build && forge test Read before writing anything: -1. `Documentation/DIN-workflow.md` — how the platform contracts relate to each other. -2. `Documentation/Model-workflow.md` — the task-contract lifecycle (Global Iterations, LMS, aggregation, slashing). +1. `Documentation/public/workflows/din-workflow.md` — how the platform contracts relate to each other. +2. `Documentation/public/workflows/model-workflow.md` — the task-contract lifecycle (Global Iterations, LMS, aggregation, slashing). 3. `Documentation/technical/upgradable-contracts/` — the proxy design the platform contracts now follow. 4. The contracts themselves (§3). @@ -103,7 +103,7 @@ The platform contracts are mid-flight in PR #13 and gated on a CLI test harness **Read first:** - Discussion #18: https://github.com/InfiniteZeroFoundation/DevNet/discussions/18 — you're cc'd on it; this section is that cc turned into a deliverable. -- PR #16: https://github.com/InfiniteZeroFoundation/DevNet/pull/16 — search the changed files for `lighthouse` / `filecoin`. The relevant ones: `dincli/services/ipfs_lighthouse.py` (the adapter), `dincli/services/ipfs.py` (provider abstraction), `Documentation/guides/ipfs.md`, `tests/test_ipfs_lighthouse.py`. You don't need to write Python — read these for how DIN's storage abstraction works and what the adapter assumed. +- PR #16: https://github.com/InfiniteZeroFoundation/DevNet/pull/16 — search the changed files for `lighthouse` / `filecoin`. The relevant ones: `dincli/services/ipfs_lighthouse.py` (the adapter), `dincli/services/ipfs.py` (provider abstraction), `Documentation/public/guides/ipfs.md`, `tests/test_ipfs_lighthouse.py`. You don't need to write Python — read these for how DIN's storage abstraction works and what the adapter assumed. - `Developer/discussion/add-filecoin-support.md` — the internal design doc whose Lighthouse recommendation PR #16 implemented. **The problem:** PR #16's live testing found that Lighthouse uploads work perfectly but **retrieval is payment-gated** — `GET gateway.lighthouse.storage/ipfs/` returns `402 Payment Required` even for the file's own uploader, and the CID is not discoverable on the public IPFS network at all (`ipfs.io`: "no providers found"). That breaks DIN's core assumption: aggregators and auditors must be able to fetch any artifact by CID. The discussion lays out the candidate paths (Storacha, Lighthouse paid tier, Filebase Filecoin bucket, FVM-native deals, hybrid). diff --git a/Developer/tasks/task_290626_1.md b/Developer/tasks/task_290626_1.md index 83f7cde..24fd0f7 100644 --- a/Developer/tasks/task_290626_1.md +++ b/Developer/tasks/task_290626_1.md @@ -375,4 +375,4 @@ use /home/azureuser/tempdir/dincli/ as temp_dir to output any test results to av - Integration test runner script or `pytest` fixture that documents the required prerequisites - At minimum: deploy flow (Phase 1–2), registration flow (Phase 3), and GI state transitions (Phase 4 — open/close phases) covered with exit-code assertions - Notes file (`tests/dincli/NOTES.md`) identifying which command sequences are SDK candidates for P4 WP 1.2 -- notes in Documentation/technical/DINCLI_TESTING_GUIDE.md fo running the tests on your local hardhat node. \ No newline at end of file +- notes in Documentation/technical/testing/dincli-testing-guide.md fo running the tests on your local hardhat node. \ No newline at end of file diff --git a/Developer/tasks/task_300626_3.md b/Developer/tasks/task_300626_3.md index 577974c..c7f2c3f 100644 --- a/Developer/tasks/task_300626_3.md +++ b/Developer/tasks/task_300626_3.md @@ -61,14 +61,14 @@ Current paths and their appropriate tier: The docs and config templates should reflect this classification: - In `.env.example` files (repo root and `hardhat/`), label `ETH_PRIVATE_KEY_` entries explicitly as development/testing variables — do not remove them, but add a comment directing production validators to the keystore path instead -- In `Documentation/guides/` and `Developer/DEVELOPMENT_SETUP.md`, wherever a guide instructs setting a raw key in `.env`, add a callout: "This pattern is for local development. If you are running a production validator node, use the encrypted keystore — see `Documentation/guides/wallet-setup.md`." +- In `Documentation/public/guides/` and `Developer/DEVELOPMENT_SETUP.md`, wherever a guide instructs setting a raw key in `.env`, add a callout: "This pattern is for local development. If you are running a production validator node, use the encrypted keystore — see `Documentation/public/guides/wallet-setup.md`." - In `dincli/docker/node/README.md`, add a note in the environment variables section distinguishing dev and production key management ### 1b. Burner wallet setup guide This is a gap that currently nowhere in the documentation addresses: the existing guides assume the operator already has a wallet and knows why to keep it separate. They do not. -Write `Documentation/guides/wallet-setup.md` with the following sections: +Write `Documentation/public/guides/wallet-setup.md` with the following sections: 1. **Why a dedicated burner wallet, not your main address.** The private key will be loaded into `dincli` to sign transactions on every command. Any compromise of the machine or the keystore file exposes every asset at that address. Use an address you are comfortable treating as disposable. @@ -78,7 +78,7 @@ Write `Documentation/guides/wallet-setup.md` with the following sections: 4. **What happens next.** Cross-reference the keystore migration guide (1d) so the operator understands how the key file gets loaded by `dincli` after setup. -This guide should be linked from `Documentation/guides/ipfs.md` and `Developer/DEVELOPMENT_SETUP.md` as a prerequisite step. +This guide should be linked from `Documentation/public/guides/ipfs.md` and `Developer/DEVELOPMENT_SETUP.md` as a prerequisite step. ### 1c. Multi-account support and security/convenience balance in `connect_wallet` @@ -116,7 +116,7 @@ Assess feasibility: what interface does OWS expose, can `dincli` enumerate accou ### 1d. Keystore migration documentation -Write `Documentation/guides/keystore-migration.md`: +Write `Documentation/public/guides/keystore-migration.md`: 1. What changed: the old `.env` / `PRIVATE_KEY=` pattern vs. the new encrypted keystore 2. How to migrate an existing key: move it into a keystore file via an `eth-account` one-liner or OWS export (show the exact command) @@ -154,7 +154,7 @@ Read `Developer/discussion/add-filecoin-support.md` in full before starting. The ### Deliverables for Part 2 - `dincli/services/ipfs_lighthouse.py` (or equivalent module name) implementing the adapter -- `Documentation/guides/ipfs.md` updated with Lighthouse provider configuration steps and the `LIGHTHOUSE_API_KEY` env var +- `Documentation/public/guides/ipfs.md` updated with Lighthouse provider configuration steps and the `LIGHTHOUSE_API_KEY` env var - Unit tests in `tests/test_ipfs_lighthouse.py` covering: provider selection from env vars, upload mock returning a CID, retrieve mock, error case when API key is missing - A brief note (inline in the PR description or a `Developer/discussion/` follow-up comment) on any gaps or surprises found in the Lighthouse/Storacha API versus what the discussion document assumed — this feeds directly into the next design step @@ -164,12 +164,12 @@ Read `Developer/discussion/add-filecoin-support.md` in full before starting. The **By end of Jul 3:** - [ ] Plaintext private key patterns removed or replaced in all docs and `.env.example` files -- [ ] Burner wallet setup guide written (`Documentation/guides/wallet-setup.md`), referencing OWS / openwallet.sh +- [ ] Burner wallet setup guide written (`Documentation/public/guides/wallet-setup.md`), referencing OWS / openwallet.sh - [ ] `dincli system connect-wallet` supports `--keystore ` with passphrase prompt — flagged for Umer review before merge -- [ ] Keystore migration guide written (`Documentation/guides/keystore-migration.md`) +- [ ] Keystore migration guide written (`Documentation/public/guides/keystore-migration.md`) - [ ] OWS integration feasibility documented in the PR description - [ ] Lighthouse provider adapter implemented and passing unit tests -- [ ] `Documentation/guides/ipfs.md` updated with Lighthouse configuration steps +- [ ] `Documentation/public/guides/ipfs.md` updated with Lighthouse configuration steps - [ ] PR description note on any API gaps or surprises vs. the discussion document --- diff --git a/Documentation/README.md b/Documentation/README.md new file mode 100644 index 0000000..668c9a1 --- /dev/null +++ b/Documentation/README.md @@ -0,0 +1,49 @@ +# DIN Protocol Documentation + +Documentation for the DIN Protocol DevNet as implemented on the `develop` branch. + +> **Scope rule:** everything in this folder describes **what exists in the code on `develop`** — not the live deployment on Optimism Sepolia (which may lag `develop`), and not planned or proposed designs. Plans, designs, proposals, and process docs live in [`Developer/`](../Developer/README.md). If a change to the code would make a document here wrong, the document belongs here; if it would still be valid, it belongs in `Developer/`. + +## Layout + +- **[`public/`](public/)** — for network participants (validators, clients, model owners). Assumes you operate the network through `dincli`; never requires reading source code. +- **[`technical/`](technical/)** — internal documentation for people modifying or auditing the code. + +## Public documentation + +| Document | Purpose | +|---|---| +| [Getting Started](public/getting-started.md) | Onboarding guide for Model_0 on the live devnet (`sepolia-op-devnet`) | +| [Setup Guide](public/setup.md) | Installing and configuring `dincli`: venv, wallet, network, logging, demo mode, IPFS | +| [CLI Reference](public/cli-reference.md) | Common `dincli` reference across all roles | +| [Manifest](public/manifest.md) | The per-model `manifest.json`: metadata, services, contract addresses | +| [Services](public/services.md) | Service files a model owner must provide (model, client, auditor, aggregator logic) | + +### Role guides — [`public/roles/`](public/roles/) + +One guide per network role: [Clients](public/roles/clients.md) · [Auditors](public/roles/auditors.md) · [Aggregators](public/roles/aggregators.md) · [Model Owners](public/roles/model-owner.md) · [DIN-Representative (dindao)](public/roles/dindao.md) + +### Workflows — [`public/workflows/`](public/workflows/) + +- [DIN Workflow](public/workflows/din-workflow.md) — platform-level contracts and how they relate (deployed once, by the DIN-Representative) +- [Model Workflow](public/workflows/model-workflow.md) — task-contract lifecycle per model: deploy → slasher authorization → genesis model → registration → Global Iterations + +### Guides — [`public/guides/`](public/guides/) + +- [Wallet Setup](public/guides/wallet-setup.md) — burner wallet and encrypted keystore setup +- [Keystore Migration](public/guides/keystore-migration.md) — migrating from plaintext `.env` keys to encrypted keystores +- [IPFS](public/guides/ipfs.md) — configuring the IPFS backend (node, Filebase, or custom provider) +- [Client Onboarding](public/guides/client-onboarding.md) — model-owner-to-client onboarding instructions + +## Technical documentation + +| Area | Contents | +|---|---| +| [ARCHITECTURE.md](technical/ARCHITECTURE.md) | System architecture reference (first complete draft tracked as P3-DOC1) | +| [`contracts/`](technical/contracts/) | Per-contract references: `DinCoordinator`, `DinToken`, `DinValidatorStake`, `DINModelRegistry`, `DINTaskCoordinator`, `DINTaskAuditor`, `DINShared` | +| [`mechanisms/`](technical/mechanisms/) | Currently implemented protocol mechanisms (e.g. [staking](technical/mechanisms/staking-mechanism.md)) | +| [`services/`](technical/services/) | Reference service internals (e.g. [client service](technical/services/clients.md)) | +| [`testing/`](technical/testing/) | [dincli testing guide](technical/testing/dincli-testing-guide.md), [containerization guide](technical/testing/containerization-guide.md) | +| [`upgradable-contracts/`](technical/upgradable-contracts/) | Transparent Proxy deployment architecture and upgrade test documentation | +| [manifest.md](technical/manifest.md) | Manifest runtime resolution and service loading internals | +| [requirements.md](technical/requirements.md) | Pinned pip requirements reference | diff --git a/Documentation/ReadMe.md b/Documentation/ReadMe.md deleted file mode 100644 index adc533b..0000000 --- a/Documentation/ReadMe.md +++ /dev/null @@ -1,47 +0,0 @@ -# DIN Protocol Documentation - -Welcome to the DIN Protocol documentation. This README serves as an index and guide for the various components of the `dincli` setup, commands, and overall workflow. - -## [Setup Guide](./setup.md) - -This guide walks you through setting up the `dincli` environment. It includes instructions for setting up a virtual environment, installing and initializing `dincli`, connecting a wallet, configuring the network, managing logging, using demo-mode, and configuring IPFS. - -## [IPFS Configuration Guide](./guides/ipfs.md) - -This guide explains the three supported IPFS modes in `dincli`: `.env`-backed, Filebase, and fully custom Python services. - -## [Common Commands](./common.md) - -This document outlines the common `dincli` commands shared across all stakeholders, including Model Owners, Clients, Aggregators, and Auditors. - -## [Model Owners Commands](./model-owner.md) - -This guide details the specific `dincli` commands and workflow utilized by Model Owners for managing tasks and models within the protocol. - -## [Clients Commands](./clients.md) - -This guide covers the specific `dincli` commands and workflow utilized by Clients. - -## [Aggregators Commands](./aggregators.md) - -This file explains the specific `dincli` commands and workflow needed by Aggregators. - -## [Auditors Commands](./auditors.md) - -This guide provides information on the specific `dincli` commands and workflow available for Auditors. - -## [Manifest Commands](./manifest.md) - -This file details the instructions for setting up and managing the Manifest file of a model. - -## [Services Guide](./services.md) - -This document contains detailed technical specifications and instructions for creating the various background service scripts required by the stakeholders. - -## [DIN Workflow](./DIN-workflow.md) - -This file provides a comprehensive overview of the platform level contracts and their interactions within the DIN Protocol ecosystem. - -## [Model Workflow](./Model-workflow.md) - -This document outlines comprehensive overview of the task level contracts and workflow. It includes complete training, auditing and aggregation workflow for a task (model) within the DIN Protocol. diff --git a/Documentation/guides/wallet-setup.md b/Documentation/guides/wallet-setup.md deleted file mode 100644 index c226eca..0000000 --- a/Documentation/guides/wallet-setup.md +++ /dev/null @@ -1,143 +0,0 @@ -# Burner Wallet Setup Guide - -How to create a dedicated burner wallet for DIN Protocol participation. - ---- - -## 1. Why a dedicated burner wallet - -`dincli` loads your private key into memory to sign every on-chain transaction. -If your machine or the keystore file is compromised, every asset at that address -is exposed. Use a **disposable** address with only the funds needed for -participation — never your primary wallet. - ---- - -## 2. Generate a burner wallet - -### Option A — `eth-account` one-liner (recommended) - -```bash -python3 -c " -from eth_account import Account -from getpass import getpass -import json - -acct = Account.create() -pw = getpass('Keystore passphrase: ') -ks = Account.encrypt(acct.key.hex(), pw) -with open('keystore.json', 'w') as f: - json.dump(ks, f) -print('Address:', acct.address) -" -``` - -This creates `keystore.json` — a standard Ethereum JSON keystore file — -in the current directory. Import it with: - -```bash -dincli system connect-wallet --keystore ./keystore.json --name validator -``` - -Then delete the temporary file: `rm keystore.json`. - -### Option B — OWS (Open Wallet Standard) - -[OWS](https://openwallet.sh/) is a chain-agnostic, agent-friendly key management -tool that stores wallets encrypted with AES-256-GCM (corroborated by MoonPay's launch -press release — see [openwallet.sh](https://openwallet.sh/)). Install -it via npm: - -```bash -npm install -g @open-wallet-standard/core -``` - -Generate a fresh wallet: - -```bash -ows wallet create --name validator -``` - -Then check `ows --help` or [docs.openwallet.sh](https://docs.openwallet.sh/) for -the current export/import surface to export the keystore to a file. Once exported, -import it into `dincli`: - -```bash -dincli system connect-wallet --keystore ./keystore.json --name validator -rm ./keystore.json -``` - -> **Note:** OWS direct signing delegation from `dincli` is under evaluation. -> OWS exposes MCP, REST, and SDK interfaces for agent access with scoped API -> tokens — `dincli` never sees the raw key. This is the preferred production -> path if the signing interface meets the protocol's requirements. For now, -> OWS is used as the keystore generation/export tool, and `dincli` holds the -> encrypted keystore locally. - ---- - -## 3. Fund the burner wallet - -You need the wallet's address with **ETH for gas** and **DIN for staking**. - -**Minimum to participate:** -- **10 DIN** to stake (`DinValidatorStake.MIN_STAKE = 10 * 10^18`) -- A small amount of **ETH** for transaction gas fees - -### Get Sepolia Optimism ETH - -Send your new address to one of these faucets: - -- [Optimism Faucet](https://console.optimism.io/faucet) -- [Chainlink Faucet](https://faucets.chain.link/optimism-sepolia) -- [LearnWeb3 Faucet](https://learnweb3.io/faucets/optimism_sepolia/) - -### Get DIN tokens - -DIN tokens are obtained by depositing ETH through the `DinCoordinator` contract -(ETH → DIN exchange). Use `dincli`: - -```bash -dincli aggregator dintoken buy 0.00001 -``` - -See [DIN-workflow.md](../DIN-workflow.md) for the complete token workflow. - ---- - -## 4. What happens next - -After funding the address, load the encrypted keystore into `dincli`: - -```bash -dincli system connect-wallet --keystore ./keystore.json --name validator -``` - -You will be prompted for the keystore passphrase. `dincli` persists the encrypted -keystore in `~/.config/dincli/wallets/wallet_validator.json` — your raw private -key is **never written to disk** in plaintext. - -**Password handling:** -- You will be prompted for your passphrase **each command** (dincli does not - cache passwords across invocations). -- To avoid re-prompting, set `DIN_WALLET_PASSWORD=` in your - `.env` file (suitable for unattended/CI runs, not for shared machines). - -For detailed migration and runtime selection, see -[keystore-migration.md](./keystore-migration.md). - ---- - -## Key management tiers - -| Path | Convenience | Security | Recommended for | -|---|---|---|---| -| `ETH_PRIVATE_KEY_` in `.env` | High (multi-acct, no prompts) | Low (plaintext on disk) | Local dev, automated testing | -| Interactive `getpass` + encrypted keystore | Medium (prompt per command; in-memory cache only within one process) | High (encrypted at rest, pw in memory only) | Production (current best) | -| `--keystore ` JSON keystore input | Medium (passphrase prompt) | High (external keystore, key never re-encoded) | Production validators w/ external key mgmt | -| `DIN_WALLET_PASSWORD` env | High (no prompts across commands) | Medium (pw plaintext in env/`.env`) | Unattended automation / CI | -| OWS delegation (if feasible) | High (named accts, no raw key in dincli) | Highest (dincli never sees the key) | Production wanting full key isolation | - ---- - -**Next:** [Keystore Migration Guide](./keystore-migration.md) diff --git a/Documentation/common.md b/Documentation/public/cli-reference.md similarity index 100% rename from Documentation/common.md rename to Documentation/public/cli-reference.md diff --git a/Documentation/GettingStarted.md b/Documentation/public/getting-started.md similarity index 99% rename from Documentation/GettingStarted.md rename to Documentation/public/getting-started.md index 31de2e5..e0b2b73 100644 --- a/Documentation/GettingStarted.md +++ b/Documentation/public/getting-started.md @@ -218,7 +218,7 @@ Before participating, ensure dincli is correctly installed and configured. Please read: -https://github.com/InfiniteZeroFoundation/DevNet/blob/main/Documentation/setup.md +https://github.com/InfiniteZeroFoundation/DevNet/blob/develop/Documentation/public/setup.md ### Initialize DIN CLI diff --git a/Documentation/guides/clients.md b/Documentation/public/guides/client-onboarding.md similarity index 100% rename from Documentation/guides/clients.md rename to Documentation/public/guides/client-onboarding.md diff --git a/Documentation/guides/ipfs.md b/Documentation/public/guides/ipfs.md similarity index 100% rename from Documentation/guides/ipfs.md rename to Documentation/public/guides/ipfs.md diff --git a/Documentation/guides/keystore-migration.md b/Documentation/public/guides/keystore-migration.md similarity index 100% rename from Documentation/guides/keystore-migration.md rename to Documentation/public/guides/keystore-migration.md diff --git a/Documentation/public/guides/wallet-setup.md b/Documentation/public/guides/wallet-setup.md new file mode 100644 index 0000000..3614233 --- /dev/null +++ b/Documentation/public/guides/wallet-setup.md @@ -0,0 +1,163 @@ +# Burner Wallet Setup Guide + +How to create a dedicated burner wallet for DIN Protocol participation. + +--- + +## 1. Why a dedicated burner wallet + +`dincli` loads your private key into memory to sign every on-chain transaction. +If your machine or the keystore file is compromised, every asset at that address +is exposed. Use a **disposable** address with only the funds needed for +participation — never your primary wallet. + +--- + +## 2. Generate a burner wallet + +### Option A — `eth-account` one-liner (recommended) + +```bash +python3 -c " +from eth_account import Account +from getpass import getpass +import json + +acct = Account.create() +pw = getpass('Keystore passphrase: ') +ks = Account.encrypt(acct.key.hex(), pw) +with open('keystore.json', 'w') as f: + json.dump(ks, f) +print('Address:', acct.address) +" +``` + +This creates `keystore.json` — a standard Ethereum JSON keystore file — in the current +directory and prints the address. **Note the address; you will connect the keystore to +`dincli` in step 3 and delete the file then.** Do not delete `keystore.json` yet. + +### Option B — OWS (Open Wallet Standard) + +[OWS](https://openwallet.sh/) is a chain-agnostic, agent-friendly key management +tool that stores wallets in a local vault (`~/.ows/`) encrypted with AES-256-GCM +(scrypt KDF). Install it via the official one-liner (a global +`npm install -g @open-wallet-standard/core` also provides the `ows` CLI): + +```bash +curl -fsSL https://docs.openwallet.sh/install.sh | bash +``` + +Generate a fresh wallet (a single mnemonic derives an Ethereum address plus +addresses for every other supported chain): + +```bash +ows wallet create --name validator +ows wallet list # shows the eip155 (Ethereum) address +``` + +**Important:** OWS does **not** export an Ethereum JSON keystore. `ows wallet export` +prints the **raw mnemonic / private key** (interactive terminal only). So there are two +ways to use an OWS-managed key with `dincli`: + +1. **Export the raw private key from OWS and import it interactively** — simple, but the key + leaves the OWS vault, which forfeits OWS's main benefit. `ows wallet export` runs **only in + an interactive terminal** (it refuses piped input) and prints the raw secret to the screen; + then connect it in step 3 by pasting the key when prompted: + ```bash + ows wallet export --wallet validator # interactive; prints the raw private key / mnemonic + ``` +2. **Signing delegation (planned, not yet shipped)** — a *future* `dincli` integration would + ask OWS to sign each transaction so the key never leaves the vault. EVM signing without key + exposure is confirmed feasible (see the feasibility spike at + `Developer/discussion/ows-delegation-feasibility.md` in the repository), + but a `--wallet-backend ows` option is a planned follow-up and is **not available today**. + +> **Note:** OWS also exposes a scoped API-key + policy model and Node/Python SDKs for +> programmatic access. Any future delegation integration **must verify and require policy-scoped +> signing — do not rely on unscoped OWS keys.** OWS lets API keys be created with no policy, and +> its EVM signer will sign arbitrary payloads without validating them, so the policy layer's +> enforcement must be proven before it is trusted. See `ows --help` or +> [docs.openwallet.sh](https://docs.openwallet.sh/) for the current CLI surface. + +--- + +## 3. Connect the wallet to `dincli` and make it active + +Import the wallet once, then set it active so later commands use it. + +**If you used Option A (`eth-account` keystore):** + +```bash +dincli system connect-wallet --keystore ./keystore.json --name validator +rm ./keystore.json # remove the temporary keystore file once imported +``` + +**If you used Option B path 1 (OWS raw-key export):** paste the exported private key when prompted: + +```bash +dincli system connect-wallet --name validator +``` + +Then make `validator` the active wallet: + +```bash +dincli system set-wallet validator +``` + +`connect-wallet` prompts for a keystore passphrase (Option A) and persists the encrypted +keystore at `~/.config/dincli/wallets/wallet_validator.json` — your raw private key is +**never written to disk in plaintext**. + +**Password handling:** +- You are prompted for your passphrase **each command** (`dincli` does not cache passwords + across invocations). +- To avoid re-prompting, set `DIN_WALLET_PASSWORD=` in your `.env` file + (suitable for unattended/CI runs, not for shared machines). + +For detailed migration and runtime selection, see +[keystore-migration.md](./keystore-migration.md). + +--- + +## 4. Fund the burner wallet + +Your wallet needs **ETH for gas** and **DIN for staking**. + +**Minimum to participate:** +- **10 DIN** to stake (`DinValidatorStake.MIN_STAKE = 10 * 10^18`) +- A small amount of **ETH** for transaction gas fees + +### Get Sepolia Optimism ETH + +Send your address to one of these faucets: + +- [Optimism Faucet](https://console.optimism.io/faucet) +- [Chainlink Faucet](https://faucets.chain.link/optimism-sepolia) +- [LearnWeb3 Faucet](https://learnweb3.io/faucets/optimism_sepolia/) + +### Get DIN tokens + +With `validator` connected and active (step 3) and holding some ETH, buy DIN by depositing +ETH through the `DinCoordinator` contract (ETH → DIN exchange): + +```bash +dincli aggregator dintoken buy 0.00001 # uses the active wallet; or add --wallet validator +``` + +See [DIN-workflow.md](../workflows/din-workflow.md) for the complete token workflow. + +--- + +## Key management tiers + +| Path | Convenience | Security | Recommended for | +|---|---|---|---| +| `ETH_PRIVATE_KEY_` in `.env` | High (multi-acct, no prompts) | Low (plaintext on disk) | Local dev, automated testing | +| Interactive `getpass` + encrypted keystore | Medium (prompt per command; in-memory cache only within one process) | High (encrypted at rest, pw in memory only) | Production (current best) | +| `--keystore ` JSON keystore input | Medium (passphrase prompt) | High (external keystore, key never re-encoded) | Production validators w/ external key mgmt | +| `DIN_WALLET_PASSWORD` env | High (no prompts across commands) | Medium (pw plaintext in env/`.env`) | Unattended automation / CI | +| OWS signing delegation (planned; not yet shipped) | High (named accts, no raw key in dincli) | Potentially high, but **unproven** — key isn't exposed to dincli, yet the tested path is unscoped and not passphrase-gated; needs verified policy-scoped signing first | Production wanting full key isolation (future) | + +--- + +**Next:** [Keystore Migration Guide](./keystore-migration.md) diff --git a/Documentation/manifest.md b/Documentation/public/manifest.md similarity index 98% rename from Documentation/manifest.md rename to Documentation/public/manifest.md index 81319e9..ffa5f67 100644 --- a/Documentation/manifest.md +++ b/Documentation/public/manifest.md @@ -1,7 +1,7 @@ # Manifest For a lower-level explanation of runtime resolution, service loading, and the -new nested `dp` configuration block, see [technical/manifest.md](technical/manifest.md). +new nested `dp` configuration block, see [technical/manifest.md](../technical/manifest.md). The manifest is a JSON file containing the metadata for your model and task. It serves as the central configuration that ties together the model, its services, and contract addresses. @@ -111,7 +111,7 @@ pip3 install -r requirements.txt ## Example Manifest -Please find the example/template manifest file at [cache_model_0/manifest.json](../cache_model_0/manifest.json) +Please find the example/template manifest file at [cache_model_0/manifest.json](../../cache_model_0/manifest.json) --- diff --git a/Documentation/aggregators.md b/Documentation/public/roles/aggregators.md similarity index 100% rename from Documentation/aggregators.md rename to Documentation/public/roles/aggregators.md diff --git a/Documentation/auditors.md b/Documentation/public/roles/auditors.md similarity index 100% rename from Documentation/auditors.md rename to Documentation/public/roles/auditors.md diff --git a/Documentation/clients.md b/Documentation/public/roles/clients.md similarity index 100% rename from Documentation/clients.md rename to Documentation/public/roles/clients.md diff --git a/Documentation/dindao.md b/Documentation/public/roles/dindao.md similarity index 100% rename from Documentation/dindao.md rename to Documentation/public/roles/dindao.md diff --git a/Documentation/model-owner.md b/Documentation/public/roles/model-owner.md similarity index 99% rename from Documentation/model-owner.md rename to Documentation/public/roles/model-owner.md index bd545cc..a033ad5 100644 --- a/Documentation/model-owner.md +++ b/Documentation/public/roles/model-owner.md @@ -66,11 +66,11 @@ dincli system get-fees **Manifest file** -The manifest is a JSON file containing the metadata for your model and task. For the full schema, field descriptions, and an example, see [manifest.md](manifest.md). +The manifest is a JSON file containing the metadata for your model and task. For the full schema, field descriptions, and an example, see [manifest.md](../manifest.md). **Service files** -The Model Owner must provide a set of service files tailored to the task. For detailed documentation on each service file and the required function signatures, see [services.md](services.md). +The Model Owner must provide a set of service files tailored to the task. For detailed documentation on each service file and the required function signatures, see [services.md](../services.md). --- diff --git a/Documentation/services.md b/Documentation/public/services.md similarity index 98% rename from Documentation/services.md rename to Documentation/public/services.md index 93703ff..9698cdb 100644 --- a/Documentation/services.md +++ b/Documentation/public/services.md @@ -124,7 +124,7 @@ Creates audit test data CIDs for auditor batches. The `testData_percentage_per_a This service defines the functions used by the clients to train and submit local models. -For the current DevNet reference implementation and the manifest-driven DP flow, see [technical/services/clients.md](technical/services/clients.md). +For the current DevNet reference implementation and the manifest-driven DP flow, see [technical/services/clients.md](../technical/services/clients.md). ### 3.1. `train_client_model_and_upload_to_ipfs(...)` diff --git a/Documentation/setup.md b/Documentation/public/setup.md similarity index 98% rename from Documentation/setup.md rename to Documentation/public/setup.md index d623347..5344e5d 100644 --- a/Documentation/setup.md +++ b/Documentation/public/setup.md @@ -38,7 +38,7 @@ pip install dincli-0.1.0-py3-none-any.whl pip install git+https://github.com/InfiniteZeroFoundation/devnet.git@main#subdirectory=dist ``` -> for any missing dependency please install it using pip. Please see a complete dependency list in [requirements.txt](../cache_model_0/requirements.txt) +> for any missing dependency please install it using pip. Please see a complete dependency list in [requirements.txt](../../cache_model_0/requirements.txt) ### Verify Installation diff --git a/Documentation/DIN-workflow.md b/Documentation/public/workflows/din-workflow.md similarity index 100% rename from Documentation/DIN-workflow.md rename to Documentation/public/workflows/din-workflow.md diff --git a/Documentation/Model-workflow.md b/Documentation/public/workflows/model-workflow.md similarity index 98% rename from Documentation/Model-workflow.md rename to Documentation/public/workflows/model-workflow.md index 5cc9fe4..fe2bbcc 100644 --- a/Documentation/Model-workflow.md +++ b/Documentation/public/workflows/model-workflow.md @@ -26,10 +26,10 @@ dincli system connect-wallet --account > [!WARNING] > `--account ` (using `ETH_PRIVATE_KEY_` from `.env`) is > **local development only**. Production validators should use the encrypted -> keystore — see [wallet-setup.md](guides/wallet-setup.md). +> keystore — see [wallet-setup.md](../guides/wallet-setup.md). > [!NOTE] -> More on wallet configuration can be found in the [DIN CLI Documentation](common.md). +> More on wallet configuration can be found in the [DIN CLI Documentation](../cli-reference.md). In current DIN Protocol, the model owner needs to deploy a taskCoordinator contract and a taskAuditor contract specific to each model. @@ -155,7 +155,7 @@ The Model Owner must provide Python service files that implement task-specific l Each service file must be uploaded and pinned to IPFS. The resulting CID is referenced in the manifest file. -> For detailed documentation on each service file and required function signatures, see [services.md](services.md). +> For detailed documentation on each service file and required function signatures, see [services.md](../services.md). ### 6.2. Create Manifest File @@ -165,7 +165,7 @@ The manifest is a JSON file containing model metadata, contract addresses, and s /tasks//task_/manifest.json ``` -> For the full manifest schema, field descriptions, and an example manifest, see [manifest.md](manifest.md). +> For the full manifest schema, field descriptions, and an example manifest, see [manifest.md](../manifest.md). ## 7. Create and Submit Genesis Model - Model Owner diff --git a/Developer/ARCHITECTURE.md b/Documentation/technical/ARCHITECTURE.md similarity index 100% rename from Developer/ARCHITECTURE.md rename to Documentation/technical/ARCHITECTURE.md diff --git a/Documentation/technical/audits/foundry-src-security-review.md b/Documentation/technical/audits/foundry-src-security-review.md new file mode 100644 index 0000000..09d9aeb --- /dev/null +++ b/Documentation/technical/audits/foundry-src-security-review.md @@ -0,0 +1,240 @@ +# Foundry `src/` Security Review — July 2026 + +**Reviewer:** Similoluwa Abidoye (@Abidoyesimze) +**Scope:** All 7 contracts in `foundry/src/` +**Pinned commit:** `d136ff3ecef06670a3b785807669c40e027bcdf3` — reviewed as-is; `develop` has moved on since (see `Developer/tasks/task_060726_4.md` §7 for why the pin is deliberate). All line numbers below refer to this commit. +**Task:** `Developer/tasks/task_060726_4.md`, Part 1 +**Companion reading:** `Documentation/DIN-workflow.md`, `Documentation/Model-workflow.md`, `Documentation/technical/upgradable-contracts/` (as of `develop@805ce9d`) + +This is a findings-only report. No contract in `foundry/src/` was modified. PoC tests are additive, in `foundry/test/SecurityFindings.t.sol`, and run with: + +```bash +cd foundry && forge test --match-contract SecurityFindingsTest -vv +``` + +Baseline confirmed before review: `forge build` and `forge test` both pass clean at the pinned commit (4/4 existing tests in `UpgradeValidation.t.sol`). + +--- + +## Summary + +| Severity | Count | +|---|---| +| Critical | 0 | +| High | 4 | +| Medium | 4 | +| Low / Informational | 8 | + +The headline risk isn't in the four upgradeable platform contracts — their proxy conversion is careful and the initializer/storage-layout checks in this report all came back clean. It's in the two **non-upgradeable task contracts** (`DINTaskCoordinator`, `DINTaskAuditor`), which run the actual federated-learning round: they have several ways for a single low-stake participant to permanently brick a Global Iteration, and no on-chain recovery once that happens (redeploy is the only path, per the documented "task contracts are disposable" design — but that design assumes *planned* redeployment, not mid-round griefing). + +--- + +## High Severity + +### H-1. Sybil-cheap gas-DoS: unbounded auditor/aggregator registration feeds four O(n) or worse loops with no pagination + +**Contracts / functions:** +- `DINTaskAuditor.sol`: `_activeAuditorPool()` (L275-297), `slashAuditors()` (L620-659), `finalizeEvaluation()` (L552-612) +- `DINTaskCoordinator.sol`: `_activeAggregatorPool()` (L410-432), `slashAggregators()` (L678-755), `finalizeT1Aggregation()` (L549-582), `finalizeT2Aggregation()` (L627-660) + +**This is the seeded lead, confirmed real — not refuted.** + +`registerDINAuditor()` and `registerDINaggregator()` have **no registration cap**, unlike client submissions which are capped at `MAX_LM_SUBMISSIONS = 10000`. Every one of the functions above iterates over the full historical registrant list (`dinAuditors[_GI]` / `dinAggregators[_GI]`), and several nest that iteration inside a second loop over batches and a third over models-per-batch: + +```solidity +// DINTaskAuditor.slashAuditors — triple-nested, no chunking +for (uint b = 0; b < batchCount; b++) { // O(batches) + for (uint a = 0; a < batch.auditors.length; a++) { // O(auditorsPerBatch) + for (uint m = 0; m < batch.modelIndexes.length; m++) { // O(modelsPerBatch) + if (!hasAuditedLM[...]) { missedVote = true; break; } + } + if (missedVote) dinvalidatorStakeContract.slash(...); // external call + SSTORE + event + } +} +``` + +batches scales linearly with registered auditors (`batches ≈ registeredAuditors / auditorsPerBatch`), so total loop body executions are `O(registeredAuditors × modelsPerBatch)`, and every "missed vote" auditor costs one more external `slash()` call (~50k+ gas each with the SSTORE + event in `DinValidatorStake`). + +**Concrete cost math, using the contract's own documented parameters** (`Params` struct comments in `DINTaskAuditor.sol` L36-43 literally say "demo: 3… spec: 10" / "demo: 3… spec: 100"): + +- At spec scale (`auditorsPerBatch=10`, `modelsPerBatch=100`): **one single batch** already requires 1,000 `hasAuditedLM` reads before any slashing even starts. 50 batches (500 registered auditors — a modest number for a "decentralized" network) is 50,000 loop iterations plus up to 500 external `slash()` calls. That is well past any realistic L2 block gas limit (Optimism's is ~30M as of this review); the transaction reverts with out-of-gas every time it's retried, and there is **no smaller-batch entry point to work around it** — `slashAuditors`/`slashAggregators`/`finalizeT1Aggregation`/`finalizeT2Aggregation`/`finalizeEvaluation` all process the *entire* GI in one call. +- **And it's cheap to trigger deliberately.** `MIN_STAKE` is 10 DIN, bought at the default rate of 1,000,000 DIN/ETH (`DinCoordinator.initialize`, L58) — **0.00001 ETH per Sybil identity**. Registering 1,000 Sybil auditor addresses costs ~0.01 ETH plus L2 gas, and since staked DIN isn't consumed by registering (only by being slashed), the attacker can unstake and reuse it after the 7-day unbonding window. This is not a scaling accident that only bites at organic success — it's a cheap, repeatable attack a single actor can execute against any model's GI today. + +**Failure scenario:** An attacker registers a few hundred throwaway addresses as auditors for a target model's GI (trivial cost, no collusion needed with anyone else). Once the model owner calls `slashAuditors()` (or `finalizeEvaluation()`, or the coordinator's `slashAggregators()`/`finalizeT1Aggregation()`/`finalizeT2Aggregation()` with a matching flood of aggregator Sybils), the call runs out of gas every time. The GI is now stuck at that phase permanently — `DINTaskCoordinator`/`DINTaskAuditor` are not upgradeable, so there is no way to patch around it; the model owner's only recourse is to abandon the GI and redeploy new task contracts (losing all state and honest participants' in-flight work for that round). + +**Recommendation:** Cap registrant counts the same way `MAX_LM_SUBMISSIONS` caps client submissions, and/or add pagination to every finalize/slash function (e.g. `slashAuditors(uint _GI, uint startBatch, uint endBatch)`), tracking a `lastProcessedBatch` cursor so a GI can be advanced through many smaller transactions instead of one unbounded one. + +--- + +### H-2. Zero-CID sentinel collision permanently bricks `finalizeT1Aggregation` / `finalizeT2Aggregation` for the whole GI + +**Contract / functions:** `DINTaskCoordinator.sol`, `finalizeT1Aggregation()` (L549-582), `finalizeT2Aggregation()` (L627-660), and the corresponding `submitT1Aggregation()` / `submitT2Aggregation()` which accept an unvalidated `_aggregationCID`. + +**Confirmed exploitable — PoC: `test_finalizeT1Aggregation_zeroCID_bricksEntireGI` in `foundry/test/SecurityFindings.t.sol`.** + +Both finalize functions use `bytes32(0)` as a sentinel for "nobody submitted anything in this batch": + +```solidity +bytes32 winningCID = ""; +uint maxVotes = 0; +for (uint j = 0; j < b.aggregators.length; j++) { + ... + if (votes > maxVotes) { maxVotes = votes; winningCID = cid; } +} +if (winningCID == bytes32(0)) revert TC_NoSubmissions(); // <-- ambiguous +``` + +`submitT1Aggregation`/`submitT2Aggregation` never validate `_aggregationCID != bytes32(0)`. Any assigned aggregator can submit `bytes32(0)` as their own genuine vote. If that vote ends up as the batch's plurality (trivially true if they're the only one of the 3 assigned aggregators who submits before the window closes — a routine "not everyone responded" scenario, not an edge case), `winningCID` legitimately equals `bytes32(0)` and the code cannot distinguish "a real submission of the zero CID" from "no one submitted." The function reverts. + +Because `finalizeT1Aggregation`/`finalizeT2Aggregation` loop over **every batch in the GI in one transaction** and revert the whole call on the first bad batch, **one poisoned batch blocks finalization for every other (honest) batch too**. There is no admin override to skip a stuck batch, and the GI cannot progress past `T1AggregationStarted`/`T2AggregationStarted` — permanently, since these contracts aren't upgradeable. + +**Failure scenario (see PoC):** 3 aggregators are assigned to a Tier-1 batch. Two never submit (they don't have to be malicious — just slow, offline, or the model owner closes the window before they respond). The third submits `bytes32(0)`. It is the only submission, so it's unambiguously the plurality winner. `finalizeT1Aggregation` reverts with `TC_NoSubmissions` even though a legitimate submission exists, and stays reverting on every retry. + +**Recommendation:** Reject `_aggregationCID == bytes32(0)` at submission time in both `submitT1Aggregation` and `submitT2Aggregation` (fail fast, cheaply, on the individual submitter rather than the whole GI), and/or track submission presence with an explicit `bool hasSubmissions` per batch rather than inferring it from the winning CID's value. + +--- + +### H-3. No quorum enforced before Tier-1/Tier-2 aggregation is accepted as "final" — a single aggregator's output can become the consensus result + +**Contract / functions:** `DINTaskCoordinator.sol`, `finalizeT1Aggregation()` (L549-582), `finalizeT2Aggregation()` (L627-660). + +Compare this to `DINTaskAuditor`, which explicitly enforces `minEligibilityQuorum`/`minScoreQuorum` before treating a vote as final (`_tryFinalizeEligibility`, L487-489; `finalizeEvaluation`, L596). `finalizeT1Aggregation`/`finalizeT2Aggregation` have **no equivalent check**. The only condition for accepting a "winning" CID is `winningCID != bytes32(0)` — i.e., *someone* submitted *something*. If only 1 of the 3 assigned aggregators submits before the model owner closes the window (no malice required — the other two may simply not have finished their off-chain aggregation yet), that lone submission is accepted as the batch's final, majority-agreed result with zero cross-validation from any other party. + +**Failure scenario:** A malicious or buggy aggregator submits a garbage/incorrect CID and is the only one of their batch to submit in time. `finalizeT1Aggregation` accepts it as `finalCID` with no dissent recorded anywhere, and that CID flows into Tier-2 aggregation and ultimately becomes (part of) the new global model — silently, with no on-chain signal that only 1-of-3 parties actually agreed. The entire point of a multi-aggregator batch (fault tolerance / cross-validation) is defeated by the absence of a minimum-participation check. + +**Recommendation:** Require a minimum submission count (e.g. `submitted.length >= T1_AGGREGATORS_PER_BATCH / 2 + 1`, mirroring the auditor contract's quorum pattern) before `finalizeT1Aggregation`/`finalizeT2Aggregation` will accept a winning CID; revert (or mark the batch as "unresolved" for out-of-band handling) otherwise. + +--- + +### H-4. Predictable, grindable pseudo-randomness in auditor/aggregator batch shuffling enables collusion + +**Contracts / functions:** +- `DINTaskAuditor.sol`: `_shuffleAddressArray()` (L263-273), `_shuffleUintArray()` (L299-308), both used by `createAuditorsBatches()` +- `DINTaskCoordinator.sol`: `_shuffleAddressArray()` (L379-389), `_shuffleUintArray()` (L391-400), both used by `autoCreateTier1AndTier2()` + +**This is the seeded lead, confirmed real — not refuted.** + +Both contracts derive their Fisher-Yates shuffle entropy from `blockhash(block.number - 1)` (address shuffle) and `block.timestamp` (model-index shuffle, further combined with `msg.sender`, which is always the same fixed coordinator address per model — contributing no real entropy). Both inputs are public and known *before* the batch-creation transaction is even submitted, since the previous block is already final by the time anyone calls `createAuditorsBatches`/`autoCreateTier1AndTier2`. + +The caller of these functions (the model owner, since both are gated `onlyOwner`/routed through the owner-only coordinator call) can therefore **compute the resulting batch assignment offline before submitting**, and can choose *when* to submit (i.e., wait for a block whose hash produces a favorable shuffle) since retrying costs only gas on an L2. Combined with H-1's cheap Sybil registration: + +**Failure scenario:** A model owner registers a handful of Sybil-controlled auditor addresses alongside honest ones. Before calling `createAuditorsBatches`, they simulate the shuffle for the current `blockhash(block.number - 1)` locally; if the resulting batch doesn't cluster ≥2 of their Sybils into the same batch as their target model (default `auditorsPerBatch=3`, `minEligibilityQuorum=2` — only 2-of-3 needed to control a batch's eligibility outcome), they simply wait for the next block and recompute. Once a favorable block arrives, they submit. Their colluding auditors then rubber-stamp their own (possibly low-quality or malicious) submitted model as eligible, defeating the purpose of independent auditor review. The same grinding applies to `autoCreateTier1AndTier2`'s aggregator/model shuffle. + +**Recommendation:** Replace blockhash/timestamp entropy with a source the caller cannot pre-compute against before choosing whether to submit — e.g. commit-reveal (commit to a seed one block before creating batches, so the batch-creator can't selectively decide post-hoc) or a VRF (Chainlink VRF or similar). At minimum, do not let the same address that benefits from the shuffle outcome (the model owner) be the one who chooses the exact block it executes in. + +--- + +## Medium Severity + +### M-1. No commit-reveal on aggregation/scoring submissions — "copy the leader" free-riding + +**Contracts / functions:** `DINTaskCoordinator.submitT1Aggregation()` / `submitT2Aggregation()` (L520-543, L600-622); `DINTaskAuditor.setAuditScorenEligibility()` (L515-544). + +Votes/scores/CIDs are submitted in the clear and tallied by direct value match; there is no commit-then-reveal step. Any participant who is not the first to submit for a given batch/model can read every prior submission from public contract state before deciding what to submit themselves. A lazy or dishonest aggregator can copy another party's already-submitted CID instead of doing the aggregation work, guaranteeing they "match consensus" and avoid `AGG_T1_BAD_CONSENSUS`/`AGG_T2_BAD_CONSENSUS` slashing while contributing nothing. The same applies to an auditor submitting last on `setAuditScorenEligibility` — they can see the running vote tally (`_tryFinalizeEligibility` is invoked after every vote, so intermediate state is observable) and simply match the emerging majority. + +This doesn't cause direct fund loss, but it undermines the core assumption that agreement among independently-computed results is meaningful signal — with copying possible, "consensus" can be manufactured by a single honest party plus N idle followers. + +**Recommendation:** Commit-reveal: submitters post `keccak256(cid, salt)` during the submission window, then reveal `(cid, salt)` in a second window after submissions close. Standard mitigation for this exact class of on-chain "peek and copy" issue. + +--- + +### M-2. `DINModelRegistry.disableModel()` kill-switch doesn't reach the live task contracts + +**Contracts / functions:** `DINModelRegistry.sol`, `disableModel()`/`enableModel()` (L404-416); contrast with `DINTaskCoordinator.sol` and `DINTaskAuditor.sol`, neither of which references `DINModelRegistry` at all. + +`modelDisabled[modelId]` only gates `requestManifestUpdate` (via the `notDisabled` modifier) inside the registry itself. It has **zero effect** on the model's actual `DINTaskCoordinator`/`DINTaskAuditor` — those contracts have no dependency on, or awareness of, the registry. A model that DIN-Representative has disabled (e.g., because it was found to be malicious, or is slashing honest validators due to a bug) can keep running full GIs — registration, LMS, evaluation, aggregation, slashing — completely unaffected. + +**Failure scenario:** DIN-Representative discovers a model's task contracts have a bug that's wrongfully slashing honest auditors (or is otherwise harmful) and calls `disableModel()` expecting it to function as an emergency stop. It doesn't — the GI in progress continues, and honest validators keep getting exposed to (and slashed by) the disabled model until the model owner voluntarily stops calling functions on it (which they have no obligation to do, especially if they're the malicious party). + +**Recommendation:** If `disableModel` is meant to be an emergency stop (the task's own contract table calls it a "kill-switch"), either (a) have the task contracts check `DINModelRegistry.modelDisabled(modelId)` before allowing state-changing calls (requires wiring the model ID and registry address into the task contracts — a larger change, out of scope for this findings-only pass, but worth a design ticket), or (b) explicitly document that `disableModel` is registry-metadata-only and does not stop a live GI, so operators don't rely on it as a circuit breaker it isn't. + +--- + +### M-3. `DINTaskAuditor.slashAuditors()` has no internal GI-state gate + +**Contract / function:** `DINTaskAuditor.sol`, `slashAuditors()` (L620-659). + +Every other state-changing function in `DINTaskAuditor` independently re-checks `dintaskcoordinatorContract.GIstate()` before acting (`createAuditorsBatches` requires `LMSclosed`; `setTestDataAssignedFlag` requires `AuditorsBatchesCreated`; `finalizeEvaluation` requires `LMSevaluationStarted`). `slashAuditors()` breaks that pattern — it only checks `onlyTaskCoordinator` + `onlyCurrentGI`, with no GI-state precondition of its own. Correctness currently depends entirely on `DINTaskCoordinator.slashAuditors()` (L665-671) gating the call to `GIstate == T2AggregationDone` — i.e. a single point of trust with no defense-in-depth, unlike everywhere else in this pair of contracts. + +Today this isn't independently exploitable (the coordinator's gate holds), but it's the one function in the file that would let a future change to `DINTaskCoordinator` (or a bug in it) trigger slashing at the wrong phase with no second check catching it. + +**Recommendation:** Add an explicit `GIstates` check inside `DINTaskAuditor.slashAuditors()` itself, consistent with every other function in the contract. + +--- + +### M-4. `DinValidatorStake`'s `Jailed` status is dead code + +**Contract / function:** `DinValidatorStake.sol` — `ValidatorStatus.Jailed` (L46), `jailedUntil` field (L54), read at L269 and L324-325. + +No function anywhere in `DinValidatorStake.sol` (or any other contract in scope) ever writes a non-zero value to `jailedUntil`, or sets `status = ValidatorStatus.Jailed`, except the restore-path inside `unblacklistValidator()` — which itself only re-enters `Jailed` if `jailedUntil > block.timestamp`, a condition that can never be true since nothing ever sets `jailedUntil`. The entire jailing mechanism referenced in the enum and struct is unreachable. + +This isn't exploitable, but it either indicates a missing feature (a `jail()` function was planned/removed but the supporting state/logic was left behind) or dead state that should be removed. Worth a product/eng decision rather than silent removal, since `Documentation/DIN-workflow.md`/`Model-workflow.md` don't mention jailing as a current mechanism either. + +**Recommendation:** Confirm with the team whether jailing is planned; if not, remove `Jailed`/`jailedUntil` to avoid confusing future readers (flagging only — no change made here per the findings-only boundary). + +--- + +## Low / Informational + +| # | Finding | Location | +|---|---|---| +| L-1 | `stake()` calls `DIN_TOKEN.safeTransferFrom` (external call) before updating `activeStake` — violates checks-effects-interactions. Mitigated today by `nonReentrant` + `DIN_TOKEN` being a trusted, protocol-deployed contract, but worth fixing on principle. | `DinValidatorStake.sol` L113-126 | +| L-2 | `depositAndMint()` has no minimum-deposit / non-zero-mint check. If the owner ever sets `dinPerEth` to a value that isn't a clean multiple of `1e18`, a small enough `msg.value` can round `mintAmount` to 0 via integer division while the ETH is still retained by the contract. | `DinCoordinator.sol` L64-71 | +| L-3 | `requestModelRegistration` doesn't refund `msg.value` above the required fee — any overpayment is silently kept. | `DINModelRegistry.sol` L155-194 | +| L-4 | `rejectModel()` never refunds the fee paid in the corresponding `requestModelRegistration` — a rejected requester loses their fee permanently. Confirm this is the intended design (anti-spam fee) rather than an oversight. | `DINModelRegistry.sol` L244-254 | +| L-5 | `withdrawFees(address payable to)` has no zero-address check; calling it with `to == address(0)` silently burns the entire fee balance (owner-only footgun, not exploitable by a third party). | `DINModelRegistry.sol` L472-477 | +| L-6 | `DINTaskCoordinator`/`DINTaskAuditor` constructors accept `dinvalidatorStakeContract_address` / `dintaskcoordinator_contract_address` with no zero-address check. Self-inflicted misconfiguration risk only (deployer controls the args), not attacker-triggered. | `DINTaskCoordinator.sol` L87-92, `DINTaskAuditor.sol` L148-167 | +| L-7 | `totalDepositedRewards` and the `RewardDeposited` event are declared but never written/emitted anywhere; the contract also has no `receive()`/`fallback()`, so any ETH that ever reaches this contract (e.g. via a forced `selfdestruct` send) would be permanently unwithdrawable. Dead code / incomplete feature. | `DINTaskAuditor.sol` L16, L102 | +| L-8 | `updateDinPerEth()` / `updateValidatorStakeContract()` take effect immediately with no timelock, giving depositors a front-run/back-run arbitrage window around rate changes. Inherent to the documented "tentative workaround" centralized exchange-rate design — flagged for completeness, not a new issue. | `DinCoordinator.sol` L106-120 | + +--- + +## Proxy-Specific Checks — the 4 Upgradeable Platform Contracts + +`DinToken`, `DinCoordinator`, `DinValidatorStake`, `DINModelRegistry`. Checked per `Documentation/technical/upgradable-contracts/hardhat/README.md` and the task's minimum-coverage list. + +### Initializer protection — PASS, verified at runtime, not just in source + +The task explicitly asked whether `_disableInitializers()` is *effective at runtime*, not just present in the source. Verified with two independent PoCs per contract (8 tests total, all passing — `foundry/test/SecurityFindings.t.sol`): + +1. **Direct call to a freshly-deployed implementation's `initialize()` reverts** (`test_implementation_rejectsDirectInitialize_*`) — confirms `_disableInitializers()` in the constructor actually locks the implementation, closing the classic "attacker calls `initialize()` on the implementation and claims ownership of it" path. Not exploitable against the live protocol today regardless (the implementation holds no funds or routing of its own — only the proxy does), but this is exactly the kind of thing that's cheap to verify and expensive to get wrong. +2. **A second `initialize()` call through the proxy reverts** (`test_proxy_rejectsDoubleInitialize_*`) — confirms the `initializer` modifier's one-shot guard survives the full deploy-via-proxy path, not just in isolation. + +No initializer front-running window exists either: per the architecture doc, `initCalldata` is `delegatecall`ed atomically during the `TransparentUpgradeableProxy` constructor, so there's no block where the proxy exists uninitialized. + +### Storage layout — PASS, confirmed via `forge inspect storage-layout` + +All four contracts' *own* declared state starts at slot 0, and `uint256[50] private __gap` is correctly the last entry in every one: + +``` +DinToken: slot 0 coordinator → slot 1 __gap[50] +DinCoordinator: slot 0 dinToken, 1 dinValidatorStakeContract, 2 dinPerEth → slot 3 __gap[50] +DinValidatorStake: slot 0 DIN_TOKEN, 1 DIN_COORDINATOR, 2 slasherContracts, 3 validators → slot 4 __gap[50] +DINModelRegistry: slot 0 dinValidatorStake … slot 10 modelDisabled → slot 11 __gap[50] +``` + +This is cleaner than it would be for a typical upgradeable contract because OZ v5's upgradeable base contracts (`OwnableUpgradeable`, `ERC20Upgradeable`, `Initializable`) use ERC-7201 namespaced storage — their fields live at `keccak256`-derived slots, not sequential ones — so inherited base storage cannot collide with each contract's own declared variables regardless of gap placement. The `__gap[50]` still matters for future *same-contract* V2 additions (per the design doc's stated purpose), and its position is correct in all four. The pre-existing `foundry/test/UpgradeValidation.t.sol` (`Upgrades.validateImplementation`, all 4 passing at baseline) already covers this class of check going forward — this section is corroborating evidence, not a new test. + +### Access control on the upgrade path — no in-scope finding + +Transparent Proxy puts upgrade authority in the auto-deployed `ProxyAdmin`, entirely outside the implementation contracts reviewed here — a bad implementation can't touch its own upgrade authority (that's the documented reason Transparent Proxy was chosen over UUPS). Nothing in `foundry/src/` grants any address an upgrade capability; there is no `_authorizeUpgrade` or equivalent to audit. `ProxyAdmin` custody (currently the DIN-Representative EOA per the design doc) is a governance/key-management concern, not a contract-code one — out of scope for a source-code review. + +### Constructor logic silently dropped under the proxy pattern — no finding + +All four constructors contain exactly one line, `_disableInitializers()`, with `@custom:oz-upgrades-unsafe-allow constructor` annotations. There is no other constructor logic in any of the four contracts that could silently stop running once behind a proxy — all real setup was correctly moved into `initialize()` per the conversion pattern documented in the architecture README. Checked explicitly per the task's ask; nothing to flag. + +--- + +## Seeded Leads — Explicitly Confirmed or Refuted + +1. **`DINTaskAuditor.slashAuditors()` nested iteration (batches × auditors × models) may not fit in a block at production scale — is this a real DoS?** **Confirmed real.** See H-1. Not hypothetical: cheap to trigger deliberately via Sybil registration (≈0.00001 ETH per identity at the default exchange rate), no collusion needed. +2. **`DINTaskAuditor._activeAuditorPool()` unbounded iteration over all registered auditors.** **Confirmed real**, same root cause as #1 (no registration cap) — folded into H-1 along with its `DINTaskCoordinator._activeAggregatorPool()` counterpart, which has the identical shape and wasn't in the seed list but is exposed to the same attack. +3. **blockhash/timestamp shuffling in auditor selection — predictable by block producers; does slashing/selection fairness depend on unpredictability?** **Confirmed real.** See H-4. It matters more than block-producer-level MEV — the *caller* (model owner, who is not disinterested) can grind the timing of their own transaction against public, pre-known entropy, which is a lower bar than needing block-producer collusion. + +--- + +## What I'd do differently with more time + +- Extend the PoC suite with a real (not just closed-form) gas measurement at spec-scale parameters (`auditorsPerBatch=10`, `modelsPerBatch=100`, ~500 registered auditors) to get an exact gas number instead of the order-of-magnitude estimate in H-1 — the flow to reach `slashAuditors`/`slashAggregators` requires driving a GI through 8+ phase transitions per participant, which is straightforward but time-consuming to script at that scale. +- M-1 (copy-the-leader free-riding) and H-3 (missing quorum) interact: a full fix probably wants to land together (commit-reveal naturally gives you a place to also enforce "N-of-M revealed before finalize is callable"). +- Didn't attempt fuzzing `DinCoordinator`'s exchange-rate math or `DinValidatorStake`'s stake accounting (suggested in the task) — manual review didn't turn up an obvious overflow/precision target beyond L-2, and Solidity 0.8's built-in overflow checks close off the classic wraparound class. Would still run `forge fuzz` against `slash()`'s active/pending-withdrawal split accounting given more time, since that's the one place doing subtraction across two balances in the same function. diff --git a/Documentation/technical/DINModelRegistry.md b/Documentation/technical/contracts/DINModelRegistry.md similarity index 100% rename from Documentation/technical/DINModelRegistry.md rename to Documentation/technical/contracts/DINModelRegistry.md diff --git a/Documentation/technical/DINShared.md b/Documentation/technical/contracts/DINShared.md similarity index 100% rename from Documentation/technical/DINShared.md rename to Documentation/technical/contracts/DINShared.md diff --git a/Documentation/technical/DINTaskAuditor.md b/Documentation/technical/contracts/DINTaskAuditor.md similarity index 100% rename from Documentation/technical/DINTaskAuditor.md rename to Documentation/technical/contracts/DINTaskAuditor.md diff --git a/Documentation/technical/DINTaskCoordinator.md b/Documentation/technical/contracts/DINTaskCoordinator.md similarity index 100% rename from Documentation/technical/DINTaskCoordinator.md rename to Documentation/technical/contracts/DINTaskCoordinator.md diff --git a/Documentation/technical/DinCoordinator.md b/Documentation/technical/contracts/DinCoordinator.md similarity index 100% rename from Documentation/technical/DinCoordinator.md rename to Documentation/technical/contracts/DinCoordinator.md diff --git a/Documentation/technical/DinToken.md b/Documentation/technical/contracts/DinToken.md similarity index 100% rename from Documentation/technical/DinToken.md rename to Documentation/technical/contracts/DinToken.md diff --git a/Documentation/technical/DinValidatorStake.md b/Documentation/technical/contracts/DinValidatorStake.md similarity index 99% rename from Documentation/technical/DinValidatorStake.md rename to Documentation/technical/contracts/DinValidatorStake.md index 9a83847..d7c6182 100644 --- a/Documentation/technical/DinValidatorStake.md +++ b/Documentation/technical/contracts/DinValidatorStake.md @@ -1,6 +1,6 @@ # DinValidatorStake — Technical Documentation -Technical documentation for [`hardhat/contracts/DinValidatorStake.sol`](hardhat/contracts/DinValidatorStake.sol). +Technical documentation for [`hardhat/contracts/DinValidatorStake.sol`](../../../hardhat/contracts/DinValidatorStake.sol). ## Overview diff --git a/Documentation/technical/manifest.md b/Documentation/technical/manifest.md index ee131ca..359584b 100644 --- a/Documentation/technical/manifest.md +++ b/Documentation/technical/manifest.md @@ -202,7 +202,7 @@ Primary local sources: - example manifest: [cache_model_0/manifest.json](/home/azureuser/projects/devnet/cache_model_0/manifest.json) - runtime object: [dincli/services/runtime.py](/home/azureuser/projects/devnet/dincli/services/runtime.py) - service loading path: [dincli/cli/client.py](/home/azureuser/projects/devnet/dincli/cli/client.py) -- existing higher-level manifest doc: [Documentation/manifest.md](/home/azureuser/projects/devnet/Documentation/manifest.md) +- existing higher-level manifest doc: [Documentation/public/manifest.md](/home/azureuser/projects/devnet/Documentation/public/manifest.md) - client-service technical doc: [Documentation/technical/services/clients.md](/home/azureuser/projects/devnet/Documentation/technical/services/clients.md) - DP design notes: [Developer/issues/DifferentialPrivacy.md](/home/azureuser/projects/devnet/Developer/issues/DifferentialPrivacy.md) diff --git a/Documentation/technical/DINCLI_Containerizaton_Guide.md b/Documentation/technical/testing/containerization-guide.md similarity index 96% rename from Documentation/technical/DINCLI_Containerizaton_Guide.md rename to Documentation/technical/testing/containerization-guide.md index 678291f..1f7389d 100644 --- a/Documentation/technical/DINCLI_Containerizaton_Guide.md +++ b/Documentation/technical/testing/containerization-guide.md @@ -81,8 +81,8 @@ Here is how to build your images, run the `din-node` container, and monitor Dock docker compose exec din-node dincli system configure-demo --mode yes ``` > **Production validators:** Import an encrypted keystore instead of demo mode. - > See [wallet-setup.md](../guides/wallet-setup.md) and - > [keystore-migration.md](../guides/keystore-migration.md). + > See [wallet-setup.md](../../public/guides/wallet-setup.md) and + > [keystore-migration.md](../../public/guides/keystore-migration.md). --- diff --git a/Documentation/technical/DINCLI_TESTING_GUIDE.md b/Documentation/technical/testing/dincli-testing-guide.md similarity index 100% rename from Documentation/technical/DINCLI_TESTING_GUIDE.md rename to Documentation/technical/testing/dincli-testing-guide.md diff --git a/README.md b/README.md index 17a42db..3b8c661 100644 --- a/README.md +++ b/README.md @@ -43,19 +43,19 @@ The DevNet is live. Validator nodes are running 24/7 from Japan to Canada. This New here? Start with the validator setup guide: -[Getting Started →](https://github.com/InfiniteZeroFoundation/DevNet/blob/main/Documentation/GettingStarted.md) +[Getting Started →](https://github.com/InfiniteZeroFoundation/DevNet/blob/develop/Documentation/public/getting-started.md) --- ## Install dincli -Full installation guide: [dincli Documentation](Documentation/setup.md) +Full installation guide: [dincli Documentation](Documentation/public/setup.md) --- ## Configure IPFS -Full configuration guide: [dincli Documentation](Documentation/setup.md) +Full configuration guide: [dincli Documentation](Documentation/public/setup.md) --- @@ -82,7 +82,7 @@ If you care about privacy-preserving ML, decentralised systems, or just building ## Learn More - [White Paper](https://github.com/InfiniteZeroFoundation/White-Paper) -- [Documentation](https://github.com/InfiniteZeroFoundation/DevNet/blob/main/Documentation/GettingStarted.md) +- [Documentation](https://github.com/InfiniteZeroFoundation/DevNet/blob/develop/Documentation/public/getting-started.md) - [Support the network on Giveth](https://giveth.io/project/infinitezero-network) - [Say hello](mailto:abrahamnash@protonmail.com) diff --git a/dincli/cli/contract_utils.py b/dincli/cli/contract_utils.py index ce924ce..e391c01 100644 --- a/dincli/cli/contract_utils.py +++ b/dincli/cli/contract_utils.py @@ -1,159 +1,19 @@ -import json -import os +""" +Backward-compatibility shim. -# Minimal ABIs -erc20_abi = [ - { - "constant": True, - "inputs": [], - "name": "decimals", - "outputs": [{"name": "", "type": "uint8"}], - "payable": False, - "stateMutability": "view", - "type": "function", - }, - { - "constant": True, - "inputs": [{"name": "owner", "type": "address"}], - "name": "balanceOf", - "outputs": [{"name": "", "type": "uint256"}], - "payable": False, - "stateMutability": "view", - "type": "function", - }, - { - "constant": True, - "inputs": [ - {"name": "owner", "type": "address"}, - {"name": "spender", "type": "address"}, - ], - "name": "allowance", - "outputs": [{"name": "", "type": "uint256"}], - "payable": False, - "stateMutability": "view", - "type": "function", - }, - { - "constant": False, - "inputs": [ - {"name": "spender", "type": "address"}, - {"name": "value", "type": "uint256"}, - ], - "name": "approve", - "outputs": [{"name": "", "type": "bool"}], - "payable": False, - "stateMutability": "nonpayable", - "type": "function", - }, - # --- Added: transfer --- - { - "constant": False, - "inputs": [ - {"name": "to", "type": "address"}, - {"name": "value", "type": "uint256"}, - ], - "name": "transfer", - "outputs": [{"name": "", "type": "bool"}], - "payable": False, - "stateMutability": "nonpayable", - "type": "function", - }, - # --- Added: transferFrom --- - { - "constant": False, - "inputs": [ - {"name": "from", "type": "address"}, - {"name": "to", "type": "address"}, - {"name": "value", "type": "uint256"}, - ], - "name": "transferFrom", - "outputs": [{"name": "", "type": "bool"}], - "payable": False, - "stateMutability": "nonpayable", - "type": "function", - } +The implementation moved to ``dincli.sdk.contracts`` as part of the SDK +extraction (issue #20). This module re-exports it so existing +``from dincli.cli.contract_utils import ...`` call sites keep working. +New code should import from ``dincli.sdk.contracts``. +""" +from dincli.sdk.contracts import ( # noqa: F401 + erc20_abi, + router_abi, + get_contract_instance, +) +__all__ = [ + "erc20_abi", + "router_abi", + "get_contract_instance", ] -router_abi = [ - # For estimating input ETH needed - { - "inputs": [ - {"internalType": "uint256", "name": "amountOut", "type": "uint256"}, - {"internalType": "address[]", "name": "path", "type": "address[]"} - ], - "name": "getAmountsIn", - "outputs": [{"internalType": "uint256[]", "name": "amounts", "type": "uint256[]"}], - "stateMutability": "view", - "type": "function" - }, - # For executing the swap (ETH -> exact USDT) - { - "inputs": [ - {"internalType": "uint256", "name": "amountOut", "type": "uint256"}, - {"internalType": "address[]", "name": "path", "type": "address[]"}, - {"internalType": "address", "name": "to", "type": "address"}, - {"internalType": "uint256", "name": "deadline", "type": "uint256"} - ], - "name": "swapETHForExactTokens", - "outputs": [{"internalType": "uint256[]", "name": "amounts", "type": "uint256[]"}], - "stateMutability": "payable", - "type": "function" - } -] - - -def get_contract_instance( - artifact_path: str, - network: str, - address: str | None = None -) : - """ - Load a contract instance from an artifact (Hardhat format). - - Args: - artifact_path: Path to JSON artifact (must have "abi") - network: Target network (e.g., "local", "sepolia") - address: If provided, returns deployed contract. If None, returns deployable contract (requires "bytecode"). - - Returns: - web3.contract.Contract - """ - - from dincli.cli.utils import get_w3 - w3 = get_w3(network) - - if not os.path.isfile(artifact_path): - raise FileNotFoundError( - f"Contract artifact not found at: {artifact_path}\n" - "Tip: Ensure the contract is compiled and the path is correct." - ) - - try: - with open(artifact_path) as f: - data = json.load(f) - except json.JSONDecodeError as e: - raise ValueError( - f"Failed to parse JSON in artifact: {artifact_path}\n" - f"Reason: {str(e)}\n" - "Tip: This may indicate a corrupted or incomplete build. Try recompiling the contract." - ) from e - - if "abi" not in data: - raise ValueError(f"Artifact {artifact_path} missing 'abi' field") - abi = data["abi"] - - if address: - # Interaction mode: only ABI needed - return w3.eth.contract(address=address, abi=abi) - else: - # Deployment mode: bytecode required - if isinstance(data["bytecode"], dict): - bytecode = data["bytecode"].get("object") - else: - bytecode = data.get("bytecode") - if not bytecode: - raise ValueError( - f"Artifact {artifact_path} missing 'bytecode' — required for deployment.\n" - "Tip: Use `dincli system dump-abi --bytecode` to include it." - ) - return w3.eth.contract(abi=abi, bytecode=bytecode) diff --git a/dincli/cli/log.py b/dincli/cli/log.py index 0bb5d13..67c0a4d 100644 --- a/dincli/cli/log.py +++ b/dincli/cli/log.py @@ -1,35 +1,2 @@ -import json -import logging -from pathlib import Path - -from platformdirs import user_config_dir - -CONFIG_DIR = Path(user_config_dir("dincli")) -CONFIG_FILE = CONFIG_DIR / "config.json" - -def load_config(): - if CONFIG_FILE.exists(): - with open(CONFIG_FILE, "r") as f: - try: - return json.load(f) - except json.JSONDecodeError: - print(f"Error decoding config file at {CONFIG_FILE}. Returning empty config.") - return {} - else: - print(f"No config found at {CONFIG_FILE}") - return {} - -def get_config(key, default=None): - config = load_config() - return config.get(key, default) - -# Initialize logging -log_level_str = get_config("log_level", default="INFO") -log_level = getattr(logging, log_level_str.upper(), logging.INFO) - -logging.basicConfig( - level=log_level, - format="%(asctime)s - %(name)s - %(levelname)s - %(message)s" -) - -logger = logging.getLogger("dincli") \ No newline at end of file +import logging # re-exported: dincli.cli.context does `from dincli.cli.log import logger, logging` +from dincli.sdk.log import logger diff --git a/dincli/cli/system.py b/dincli/cli/system.py index 8091121..607a53e 100644 --- a/dincli/cli/system.py +++ b/dincli/cli/system.py @@ -20,13 +20,12 @@ CONFIG_FILE, WORKER_CACHE_DIR, print_tx_info, SUPPORTED_IPFS_PROVIDERS, _get_password, - _confirm_or_exit, - _clean_optional_string, + _confirm_or_exit, _clean_optional_string, get_config, get_demo_private_key, get_env_key, load_cid_services, load_config, load_din_info, normalize_ipfs_provider, resolve_ipfs_config, resolve_task_coordinator_address, - save_config, + save_config, validate_account_name, wallet_path_for_name, atomic_write_wallet, resolve_wallet_path, list_accounts, get_active_account_name, @@ -72,7 +71,7 @@ def system( ), ): # If the subcommand is one that doesn't need an account, we skip the default setup logic - if ctx.invoked_subcommand in ["connect-wallet", "init", "welcome", "where", "configure-network", "configure-demo", "read_wallet", "show_index", "din-info", "configure-logging", "dump-abi", "reset-all", "todo", "dataset", "send-eth", "run-worker-counting", "run-node-counting", "list-accounts", "set-wallet"]: + if ctx.invoked_subcommand in ["connect-wallet", "init", "welcome", "where", "configure-network", "configure-demo", "read-wallet", "show-index", "din-info", "configure-logging", "dump-abi", "reset-all", "todo", "dataset", "send-eth", "run-worker-counting", "run-node-counting", "list-accounts", "set-wallet"]: return effective_network, w3, account, console = ctx.obj.get_en_w3_account_console() @@ -288,6 +287,7 @@ def connect_wallet(ctx: typer.Context, account: Optional[int] = typer.Option(None, "--account", "-a", help="Hardhat dev account index (0-69)"), keystore: Optional[Path] = typer.Option(None, "--keystore", help="Import a standard Ethereum JSON keystore file"), name: Optional[str] = typer.Option("default", "--name", "-n", help="Label for the saved keystore (default 'default')"), + yes: bool = typer.Option(False, "--yes", "-y", help="Skip the confirmation prompt when overwriting an existing named wallet"), ): """ Connect a wallet to DIN CLI. @@ -309,10 +309,14 @@ def connect_wallet(ctx: typer.Context, dincli system connect-wallet --account 3 Encrypt and store the user's wallet for DIN CLI. - In demo mode (--yes), stores plaintext key for Hardhat testing. + In configured demo mode, stores plaintext key for Hardhat testing. """ console = ctx.obj.console + # Direct callers (e.g. tests) may omit --yes; Typer then passes an OptionInfo + # object (truthy), which would silently bypass the overwrite guard below. + # Normalize to a real bool so the guard behaves the same on direct calls. + yes = yes if isinstance(yes, bool) else False # Validate account name try: @@ -329,18 +333,19 @@ def connect_wallet(ctx: typer.Context, (keystore, "keystore file"), ] provided_methods = [n for val, n in auth_methods if val is not None] - + if len(provided_methods) > 1: console.print(f"[red]❌ Please specify only one of: {', '.join(provided_methods)}.[/red]") raise typer.Exit(1) - console.print(f"[green] ⚙️ Connecting wallet... to new account[/green]") - console.print(f"[cyan]Saving Wallet as:[/cyan] {resolved_name}") - demo_mode = get_config("demo_mode") source = "created" - # --- keystore import path --- + # --- keystore import: non-secret validation only --- + # Runs before the overwrite guard so a missing/malformed keystore fails fast + # without prompting, and so no secret (passphrase) is requested before the + # overwrite decision. The passphrase/decrypt happen after the guard, below. + inner_keystore = None if keystore is not None: keystore = keystore.expanduser() if not keystore.exists(): @@ -352,21 +357,38 @@ def connect_wallet(ctx: typer.Context, except (json.JSONDecodeError, OSError) as e: console.print(f"[red]❌ Failed to read keystore file: {e}[/red]") raise typer.Exit(1) - passphrase = getpass("Keystore passphrase: ") try: - inner_ks = _extract_keystore(imported_ks) + inner_keystore = _extract_keystore(imported_ks) except ValueError: console.print("[red]❌ File does not contain a recognisable keystore.[/red]") raise typer.Exit(1) + source = "imported" + + # --- overwrite guard --- + # Confirm before overwriting an existing named keystore, and before any secret + # prompt. Keys on the exact file to be written (a legacy wallet.json for the + # 'default' name is a non-destructive migration and intentionally not guarded). + target_path = wallet_path_for_name(resolved_name) + if target_path.exists() and not yes: + console.print(f"[yellow]A wallet named '{resolved_name}' already exists at {target_path}.[/yellow]") + if not typer.confirm("Overwrite it?"): + console.print("[yellow]Aborted. Existing wallet left unchanged.[/yellow]") + raise typer.Exit(0) + + console.print(f"[green] ⚙️ Connecting wallet... to new account[/green]") + console.print(f"[cyan]Saving as:[/cyan] {resolved_name}") + + # --- resolve secret material --- + if keystore is not None: + # Uses the keystore already extracted above — no re-read/re-parse. + passphrase = getpass("Keystore passphrase: ") try: - decrypted_key = Account.decrypt(inner_ks, passphrase) + decrypted_key = Account.decrypt(inner_keystore, passphrase) except ValueError: console.print("[red]❌ Wrong passphrase or corrupted keystore.[/red]") raise typer.Exit(1) acct = Account.from_key(decrypted_key) address = acct.address - inner_keystore = inner_ks - source = "imported" elif account is not None and demo_mode: # Load from demo accounts @@ -444,7 +466,10 @@ def connect_wallet(ctx: typer.Context, if keystore is not None: ks_payload = inner_keystore else: - password = _get_password(resolved_name, False) + # is_new_wallet=True: never reuse a stale in-memory-cached password when + # creating/overwriting a wallet; force the create/confirm flow (env var + # DIN_WALLET_PASSWORD is still honored for automation). + password = _get_password(resolved_name, False, is_new_wallet=True) if password == "": password = getpass("Create wallet password: ") confirm = getpass("Confirm password: ") @@ -470,13 +495,15 @@ def connect_wallet(ctx: typer.Context, @app.command("read-wallet") def read_wallet(ctx: typer.Context, - name: str = typer.Option(..., "--name", "-n", help="Wallet name") + name: Optional[str] = typer.Option(None, "--name", "-n", help="Wallet name") ): """ Read and display wallet info. In demo mode, shows private key. Otherwise, shows only address (after decrypting). """ console = ctx.obj.console + if not isinstance(name, str): + name = None if name is not None: wallet_to_read_name = name @@ -503,7 +530,7 @@ def read_wallet(ctx: typer.Context, return try: # to fix to read other wallets - console.print(f"[yellow]Address:[/yellow] {ctx.obj.account.address}") + console.print(f"[green]Address:[/green] {ctx.obj.account.address}") console.print("[green]✅ Wallet decrypted successfully.[/green]") except Exception as e: console.print(f"[red]❌ {e}[/red]") diff --git a/dincli/cli/utils.py b/dincli/cli/utils.py index 0319e90..8a08faa 100644 --- a/dincli/cli/utils.py +++ b/dincli/cli/utils.py @@ -19,21 +19,25 @@ _PASSWORD_TTL_DEFAULT = 900 _PASSWORD_CACHE: dict[str, tuple[str, float]] = {} +# Sentinel so callers can inject an already-fetched DIN_WALLET_PASSWORD value +# (fetched once per unlock) while callers that omit it self-fetch as before. +_UNSET = object() + from dincli.cli.contract_utils import get_contract_instance from dincli.cli.log import logger from dincli.services.cid_utils import get_cid_from_bytes32 console = Console() -CONFIG_DIR = Path(user_config_dir("dincli")) -CACHE_DIR = Path(user_cache_dir("dincli")) - -# Sibling cache used for docker-only artifacts (e.g. pip-installed client -# packages). Kept separate from CACHE_DIR so containers never need read/write -# access to the manifest/service/wallet cache that dincli itself manages. -WORKER_CACHE_DIR = Path(user_cache_dir("dincli-worker")) - -CONFIG_FILE = CONFIG_DIR / "config.json" +from dincli.sdk.config import ( + CONFIG_DIR, CACHE_DIR, WORKER_CACHE_DIR, CONFIG_FILE, + ALLOWED_NETWORKS, SUPPORTED_IPFS_PROVIDERS, LEGACY_IPFS_PROVIDER_ALIASES, + FILEBASE_IPFS_ADD_URL, FILEBASE_IPFS_CAT_URL, FILEBASE_IPFS_PIN_URL, + IPFSConfig, save_config, load_config, get_config, _clean_optional_string, + normalize_ipfs_provider, resolve_network, resolve_ipfs_config, + get_env_key, set_env_key, resolve_network_value, +) +from dincli.sdk.web3 import get_w3 WALLET_FILE = CONFIG_DIR / "wallet.json" WALLETS_DIR = CONFIG_DIR / "wallets" @@ -112,230 +116,6 @@ def _cleanup_stale_session() -> None: pass -ALLOWED_NETWORKS = ["local", "sepolia_devnet", "sepolia_op_devnet", "mainnet"] # "sepolia_testnet" -SUPPORTED_IPFS_PROVIDERS = ("env", "filebase", "custom") - -LEGACY_IPFS_PROVIDER_ALIASES = { - "": "env", - "default": "env", - "env": "env", - "ipfs node": "env", - "ipfs-node": "env", - "node": "env", -} - -FILEBASE_IPFS_ADD_URL = "https://rpc.filebase.io/api/v0/add" -FILEBASE_IPFS_CAT_URL = "https://rpc.filebase.io/api/v0/cat" -FILEBASE_IPFS_PIN_URL = "https://rpc.filebase.io/api/v0/pin/add" - - -# Optional: only import dotenv if needed -try: - from dotenv import dotenv_values - HAS_DOTENV = True -except ImportError: - HAS_DOTENV = False - - -@dataclass(frozen=True) -class IPFSConfig: - provider: str = "env" - api_url_add: Optional[str] = None - api_url_retrieve: Optional[str] = None - api_key: Optional[str] = None - api_secret: Optional[str] = None - service_path: Optional[Path] = None - - -def save_config(data): - CONFIG_DIR.mkdir(parents=True, exist_ok=True) - with open(CONFIG_FILE, "w") as f: - json.dump(data, f, indent=4) - logger.debug(f"Config saved to {CONFIG_FILE}") - - -def load_config(): - if CONFIG_FILE.exists(): - logger.debug(f"Loading config from {CONFIG_FILE}") - with open(CONFIG_FILE, "r") as f: - try: - return json.load(f) - except json.JSONDecodeError: - logger.error(f"Error decoding config file at {CONFIG_FILE}. Returning empty config.") - return {} - else: - logger.warning(f"No config found at {CONFIG_FILE}") - return {} - - -def get_config(key, default=None): - config = load_config() - return config.get(key, default) - - -def _clean_optional_string(value: Optional[str]) -> Optional[str]: - if not isinstance(value, str): - return None - - stripped = value.strip() - return stripped or None - - -def normalize_ipfs_provider(provider: Optional[str]) -> str: - if provider is None: - return "env" - - normalized = provider.strip().lower() - return LEGACY_IPFS_PROVIDER_ALIASES.get(normalized, normalized) - -def resolve_network(cli_network: str | None = None, default: str = "local") -> str: - """ - Resolve network: use CLI arg if provided, else config, else default. - """ - # 1. CLI takes highest precedence - if cli_network is not None: - if cli_network not in ALLOWED_NETWORKS: - raise ValueError(f"Invalid network: {cli_network}. Must be one of: {ALLOWED_NETWORKS}") - return cli_network - - # 3. Check global config - from_config = get_config("network") - if from_config and isinstance(from_config, str) and from_config.strip(): - return from_config.strip() - - # 4. Fallback - return default - -def resolve_ipfs_config(): - """ - Resolve the effective IPFS runtime configuration. - """ - config = load_config() - configured_provider = config.get("ipfs_provider") - provider = normalize_ipfs_provider(configured_provider) if configured_provider else normalize_ipfs_provider(get_env_key("IPFS_PROVIDER", verbose=False)) - raw_service_path = _clean_optional_string(config.get("ipfs_service_path")) - - api_key = _clean_optional_string(config.get(f"ipfs_api_key_{provider}")) - if not api_key and provider == "filebase": - api_key = _clean_optional_string(config.get("ipfs_api_key")) - - return IPFSConfig( - provider=provider, - api_url_add=_clean_optional_string(get_env_key("IPFS_API_URL_ADD", verbose=False)), - api_url_retrieve=_clean_optional_string(get_env_key("IPFS_API_URL_RETRIEVE", verbose=False)), - api_key=api_key, - api_secret=_clean_optional_string(config.get("ipfs_api_secret")), - service_path=Path(raw_service_path).expanduser().resolve() if raw_service_path else None, - ) - - -def get_env_key(key: str, default: Optional[str] = None, verbose: bool = True) -> Optional[str]: - """ - Get a key from: - 1. Current environment (e.g., SEPOLIA_RPC_URL) - 2. ./ .env file - 3. Default fallback - """ - # 1. Already in environment? (e.g., from shell or parent process) - if key in os.environ: - return os.environ[key] - - # 2. Load from .env in current directory (if available) - env_path = Path(os.getcwd()) / ".env" - if HAS_DOTENV and env_path.exists(): - # Load .env into a dict (doesn't pollute os.environ) - values = dotenv_values(dotenv_path=env_path) - if key not in values and default is None: - if verbose: - console.print(f"[bold red] ❌ {key} not found in {os.getcwd()}/.env file[/bold red]") - return values.get(key, default) - - if not HAS_DOTENV and verbose: - console.print("[yellow]Warning: python-dotenv not installed. Cannot save to .env[/yellow]") - - return default - - -def set_env_key(key: str, value: str): - """ - Set a key in the .env file. - """ - if not HAS_DOTENV: - console.print("[yellow]Warning: python-dotenv not installed. Cannot save to .env[/yellow]") - return - - env_path = Path(os.getcwd()) / ".env" - - try: - from dotenv import set_key - - # Create file if it doesn't exist - if not env_path.exists(): - env_path.touch() - set_key(env_path, key, value) - except Exception as e: - console.print(f"[red]Error saving to .env: {e}[/red]") - - -def resolve_network_value( - network: str, - key: str, - default: Optional[str] = None -) -> str: - """ - Resolve a network-specific config value with priority: - 1. .env in current directory (e.g., SEPOLIA_RPC_URL) - 2. Global user config (~/.din/config.json → config["networks"][network][key]) - 3. Fallback default (if provided) - - Example: - resolve_network_value("sepolia", "rpc_url") - → checks SEPOLIA_RPC_URL in .env, then config - """ - if not network or not key: - raise ValueError("network and key must be non-empty strings") - - # Normalize key to uppercase for .env (e.g., "rpc_url" → "RPC_URL") - env_key_suffix = key.upper() - env_var_name = f"{network.upper()}_{env_key_suffix}" - - - # ✅ 1. Check .env in current working directory - resolved_env_var_name = get_env_key(env_var_name) - if resolved_env_var_name: - return resolved_env_var_name - - - # ✅ 2. Check global user config: ~/.din/config.json - config = load_config() - user_networks = config.get("networks", {}) - if network in user_networks and key in user_networks[network]: - return user_networks[network][key] - - # ✅ 3. Fallback to provided default or raise error - if default is not None: - return default - - raise KeyError( - f"Could not resolve '{key}' for network '{network}'.\n" - f"→ Checked .env for '{env_var_name}'\n" - f"→ Checked config.json → networks.{network}.{key}\n" - f"→ No fallback provided." - ) - - -def get_w3(effective_network): - rpc_url = resolve_network_value(effective_network,"rpc_url") - try: - - w3 = Web3(Web3.HTTPProvider(rpc_url)) - if not w3.is_connected(): - raise ConnectionError(f"Could not connect to Ethereum node at {rpc_url}") - return w3 - except Exception as e: - raise ConnectionError(f"Could not connect to Ethereum node for network '{effective_network}': {e}") from e - - def get_demo_private_key(account_index: int) -> str: """Load private key for Hardhat dev account by index.""" # Path to accounts.json (relative to dincli package) @@ -406,51 +186,61 @@ def load_account(name: str = "default") -> Account: keystore_data = _extract_keystore(data) - password = _get_password(name) + # Fetch DIN_WALLET_PASSWORD once and thread it through the password helpers so a + # single unlock parses .env once rather than twice (get_env_key has no memoization). + env_pass = get_env_key("DIN_WALLET_PASSWORD") + + password = _get_password(name, env_pass=env_pass) try: private_key = Account.decrypt(keystore_data, password) - - _cache_password_in_memory(name, password) + _cache_password_in_memory(name, password, env_pass=env_pass) _cleanup_stale_session() return Account.from_key(private_key) except ValueError: - if _clear_memory_cache(name): console.print("[yellow]Cached password failed, prompting...[/yellow]") password = getpass("Enter wallet password: ") try: private_key = Account.decrypt(keystore_data, password) - _cache_password_in_memory(name, password) + _cache_password_in_memory(name, password, env_pass=env_pass) _cleanup_stale_session() return Account.from_key(private_key) - except ValueError: pass raise ValueError("Invalid password or corrupted keystore.") -def _get_password(name: str = "default", prompt: bool = True) -> str: +def _get_password(name: str = "default", prompt: bool = True, + is_new_wallet: bool = False, env_pass=_UNSET) -> str: """ Get password from: 1. DIN_WALLET_PASSWORD env var 2. In-memory cache (keyed by name) 3. Interactive prompt + is_new_wallet: when True, the in-memory cache is skipped entirely so a freshly + created/overwritten wallet is never silently encrypted with a stale cached + password. The DIN_WALLET_PASSWORD env var is still honored (deliberate + automation path). + env_pass: an already-fetched DIN_WALLET_PASSWORD value, to avoid re-parsing .env + (see load_account). Omit (_UNSET) to self-fetch. """ _cleanup_stale_session() # 1. Environment variable - env_pass = get_env_key("DIN_WALLET_PASSWORD") + if env_pass is _UNSET: + env_pass = get_env_key("DIN_WALLET_PASSWORD") if env_pass: return env_pass # 2. Session cache - now = time.time() - entry = _PASSWORD_CACHE.get(name) - if entry is not None: - cached_pw, expiry = entry - if now < expiry: - return cached_pw - del _PASSWORD_CACHE[name] + if not is_new_wallet: + now = time.time() + entry = _PASSWORD_CACHE.get(name) + if entry is not None: + cached_pw, expiry = entry + if now < expiry: + return cached_pw + del _PASSWORD_CACHE[name] # 3. Prompt if prompt: @@ -458,8 +248,11 @@ def _get_password(name: str = "default", prompt: bool = True) -> str: return "" -def _cache_password_in_memory(name: str, password: str) -> None: - if get_env_key("DIN_WALLET_PASSWORD"): + +def _cache_password_in_memory(name: str, password: str, env_pass=_UNSET) -> None: + if env_pass is _UNSET: + env_pass = get_env_key("DIN_WALLET_PASSWORD") + if env_pass: return ttl = int(os.environ.get("DIN_PASSWORD_TTL", _PASSWORD_TTL_DEFAULT)) _PASSWORD_CACHE[name] = (password, time.time() + ttl) diff --git a/dincli/dind/__init__.py b/dincli/dind/__init__.py new file mode 100644 index 0000000..e5adde0 --- /dev/null +++ b/dincli/dind/__init__.py @@ -0,0 +1,8 @@ +"""DIN Daemon (dind) — always-on process framework (P4-1.1). + +The daemon drives the protocol event loop, job queue, health endpoint, and +graceful shutdown. It imports from ``dincli.sdk`` (config, log, errors) never +from ``dincli.cli``. +""" + +__version__ = "0.1.0" diff --git a/dincli/dind/config.py b/dincli/dind/config.py new file mode 100644 index 0000000..bb1471f --- /dev/null +++ b/dincli/dind/config.py @@ -0,0 +1,69 @@ +"""Shared resolver for dind lifecycle commands. + +Precedence: --flag > env > config.json > default. + +Used by start, stop, and status so a lifecycle command can never target the +wrong daemon. +""" + +import os +from pathlib import Path + +from dincli.sdk.config import CACHE_DIR, load_config + +HEALTH_HOST_DEFAULT = "127.0.0.1" +HEALTH_PORT_DEFAULT = 8787 +DEFAULT_STATE_DIR = CACHE_DIR / "dind" + + +def resolve_state_dir(flag: str | None = None) -> Path: + if flag: + return Path(flag).expanduser().resolve() + + env_val = os.environ.get("DIN_DIND_STATE_DIR") + if env_val: + return Path(env_val).expanduser().resolve() + + config = load_config() + config_val = config.get("dind_state_dir") + if config_val: + return Path(config_val).expanduser().resolve() + + return DEFAULT_STATE_DIR + + +def resolve_health_host(flag: str | None = None) -> str: + if flag: + return flag + + env_val = os.environ.get("DIN_DIND_HEALTH_HOST") + if env_val: + return env_val + + config = load_config() + config_val = config.get("dind_health_host") + if config_val: + return config_val + + return HEALTH_HOST_DEFAULT + + +def resolve_health_port(flag: int | None = None) -> int: + if flag is not None: + return flag + + env_val = os.environ.get("DIN_DIND_HEALTH_PORT") + if env_val and env_val.strip(): + return int(env_val) + + config = load_config() + config_val = config.get("dind_health_port") + if config_val is not None: + return int(config_val) + + return HEALTH_PORT_DEFAULT + + +def validate_health_port(port: int) -> None: + if not (1 <= port <= 65535): + raise ValueError(f"Health port must be 1-65535, got {port}") diff --git a/dincli/dind/daemon.py b/dincli/dind/daemon.py new file mode 100644 index 0000000..e09f654 --- /dev/null +++ b/dincli/dind/daemon.py @@ -0,0 +1,67 @@ +"""Scheduler loop with injectable stop_event/max_ticks for testability. + +Heartbeat = direct ``daemon_meta.last_tick`` update (not a queue row). +Each tick: heartbeat, then claim/dispatch one pending job. +""" + +import json +import logging +import time +from datetime import datetime, timezone +from threading import Event + +from dincli.dind.jobs import JOB_HANDLERS, Job +from dincli.dind.state import StateStore + +logger = logging.getLogger("dincli") + + +class DaemonLoop: + def __init__( + self, + state: StateStore, + stop_event: Event, + tick_interval: float = 1.0, + max_ticks: int | None = None, + ): + self.state = state + self._stop = stop_event + self.tick_interval = tick_interval + self.max_ticks = max_ticks + self.tick_count = 0 + + def run(self) -> None: + while not self._stop.is_set(): + if self.max_ticks is not None and self.tick_count >= self.max_ticks: + break + + self._tick() + self.tick_count += 1 + + self._stop.wait(self.tick_interval) + + def _tick(self) -> None: + now = datetime.now(timezone.utc).isoformat() + self.state.set_meta("last_tick", now) + + row = self.state.claim_next() + if row is None: + return + + job = Job( + type=row["type"], + payload=json.loads(row.get("payload", "{}")), + id=row["id"], + ) + + handler = JOB_HANDLERS.get(job.type) + if handler is None: + self.state.fail_job(job.id, f"No handler for job type: {job.type}") + return + + try: + handler(job) + self.state.complete_job(job.id) + self.state.set_meta("last_success", now) + except Exception as e: + self.state.fail_job(job.id, str(e)) diff --git a/dincli/dind/examples/com.din.dind.plist b/dincli/dind/examples/com.din.dind.plist new file mode 100644 index 0000000..cf43153 --- /dev/null +++ b/dincli/dind/examples/com.din.dind.plist @@ -0,0 +1,30 @@ + + + + + Label + com.din.dind + ProgramArguments + + dind + start + + RunAtLoad + + KeepAlive + + StandardOutPath + /usr/local/var/log/dind.log + StandardErrorPath + /usr/local/var/log/dind.err + EnvironmentVariables + + DIN_DIND_STATE_DIR + /usr/local/var/dind + DIN_DIND_HEALTH_HOST + 127.0.0.1 + DIN_DIND_HEALTH_PORT + 8787 + + + diff --git a/dincli/dind/examples/dind.service b/dincli/dind/examples/dind.service new file mode 100644 index 0000000..402a213 --- /dev/null +++ b/dincli/dind/examples/dind.service @@ -0,0 +1,22 @@ +[Unit] +Description=DIN Daemon (dind) — Decentralized Intelligence Network +Documentation=https://github.com/InfiniteZeroFoundation/DevNet +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=%i +EnvironmentFile=-%E/dind/%i.env +ExecStart=dind start +ExecReload=/bin/kill -HUP $MAINPID +Restart=on-failure +RestartSec=5 +KillSignal=SIGTERM +TimeoutStopSec=30 +StandardOutput=journal +StandardError=journal +SyslogIdentifier=dind + +[Install] +WantedBy=multi-user.target diff --git a/dincli/dind/health.py b/dincli/dind/health.py new file mode 100644 index 0000000..d782079 --- /dev/null +++ b/dincli/dind/health.py @@ -0,0 +1,111 @@ +"""HTTP /health endpoint — stdlib ThreadingHTTPServer, one JSON handler. + +Records the actually-bound host/port in ``daemon_meta`` so ``status`` can +find the endpoint even with ephemeral ports. +""" + +import json +import logging +import os +import shutil +import time +from datetime import datetime, timezone +from http.server import HTTPServer, BaseHTTPRequestHandler + +from dincli.dind.state import StateStore + +logger = logging.getLogger("dincli") + + +class _ThreadingHTTPServer(HTTPServer): + daemon_threads = True + + +class HealthHandler(BaseHTTPRequestHandler): + state: StateStore = None + start_time: float = 0.0 + stale_seconds: float = 30.0 + + def log_message(self, format, *args): + pass + + def do_GET(self): + if self.path != "/health": + self.send_error(404) + return + + try: + payload = self._build_payload() + body = json.dumps(payload).encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + except Exception: + logger.exception("Health endpoint error") + self.send_error(500) + + def _build_payload(self) -> dict: + now_ts = time.time() + uptime_s = int(now_ts - self.start_time) + pid = os.getpid() + + last_tick = self.state.get_meta("last_tick") + last_success = self.state.get_meta("last_success") + + status = "healthy" + if last_tick: + tick_dt = datetime.fromisoformat(last_tick) + age = (datetime.now(timezone.utc) - tick_dt).total_seconds() + if age > self.stale_seconds: + status = "degraded" + + queue = self.state.get_job_counts() + + state_dir = str(self.state.db_path.parent) + try: + disk = shutil.disk_usage(state_dir) + disk_free = disk.free + disk_total = disk.total + except OSError: + disk_free = None + disk_total = None + + return { + "status": status, + "uptime_s": uptime_s, + "pid": pid, + "last_tick": last_tick, + "last_success": last_success, + "queue": queue, + "resources": { + "cpu_count": os.cpu_count(), + "disk_free_bytes": disk_free, + "disk_total_bytes": disk_total, + }, + } + + +class HealthServer: + def __init__(self, host: str, port: int, state: StateStore): + self.host = host + self.port = port + self.state = state + + def run(self) -> None: + HealthHandler.state = self.state + HealthHandler.start_time = time.time() + + self.server = _ThreadingHTTPServer((self.host, self.port), HealthHandler) + + actual_host, actual_port = self.server.server_address + self.state.set_meta("health_host", actual_host) + self.state.set_meta("health_port", str(actual_port)) + + logger.info("Health server listening on %s:%d", actual_host, actual_port) + self.server.serve_forever(poll_interval=0.5) + + def shutdown(self) -> None: + if hasattr(self, "server") and self.server: + self.server.shutdown() diff --git a/dincli/dind/jobs.py b/dincli/dind/jobs.py new file mode 100644 index 0000000..ad06aec --- /dev/null +++ b/dincli/dind/jobs.py @@ -0,0 +1,37 @@ +"""Job dataclass, status enum, and handler registry.""" + +from dataclasses import dataclass, field +from enum import Enum +from typing import Callable + + +class JobStatus(str, Enum): + PENDING = "pending" + RUNNING = "running" + DONE = "done" + FAILED = "failed" + + +@dataclass +class Job: + type: str + payload: dict = field(default_factory=dict) + id: int | None = None + status: JobStatus = JobStatus.PENDING + attempts: int = 0 + + +JOB_HANDLERS: dict[str, Callable] = {} + + +def register_handler(job_type: str): + def decorator(fn): + JOB_HANDLERS[job_type] = fn + return fn + + return decorator + + +@register_handler("demo") +def demo_handler(job: Job) -> None: + pass diff --git a/dincli/dind/logging.py b/dincli/dind/logging.py new file mode 100644 index 0000000..142a51b --- /dev/null +++ b/dincli/dind/logging.py @@ -0,0 +1,69 @@ +"""Structured JSON logging for the dind daemon (T0.2d). + +JsonFormatter emits line-delimited JSON with fields: +ts, level, logger, msg, (+ context: role, network, model_id, gi, job_id, error_code). + +configure_logging(mode) is idempotent — repeat calls don't stack handlers. +It attaches to the "dincli" logger (which otherwise propagates to the root +logger's text handler installed by sdk/log.py), sets propagate=False so +records fire only the JSON handler, and captures/restores the level. + +Error details reuse sdk.errors sanitize_details for secrets safety. +""" + +import json +import logging +from datetime import datetime, timezone + +from dincli.sdk.config import CONFIG_FILE + +_CONTEXT_FIELDS = ("role", "network", "model_id", "gi", "job_id", "error_code") +_DAEMON_HANDLER_MARKER = "__dind_json_handler__" + + +def _resolve_log_level() -> int: + if CONFIG_FILE.exists(): + try: + with open(CONFIG_FILE, "r") as f: + config = json.load(f) + level_str = config.get("log_level", "INFO") + return getattr(logging, level_str.upper(), logging.INFO) + except (json.JSONDecodeError, OSError): + return logging.INFO + return logging.INFO + + +class JsonFormatter(logging.Formatter): + def format(self, record: logging.LogRecord) -> str: + obj = { + "ts": datetime.now(timezone.utc).isoformat(), + "level": record.levelname, + "logger": record.name, + "msg": record.getMessage(), + } + for key in _CONTEXT_FIELDS: + val = getattr(record, key, None) + if val is not None: + obj[key] = val + + if record.exc_info and record.exc_info[1]: + obj["exception"] = str(record.exc_info[1]) + + return json.dumps(obj) + + +def configure_logging(mode: str = "json") -> None: + logger = logging.getLogger("dincli") + + for h in list(logger.handlers): + if isinstance(h.formatter, JsonFormatter): + logger.removeHandler(h) + + if mode == "json": + level = _resolve_log_level() + handler = logging.StreamHandler() + handler.setFormatter(JsonFormatter()) + setattr(handler, _DAEMON_HANDLER_MARKER, True) + logger.addHandler(handler) + logger.propagate = False + logger.setLevel(level) diff --git a/dincli/dind/main.py b/dincli/dind/main.py new file mode 100644 index 0000000..f27f32f --- /dev/null +++ b/dincli/dind/main.py @@ -0,0 +1,209 @@ +"""dind Typer app — start | stop | status (P4-1.1 scaffold). + +All three commands accept --state-dir so lifecycle ops can't target the +wrong daemon. +""" + +import json +import signal +import threading +import time +import urllib.request +from datetime import datetime, timezone +from pathlib import Path + +import typer + +from dincli import __version__ +from dincli.dind.config import ( + resolve_health_host, + resolve_health_port, + resolve_state_dir, + validate_health_port, +) +from dincli.dind.paths import StateDirs +from dincli.dind.process import ( + is_process_running, + read_pid, + remove_pid, + send_signal, + write_pid, +) + +app = typer.Typer( + help=f"DIN Daemon (dind) v{__version__} — always-on process framework.", + pretty_exceptions_enable=False, +) + +_STATE_DIR_HELP = "Path to dind state directory (pid, db). Env: DIN_DIND_STATE_DIR. Default: XDG_CACHE/dincli/dind" + +STATE_DIR = typer.Option( + None, "--state-dir", help=_STATE_DIR_HELP, +) + + +@app.command() +def start( + state_dir: str | None = STATE_DIR, + health_host: str | None = typer.Option( + None, + "--health-host", + help="Health bind host. Env: DIN_DIND_HEALTH_HOST. Default: 127.0.0.1", + ), + health_port: int | None = typer.Option( + None, + "--health-port", + help="Health bind port. Env: DIN_DIND_HEALTH_PORT. Default: 8787", + ), +) -> None: + """Start the dind daemon (foreground).""" + resolved = resolve_state_dir(state_dir) + paths = StateDirs(resolved) + + existing_pid = read_pid(paths.pid_path) + if existing_pid is not None and is_process_running(existing_pid): + typer.echo( + f"dind already running (PID {existing_pid}) in {resolved}", + err=True, + ) + raise typer.Exit(1) + + if existing_pid is not None: + remove_pid(paths.pid_path) + + host = resolve_health_host(health_host) + port = resolve_health_port(health_port) + validate_health_port(port) + + from dincli.dind.daemon import DaemonLoop + from dincli.dind.health import HealthServer + from dincli.dind.logging import configure_logging + from dincli.dind.signals import install_shutdown_handlers + from dincli.dind.state import StateStore + + configure_logging("json") + + import logging + logger = logging.getLogger("dincli") + + write_pid(paths.pid_path) + + stop_event = threading.Event() + install_shutdown_handlers(stop_event) + + state = StateStore(paths.db_path) + state.set_meta("started_at", datetime.now(timezone.utc).isoformat()) + state.reset_running_jobs() + + health = HealthServer(host, port, state) + health_thread = threading.Thread(target=health.run, daemon=True) + health_thread.start() + + loop = DaemonLoop(state, stop_event) + try: + logger.info("dind daemon started (state=%s)", resolved) + loop.run() + finally: + logger.info("Shutting down dind daemon...") + + state.reset_running_jobs() + shutdown_count_str = state.get_meta("shutdown_count") or "0" + state.set_meta("shutdown_count", str(int(shutdown_count_str) + 1)) + + health.shutdown() + health_thread.join(timeout=5) + + remove_pid(paths.pid_path) + state.close() + + logger.info("dind daemon shut down") + + +@app.command() +def stop( + state_dir: str | None = STATE_DIR, + timeout: int = typer.Option( + 30, "--timeout", "-t", + help="Seconds to wait for the daemon to stop after SIGTERM.", + ), +) -> None: + """Stop a running dind daemon via its PID file.""" + resolved = resolve_state_dir(state_dir) + paths = StateDirs(resolved) + + pid = read_pid(paths.pid_path) + if pid is None: + typer.echo( + f"No PID file found at {paths.pid_path}. Is dind running?", + err=True, + ) + raise typer.Exit(1) + + if not is_process_running(pid): + typer.echo(f"PID {pid} is stale — cleaning up.") + remove_pid(paths.pid_path) + return + + send_signal(pid, signal.SIGTERM) + typer.echo(f"Sent SIGTERM to PID {pid}. Waiting up to {timeout}s...") + + for _ in range(timeout): + if not is_process_running(pid): + typer.echo("dind stopped.") + return + time.sleep(1) + + typer.echo(f"dind did not stop within {timeout}s.", err=True) + raise typer.Exit(1) + + +@app.command() +def status( + state_dir: str | None = STATE_DIR, +) -> None: + """Check whether a dind daemon is running (PID + optional /health).""" + resolved = resolve_state_dir(state_dir) + paths = StateDirs(resolved) + + pid = read_pid(paths.pid_path) + if pid is None: + typer.echo("dind is not running (no PID file).") + return + + running = is_process_running(pid) + if not running: + typer.echo( + f"dind is stopped (stale PID {pid} in {paths.pid_path})." + ) + return + + typer.echo(f"dind is running PID {pid} state-dir {resolved}") + + try: + from dincli.dind.state import StateStore + + store = StateStore(paths.db_path) + health_host = store.get_meta("health_host") or resolve_health_host() + health_port_str = store.get_meta("health_port") + health_port = ( + int(health_port_str) + if health_port_str + else resolve_health_port() + ) + + url = f"http://{health_host}:{health_port}/health" + with urllib.request.urlopen(url, timeout=5) as resp: + health = json.loads(resp.read()) + + typer.echo(f" Health: {health['status']}") + typer.echo(f" Uptime: {health['uptime_s']}s") + typer.echo( + f" Pending: {health['queue']['pending']} " + f"Running: {health['queue']['running']} " + f"Failed: {health['queue']['failed']}" + ) + typer.echo( + f" CPU count: {health['resources']['cpu_count']}" + ) + except Exception: + typer.echo(" (health endpoint unavailable)") diff --git a/dincli/dind/paths.py b/dincli/dind/paths.py new file mode 100644 index 0000000..688df28 --- /dev/null +++ b/dincli/dind/paths.py @@ -0,0 +1,15 @@ +from pathlib import Path + + +class StateDirs: + def __init__(self, state_dir: str | Path): + self.state_dir = Path(state_dir) + self.state_dir.mkdir(parents=True, exist_ok=True) + + @property + def db_path(self) -> Path: + return self.state_dir / "dind.db" + + @property + def pid_path(self) -> Path: + return self.state_dir / "dind.pid" diff --git a/dincli/dind/process.py b/dincli/dind/process.py new file mode 100644 index 0000000..2195202 --- /dev/null +++ b/dincli/dind/process.py @@ -0,0 +1,39 @@ +"""PID file read/write, liveness detection, signal dispatch.""" + +import os +import signal +from pathlib import Path + + +def read_pid(pid_path: Path) -> int | None: + if not pid_path.exists(): + return None + content = pid_path.read_text().strip() + if not content: + return None + try: + return int(content) + except ValueError: + return None + + +def write_pid(pid_path: Path) -> None: + pid_path.parent.mkdir(parents=True, exist_ok=True) + pid_path.write_text(str(os.getpid())) + + +def remove_pid(pid_path: Path) -> None: + if pid_path.exists(): + pid_path.unlink(missing_ok=True) + + +def is_process_running(pid: int) -> bool: + try: + os.kill(pid, 0) + return True + except (ProcessLookupError, PermissionError): + return False + + +def send_signal(pid: int, sig: int = signal.SIGTERM) -> None: + os.kill(pid, sig) diff --git a/dincli/dind/signals.py b/dincli/dind/signals.py new file mode 100644 index 0000000..0aeec0b --- /dev/null +++ b/dincli/dind/signals.py @@ -0,0 +1,21 @@ +"""Signal handlers for graceful daemon shutdown. + +Installs SIGTERM/SIGINT handlers that set a threading.Event. Must be called +from the main thread (signal.signal restriction). +""" + +import logging +import signal +from threading import Event + +logger = logging.getLogger("dincli") + + +def install_shutdown_handlers(stop_event: Event) -> None: + def _handler(signum, frame): + sig_name = signal.Signals(signum).name + logger.info("Received %s, initiating graceful shutdown", sig_name) + stop_event.set() + + signal.signal(signal.SIGTERM, _handler) + signal.signal(signal.SIGINT, _handler) diff --git a/dincli/dind/state.py b/dincli/dind/state.py new file mode 100644 index 0000000..06ab2e4 --- /dev/null +++ b/dincli/dind/state.py @@ -0,0 +1,173 @@ +"""SQLite state store — jobs queue + daemon metadata. + +WAL mode for safe concurrency. Per-thread connections. Retention cap on +done/failed history from day one. + +Tables: + jobs (id, type, status, payload, attempts, created_at, updated_at, last_error) + daemon_meta (key, value) +""" + +import json +import sqlite3 +import threading +from datetime import datetime, timezone +from pathlib import Path + + +SCHEMA = """ +CREATE TABLE IF NOT EXISTS jobs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + type TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending', + payload TEXT NOT NULL DEFAULT '{}', + attempts INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + last_error TEXT +); + +CREATE TABLE IF NOT EXISTS daemon_meta ( + key TEXT PRIMARY KEY, + value TEXT +); +""" + + +class StateStore: + def __init__(self, db_path: Path, retention_limit: int = 1000): + self.db_path = db_path + self.retention_limit = retention_limit + self._local = threading.local() + + def _get_conn(self) -> sqlite3.Connection: + if not hasattr(self._local, "conn") or self._local.conn is None: + conn = sqlite3.connect(str(self.db_path)) + conn.execute("PRAGMA journal_mode=WAL") + conn.execute("PRAGMA foreign_keys=ON") + conn.row_factory = sqlite3.Row + conn.executescript(SCHEMA) + self._local.conn = conn + return self._local.conn + + def close(self) -> None: + if hasattr(self._local, "conn") and self._local.conn: + self._local.conn.close() + self._local.conn = None + + # ── daemon_meta ────────────────────────────────────────────────────── + + def set_meta(self, key: str, value: str) -> None: + conn = self._get_conn() + conn.execute( + "INSERT INTO daemon_meta(key, value) VALUES (?, ?) " + "ON CONFLICT(key) DO UPDATE SET value = excluded.value", + (key, value), + ) + conn.commit() + + def get_meta(self, key: str) -> str | None: + conn = self._get_conn() + row = conn.execute( + "SELECT value FROM daemon_meta WHERE key = ?", (key,) + ).fetchone() + return row["value"] if row else None + + # ── jobs ───────────────────────────────────────────────────────────── + + def enqueue(self, job_type: str, payload: dict | None = None) -> int: + now = datetime.now(timezone.utc).isoformat() + conn = self._get_conn() + cursor = conn.execute( + "INSERT INTO jobs(type, status, payload, created_at, updated_at) " + "VALUES (?, 'pending', ?, ?, ?)", + (job_type, json.dumps(payload or {}), now, now), + ) + conn.commit() + return cursor.lastrowid + + def claim_next(self) -> dict | None: + conn = self._get_conn() + conn.execute("BEGIN IMMEDIATE") + row = conn.execute( + "SELECT * FROM jobs WHERE status = 'pending' ORDER BY id ASC LIMIT 1" + ).fetchone() + if row is None: + conn.commit() + return None + now = datetime.now(timezone.utc).isoformat() + conn.execute( + "UPDATE jobs SET status = 'running', updated_at = ?, " + "attempts = attempts + 1 WHERE id = ?", + (now, row["id"]), + ) + conn.commit() + row = conn.execute( + "SELECT * FROM jobs WHERE id = ?", (row["id"],) + ).fetchone() + return dict(row) + + def complete_job(self, job_id: int) -> None: + now = datetime.now(timezone.utc).isoformat() + conn = self._get_conn() + conn.execute( + "UPDATE jobs SET status = 'done', updated_at = ? WHERE id = ?", + (now, job_id), + ) + conn.commit() + self._retain() + + def fail_job(self, job_id: int, error: str) -> None: + now = datetime.now(timezone.utc).isoformat() + conn = self._get_conn() + conn.execute( + "UPDATE jobs SET status = 'failed', updated_at = ?, last_error = ? " + "WHERE id = ?", + (now, error, job_id), + ) + conn.commit() + self._retain() + + def reset_running_jobs(self) -> None: + now = datetime.now(timezone.utc).isoformat() + conn = self._get_conn() + conn.execute( + "UPDATE jobs SET status = 'pending', updated_at = ?, " + "last_error = 'interrupted@shutdown' " + "WHERE status = 'running'", + (now,), + ) + conn.commit() + + def get_job_counts(self) -> dict: + conn = self._get_conn() + queries = { + "pending": "WHERE status = 'pending'", + "running": "WHERE status = 'running'", + "failed": "WHERE status = 'failed'", + } + result = {} + for key, clause in queries.items(): + row = conn.execute( + f"SELECT COUNT(*) as c FROM jobs {clause}" + ).fetchone() + result[key] = row["c"] + return result + + # ── retention ──────────────────────────────────────────────────────── + + def _retain(self) -> None: + conn = self._get_conn() + total = conn.execute( + "SELECT COUNT(*) as c FROM jobs WHERE status IN ('done', 'failed')" + ).fetchone()["c"] + if total > self.retention_limit: + excess = total - self.retention_limit + conn.execute( + "DELETE FROM jobs WHERE id IN (" + " SELECT id FROM jobs WHERE status IN ('done', 'failed') " + " ORDER BY updated_at ASC LIMIT ?" + ")", + (excess,), + ) + conn.commit() diff --git a/dincli/docker/node/Dockerfile b/dincli/docker/node/Dockerfile index 6935ee7..ea02fec 100644 --- a/dincli/docker/node/Dockerfile +++ b/dincli/docker/node/Dockerfile @@ -73,7 +73,6 @@ RUN groupadd --gid "$DIN_GID" dinuser \ USER dinuser WORKDIR /home/dinuser -# dincli is a CLI, not yet a daemon (the `dind` daemon is a future phase). To -# keep a long-lived container that operators `docker compose exec` into, the -# main process just idles. When `dind` exists this CMD becomes its entrypoint. -CMD ["sleep", "infinity"] +# dind daemon entrypoint: the long-lived container process that drives the +# protocol event loop, health endpoint, and job queue. +CMD ["dind", "start"] diff --git a/dincli/docker/node/README.md b/dincli/docker/node/README.md index 2cf1ad3..86e5c3e 100644 --- a/dincli/docker/node/README.md +++ b/dincli/docker/node/README.md @@ -267,8 +267,8 @@ $DIN_STATE_DIR/ but **not for production**. - **Production:** Import an **encrypted keystore** using `--keystore` into a named account, then select it via `--wallet` or `DIN_WALLET_NAME`. See - [wallet-setup.md](../../../Documentation/guides/wallet-setup.md) and - [keystore-migration.md](../../../Documentation/guides/keystore-migration.md). + [wallet-setup.md](../../../Documentation/public/guides/wallet-setup.md) and + [keystore-migration.md](../../../Documentation/public/guides/keystore-migration.md). > The commands below use `$DIN_STATE_DIR`. `docker compose` reads `.env` diff --git a/dincli/docker/node/docker-compose.yml b/dincli/docker/node/docker-compose.yml index 2bc6016..7a27d23 100644 --- a/dincli/docker/node/docker-compose.yml +++ b/dincli/docker/node/docker-compose.yml @@ -72,7 +72,13 @@ services: # Survive host reboots / daemon restarts; `docker compose down` still stops it. restart: unless-stopped - # Inherited from the image (CMD ["sleep", "infinity"]): dincli is a CLI, not - # yet a daemon, so the container idles and operators `exec` in. Stated here - # for clarity; remove when the `dind` daemon becomes the entrypoint. - command: ["sleep", "infinity"] + # din-node now runs the dind daemon as the entrypoint (CMD ["dind","start"] + # in the Dockerfile). The container is the long-lived daemon process. + command: ["dind", "start"] + + healthcheck: + test: ["CMD", "curl", "-f", "http://127.0.0.1:8787/health"] + interval: 15s + timeout: 5s + retries: 3 + start_period: 10s diff --git a/dincli/main.py b/dincli/main.py index 0a60a16..3514f56 100644 --- a/dincli/main.py +++ b/dincli/main.py @@ -60,6 +60,7 @@ def main( ): ctx.obj = DinContext() console = ctx.obj.console + ctx.obj.select_wallet(wallet) configured_network = ctx.obj.select_network(network).network diff --git a/dincli/sdk/__init__.py b/dincli/sdk/__init__.py new file mode 100644 index 0000000..4f5a63e --- /dev/null +++ b/dincli/sdk/__init__.py @@ -0,0 +1,15 @@ +"""DIN-SDK — shared logic layer for dincli and the dind daemon. + +All protocol logic (chain, IPFS, wallet, manifest, transactions) lives here so +both the interactive CLI and the daemon can drive it. SDK functions return typed +dataclasses and raise ``DinError`` subclasses; they never print, never call +``typer.Exit``/``sys.exit``, and never prompt. + +Design contract: ``dincli.sdk`` must import without pulling in ``typer``, +``rich``, or any ``dincli.cli.*`` module (enforced by tests/test_sdk_boundary.py). +The CLI depends on the SDK, never the reverse. + +See Developer/design/sdk-interface-proposal.md for the full interface proposal. +""" + +__version__ = "0.1.0" diff --git a/dincli/sdk/cid.py b/dincli/sdk/cid.py new file mode 100644 index 0000000..5af52a5 --- /dev/null +++ b/dincli/sdk/cid.py @@ -0,0 +1,71 @@ +""" +CID ↔ bytes32 conversion utilities. + +Pure module (no imports from dincli.cli) so both the SDK and the CLI can depend +on it without creating a circular dependency. Moved verbatim from +dincli.services.cid_utils as the first SDK clean-seed extraction (issue #20); +that path remains as a thin re-export shim for backward compatibility. +""" +import cid as _cid + + +def get_bytes32_from_cid(cid_str: str) -> str: + """ + Extracts the raw 32-byte SHA2-256 digest (hex) from a CID string. + Returns 64 hex characters (without the '1220' multihash prefix). + """ + cid_obj = _cid.make_cid(cid_str) + + # Full multihash bytes include '1220' prefix (12 = sha2-256, 20 = 32-byte length) + full_multihash_hex = cid_obj.multihash.hex() + + # Strip the first 4 hex characters ('1220') to get only the 32-byte digest + return full_multihash_hex[4:] + + +def get_cid_from_bytes32(byte32_hex: str, version: int = 1, encoding: str = "base32") -> str: + """ + Converts a raw 32-byte SHA2-256 digest (hex) back into a CID. + + Args: + byte32_hex: 64-character hex string (raw digest, NO '1220' prefix). + version: 0 or 1. + encoding: 'base32' (recommended for v1) or 'base58btc'. + + Returns: + The CID string. + """ + from multibase import encode # local import keeps top-level deps minimal + + # Reconstruct the full multihash: 12 = sha2-256, 20 = 32-byte length + multihash_hex = "1220" + str(byte32_hex) + multihash_bytes = bytes.fromhex(multihash_hex) + + # Build CIDv1 (dag-pb is standard for IPFS file data) + cid_v1 = _cid.CIDv1(codec="dag-pb", multihash=multihash_bytes) + + if version == 0: + return str(cid_v1.to_v0()) + + if version == 1: + if encoding == "base32": + # CIDv1 raw bytes: + raw_cid_bytes = b"\x01" + b"\x70" + multihash_bytes + return encode("base32", raw_cid_bytes).decode("ascii") + if encoding == "base58btc": + return str(cid_v1) + raise ValueError("encoding must be 'base32' or 'base58btc'") + + raise ValueError("version must be 0 or 1") + +def get_cidv1base32_from_cid(cid_str: str) -> str: + """ + Extracts the CIDv1 from a CID string. + """ + + # Base32 CIDv1 starts with 'bafy' and uses [a-z2-7] + if cid_str.startswith('bafy') and all(c in 'abcdefghijklmnopqrstuvwxyz234567' for c in cid_str[4:]): + return cid_str + + bytes32_hex = get_bytes32_from_cid(cid_str) + return get_cid_from_bytes32(bytes32_hex) diff --git a/dincli/sdk/config.py b/dincli/sdk/config.py new file mode 100644 index 0000000..4bfa719 --- /dev/null +++ b/dincli/sdk/config.py @@ -0,0 +1,200 @@ +import json +import logging +import os +from dataclasses import dataclass +from pathlib import Path +from typing import Optional + +from platformdirs import user_cache_dir, user_config_dir + +logger = logging.getLogger("dincli") + +CONFIG_DIR = Path(user_config_dir("dincli")) +CACHE_DIR = Path(user_cache_dir("dincli")) + +# Sibling cache used for docker-only artifacts (e.g. pip-installed client +# packages). Kept separate from CACHE_DIR so containers never need read/write +# access to the manifest/service/wallet cache that dincli itself manages. +WORKER_CACHE_DIR = Path(user_cache_dir("dincli-worker")) + +CONFIG_FILE = CONFIG_DIR / "config.json" + +ALLOWED_NETWORKS = ["local", "sepolia_devnet", "sepolia_op_devnet", "mainnet"] # "sepolia_testnet" +SUPPORTED_IPFS_PROVIDERS = ("env", "filebase", "custom") + +LEGACY_IPFS_PROVIDER_ALIASES = { + "": "env", + "default": "env", + "env": "env", + "ipfs node": "env", + "ipfs-node": "env", + "node": "env", +} + +FILEBASE_IPFS_ADD_URL = "https://rpc.filebase.io/api/v0/add" +FILEBASE_IPFS_CAT_URL = "https://rpc.filebase.io/api/v0/cat" +FILEBASE_IPFS_PIN_URL = "https://rpc.filebase.io/api/v0/pin/add" + + +# Optional: only import dotenv if needed +try: + from dotenv import dotenv_values + HAS_DOTENV = True +except ImportError: + HAS_DOTENV = False + + +@dataclass(frozen=True) +class IPFSConfig: + provider: str = "env" + api_url_add: Optional[str] = None + api_url_retrieve: Optional[str] = None + api_key: Optional[str] = None + api_secret: Optional[str] = None + service_path: Optional[Path] = None + + +def save_config(data): + CONFIG_DIR.mkdir(parents=True, exist_ok=True) + with open(CONFIG_FILE, "w") as f: + json.dump(data, f, indent=4) + logger.debug(f"Config saved to {CONFIG_FILE}") + + +def load_config(): + if CONFIG_FILE.exists(): + logger.debug(f"Loading config from {CONFIG_FILE}") + with open(CONFIG_FILE, "r") as f: + try: + return json.load(f) + except json.JSONDecodeError: + logger.error(f"Error decoding config file at {CONFIG_FILE}. Returning empty config.") + return {} + else: + logger.warning(f"No config found at {CONFIG_FILE}") + return {} + + +def get_config(key, default=None): + config = load_config() + return config.get(key, default) + + +def _clean_optional_string(value: Optional[str]) -> Optional[str]: + if not isinstance(value, str): + return None + + stripped = value.strip() + return stripped or None + + +def normalize_ipfs_provider(provider: Optional[str]) -> str: + if provider is None: + return "env" + + normalized = provider.strip().lower() + return LEGACY_IPFS_PROVIDER_ALIASES.get(normalized, normalized) + + +def resolve_network(cli_network: str | None = None, default: str = "local") -> str: + # 1. CLI takes highest precedence + if cli_network is not None: + if cli_network not in ALLOWED_NETWORKS: + raise ValueError(f"Invalid network: {cli_network}. Must be one of: {ALLOWED_NETWORKS}") + return cli_network + + # 3. Check global config + from_config = get_config("network") + if from_config and isinstance(from_config, str) and from_config.strip(): + return from_config.strip() + + # 4. Fallback + return default + + +def resolve_ipfs_config(): + config = load_config() + configured_provider = config.get("ipfs_provider") + provider = normalize_ipfs_provider(configured_provider) if configured_provider else normalize_ipfs_provider(get_env_key("IPFS_PROVIDER", verbose=False)) + raw_service_path = _clean_optional_string(config.get("ipfs_service_path")) + + api_key = _clean_optional_string(config.get(f"ipfs_api_key_{provider}")) + if not api_key and provider == "filebase": + api_key = _clean_optional_string(config.get("ipfs_api_key")) + + return IPFSConfig( + provider=provider, + api_url_add=_clean_optional_string(get_env_key("IPFS_API_URL_ADD", verbose=False)), + api_url_retrieve=_clean_optional_string(get_env_key("IPFS_API_URL_RETRIEVE", verbose=False)), + api_key=api_key, + api_secret=_clean_optional_string(config.get("ipfs_api_secret")), + service_path=Path(raw_service_path).expanduser().resolve() if raw_service_path else None, + ) + + +def get_env_key(key: str, default: Optional[str] = None, verbose: bool = True) -> Optional[str]: + # 1. Already in environment? (e.g., from shell or parent process) + if key in os.environ: + return os.environ[key] + + # 2. Load from .env in current directory (if available) + env_path = Path(os.getcwd()) / ".env" + if HAS_DOTENV and env_path.exists(): + values = dotenv_values(dotenv_path=env_path) + if key not in values and default is None: + if verbose: + logger.warning(f" ❌ {key} not found in {os.getcwd()}/.env file") + return values.get(key, default) + + if not HAS_DOTENV and verbose: + logger.warning("Warning: python-dotenv not installed. Cannot save to .env") + + return default + + +def set_env_key(key: str, value: str): + if not HAS_DOTENV: + logger.warning("Warning: python-dotenv not installed. Cannot save to .env") + return + + env_path = Path(os.getcwd()) / ".env" + + try: + from dotenv import set_key + + if not env_path.exists(): + env_path.touch() + set_key(env_path, key, value) + except Exception as e: + logger.warning(f"Error saving to .env: {e}") + + +def resolve_network_value( + network: str, + key: str, + default: Optional[str] = None +) -> str: + if not network or not key: + raise ValueError("network and key must be non-empty strings") + + env_key_suffix = key.upper() + env_var_name = f"{network.upper()}_{env_key_suffix}" + + resolved_env_var_name = get_env_key(env_var_name) + if resolved_env_var_name: + return resolved_env_var_name + + config = load_config() + user_networks = config.get("networks", {}) + if network in user_networks and key in user_networks[network]: + return user_networks[network][key] + + if default is not None: + return default + + raise KeyError( + f"Could not resolve '{key}' for network '{network}'.\n" + f"→ Checked .env for '{env_var_name}'\n" + f"→ Checked config.json → networks.{network}.{key}\n" + f"→ No fallback provided." + ) diff --git a/dincli/sdk/contracts.py b/dincli/sdk/contracts.py new file mode 100644 index 0000000..728f76c --- /dev/null +++ b/dincli/sdk/contracts.py @@ -0,0 +1,141 @@ +import json +import os + +# Minimal ABIs +erc20_abi = [ + { + "constant": True, + "inputs": [], + "name": "decimals", + "outputs": [{"name": "", "type": "uint8"}], + "payable": False, + "stateMutability": "view", + "type": "function", + }, + { + "constant": True, + "inputs": [{"name": "owner", "type": "address"}], + "name": "balanceOf", + "outputs": [{"name": "", "type": "uint256"}], + "payable": False, + "stateMutability": "view", + "type": "function", + }, + { + "constant": True, + "inputs": [ + {"name": "owner", "type": "address"}, + {"name": "spender", "type": "address"}, + ], + "name": "allowance", + "outputs": [{"name": "", "type": "uint256"}], + "payable": False, + "stateMutability": "view", + "type": "function", + }, + { + "constant": False, + "inputs": [ + {"name": "spender", "type": "address"}, + {"name": "value", "type": "uint256"}, + ], + "name": "approve", + "outputs": [{"name": "", "type": "bool"}], + "payable": False, + "stateMutability": "nonpayable", + "type": "function", + }, + { + "constant": False, + "inputs": [ + {"name": "to", "type": "address"}, + {"name": "value", "type": "uint256"}, + ], + "name": "transfer", + "outputs": [{"name": "", "type": "bool"}], + "payable": False, + "stateMutability": "nonpayable", + "type": "function", + }, + { + "constant": False, + "inputs": [ + {"name": "from", "type": "address"}, + {"name": "to", "type": "address"}, + {"name": "value", "type": "uint256"}, + ], + "name": "transferFrom", + "outputs": [{"name": "", "type": "bool"}], + "payable": False, + "stateMutability": "nonpayable", + "type": "function", + } + +] +router_abi = [ + { + "inputs": [ + {"internalType": "uint256", "name": "amountOut", "type": "uint256"}, + {"internalType": "address[]", "name": "path", "type": "address[]"} + ], + "name": "getAmountsIn", + "outputs": [{"internalType": "uint256[]", "name": "amounts", "type": "uint256[]"}], + "stateMutability": "view", + "type": "function" + }, + { + "inputs": [ + {"internalType": "uint256", "name": "amountOut", "type": "uint256"}, + {"internalType": "address[]", "name": "path", "type": "address[]"}, + {"internalType": "address", "name": "to", "type": "address"}, + {"internalType": "uint256", "name": "deadline", "type": "uint256"} + ], + "name": "swapETHForExactTokens", + "outputs": [{"internalType": "uint256[]", "name": "amounts", "type": "uint256[]"}], + "stateMutability": "payable", + "type": "function" + } +] + + +def get_contract_instance( + artifact_path: str, + network: str, + address: str | None = None +): + from dincli.sdk.web3 import get_w3 + w3 = get_w3(network) + + if not os.path.isfile(artifact_path): + raise FileNotFoundError( + f"Contract artifact not found at: {artifact_path}\n" + "Tip: Ensure the contract is compiled and the path is correct." + ) + + try: + with open(artifact_path) as f: + data = json.load(f) + except json.JSONDecodeError as e: + raise ValueError( + f"Failed to parse JSON in artifact: {artifact_path}\n" + f"Reason: {str(e)}\n" + "Tip: This may indicate a corrupted or incomplete build. Try recompiling the contract." + ) from e + + if "abi" not in data: + raise ValueError(f"Artifact {artifact_path} missing 'abi' field") + abi = data["abi"] + + if address: + return w3.eth.contract(address=address, abi=abi) + else: + if isinstance(data["bytecode"], dict): + bytecode = data["bytecode"].get("object") + else: + bytecode = data.get("bytecode") + if not bytecode: + raise ValueError( + f"Artifact {artifact_path} missing 'bytecode' — required for deployment.\n" + "Tip: Use `dincli system dump-abi --bytecode` to include it." + ) + return w3.eth.contract(abi=abi, bytecode=bytecode) diff --git a/dincli/sdk/errors.py b/dincli/sdk/errors.py new file mode 100644 index 0000000..2cbd9a2 --- /dev/null +++ b/dincli/sdk/errors.py @@ -0,0 +1,189 @@ +"""DIN-SDK domain error hierarchy + detail sanitization. + +Pure module: no imports from ``dincli.cli``, ``typer``, or ``rich``. SDK +functions raise these instead of calling ``typer.Exit``/``sys.exit``; the CLI +maps them to exit codes, the daemon maps them to job errors and keys retries +off the stable ``code``. + +See Developer/design/sdk-interface-proposal.md §4 (taxonomy) and §4a +(details sanitization). +""" +from __future__ import annotations + +from typing import Any + +# --- Reserved stable subcodes (proposal §4) --------------------------------- +# Pass as ``code=`` when raising; daemon retry policy keys off these strings. +TX_ESTIMATION_FAILED = "tx_estimation_failed" +TX_REVERTED = "tx_reverted" +TX_TIMEOUT = "tx_timeout" +TX_NONCE_CONFLICT = "tx_nonce_conflict" +TX_REPLACEMENT_UNDERPRICED = "tx_replacement_underpriced" +RECEIPT_MISSING = "receipt_missing" +RPC_UNREACHABLE = "rpc_unreachable" + + +class DinError(Exception): + """Base class for all SDK domain errors. + + ``code`` is a stable, machine-readable string that flows into the JSON + envelope's ``error.code`` (proposal §3). ``details`` is structured context + that is *sanitized on construction* (proposal §4a) — it never carries + secrets and its shape is bounded per ``code``. + """ + + code: str = "din_error" + + def __init__(self, message: str, *, code: str | None = None, + details: dict[str, Any] | None = None): + super().__init__(message) + self.message = message + if code is not None: + self.code = code + self.details = sanitize_details(self.code, details or {}) + + def to_error(self) -> dict[str, Any]: + """Serialize to the envelope's ``error`` object (proposal §3).""" + return {"code": self.code, "message": self.message, "details": self.details} + + +class ConfigError(DinError): + code = "config_error" + + +class NetworkError(DinError): + code = "network_unreachable" + + +class WalletError(DinError): + code = "wallet_error" + + +class SignerUnavailable(DinError): + code = "signer_unavailable" + + +class ManifestError(DinError): + code = "manifest_error" + + +class IpfsError(DinError): + code = "ipfs_error" + + +class ContractError(DinError): + code = "contract_error" + + +class NotFoundError(DinError): + code = "not_found" + + +class ValidationError(DinError): + code = "validation_failed" + + +class TransactionError(DinError): + code = "tx_failed" + + +class ConfirmationRequired(DinError): + code = "confirmation_required" + + +# --- Detail sanitization (proposal §4a) ------------------------------------- +# Allowlist-oriented: each error code declares the detail keys it may carry and +# their type. Anything not listed for a code is DROPPED — free-form details are +# unsafe by default. This is what keeps secrets out of long-lived daemon JSON +# logs; a generic scrubber would miss secrets nested in arbitrary values. + +_MAX_STR = 256 # cap on ordinary string values +_MAX_TAIL = 2000 # cap on stdout/stderr-style tails + +# Keys that must NEVER survive, regardless of error code. +_NEVER = frozenset({ + "private_key", "privatekey", "password", "passwd", "secret", + "mnemonic", "seed", "api_key", "api_secret", "raw_signed_tx", "signed_tx", +}) + +# Per-code allowed keys -> normalizer kind. +# Kinds: "hex" (0x string), "int", "bool", "str" (bounded), "tail" (bounded tail), "host". +_ALLOWLIST: dict[str, dict[str, str]] = { + "tx_failed": {"tx_hash": "hex", "nonce": "int", "broadcast": "bool", "reason": "str"}, + TX_ESTIMATION_FAILED: {"reason": "str", "gi": "int", "broadcast": "bool"}, + TX_REVERTED: {"tx_hash": "hex", "nonce": "int", "gi": "int", "block_number": "int", + "broadcast": "bool", "reason": "str"}, + TX_TIMEOUT: {"tx_hash": "hex", "nonce": "int", "broadcast": "bool"}, + TX_NONCE_CONFLICT: {"tx_hash": "hex", "nonce": "int", "broadcast": "bool"}, + TX_REPLACEMENT_UNDERPRICED: {"nonce": "int", "broadcast": "bool"}, + RECEIPT_MISSING: {"tx_hash": "hex", "nonce": "int", "broadcast": "bool"}, + "network_unreachable": {"endpoint_host": "host"}, + RPC_UNREACHABLE: {"endpoint_host": "host"}, + "ipfs_error": {"provider": "str", "status_code": "int", "path": "str", "stderr": "tail"}, + "manifest_error": {"path": "str", "key": "str"}, + "contract_error": {"contract": "str", "function": "str"}, + "not_found": {"kind": "str", "id": "int"}, + "validation_failed": {"field": "str", "expected": "str", "actual": "str"}, + "wallet_error": {"account": "str"}, + "signer_unavailable": {"account": "str"}, + "config_error": {"key": "str"}, + "confirmation_required": {"operation": "str"}, +} + + +def _bounded_str(value: Any, limit: int = _MAX_STR) -> str: + text = str(value) + return text if len(text) <= limit else text[:limit] + "…" + + +def _sanitize_host(value: Any) -> str: + """Strip credentials/query from a URL, keeping scheme://host[:port]/path.""" + text = str(value) + try: + from urllib.parse import urlsplit + parts = urlsplit(text) + if parts.scheme and parts.netloc: + host = parts.hostname or "" + if parts.port: + host = f"{host}:{parts.port}" + return _bounded_str(f"{parts.scheme}://{host}{parts.path}") + except Exception: + pass + return _bounded_str(text) + + +def _coerce(kind: str, value: Any) -> Any: + if kind == "int": + try: + return int(value) + except (TypeError, ValueError): + return None + if kind == "bool": + return bool(value) + if kind == "hex": + text = str(value) + return _bounded_str(text if text.startswith("0x") else "0x" + text) + if kind == "host": + return _sanitize_host(value) + if kind == "tail": + text = str(value) + return text[-_MAX_TAIL:] if len(text) > _MAX_TAIL else text + return _bounded_str(value) + + +def sanitize_details(code: str, raw: dict[str, Any]) -> dict[str, Any]: + """Return a sanitized copy of ``raw`` per the allowlist for ``code``. + + Keys not allowlisted for the code are dropped; never-allowed secret keys are + always dropped; values are coerced/bounded by their declared kind. Codes + with no schema yield an empty dict (free-form details are unsafe). + """ + allowed = _ALLOWLIST.get(code, {}) + out: dict[str, Any] = {} + for key, value in raw.items(): + if key in _NEVER or key not in allowed: + continue + coerced = _coerce(allowed[key], value) + if coerced is not None: + out[key] = coerced + return out diff --git a/dincli/sdk/ipfs.py b/dincli/sdk/ipfs.py new file mode 100644 index 0000000..b60b8f2 --- /dev/null +++ b/dincli/sdk/ipfs.py @@ -0,0 +1,254 @@ +import importlib.util +from pathlib import Path +from urllib.parse import quote + +import requests +from dincli.sdk.log import logger +from dincli.sdk.config import ( + FILEBASE_IPFS_ADD_URL, + FILEBASE_IPFS_CAT_URL, + FILEBASE_IPFS_PIN_URL, + resolve_ipfs_config, +) +from dincli.sdk.cid import get_cidv1base32_from_cid + + +def _ensure_file_exists(file_path: Path): + if not file_path.exists(): + raise FileNotFoundError(f"File not found: {file_path}") + + +def _load_custom_fn(module_path: Path, fn_name: str): + _ensure_file_exists(module_path) + + spec = importlib.util.spec_from_file_location(module_path.stem, module_path) + if spec is None or spec.loader is None: + raise ImportError(f"Unable to load module from {module_path}") + + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + if not hasattr(module, fn_name): + raise AttributeError(f"{fn_name} not found in custom service {module_path}") + + fn = getattr(module, fn_name) + if not callable(fn): + raise TypeError(f"{fn_name} in {module_path} is not callable") + + return fn + + +def _normalize_path(path: str | Path) -> Path: + safe_path = Path(path).expanduser().resolve() + dangerous_roots = (Path("/etc"), Path("/boot"), Path("/dev"), Path("/proc")) + + if any(str(safe_path).startswith(str(root)) for root in dangerous_roots): + logger.warning(f"Reading or writing through a sensitive system path: {safe_path}") + + return safe_path + + +def _provider_label(provider: str) -> str: + return { + "env": "environment-backed IPFS", + "filebase": "Filebase", + "custom": "custom IPFS service", + }.get(provider, provider) + + +def _require_custom_service_path(config): + if config.service_path is None: + raise ValueError( + "Custom IPFS provider requires 'ipfs_service_path' in dincli config. " + "Set it with `dincli system configure-ipfs --provider custom --service-path `." + ) + + return config.service_path + + +def _build_add_url(raw_url: str) -> str: + url = raw_url.rstrip("/") + return url if url.endswith("/add") else f"{url}/add" + + +def _build_retrieve_url(raw_url: str, cid: str) -> str: + url = raw_url.rstrip("/") + encoded_cid = quote(cid) + + if "{cid}" in url: + return url.format(cid=encoded_cid) + if "arg=" in url: + separator = "" if url.endswith(("=", "&", "?")) else "&" + return f"{url}{separator}arg={encoded_cid}" + if url.endswith("/cat"): + return f"{url}?arg={encoded_cid}" + + return f"{url}/cat?arg={encoded_cid}" + + +def _raise_for_http_error(response: requests.Response, action: str, provider: str): + try: + response.raise_for_status() + except requests.HTTPError as exc: + details = (response.text or "").strip() + details = details[:300] if details else "No error details returned." + # TODO(sdk-keystones): IpfsError + raise RuntimeError( + f"{provider} {action} failed [{response.status_code}]: {details}" + ) from exc + + +def _upload_via_env(config, file_path: Path) -> str: + if not config.api_url_add: + raise ValueError( + "IPFS provider 'env' requires IPFS_API_URL_ADD in the current .env or environment." + ) + + with file_path.open("rb") as handle: + response = requests.post( + _build_add_url(config.api_url_add), + files={"file": (file_path.name, handle, "application/octet-stream")}, + timeout=30, + ) + + _raise_for_http_error(response, "upload", "Environment-backed IPFS") + return response.json()["Hash"] + + +def _upload_via_filebase(config, file_path: Path) -> str: + if not config.api_key: + raise ValueError( + "Filebase IPFS provider requires 'ipfs_api_key' in dincli config." + ) + + headers = {"Authorization": f"Bearer {config.api_key}"} + + with file_path.open("rb") as handle: + response = requests.post( + FILEBASE_IPFS_ADD_URL, + files={"file": (file_path.name, handle, "application/octet-stream")}, + headers=headers, + timeout=120, + ) + + _raise_for_http_error(response, "upload", "Filebase") + cid = response.json()["Hash"] + + pin_response = requests.post( + f"{FILEBASE_IPFS_PIN_URL}?arg={quote(cid)}", + headers=headers, + timeout=10, + ) + if pin_response.status_code != 200: + logger.warning(f"Filebase pin request failed for CID {cid}: {pin_response.status_code}") + + return cid + + +def _upload_via_custom(config, file_path: Path, msg=None) -> str: + fn = _load_custom_fn(_require_custom_service_path(config), "upload_to_ipfs") + cid = fn(file_path, msg) + + if not isinstance(cid, str) or not cid.strip(): + raise TypeError("Custom upload_to_ipfs must return a non-empty CID string.") + + return cid.strip() + + +def upload_to_ipfs(file_path, msg=None): + normalized_path = _normalize_path(file_path) + _ensure_file_exists(normalized_path) + + config = resolve_ipfs_config() + provider = config.provider + + try: + if provider == "env": + logger.info("Uploading via Environment-backed IPFS...") + cid = _upload_via_env(config, normalized_path) + elif provider == "filebase": + logger.info("Uploading via Filebase...") + cid = _upload_via_filebase(config, normalized_path) + elif provider == "custom": + logger.info("Uploading via Custom IPFS provider...") + cid = _upload_via_custom(config, normalized_path, msg) + else: + raise NotImplementedError(f"Unsupported IPFS provider: {provider}") + + normalized_cid = get_cidv1base32_from_cid(cid) + if msg: + logger.info(f"{msg} with path: {normalized_path} uploaded to IPFS with CID: {normalized_cid}") + return normalized_cid + + except requests.exceptions.RequestException as exc: + raise RuntimeError(f"{_provider_label(provider)} upload failed: {exc.__class__.__name__}") from exc + + +def _retrieve_via_env(config, cid: str) -> requests.Response: + if not config.api_url_retrieve: + raise ValueError( + "IPFS provider 'env' requires IPFS_API_URL_RETRIEVE in the current .env or environment." + ) + + response = requests.post( + _build_retrieve_url(config.api_url_retrieve, cid), + stream=True, + timeout=30, + ) + _raise_for_http_error(response, "download", "Environment-backed IPFS") + return response + + +def _retrieve_via_filebase(config, cid: str) -> requests.Response: + if not config.api_key: + raise ValueError( + "Filebase IPFS provider requires 'ipfs_api_key' in dincli config." + ) + + response = requests.post( + f"{FILEBASE_IPFS_CAT_URL}?arg={quote(cid)}", + headers={"Authorization": f"Bearer {config.api_key}"}, + stream=True, + timeout=30, + ) + _raise_for_http_error(response, "download", "Filebase") + return response + + +def _write_response_to_file(response: requests.Response, destination: Path): + with destination.open("wb") as handle: + for chunk in response.iter_content(chunk_size=8192): + if chunk: + handle.write(chunk) + + +def retrieve_from_ipfs(hash_value, retrieved_file_path): + safe_path = _normalize_path(retrieved_file_path) + safe_path.parent.mkdir(parents=True, exist_ok=True) + + config = resolve_ipfs_config() + provider = config.provider + logger.info(f"Retrieving CID: {hash_value} from {_provider_label(provider)}") + + try: + if provider == "env": + response = _retrieve_via_env(config, hash_value) + _write_response_to_file(response, safe_path) + status_code = response.status_code + elif provider == "filebase": + response = _retrieve_via_filebase(config, hash_value) + _write_response_to_file(response, safe_path) + status_code = response.status_code + elif provider == "custom": + fn = _load_custom_fn(_require_custom_service_path(config), "retrieve_from_ipfs") + result = fn(hash_value, safe_path) + status_code = result if result is not None else 200 + else: + raise NotImplementedError(f"Unsupported IPFS provider: {provider}") + + logger.info(f"Retrieved to: {safe_path}") + return status_code + + except requests.exceptions.RequestException as exc: + logger.error(f"IPFS retrieval failed for {hash_value[:12]}: {exc}") + raise RuntimeError(f"Failed to retrieve CID {hash_value[:12]}") from exc diff --git a/dincli/sdk/log.py b/dincli/sdk/log.py new file mode 100644 index 0000000..e475b24 --- /dev/null +++ b/dincli/sdk/log.py @@ -0,0 +1,31 @@ +import json +import logging +from pathlib import Path + +from dincli.sdk.config import CONFIG_DIR, CONFIG_FILE + +_LOG_FORMAT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + + +def _read_log_level() -> int: + if CONFIG_FILE.exists(): + try: + with open(CONFIG_FILE, "r") as f: + config = json.load(f) + level_str = config.get("log_level", "INFO") + return getattr(logging, level_str.upper(), logging.INFO) + except (json.JSONDecodeError, OSError): + return logging.INFO + return logging.INFO + + +handler = logging.StreamHandler() +handler.setFormatter(logging.Formatter(_LOG_FORMAT)) + +logging.basicConfig( + level=_read_log_level(), + format=_LOG_FORMAT, + handlers=[handler], +) + +logger = logging.getLogger("dincli") diff --git a/dincli/sdk/web3.py b/dincli/sdk/web3.py new file mode 100644 index 0000000..0a43832 --- /dev/null +++ b/dincli/sdk/web3.py @@ -0,0 +1,15 @@ +from web3 import Web3 + +from dincli.sdk.config import resolve_network_value + + +def get_w3(effective_network): + rpc_url = resolve_network_value(effective_network, "rpc_url") + try: + w3 = Web3(Web3.HTTPProvider(rpc_url)) + if not w3.is_connected(): + raise ConnectionError(f"Could not connect to Ethereum node at {rpc_url}") + return w3 + except Exception as e: + # TODO(sdk-keystones): NetworkError + raise ConnectionError(f"Could not connect to Ethereum node for network '{effective_network}': {e}") from e diff --git a/dincli/services/cid_utils.py b/dincli/services/cid_utils.py index a6c155e..4ac3be4 100644 --- a/dincli/services/cid_utils.py +++ b/dincli/services/cid_utils.py @@ -1,70 +1,19 @@ """ -CID ↔ bytes32 conversion utilities. +Backward-compatibility shim. -Kept in a separate module (no imports from dincli.cli) so that both -dincli.cli.utils and dincli.services.ipfs can import from here without -creating a circular dependency. +The implementation moved to ``dincli.sdk.cid`` as part of the SDK extraction +(issue #20). This module re-exports it so existing +``from dincli.services.cid_utils import ...`` call sites keep working. +New code should import from ``dincli.sdk.cid``. """ -import cid as _cid - - -def get_bytes32_from_cid(cid_str: str) -> str: - """ - Extracts the raw 32-byte SHA2-256 digest (hex) from a CID string. - Returns 64 hex characters (without the '1220' multihash prefix). - """ - cid_obj = _cid.make_cid(cid_str) - - # Full multihash bytes include '1220' prefix (12 = sha2-256, 20 = 32-byte length) - full_multihash_hex = cid_obj.multihash.hex() - - # Strip the first 4 hex characters ('1220') to get only the 32-byte digest - return full_multihash_hex[4:] - - -def get_cid_from_bytes32(byte32_hex: str, version: int = 1, encoding: str = "base32") -> str: - """ - Converts a raw 32-byte SHA2-256 digest (hex) back into a CID. - - Args: - byte32_hex: 64-character hex string (raw digest, NO '1220' prefix). - version: 0 or 1. - encoding: 'base32' (recommended for v1) or 'base58btc'. - - Returns: - The CID string. - """ - from multibase import encode # local import keeps top-level deps minimal - - # Reconstruct the full multihash: 12 = sha2-256, 20 = 32-byte length - multihash_hex = "1220" + str(byte32_hex) - multihash_bytes = bytes.fromhex(multihash_hex) - - # Build CIDv1 (dag-pb is standard for IPFS file data) - cid_v1 = _cid.CIDv1(codec="dag-pb", multihash=multihash_bytes) - - if version == 0: - return str(cid_v1.to_v0()) - - if version == 1: - if encoding == "base32": - # CIDv1 raw bytes: - raw_cid_bytes = b"\x01" + b"\x70" + multihash_bytes - return encode("base32", raw_cid_bytes).decode("ascii") - if encoding == "base58btc": - return str(cid_v1) - raise ValueError("encoding must be 'base32' or 'base58btc'") - - raise ValueError("version must be 0 or 1") - -def get_cidv1base32_from_cid(cid_str: str) -> str: - """ - Extracts the CIDv1 from a CID string. - """ - - # Base32 CIDv1 starts with 'bafy' and uses [a-z2-7] - if cid_str.startswith('bafy') and all(c in 'abcdefghijklmnopqrstuvwxyz234567' for c in cid_str[4:]): - return cid_str - - bytes32_hex = get_bytes32_from_cid(cid_str) - return get_cid_from_bytes32(bytes32_hex) \ No newline at end of file +from dincli.sdk.cid import ( # noqa: F401 + get_bytes32_from_cid, + get_cid_from_bytes32, + get_cidv1base32_from_cid, +) + +__all__ = [ + "get_bytes32_from_cid", + "get_cid_from_bytes32", + "get_cidv1base32_from_cid", +] diff --git a/dincli/services/ipfs.py b/dincli/services/ipfs.py index 3e27346..d496859 100644 --- a/dincli/services/ipfs.py +++ b/dincli/services/ipfs.py @@ -1,255 +1,22 @@ -import importlib.util -from pathlib import Path - -import requests -from urllib.parse import quote -from rich.console import Console -from dincli.cli.log import logger -from dincli.cli.utils import ( - FILEBASE_IPFS_ADD_URL, - FILEBASE_IPFS_CAT_URL, - FILEBASE_IPFS_PIN_URL, - resolve_ipfs_config, +""" +Backward-compatibility shim. + +The implementation moved to ``dincli.sdk.ipfs`` as part of the SDK extraction +(issue #20). This module re-exports it so existing +``from dincli.services.ipfs import ...`` call sites keep working. +New code should import from ``dincli.sdk.ipfs``. +""" +from dincli.sdk.ipfs import ( # noqa: F401 + upload_to_ipfs, + retrieve_from_ipfs, + _ensure_file_exists, + _load_custom_fn, + _normalize_path, + _provider_label, + _require_custom_service_path, ) -from dincli.services.cid_utils import get_cidv1base32_from_cid - -console = Console() - -def _ensure_file_exists(file_path: Path): - if not file_path.exists(): - raise FileNotFoundError(f"File not found: {file_path}") - - -def _load_custom_fn(module_path: Path, fn_name: str): - _ensure_file_exists(module_path) - - spec = importlib.util.spec_from_file_location(module_path.stem, module_path) - if spec is None or spec.loader is None: - raise ImportError(f"Unable to load module from {module_path}") - - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - - if not hasattr(module, fn_name): - raise AttributeError(f"{fn_name} not found in custom service {module_path}") - - fn = getattr(module, fn_name) - if not callable(fn): - raise TypeError(f"{fn_name} in {module_path} is not callable") - - return fn - - -def _normalize_path(path: str | Path) -> Path: - safe_path = Path(path).expanduser().resolve() - dangerous_roots = (Path("/etc"), Path("/boot"), Path("/dev"), Path("/proc")) - - if any(str(safe_path).startswith(str(root)) for root in dangerous_roots): - logger.warning(f"Reading or writing through a sensitive system path: {safe_path}") - - return safe_path - - -def _provider_label(provider: str) -> str: - return { - "env": "environment-backed IPFS", - "filebase": "Filebase", - "custom": "custom IPFS service", - }.get(provider, provider) - - -def _require_custom_service_path(config): - if config.service_path is None: - raise ValueError( - "Custom IPFS provider requires 'ipfs_service_path' in dincli config. " - "Set it with `dincli system configure-ipfs --provider custom --service-path `." - ) - - return config.service_path - - -def _build_add_url(raw_url: str) -> str: - url = raw_url.rstrip("/") - return url if url.endswith("/add") else f"{url}/add" - - -def _build_retrieve_url(raw_url: str, cid: str) -> str: - url = raw_url.rstrip("/") - encoded_cid = quote(cid) - - if "{cid}" in url: - return url.format(cid=encoded_cid) - if "arg=" in url: - separator = "" if url.endswith(("=", "&", "?")) else "&" - return f"{url}{separator}arg={encoded_cid}" - if url.endswith("/cat"): - return f"{url}?arg={encoded_cid}" - - return f"{url}/cat?arg={encoded_cid}" - - -def _raise_for_http_error(response: requests.Response, action: str, provider: str): - try: - response.raise_for_status() - except requests.HTTPError as exc: - details = (response.text or "").strip() - details = details[:300] if details else "No error details returned." - raise RuntimeError( - f"{provider} {action} failed [{response.status_code}]: {details}" - ) from exc - - -def _upload_via_env(config, file_path: Path) -> str: - if not config.api_url_add: - raise ValueError( - "IPFS provider 'env' requires IPFS_API_URL_ADD in the current .env or environment." - ) - - with file_path.open("rb") as handle: - response = requests.post( - _build_add_url(config.api_url_add), - files={"file": (file_path.name, handle, "application/octet-stream")}, - timeout=30, - ) - - _raise_for_http_error(response, "upload", "Environment-backed IPFS") - return response.json()["Hash"] - - -def _upload_via_filebase(config, file_path: Path) -> str: - if not config.api_key: - raise ValueError( - "Filebase IPFS provider requires 'ipfs_api_key' in dincli config." - ) - - headers = {"Authorization": f"Bearer {config.api_key}"} - - with file_path.open("rb") as handle: - response = requests.post( - FILEBASE_IPFS_ADD_URL, - files={"file": (file_path.name, handle, "application/octet-stream")}, - headers=headers, - timeout=120, - ) - - _raise_for_http_error(response, "upload", "Filebase") - cid = response.json()["Hash"] - - pin_response = requests.post( - f"{FILEBASE_IPFS_PIN_URL}?arg={quote(cid)}", - headers=headers, - timeout=10, - ) - if pin_response.status_code != 200: - logger.warning(f"Filebase pin request failed for CID {cid}: {pin_response.status_code}") - - return cid - - -def _upload_via_custom(config, file_path: Path, msg=None) -> str: - fn = _load_custom_fn(_require_custom_service_path(config), "upload_to_ipfs") - cid = fn(file_path, msg) - - if not isinstance(cid, str) or not cid.strip(): - raise TypeError("Custom upload_to_ipfs must return a non-empty CID string.") - - return cid.strip() - - -def upload_to_ipfs(file_path, msg=None): - normalized_path = _normalize_path(file_path) - _ensure_file_exists(normalized_path) - - config = resolve_ipfs_config() - provider = config.provider - - try: - if provider == "env": - console.print("[bold green]Uploading via Environment-backed IPFS...[/bold green]") - cid = _upload_via_env(config, normalized_path) - elif provider == "filebase": - console.print("[bold green]Uploading via Filebase...[/bold green]") - cid = _upload_via_filebase(config, normalized_path) - elif provider == "custom": - console.print("[bold green]Uploading via Custom IPFS provider...[/bold green]") - cid = _upload_via_custom(config, normalized_path, msg) - else: - raise NotImplementedError(f"Unsupported IPFS provider: {provider}") - - normalized_cid = get_cidv1base32_from_cid(cid) - if msg: - logger.info(f"{msg} with path: {normalized_path} uploaded to IPFS with CID: {normalized_cid}") - return normalized_cid - - except requests.exceptions.RequestException as exc: - raise RuntimeError(f"{_provider_label(provider)} upload failed: {exc.__class__.__name__}") from exc - - -def _retrieve_via_env(config, cid: str) -> requests.Response: - if not config.api_url_retrieve: - raise ValueError( - "IPFS provider 'env' requires IPFS_API_URL_RETRIEVE in the current .env or environment." - ) - - response = requests.post( - _build_retrieve_url(config.api_url_retrieve, cid), - stream=True, - timeout=30, - ) - _raise_for_http_error(response, "download", "Environment-backed IPFS") - return response - - -def _retrieve_via_filebase(config, cid: str) -> requests.Response: - if not config.api_key: - raise ValueError( - "Filebase IPFS provider requires 'ipfs_api_key' in dincli config." - ) - - response = requests.post( - f"{FILEBASE_IPFS_CAT_URL}?arg={quote(cid)}", - headers={"Authorization": f"Bearer {config.api_key}"}, - stream=True, - timeout=30, - ) - _raise_for_http_error(response, "download", "Filebase") - return response - - -def _write_response_to_file(response: requests.Response, destination: Path): - with destination.open("wb") as handle: - for chunk in response.iter_content(chunk_size=8192): - if chunk: - handle.write(chunk) - - -def retrieve_from_ipfs(hash_value, retrieved_file_path): - safe_path = _normalize_path(retrieved_file_path) - safe_path.parent.mkdir(parents=True, exist_ok=True) - - config = resolve_ipfs_config() - provider = config.provider - logger.info(f"Retrieving CID: {hash_value} from {_provider_label(provider)}") - - try: - if provider == "env": - response = _retrieve_via_env(config, hash_value) - _write_response_to_file(response, safe_path) - status_code = response.status_code - elif provider == "filebase": - response = _retrieve_via_filebase(config, hash_value) - _write_response_to_file(response, safe_path) - status_code = response.status_code - elif provider == "custom": - fn = _load_custom_fn(_require_custom_service_path(config), "retrieve_from_ipfs") - result = fn(hash_value, safe_path) - status_code = result if result is not None else 200 - else: - raise NotImplementedError(f"Unsupported IPFS provider: {provider}") - - logger.info(f"Retrieved to: {safe_path}") - return status_code - except requests.exceptions.RequestException as exc: - logger.error(f"IPFS retrieval failed for {hash_value[:12]}: {exc}") - raise RuntimeError(f"Failed to retrieve CID {hash_value[:12]}") from exc +__all__ = [ + "upload_to_ipfs", + "retrieve_from_ipfs", +] diff --git a/foundry/test/SecurityFindings.t.sol b/foundry/test/SecurityFindings.t.sol new file mode 100644 index 0000000..d5f1e55 --- /dev/null +++ b/foundry/test/SecurityFindings.t.sol @@ -0,0 +1,307 @@ +// SPDX-License-Identifier: UNLICENSED +pragma solidity ^0.8.28; + +// ───────────────────────────────────────────────────────────────────────────── +// PoC tests for the 2026-07 security review (Developer/audits/2026-07_foundry-src-security-review.md). +// Pinned to commit d136ff3. These are additive, read/observe only — no +// contract source under test is modified. +// Run: forge test --match-contract SecurityFindingsTest -vv +// ───────────────────────────────────────────────────────────────────────────── + +import {Test} from "forge-std/Test.sol"; +import {TransparentUpgradeableProxy} from "@openzeppelin/contracts/proxy/transparent/TransparentUpgradeableProxy.sol"; + +import {DinToken} from "../src/DinToken.sol"; +import {DinCoordinator} from "../src/DinCoordinator.sol"; +import {DinValidatorStake} from "../src/DinValidatorStake.sol"; +import {DINModelRegistry} from "../src/DINModelRegistry.sol"; +import {DINTaskCoordinator} from "../src/DINTaskCoordinator.sol"; +import {DINTaskAuditor} from "../src/DINTaskAuditor.sol"; +import {GIstates} from "../src/DINShared.sol"; + +contract SecurityFindingsTest is Test { + // ───────────────────────────────────────────────────────────────────── + // Shared platform fixture (6-step deployment order from + // Documentation/technical/upgradable-contracts/hardhat/README.md §5) + // ───────────────────────────────────────────────────────────────────── + + DinToken tokenImpl; + DinCoordinator coordinatorImpl; + DinValidatorStake stakeImpl; + DINModelRegistry registryImpl; + + DinToken token; + DinCoordinator coordinator; + DinValidatorStake stake; + DINModelRegistry registry; + + address admin = makeAddr("admin"); + + function _deployPlatform() internal { + vm.startPrank(admin); + + // 1. DinToken proxy + tokenImpl = new DinToken(); + TransparentUpgradeableProxy tokenProxy = new TransparentUpgradeableProxy( + address(tokenImpl), + admin, + abi.encodeCall(DinToken.initialize, ()) + ); + token = DinToken(address(tokenProxy)); + + // 2. DinCoordinator proxy + coordinatorImpl = new DinCoordinator(); + TransparentUpgradeableProxy coordinatorProxy = new TransparentUpgradeableProxy( + address(coordinatorImpl), + admin, + abi.encodeCall(DinCoordinator.initialize, (address(token))) + ); + coordinator = DinCoordinator(address(coordinatorProxy)); + + // 3. one-shot wiring + token.setCoordinator(address(coordinator)); + + // 4. DinValidatorStake proxy + stakeImpl = new DinValidatorStake(); + TransparentUpgradeableProxy stakeProxy = new TransparentUpgradeableProxy( + address(stakeImpl), + admin, + abi.encodeCall( + DinValidatorStake.initialize, + (address(token), address(coordinator)) + ) + ); + stake = DinValidatorStake(address(stakeProxy)); + + // 5. wire stake into coordinator + coordinator.updateValidatorStakeContract(address(stake)); + + // 6. DINModelRegistry proxy + registryImpl = new DINModelRegistry(); + TransparentUpgradeableProxy registryProxy = new TransparentUpgradeableProxy( + address(registryImpl), + admin, + abi.encodeCall(DINModelRegistry.initialize, (address(stake))) + ); + registry = DINModelRegistry(address(registryProxy)); + + vm.stopPrank(); + } + + // ───────────────────────────────────────────────────────────────────── + // Proxy-specific: initializer protection + // + // Confirms _disableInitializers() is effective at RUNTIME, not just + // present in source — i.e. an attacker cannot call initialize() directly + // on the deployed implementation address and claim ownership of it. + // (Not exploitable against the real protocol today, since the + // implementation holds no funds/routing of its own — but see the + // finding writeup for why this still matters operationally.) + // ───────────────────────────────────────────────────────────────────── + + function test_implementation_rejectsDirectInitialize_DinToken() public { + DinToken impl = new DinToken(); + vm.expectRevert(); + impl.initialize(); + } + + function test_implementation_rejectsDirectInitialize_DinCoordinator() public { + DinCoordinator impl = new DinCoordinator(); + vm.expectRevert(); + impl.initialize(makeAddr("fakeToken")); + } + + function test_implementation_rejectsDirectInitialize_DinValidatorStake() public { + DinValidatorStake impl = new DinValidatorStake(); + vm.expectRevert(); + impl.initialize(makeAddr("fakeToken"), makeAddr("fakeCoordinator")); + } + + function test_implementation_rejectsDirectInitialize_DINModelRegistry() public { + DINModelRegistry impl = new DINModelRegistry(); + vm.expectRevert(); + impl.initialize(makeAddr("fakeStake")); + } + + function test_proxy_rejectsDoubleInitialize_DinToken() public { + _deployPlatform(); + vm.expectRevert(); + token.initialize(); + } + + function test_proxy_rejectsDoubleInitialize_DinCoordinator() public { + _deployPlatform(); + vm.expectRevert(); + coordinator.initialize(makeAddr("someToken")); + } + + function test_proxy_rejectsDoubleInitialize_DinValidatorStake() public { + _deployPlatform(); + vm.expectRevert(); + stake.initialize(makeAddr("t"), makeAddr("c")); + } + + function test_proxy_rejectsDoubleInitialize_DINModelRegistry() public { + _deployPlatform(); + vm.expectRevert(); + registry.initialize(makeAddr("s")); + } + + // ───────────────────────────────────────────────────────────────────── + // Finding: zero-CID sentinel collision permanently DoSes + // finalizeT1Aggregation / finalizeT2Aggregation for the whole GI. + // + // T1/T2 finalization uses `winningCID == bytes32(0)` to detect "no + // submissions for this batch" and revert. Any assigned aggregator can + // submit `bytes32(0)` as their own aggregation CID; if that CID ends up + // as the batch's plurality vote (e.g. the only submission in that + // batch), finalization reverts for the ENTIRE GI, indistinguishable + // from "nobody submitted" — even though a submission genuinely exists. + // Task contracts are not upgradeable, so this has no on-chain recovery + // path: the GI is bricked and the model owner must abandon the round. + // ───────────────────────────────────────────────────────────────────── + + DINTaskCoordinator tc; + DINTaskAuditor ta; + + address modelOwner = makeAddr("modelOwner"); + address auditor1 = makeAddr("auditor1"); + address auditor2 = makeAddr("auditor2"); + address client1 = makeAddr("client1"); + address client2 = makeAddr("client2"); + address client3 = makeAddr("client3"); + address agg1 = makeAddr("agg1"); + address agg2 = makeAddr("agg2"); + address agg3 = makeAddr("agg3"); + + function _fundAndStake(address who) internal { + vm.deal(who, 1 ether); + vm.prank(who); + coordinator.depositAndMint{value: 0.001 ether}(); + vm.startPrank(who); + token.approve(address(stake), type(uint256).max); + stake.stake(10 ether); // MIN_STAKE + vm.stopPrank(); + } + + function _deployTaskPair() internal { + vm.startPrank(modelOwner); + tc = new DINTaskCoordinator(address(stake)); + ta = new DINTaskAuditor(address(stake), address(tc)); + tc.setDINTaskAuditorContract(address(ta)); + vm.stopPrank(); + + // DIN-Representative authorises both as slashers + vm.startPrank(admin); + coordinator.addSlasherContract(address(tc)); + coordinator.addSlasherContract(address(ta)); + vm.stopPrank(); + + vm.startPrank(modelOwner); + tc.setDINTaskCoordinatorAsSlasher(); + tc.setDINTaskAuditorAsSlasher(); + tc.setGenesisModelIpfsHash(bytes32(uint256(1))); + tc.startGI(1); + vm.stopPrank(); + } + + function _runToT1AggregationStarted() internal { + _deployPlatform(); + _deployTaskPair(); + + _fundAndStake(auditor1); + _fundAndStake(auditor2); + _fundAndStake(agg1); + _fundAndStake(agg2); + _fundAndStake(agg3); + + vm.startPrank(modelOwner); + tc.startDINaggregatorsRegistration(1); + vm.stopPrank(); + + vm.prank(agg1); + tc.registerDINaggregator(1); + vm.prank(agg2); + tc.registerDINaggregator(1); + vm.prank(agg3); + tc.registerDINaggregator(1); + + vm.startPrank(modelOwner); + tc.closeDINaggregatorsRegistration(1); + tc.startDINauditorsRegistration(1); + vm.stopPrank(); + + // auditorsPerBatch defaults to 3 (demo Params in the DINTaskAuditor + // constructor) so all 3 auditors must register before the window closes. + address auditor3 = makeAddr("auditor3"); + _fundAndStake(auditor3); + + vm.prank(auditor1); + ta.registerDINAuditor(1); + vm.prank(auditor2); + ta.registerDINAuditor(1); + vm.prank(auditor3); + ta.registerDINAuditor(1); + + vm.startPrank(modelOwner); + tc.closeDINauditorsRegistration(1); + tc.startLMsubmissions(1); + vm.stopPrank(); + + vm.prank(client1); + ta.submitLocalModel(bytes32(uint256(100)), 1); + vm.prank(client2); + ta.submitLocalModel(bytes32(uint256(200)), 1); + vm.prank(client3); + ta.submitLocalModel(bytes32(uint256(300)), 1); + + vm.startPrank(modelOwner); + tc.closeLMsubmissions(1); + tc.createAuditorsBatches(1); + tc.setTestDataAssignedFlag(1, true); + tc.startLMsubmissionsEvaluation(1); + vm.stopPrank(); + + // Both models pass: all 3 auditors vote eligible + score 100 on both models. + (, address[] memory batchAuditors, uint[] memory modelIdxs,) = ta.getAuditorsBatch(1, 0); + for (uint i = 0; i < batchAuditors.length; i++) { + for (uint m = 0; m < modelIdxs.length; m++) { + vm.prank(batchAuditors[i]); + ta.setAuditScorenEligibility(1, 0, modelIdxs[m], 100, true); + } + } + + vm.startPrank(modelOwner); + tc.closeLMsubmissionsEvaluation(1); + tc.autoCreateTier1AndTier2(1); + tc.startT1Aggregation(1); + vm.stopPrank(); + } + + function test_finalizeT1Aggregation_zeroCID_bricksEntireGI() public { + _runToT1AggregationStarted(); + + (, address[] memory t1aggs,,,) = tc.getTier1Batch(1, 0); + assertEq(t1aggs.length, 3, "sanity: T1 batch should have 3 aggregators"); + + // Only ONE of the three assigned aggregators submits, and submits + // the zero CID. The other two never submit before the model owner + // closes the window (a routine "not everyone responded in time" + // scenario, not an edge case). + vm.prank(t1aggs[0]); + tc.submitT1Aggregation(1, 0, bytes32(0)); + + // This is the ONLY submission in the batch, so it is unambiguously + // the plurality winner (1 vote > 0). Finalization should therefore + // either succeed with finalCID == bytes32(0), or explicitly reject + // zero-CID submissions at submit time. Instead: + vm.expectRevert(); // TC_NoSubmissions — wrongly conflates "no one voted" with "the winning vote was 0x0" + tc.finalizeT1Aggregation(1); + + // The GI is now stuck: GIstate is still T1AggregationStarted, the + // task contracts are not upgradeable, and there is no admin + // function to skip/override a single batch. Every subsequent call + // to finalizeT1Aggregation reverts the same way, forever. + assertEq(uint256(tc.GIstate()), uint256(GIstates.T1AggregationStarted)); + } +} diff --git a/pyproject.toml b/pyproject.toml index c7c8ee9..6895ffe 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -26,6 +26,7 @@ dependencies = [ [project.scripts] dincli = "dincli.main:app" +dind = "dincli.dind.main:app" # ✅ Let setuptools auto-find packages (including 'dincli') [tool.setuptools.packages.find] @@ -33,7 +34,8 @@ where = ["."] # look in project root include = ["dincli*"] # only include dincli and subpackages [tool.setuptools.package-data] -dincli = ["config/*.json", "abis/*", "docker/worker/*", "services/*.py"] +"dincli" = ["config/*.json", "abis/*", "docker/worker/*", "services/*.py"] +"dincli.dind" = ["examples/*"] [tool.pytest.ini_options] markers = [ diff --git a/tests/test_connect_wallet.py b/tests/test_connect_wallet.py index 059c3b4..8712fff 100644 --- a/tests/test_connect_wallet.py +++ b/tests/test_connect_wallet.py @@ -6,6 +6,7 @@ from pathlib import Path from types import SimpleNamespace from unittest import mock +from rich.console import Console import pytest import typer @@ -14,6 +15,7 @@ from dincli.cli import system as system_mod from dincli.cli import utils as utils_mod +from dincli.sdk import config as sdk_config from dincli.main import app as main_app @@ -21,7 +23,6 @@ DUMMY_KEY_1 = "0xfedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210" DUMMY_PW = "test-password" - @pytest.fixture def temp_config(tmp_path): config_dir = tmp_path / "config" @@ -33,11 +34,30 @@ def temp_config(tmp_path): orig_config = utils_mod.CONFIG_DIR orig_cache = utils_mod.CACHE_DIR orig_wallets = utils_mod.WALLETS_DIR + + sys_orig_config = getattr(system_mod, "CONFIG_DIR", None) + sys_orig_cache = getattr(system_mod, "CACHE_DIR", None) + sys_orig_config_file = getattr(system_mod, "CONFIG_FILE", None) + sys_orig_wallet_file = getattr(system_mod, "WALLET_FILE", None) + sys_orig_worker_cache = getattr(system_mod, "WORKER_CACHE_DIR", None) + utils_mod.CONFIG_DIR = config_dir utils_mod.CACHE_DIR = cache_dir utils_mod.WALLETS_DIR = wallets_dir utils_mod.WALLET_FILE = config_dir / "wallet.json" utils_mod.LEGACY_WALLET_FILE = config_dir / "wallet.json" + + if sys_orig_config is not None: + system_mod.CONFIG_DIR = config_dir + if sys_orig_cache is not None: + system_mod.CACHE_DIR = cache_dir + if sys_orig_config_file is not None: + system_mod.CONFIG_FILE = config_dir / "config.json" + if sys_orig_wallet_file is not None: + system_mod.WALLET_FILE = config_dir / "wallet.json" + if sys_orig_worker_cache is not None: + system_mod.WORKER_CACHE_DIR = cache_dir / "worker" + try: yield { "config_dir": config_dir, @@ -51,6 +71,17 @@ def temp_config(tmp_path): utils_mod.WALLET_FILE = orig_config / "wallet.json" utils_mod.LEGACY_WALLET_FILE = orig_config / "wallet.json" + if sys_orig_config is not None: + system_mod.CONFIG_DIR = sys_orig_config + if sys_orig_cache is not None: + system_mod.CACHE_DIR = sys_orig_cache + if sys_orig_config_file is not None: + system_mod.CONFIG_FILE = sys_orig_config_file + if sys_orig_wallet_file is not None: + system_mod.WALLET_FILE = sys_orig_wallet_file + if sys_orig_worker_cache is not None: + system_mod.WORKER_CACHE_DIR = sys_orig_worker_cache + class DummyConsole: def __init__(self): @@ -62,7 +93,7 @@ def print(self, *args, **kwargs): class DummyCtxObj: def __init__(self, wallet_name=None, resolved_wallet_name="default"): - self.console = DummyConsole() + self.console = Console() self.wallet_name = wallet_name self._resolved_wallet_name = resolved_wallet_name @@ -408,7 +439,7 @@ def test_todo_shows_named_wallet(self, temp_config, monkeypatch): (temp_config["config_dir"] / "config.json").write_text('{"network": "local", "log_level": "info", "demo_mode": false}') ctx = SimpleNamespace(obj=DummyCtxObj()) - ctx.obj.console.messages.clear() + #ctx.obj.console.messages.clear() system_mod.todo(ctx) @@ -579,17 +610,18 @@ class TestFixERegression: def test_set_wallet_persists_config(self, monkeypatch, temp_config): config_file = temp_config["config_dir"] / "config.json" config_file.write_text("{}") - orig_config_file = utils_mod.CONFIG_FILE - utils_mod.CONFIG_FILE = config_file + orig_config_file = sdk_config.CONFIG_FILE + sdk_config.CONFIG_FILE = config_file monkeypatch.setattr(system_mod, "resolve_ipfs_config", lambda: SimpleNamespace(provider="env", api_url_add=None, api_url_retrieve=None, api_key=None, api_secret=None, service_path=None)) try: + (temp_config["wallets_dir"] / "wallet_validator.json").write_text('{"version": 1, "address": "0xTest"}') result = CliRunner().invoke(main_app, ["system", "set-wallet", "validator"]) assert result.exit_code == 0 assert "Default wallet set to 'validator'" in result.output - config = utils_mod.load_config() + config = sdk_config.load_config() assert config.get("wallet_name") == "validator" finally: - utils_mod.CONFIG_FILE = orig_config_file + sdk_config.CONFIG_FILE = orig_config_file class TestSkipListRouting: @@ -610,7 +642,163 @@ def test_show_index_no_wallet_reaches_command_body(self): def test_set_wallet_no_wallet_reaches_command_body(self): result = CliRunner().invoke(main_app, ["system", "set-wallet", "test-name"]) - assert result.exit_code == 0 + # Command body is reached (not blocked by the system callback); + # exits 1 because wallet "test-name" does not exist on disk. + assert result.exit_code == 1 + assert "not found" in result.output.lower() + + +def _write_encrypted_wallet(wallets_dir, name, key, pw): + """Write a wrapper-schema encrypted keystore to wallet_.json; return the address.""" + ks = Account.encrypt(key, pw) + acct = Account.from_key(key) + wrapper = {"version": 1, "address": acct.address, "keystore": ks, + "source": "created", "name": name} + (wallets_dir / f"wallet_{name}.json").write_text(json.dumps(wrapper)) + return acct.address + + +class TestConnectWalletOverwriteGuard: + """Review fix #2: confirm before overwriting an existing named keystore.""" + + def _base_monkeypatch(self, monkeypatch): + monkeypatch.setattr(system_mod, "get_config", lambda key, default=None: False) + monkeypatch.setattr(system_mod, "load_config", lambda: {}) + monkeypatch.setattr(system_mod, "getpass", lambda prompt: DUMMY_PW) + # Deterministic: no DIN_WALLET_PASSWORD from the ambient .env. + monkeypatch.setattr(utils_mod, "get_env_key", lambda *a, **k: None) + + def test_omitted_yes_prompts_and_aborts(self, temp_config, monkeypatch): + # `yes` omitted entirely -> Typer passes a truthy OptionInfo; the guard must + # still fire (normalization). Declining leaves the existing wallet untouched. + _write_encrypted_wallet(temp_config["wallets_dir"], "prod", DUMMY_KEY_0, DUMMY_PW) + before = (temp_config["wallets_dir"] / "wallet_prod.json").read_text() + self._base_monkeypatch(monkeypatch) + monkeypatch.setattr(typer, "confirm", lambda *a, **k: False) + + ctx = make_ctx() + with pytest.raises(typer.Exit): + system_mod.connect_wallet( + ctx=ctx, privatekey=DUMMY_KEY_1, key_file=None, + account=None, keystore=None, name="prod", # `yes` intentionally omitted + ) + + after = (temp_config["wallets_dir"] / "wallet_prod.json").read_text() + assert after == before + + def test_explicit_yes_false_confirm_declined_aborts(self, temp_config, monkeypatch): + _write_encrypted_wallet(temp_config["wallets_dir"], "prod", DUMMY_KEY_0, DUMMY_PW) + before = (temp_config["wallets_dir"] / "wallet_prod.json").read_text() + self._base_monkeypatch(monkeypatch) + monkeypatch.setattr(typer, "confirm", lambda *a, **k: False) + + ctx = make_ctx() + with pytest.raises(typer.Exit): + system_mod.connect_wallet( + ctx=ctx, privatekey=DUMMY_KEY_1, key_file=None, + account=None, keystore=None, name="prod", yes=False, + ) + after = (temp_config["wallets_dir"] / "wallet_prod.json").read_text() + assert after == before + + def test_confirm_accepted_replaces(self, temp_config, monkeypatch): + old_addr = _write_encrypted_wallet(temp_config["wallets_dir"], "prod", DUMMY_KEY_0, DUMMY_PW) + self._base_monkeypatch(monkeypatch) + monkeypatch.setattr(typer, "confirm", lambda *a, **k: True) + + ctx = make_ctx() + system_mod.connect_wallet( + ctx=ctx, privatekey=DUMMY_KEY_1, key_file=None, + account=None, keystore=None, name="prod", yes=False, + ) + data = json.loads((temp_config["wallets_dir"] / "wallet_prod.json").read_text()) + assert data["address"] == Account.from_key(DUMMY_KEY_1).address + assert data["address"] != old_addr + + def test_yes_true_skips_confirm(self, temp_config, monkeypatch): + _write_encrypted_wallet(temp_config["wallets_dir"], "prod", DUMMY_KEY_0, DUMMY_PW) + self._base_monkeypatch(monkeypatch) + called = [] + monkeypatch.setattr(typer, "confirm", lambda *a, **k: called.append(True) or True) + + ctx = make_ctx() + system_mod.connect_wallet( + ctx=ctx, privatekey=DUMMY_KEY_1, key_file=None, + account=None, keystore=None, name="prod", yes=True, + ) + assert called == [] # confirm never invoked + data = json.loads((temp_config["wallets_dir"] / "wallet_prod.json").read_text()) + assert data["address"] == Account.from_key(DUMMY_KEY_1).address + + def test_doomed_keystore_input_no_prompt(self, temp_config, monkeypatch): + # A missing keystore must fail fast BEFORE the overwrite prompt (validation + # precedes the guard), so `typer.confirm` is never reached. + _write_encrypted_wallet(temp_config["wallets_dir"], "prod", DUMMY_KEY_0, DUMMY_PW) + self._base_monkeypatch(monkeypatch) + called = [] + monkeypatch.setattr(typer, "confirm", lambda *a, **k: called.append(True) or True) + + ctx = make_ctx() + with pytest.raises(typer.Exit): + system_mod.connect_wallet( + ctx=ctx, privatekey=None, key_file=None, account=None, + keystore=Path("/nonexistent/ks.json"), name="prod", yes=False, + ) + assert called == [] + + +class TestNewWalletPasswordNotCached: + """Review fix #3: creating/overwriting a wallet must not reuse a stale cached password.""" + + def test_create_ignores_cached_password(self, temp_config, monkeypatch): + utils_mod._PASSWORD_CACHE.clear() + # Seed a stale cached password for the same name, as a prior load_account would. + utils_mod._PASSWORD_CACHE["prod"] = ("stale-cached-pw", utils_mod.time.time() + 10_000) + + monkeypatch.setattr(system_mod, "get_config", lambda key, default=None: False) + monkeypatch.setattr(system_mod, "load_config", lambda: {}) + monkeypatch.setattr(utils_mod, "get_env_key", lambda *a, **k: None) + monkeypatch.setattr(system_mod, "getpass", lambda prompt: "fresh-pw") + + ctx = make_ctx() + system_mod.connect_wallet( + ctx=ctx, privatekey=DUMMY_KEY_0, key_file=None, + account=None, keystore=None, name="prod", yes=True, + ) + + wrapper = json.loads((temp_config["wallets_dir"] / "wallet_prod.json").read_text()) + expected = Account.from_key(DUMMY_KEY_0).address + # Encrypted with the freshly-entered password, NOT the stale cached one. + assert Account.from_key(Account.decrypt(wrapper["keystore"], "fresh-pw")).address == expected + with pytest.raises(ValueError): + Account.decrypt(wrapper["keystore"], "stale-cached-pw") + + +class TestSingleEnvParseOnUnlock: + """Review fix #5: one DIN_WALLET_PASSWORD fetch (=> one .env parse) per unlock.""" + + def test_load_account_fetches_env_password_once(self, temp_config, monkeypatch): + utils_mod._PASSWORD_CACHE.clear() + ks = Account.encrypt(DUMMY_KEY_0, DUMMY_PW) + acct = Account.from_key(DUMMY_KEY_0) + wrapper = {"version": 1, "address": acct.address, "keystore": ks, + "source": "created", "name": "prod"} + (temp_config["wallets_dir"] / "wallet_prod.json").write_text(json.dumps(wrapper)) + + calls = [] + + def counting_get_env_key(key, *args, **kwargs): + calls.append(key) + return None + + monkeypatch.setattr(utils_mod, "get_env_key", counting_get_env_key) + monkeypatch.setattr(utils_mod, "getpass", lambda prompt: DUMMY_PW) + monkeypatch.setattr(utils_mod, "_cleanup_stale_session", lambda: None) + + loaded = utils_mod.load_account(name="prod") + assert loaded.address == acct.address + assert calls.count("DIN_WALLET_PASSWORD") == 1 _PASSWORD_TTL_DEFAULT = 900 + diff --git a/tests/test_dind_boundary.py b/tests/test_dind_boundary.py new file mode 100644 index 0000000..e9f4383 --- /dev/null +++ b/tests/test_dind_boundary.py @@ -0,0 +1,31 @@ +"""Import-boundary test: dincli.dind pulls in no dincli.cli.* (P4-1.1). + +Runs in a fresh subprocess so earlier tests that imported UI modules can't +mask a violation. +""" + +import subprocess +import sys +import textwrap + + +def test_dind_imports_no_cli(): + script = textwrap.dedent(""" + import importlib, pkgutil, sys + import dincli.dind + for m in pkgutil.walk_packages(dincli.dind.__path__, dincli.dind.__name__ + "."): + importlib.import_module(m.name) + loaded = set(sys.modules) + bad_cli = sorted( + m for m in loaded + if m == "dincli.cli" or m.startswith("dincli.cli.") + ) + assert not bad_cli, f"dind imported dincli.cli modules: {bad_cli}" + print("ok") + """) + result = subprocess.run( + [sys.executable, "-c", script], capture_output=True, text=True + ) + assert result.returncode == 0, ( + f"stdout={result.stdout!r} stderr={result.stderr!r}" + ) diff --git a/tests/test_dind_config.py b/tests/test_dind_config.py new file mode 100644 index 0000000..51f368d --- /dev/null +++ b/tests/test_dind_config.py @@ -0,0 +1,74 @@ +"""Tests for dind/config.py resolver precedence.""" + +import os +from pathlib import Path + +from dincli.dind import config as dconf + + +def test_state_dir_default(): + result = dconf.resolve_state_dir(None) + assert result == dconf.DEFAULT_STATE_DIR + + +def test_state_dir_flag(tmp_path): + sd = tmp_path / "custom-state" + result = dconf.resolve_state_dir(str(sd)) + assert result == sd.resolve() + + +def test_state_dir_env(monkeypatch, tmp_path): + sd = tmp_path / "env-state" + monkeypatch.setenv("DIN_DIND_STATE_DIR", str(sd)) + result = dconf.resolve_state_dir(None) + assert result == sd.resolve() + + +def test_state_dir_flag_wins_over_env(monkeypatch, tmp_path): + monkeypatch.setenv("DIN_DIND_STATE_DIR", "/env/path") + result = dconf.resolve_state_dir(str(tmp_path / "flag")) + assert result != Path("/env/path") + + +def test_health_host_default(): + assert dconf.resolve_health_host(None) == "127.0.0.1" + + +def test_health_host_flag(): + assert dconf.resolve_health_host("0.0.0.0") == "0.0.0.0" + + +def test_health_host_env(monkeypatch): + monkeypatch.setenv("DIN_DIND_HEALTH_HOST", "10.0.0.1") + assert dconf.resolve_health_host(None) == "10.0.0.1" + + +def test_health_port_default(): + assert dconf.resolve_health_port(None) == 8787 + + +def test_health_port_flag(): + assert dconf.resolve_health_port(9090) == 9090 + + +def test_health_port_env(monkeypatch): + monkeypatch.setenv("DIN_DIND_HEALTH_PORT", "1234") + assert dconf.resolve_health_port(None) == 1234 + + +def test_validate_health_port_ok(): + dconf.validate_health_port(1) + dconf.validate_health_port(65535) + dconf.validate_health_port(8080) + + +def test_validate_health_port_rejects_zero(): + import pytest + with pytest.raises(ValueError, match="1-65535"): + dconf.validate_health_port(0) + + +def test_validate_health_port_rejects_too_high(): + import pytest + with pytest.raises(ValueError, match="1-65535"): + dconf.validate_health_port(99999) diff --git a/tests/test_dind_daemon.py b/tests/test_dind_daemon.py new file mode 100644 index 0000000..8779661 --- /dev/null +++ b/tests/test_dind_daemon.py @@ -0,0 +1,67 @@ +"""Tests for dind/daemon.py — event loop with heartbeat + demo job.""" + +import threading +from dincli.dind.daemon import DaemonLoop +from dincli.dind.state import StateStore + + +def test_daemon_loop_heartbeat_advances(tmp_path): + store = StateStore(tmp_path / "test.db") + stop = threading.Event() + + loop = DaemonLoop(store, stop, tick_interval=0.01, max_ticks=3) + loop.run() + + last_tick = store.get_meta("last_tick") + assert last_tick is not None + store.close() + + +def test_daemon_loop_runs_demo_job(tmp_path): + store = StateStore(tmp_path / "test.db") + store.enqueue("demo") + stop = threading.Event() + + loop = DaemonLoop(store, stop, tick_interval=0.01, max_ticks=3) + loop.run() + + last_success = store.get_meta("last_success") + assert last_success is not None + + counts = store.get_job_counts() + assert counts["pending"] == 0 + assert counts["running"] == 0 + store.close() + + +def test_daemon_loop_stops_on_event(tmp_path): + store = StateStore(tmp_path / "test.db") + stop = threading.Event() + stop.set() + + loop = DaemonLoop(store, stop, tick_interval=0.01, max_ticks=1000) + loop.run() + + assert loop.tick_count == 0 + store.close() + + +def test_daemon_loop_unknown_job_type_fails(tmp_path): + store = StateStore(tmp_path / "test.db") + store.enqueue("nonexistent_handler") + stop = threading.Event() + + loop = DaemonLoop(store, stop, tick_interval=0.01, max_ticks=3) + loop.run() + + row = store.claim_next() + assert row is None + + import sqlite3 + conn = sqlite3.connect(str(tmp_path / "test.db")) + failed = conn.execute( + "SELECT * FROM jobs WHERE type = 'nonexistent_handler' AND status = 'failed'" + ).fetchone() + conn.close() + assert failed is not None + store.close() diff --git a/tests/test_dind_health.py b/tests/test_dind_health.py new file mode 100644 index 0000000..133f243 --- /dev/null +++ b/tests/test_dind_health.py @@ -0,0 +1,78 @@ +"""Tests for dind/health.py — payload shape + degraded detection.""" + +import json +import time +from io import BytesIO + +from dincli.dind.health import HealthHandler +from dincli.dind.state import StateStore + + +class FakeServer: + server_address = ("127.0.0.1", 12345) + + +def _make_handler(store, path="/health"): + handler = HealthHandler.__new__(HealthHandler) + handler.state = store + handler.start_time = time.time() + handler.path = path + handler.requestline = "GET " + path + " HTTP/1.0" + handler.request_version = "HTTP/1.0" + handler.command = "GET" + handler.client_address = ("", 0) + handler.server = FakeServer() + handler._headers_buffer = [] + wfile = BytesIO() + handler.wfile = wfile + return handler, wfile + + +def test_health_payload_shape(tmp_path): + store = StateStore(tmp_path / "test.db") + store.set_meta("last_tick", "2025-01-01T00:00:00+00:00") + + handler, wfile = _make_handler(store) + + handler.do_GET() + response = wfile.getvalue() + assert b"HTTP/1.0 200" in response or b"200" in response + + parts = response.split(b"\r\n\r\n", 1) + body = json.loads(parts[1]) + + assert body["status"] == "degraded" + assert "uptime_s" in body + assert "pid" in body + assert "queue" in body + assert "pending" in body["queue"] + assert "cpu_count" in body["resources"] + assert "disk_free_bytes" in body["resources"] + + store.close() + + +def test_health_healthy(tmp_path): + from datetime import datetime, timezone + + store = StateStore(tmp_path / "test.db") + store.set_meta("last_tick", datetime.now(timezone.utc).isoformat()) + + handler, wfile = _make_handler(store) + + handler.do_GET() + response = wfile.getvalue() + parts = response.split(b"\r\n\r\n", 1) + body = json.loads(parts[1]) + assert body["status"] == "healthy" + store.close() + + +def test_health_404_on_other_paths(tmp_path): + store = StateStore(tmp_path / "test.db") + handler, wfile = _make_handler(store, path="/other") + + handler.do_GET() + response = wfile.getvalue() + assert b"404" in response + store.close() diff --git a/tests/test_dind_logging.py b/tests/test_dind_logging.py new file mode 100644 index 0000000..373a35f --- /dev/null +++ b/tests/test_dind_logging.py @@ -0,0 +1,101 @@ +"""Tests for dind/logging.py — JsonFormatter + idempotent configure_logging.""" + +import json +import logging + +from dincli.dind.logging import JsonFormatter, configure_logging + + +def test_json_formatter_emits_valid_json(): + fmt = JsonFormatter() + record = logging.LogRecord( + "dincli", logging.INFO, "", 0, "hello world", (), None + ) + output = fmt.format(record) + parsed = json.loads(output) + assert parsed["level"] == "INFO" + assert parsed["logger"] == "dincli" + assert parsed["msg"] == "hello world" + assert "ts" in parsed + + +def test_json_formatter_context_extra(): + fmt = JsonFormatter() + record = logging.LogRecord( + "dincli", logging.WARNING, "", 0, "test message", (), None + ) + record.job_id = 42 + record.role = "aggregator" + output = fmt.format(record) + parsed = json.loads(output) + assert parsed["job_id"] == 42 + assert parsed["role"] == "aggregator" + + +def test_json_formatter_exception(): + fmt = JsonFormatter() + try: + raise ValueError("boom") + except ValueError: + record = logging.LogRecord( + "dincli", logging.ERROR, "", 0, "fail", (), None + ) + import sys + record.exc_info = sys.exc_info() + + output = fmt.format(record) + parsed = json.loads(output) + assert "exception" in parsed + assert "boom" in parsed["exception"] + + +def test_configure_logging_idempotent(): + logger = logging.getLogger("dincli") + initial_handlers = len(logger.handlers) + initial_propagate = logger.propagate + + configure_logging("json") + after_first = len(logger.handlers) + + configure_logging("json") + after_second = len(logger.handlers) + + assert after_first == after_second + assert logger.propagate is False + + logger.handlers.clear() + logger.propagate = initial_propagate + + +def test_configure_logging_sets_info_level_by_default(): + logger = logging.getLogger("dincli") + logger.setLevel(logging.WARNING) + + configure_logging("json") + + assert logger.getEffectiveLevel() <= logging.INFO + + logger.handlers.clear() + logger.propagate = True + logger.setLevel(logging.NOTSET) + + +def test_configure_logging_respects_config_file(monkeypatch, tmp_path): + import json + + config_file = tmp_path / "config.json" + config_file.write_text(json.dumps({"log_level": "DEBUG"}), encoding="utf-8") + + from dincli.dind import logging as dlog + monkeypatch.setattr(dlog, "CONFIG_FILE", config_file) + + logger = logging.getLogger("dincli") + logger.setLevel(logging.WARNING) + + configure_logging("json") + + assert logger.getEffectiveLevel() == logging.DEBUG + + logger.handlers.clear() + logger.propagate = True + logger.setLevel(logging.NOTSET) diff --git a/tests/test_dind_process.py b/tests/test_dind_process.py new file mode 100644 index 0000000..d419b1c --- /dev/null +++ b/tests/test_dind_process.py @@ -0,0 +1,59 @@ +"""Tests for dind/process.py — PID file read/write/stale detection.""" + +import os +import signal + +from dincli.dind.process import ( + is_process_running, + read_pid, + remove_pid, + send_signal, + write_pid, +) + + +def test_write_read_remove_pid(tmp_path): + pid_path = tmp_path / "dind.pid" + write_pid(pid_path) + + pid = read_pid(pid_path) + assert pid == os.getpid() + + remove_pid(pid_path) + assert not pid_path.exists() + + +def test_read_pid_none_when_missing(tmp_path): + assert read_pid(tmp_path / "nonexistent") is None + + +def test_read_pid_none_when_empty(tmp_path): + pid_path = tmp_path / "empty.pid" + pid_path.write_text("") + assert read_pid(pid_path) is None + + +def test_read_pid_none_when_garbage(tmp_path): + pid_path = tmp_path / "garbage.pid" + pid_path.write_text("abc") + assert read_pid(pid_path) is None + + +def test_is_process_running_self(): + assert is_process_running(os.getpid()) is True + + +def test_is_process_running_dead(): + import subprocess + + proc = subprocess.Popen(["true"]) + proc.wait() + assert is_process_running(proc.pid) is False + + +def test_is_process_running_nonexistent(): + assert is_process_running(99999999) is False + + +def test_remove_pid_missing_is_noop(tmp_path): + remove_pid(tmp_path / "nonexistent") diff --git a/tests/test_dind_state.py b/tests/test_dind_state.py new file mode 100644 index 0000000..9101f83 --- /dev/null +++ b/tests/test_dind_state.py @@ -0,0 +1,115 @@ +"""Tests for dind/state.py — SQLite enqueue/claim/complete/fail/retention.""" + + +from dincli.dind.state import StateStore + + +def test_enqueue_claim_complete(tmp_path): + store = StateStore(tmp_path / "test.db") + jid = store.enqueue("demo", {"x": 1}) + assert jid == 1 + + row = store.claim_next() + assert row is not None + assert row["type"] == "demo" + assert row["status"] == "running" + assert row["attempts"] == 1 + + store.complete_job(jid) + + row2 = store.claim_next() + assert row2 is None + + store.close() + + +def test_enqueue_fail(tmp_path): + store = StateStore(tmp_path / "test.db") + jid = store.enqueue("fail_job") + row = store.claim_next() + assert row["id"] == jid + assert row["status"] == "running" + + store.fail_job(jid, "something went wrong") + + row = store.claim_next() + assert row is None + + store.close() + + +def test_reset_running_jobs(tmp_path): + store = StateStore(tmp_path / "test.db") + jid = store.enqueue("demo") + store.claim_next() + + counts = store.get_job_counts() + assert counts["running"] == 1 + + store.reset_running_jobs() + + counts = store.get_job_counts() + assert counts["running"] == 0 + assert counts["pending"] == 1 + + row = store.claim_next() + assert row["last_error"] == "interrupted@shutdown" + assert row["attempts"] == 2 + + store.close() + + +def test_meta_round_trip(tmp_path): + store = StateStore(tmp_path / "test.db") + store.set_meta("key1", "val1") + assert store.get_meta("key1") == "val1" + assert store.get_meta("nonexistent") is None + + store.set_meta("key1", "val2") + assert store.get_meta("key1") == "val2" + store.close() + + +def test_job_counts(tmp_path): + store = StateStore(tmp_path / "test.db") + store.enqueue("a") + store.enqueue("b") + counts = store.get_job_counts() + assert counts["pending"] == 2 + assert counts["running"] == 0 + assert counts["failed"] == 0 + store.close() + + +def test_retention_caps_history(tmp_path): + store = StateStore(tmp_path / "test.db", retention_limit=3) + for i in range(10): + jid = store.enqueue(f"job_{i}") + store.claim_next() + store.complete_job(jid) + + import sqlite3 + conn = sqlite3.connect(str(tmp_path / "test.db")) + total = conn.execute( + "SELECT COUNT(*) FROM jobs WHERE status IN ('done', 'failed')" + ).fetchone()[0] + conn.close() + assert total <= 3 + + store.close() + + +def test_restart_recovery(tmp_path): + store = StateStore(tmp_path / "test.db") + store.enqueue("demo") + store.claim_next() + store.close() + + store2 = StateStore(tmp_path / "test.db") + counts = store2.get_job_counts() + assert counts["running"] == 1 + store2.reset_running_jobs() + counts = store2.get_job_counts() + assert counts["running"] == 0 + assert counts["pending"] == 1 + store2.close() diff --git a/tests/test_dintoken.py b/tests/test_dintoken.py index 3a95a66..7838076 100644 --- a/tests/test_dintoken.py +++ b/tests/test_dintoken.py @@ -3,7 +3,7 @@ import pytest import typer from typer.testing import CliRunner - +from rich.console import Console from dincli.cli import dintoken from dincli.main import app as main_app @@ -52,15 +52,15 @@ def approve(self, address, amount): class DummyStakeFunctions: def __init__(self, stake): - self.stake = stake + self._stake = stake def stake(self, amount): - self.stake.stakes.append(amount) + self._stake.stakes.append(amount) return ("stake", amount) def getStake(self, address): - self.stake.get_stake_calls.append(address) - return DummyCall(self.stake.current_stake) + self._stake.get_stake_calls.append(address) + return DummyCall(self._stake.current_stake) class DummyCoordinatorFunctions: @@ -98,7 +98,7 @@ def __init__(self): class DummyContextObj: def __init__(self, token_balance=100 * 10**18, current_stake=12 * 10**18): - self.console = DummyConsole() + self.console = Console() self.w3 = DummyWeb3() self.account = SimpleNamespace(address="0xAccount") self.token = DummyToken(token_balance) @@ -106,6 +106,7 @@ def __init__(self, token_balance=100 * 10**18, current_stake=12 * 10**18): self.coordinator = DummyCoordinator() def get_en_w3_account_console(self): + self.console.print(f"[bold green]✓ Active Wallet:[/bold green] {self.account.address}") return "local", self.w3, self.account, self.console def get_deployed_din_token_contract(self): diff --git a/tests/test_ipfs_config.py b/tests/test_ipfs_config.py index 17021fc..6f9f5e1 100644 --- a/tests/test_ipfs_config.py +++ b/tests/test_ipfs_config.py @@ -2,6 +2,8 @@ from pathlib import Path from dincli.cli import utils +from dincli.sdk import config as sdk_config +from dincli.sdk import ipfs as sdk_ipfs from dincli.services import ipfs @@ -12,8 +14,8 @@ def _write_config(config_file: Path, data: dict): def test_resolve_ipfs_config_defaults_to_env_provider(monkeypatch, tmp_path): config_file = tmp_path / "config.json" - monkeypatch.setattr(utils, "CONFIG_DIR", tmp_path) - monkeypatch.setattr(utils, "CONFIG_FILE", config_file) + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) monkeypatch.chdir(tmp_path) _write_config(config_file, {"ipfs_provider": "ipfs node"}) @@ -23,7 +25,7 @@ def test_resolve_ipfs_config_defaults_to_env_provider(monkeypatch, tmp_path): encoding="utf-8", ) - resolved = utils.resolve_ipfs_config() + resolved = sdk_config.resolve_ipfs_config() assert resolved.provider == "env" assert resolved.api_url_add == "http://127.0.0.1:5001/api/v0" @@ -35,10 +37,10 @@ def test_upload_to_ipfs_uses_env_provider_by_default(monkeypatch, tmp_path): payload = tmp_path / "payload.bin" payload.write_bytes(b"payload") - monkeypatch.setattr(utils, "CONFIG_DIR", tmp_path) - monkeypatch.setattr(utils, "CONFIG_FILE", config_file) + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) monkeypatch.chdir(tmp_path) - monkeypatch.setattr(ipfs, "get_cidv1base32_from_cid", lambda cid: f"normalized-{cid}") + monkeypatch.setattr(sdk_ipfs, "get_cidv1base32_from_cid", lambda cid: f"normalized-{cid}") _write_config(config_file, {}) (tmp_path / ".env").write_text( @@ -62,9 +64,9 @@ def fake_post(url, **kwargs): calls.append(url) return DummyResponse() - monkeypatch.setattr(ipfs.requests, "post", fake_post) + monkeypatch.setattr(sdk_ipfs.requests, "post", fake_post) - cid = ipfs.upload_to_ipfs(payload) + cid = sdk_ipfs.upload_to_ipfs(payload) assert cid == "normalized-cid123" assert calls == ["http://127.0.0.1:5001/api/v0/add"] @@ -89,10 +91,10 @@ def test_custom_provider_delegates_upload_and_retrieve(monkeypatch, tmp_path): encoding="utf-8", ) - monkeypatch.setattr(utils, "CONFIG_DIR", tmp_path) - monkeypatch.setattr(utils, "CONFIG_FILE", config_file) + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) monkeypatch.chdir(tmp_path) - monkeypatch.setattr(ipfs, "get_cidv1base32_from_cid", lambda cid: f"normalized-{cid}") + monkeypatch.setattr(sdk_ipfs, "get_cidv1base32_from_cid", lambda cid: f"normalized-{cid}") _write_config( config_file, @@ -102,9 +104,113 @@ def test_custom_provider_delegates_upload_and_retrieve(monkeypatch, tmp_path): }, ) - cid = ipfs.upload_to_ipfs(payload) - status = ipfs.retrieve_from_ipfs("abc123", output) + cid = sdk_ipfs.upload_to_ipfs(payload) + status = sdk_ipfs.retrieve_from_ipfs("abc123", output) assert cid == "normalized-custom-cid" assert status == 204 assert output.read_text(encoding="utf-8") == "retrieved:abc123" + + +def test_resolve_ipfs_config_returns_filebase_provider(monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + + _write_config(config_file, { + "ipfs_provider": "filebase", + "ipfs_api_key_filebase": "fb-test-key", + }) + + resolved = sdk_config.resolve_ipfs_config() + + assert resolved.provider == "filebase" + assert resolved.api_key == "fb-test-key" + + +def test_resolve_ipfs_config_uses_ipfs_provider_env_var(monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + (tmp_path / ".env").write_text("IPFS_PROVIDER=filebase\n", encoding="utf-8") + + _write_config(config_file, {}) + + resolved = sdk_config.resolve_ipfs_config() + + assert resolved.provider == "filebase" + + +def test_config_provider_wins_over_env_var(monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + (tmp_path / ".env").write_text("IPFS_PROVIDER=custom\n", encoding="utf-8") + + _write_config(config_file, {"ipfs_provider": "filebase"}) + + resolved = sdk_config.resolve_ipfs_config() + + assert resolved.provider == "filebase" + + +def test_filebase_legacy_flat_api_key_still_resolves(monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + + _write_config(config_file, { + "ipfs_provider": "filebase", + "ipfs_api_key": "legacy-flat-key", + }) + + resolved = sdk_config.resolve_ipfs_config() + + assert resolved.provider == "filebase" + assert resolved.api_key == "legacy-flat-key" + + +def test_cross_provider_isolation_filebase_key_not_reused_for_other_provider(monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + + _write_config(config_file, { + "ipfs_provider": "custom", + "ipfs_api_key_filebase": "fb-key", + }) + + resolved = sdk_config.resolve_ipfs_config() + + assert resolved.provider == "custom" + assert resolved.api_key is None + + +def test_env_provider_unaffected_by_provider_key_config(monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + + _write_config(config_file, { + "ipfs_provider": "ipfs node", + "ipfs_api_key": "legacy-flat-key", + "ipfs_api_key_filebase": "fb-key", + }) + (tmp_path / ".env").write_text( + "IPFS_API_URL_ADD=http://127.0.0.1:5001/api/v0\n" + "IPFS_API_URL_RETRIEVE=http://127.0.0.1:5001/api/v0\n", + encoding="utf-8", + ) + + resolved = sdk_config.resolve_ipfs_config() + + assert resolved.provider == "env" + assert resolved.api_url_add == "http://127.0.0.1:5001/api/v0" + assert resolved.api_url_retrieve == "http://127.0.0.1:5001/api/v0" + assert resolved.api_key is None diff --git a/tests/test_sdk_boundary.py b/tests/test_sdk_boundary.py new file mode 100644 index 0000000..4e3d53c --- /dev/null +++ b/tests/test_sdk_boundary.py @@ -0,0 +1,74 @@ +"""SDK layer contract tests (issue #20). + +- Import-boundary: ``dincli.sdk`` (and every submodule) must import without + pulling in ``typer``, ``rich``, or any ``dincli.cli.*`` module. Run in a fresh + subprocess so an earlier test that imported those doesn't mask a violation. +- Error taxonomy + allowlist detail sanitization (proposal §4/§4a). +""" +import subprocess +import sys +import textwrap + +from dincli.sdk import errors +from dincli.sdk.errors import DinError, TransactionError, TX_REVERTED, sanitize_details + + +def test_sdk_imports_no_cli_or_ui(): + """import dincli.sdk.* pulls in no typer / rich / dincli.cli — enforces the + 'CLI depends on SDK, never the reverse' rule. Fresh subprocess required.""" + script = textwrap.dedent( + """ + import importlib, pkgutil, sys + import dincli.sdk + for m in pkgutil.walk_packages(dincli.sdk.__path__, dincli.sdk.__name__ + "."): + importlib.import_module(m.name) + loaded = set(sys.modules) + bad_ui = {"typer", "rich"} & loaded + bad_cli = sorted(m for m in loaded if m == "dincli.cli" or m.startswith("dincli.cli.")) + assert not bad_ui, f"SDK pulled in UI libs: {sorted(bad_ui)}" + assert not bad_cli, f"SDK imported dincli.cli modules: {bad_cli}" + print("ok") + """ + ) + result = subprocess.run( + [sys.executable, "-c", script], capture_output=True, text=True + ) + assert result.returncode == 0, f"stdout={result.stdout!r} stderr={result.stderr!r}" + + +def test_error_carries_stable_code_and_message(): + err = DinError("boom") + assert err.code == "din_error" + assert err.message == "boom" + assert err.to_error() == {"code": "din_error", "message": "boom", "details": {}} + + +def test_subcode_override(): + err = TransactionError("reverted", code=TX_REVERTED, details={"tx_hash": "abc", "nonce": 7}) + assert err.code == "tx_reverted" + assert err.details["tx_hash"] == "0xabc" # hex-normalized + assert err.details["nonce"] == 7 + + +def test_sanitize_drops_secrets_and_unlisted_keys(): + dirty = { + "tx_hash": "0xdead", + "password": "hunter2", # never allowed + "private_key": "0xbeef", # never allowed + "unexpected": "whatever", # not in allowlist for this code + "nonce": "12", # coerced to int + } + clean = sanitize_details(TX_REVERTED, dirty) + assert clean == {"tx_hash": "0xdead", "nonce": 12} + + +def test_sanitize_strips_url_credentials(): + clean = sanitize_details( + errors.RPC_UNREACHABLE, + {"endpoint_host": "https://user:key@rpc.example.com:8545/v1?token=abc"}, + ) + assert clean["endpoint_host"] == "https://rpc.example.com:8545/v1" + + +def test_unknown_code_yields_empty_details(): + assert sanitize_details("no_such_code", {"anything": "x"}) == {} diff --git a/tests/test_sdk_config.py b/tests/test_sdk_config.py new file mode 100644 index 0000000..6f8e27f --- /dev/null +++ b/tests/test_sdk_config.py @@ -0,0 +1,134 @@ +import json +from pathlib import Path + +import pytest +from dincli.sdk import config as sdk_config + + +def _write_json(path: Path, data: dict): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(data), encoding="utf-8") + + +class TestSaveLoadConfig: + def test_round_trip(self, monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + + data = {"ipfs_provider": "filebase", "log_level": "DEBUG"} + sdk_config.save_config(data) + + loaded = sdk_config.load_config() + assert loaded == data + + def test_load_config_missing_file(self, monkeypatch, tmp_path): + config_file = tmp_path / "nonexistent" / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", config_file.parent) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + + loaded = sdk_config.load_config() + assert loaded == {} + + def test_load_config_invalid_json(self, monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + config_file.write_text("not valid json", encoding="utf-8") + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + + loaded = sdk_config.load_config() + assert loaded == {} + + def test_get_config_existing_key(self, monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + _write_json(config_file, {"log_level": "DEBUG"}) + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + + assert sdk_config.get_config("log_level") == "DEBUG" + + def test_get_config_missing_key_default(self, monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + _write_json(config_file, {}) + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + + assert sdk_config.get_config("nonexistent", default=42) == 42 + + +class TestResolveNetworkValue: + def test_env_wins_over_config(self, monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + + _write_json(config_file, {"networks": {"local": {"rpc_url": "http://config.local:8545"}}}) + (tmp_path / ".env").write_text("LOCAL_RPC_URL=http://env.local:8545\n", encoding="utf-8") + + result = sdk_config.resolve_network_value("local", "rpc_url") + assert result == "http://env.local:8545" + + def test_config_used_when_env_missing(self, monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + + _write_json(config_file, {"networks": {"local": {"rpc_url": "http://config.local:8545"}}}) + (tmp_path / ".env").write_text("", encoding="utf-8") + + result = sdk_config.resolve_network_value("local", "rpc_url") + assert result == "http://config.local:8545" + + def test_default_falls_back(self, monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + + _write_json(config_file, {}) + (tmp_path / ".env").write_text("", encoding="utf-8") + + result = sdk_config.resolve_network_value("local", "rpc_url", default="http://default:8545") + assert result == "http://default:8545" + + def test_raises_key_error_when_nothing_found(self, monkeypatch, tmp_path): + config_file = tmp_path / "config.json" + monkeypatch.setattr(sdk_config, "CONFIG_DIR", tmp_path) + monkeypatch.setattr(sdk_config, "CONFIG_FILE", config_file) + monkeypatch.chdir(tmp_path) + + _write_json(config_file, {}) + (tmp_path / ".env").write_text("", encoding="utf-8") + + with pytest.raises(KeyError, match="Could not resolve"): + sdk_config.resolve_network_value("local", "nonexistent_key") + + +class TestNormalizeIpfsProvider: + def test_none_returns_env(self): + assert sdk_config.normalize_ipfs_provider(None) == "env" + + def test_empty_string_alias_to_env(self): + assert sdk_config.normalize_ipfs_provider("") == "env" + + def test_default_alias_to_env(self): + assert sdk_config.normalize_ipfs_provider("default") == "env" + + def test_ipfs_node_alias_to_env(self): + assert sdk_config.normalize_ipfs_provider("ipfs node") == "env" + assert sdk_config.normalize_ipfs_provider("ipfs-node") == "env" + assert sdk_config.normalize_ipfs_provider("node") == "env" + + def test_env_passes_through(self): + assert sdk_config.normalize_ipfs_provider("env") == "env" + + def test_filebase_passes_through(self): + assert sdk_config.normalize_ipfs_provider("filebase") == "filebase" + + def test_custom_passes_through(self): + assert sdk_config.normalize_ipfs_provider("custom") == "custom" + + def test_unknown_provider_passes_through(self): + assert sdk_config.normalize_ipfs_provider("some-unknown-provider") == "some-unknown-provider" diff --git a/tests/test_sdk_web3.py b/tests/test_sdk_web3.py new file mode 100644 index 0000000..da7b59e --- /dev/null +++ b/tests/test_sdk_web3.py @@ -0,0 +1,32 @@ +from unittest.mock import MagicMock + +import pytest +from dincli.sdk import web3 as sdk_web3 + + +def test_get_w3_raises_connection_error_on_unreachable(monkeypatch): + monkeypatch.setattr(sdk_web3, "resolve_network_value", lambda *a, **kw: "http://unreachable:8545") + + fake_w3 = MagicMock() + fake_w3.is_connected.return_value = False + MockWeb3 = MagicMock() + MockWeb3.HTTPProvider = MagicMock(return_value=MagicMock()) + MockWeb3.return_value = fake_w3 + monkeypatch.setattr(sdk_web3, "Web3", MockWeb3) + + with pytest.raises(ConnectionError, match="Could not connect to Ethereum node at http://unreachable:8545"): + sdk_web3.get_w3("testnet") + + +def test_get_w3_returns_web3_instance_on_success(monkeypatch): + monkeypatch.setattr(sdk_web3, "resolve_network_value", lambda *a, **kw: "http://reachable:8545") + + fake_w3 = MagicMock() + fake_w3.is_connected.return_value = True + MockWeb3 = MagicMock() + MockWeb3.HTTPProvider = MagicMock(return_value=MagicMock()) + MockWeb3.return_value = fake_w3 + monkeypatch.setattr(sdk_web3, "Web3", MockWeb3) + + result = sdk_web3.get_w3("testnet") + assert result is fake_w3