v1.3.3: Release — Collab integration, UI improvements, CLI enhancements, and agent features#250
Merged
Conversation
…ration
pause_controller starts paused by default and is intended to be driven by
the UI / training loop, not the data path. Calling _wait_if_paused() from
__iter__ and __next__ meant any script that iterated the loader before an
external resume would hang forever — even at num_workers=0. Workers made
the failure mode look like a worker bug (leaked semaphores, freezes under
load), but the same hang reproduced with no workers at all.
Pause is a training-loop concern; the loader should just deliver bytes.
_wait_if_paused() itself is preserved so training loops can still call it
explicitly at safe points (between optimizer steps).
Verified:
- weightslab/tests/backend/test_data_loader_interface.py: 9/9 pass
(incl. test_dataloader_interface_uses_multiple_workers,
test_multiple_workers_parallelize_preprocessing)
- Wider tests/backend + tests/components sweep: 108 pass, 2 unrelated
pre-existing failures in test_ui_docker_bridge (cert script + Windows
path test on Linux)
- ws-detection example with num_workers={0,2,4} on CPU: clean runs,
W=2 ~21% faster than W=0, no hangs or crashes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
a4cf489 removed DataLoaderInterface._wait_if_paused's call sites to fix the num_workers>0 startup deadlock, but that was also the only place enforcing the explicit pause_at_step hyperparam. Re-add the trigger in GuardContext.__enter__ (training only), before the architecture lock so the pause blocks lock-free. pause() zeroes pause_at_step, so it fires once. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Its call sites were removed in a4cf489 (deadlock fix) and its pause_at_step trigger was relocated to GuardContext.__enter__ in the previous commit, so the method is now unused. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- main.py: refactor train loop to the infinite-generator form (re-shuffles each epoch) and default both loaders to num_workers=2 (GPU sweep: ~+76% throughput vs workers=0, sweet spot; W=4 regresses). - bench.py + run_bench.sh: configurable num_workers/epochs/wall-time harness with WL_BENCH_NO_VAL for clean throughput runs; forces the model onto the target device after watch_or_edit (see device note below). - config.yaml: local run tweaks. Note: watch_or_edit(flag="model", device=...) currently drops its device= kwarg (src.py model branch returns the proxy without honoring it), so the bench applies an explicit .to(device) workaround; the framework path is still unfixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…terface Both were only referenced by _wait_if_paused, removed in 41b76b8. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These were local correctness-check tooling for the num_workers sweep, not part of the example. Kept on disk locally (untracked), removed from version control. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
main.py (runnable train-loop fix) moves to a dedicated fix branch off dev; config.yaml carried machine-local tweaks. Both are kept locally but removed from this branch so it contains only the parallelism/pause framework changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
An earlier merge left main.py's train loop in a non-runnable state. Restore a working loop (infinite-generator batching that re-shuffles each epoch, per-sample loss/IoU via the criterion dict) and default the loaders to num_workers=2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix ws-detection example: restore runnable training loop
…llel+distributed+plightning+networkfs Framework intergrations/parallel+distributed+plightning+networkfs
* fix oom bug on break by slice The break-by-slice handler called get_signal_history_per_sample(), which inflated the entire per-sample signal history into nested dicts and then triple-looped over it (~609 MB spike per slice query -> OOM). Separately, query_per_sample() compared sample ids as int (stored) vs str (queried), so the cheap path would have silently returned 0 rows. Query the compact per-signal arrays directly via query_per_sample, normalize the id compare to str, and derive audit_mode from the eval-marker hash. A 200-sample slice over 2.1M entries now peaks at +0.6 MB instead of +609 MB (~1000x lighter). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix BBS feature to make the main computation part on our side --------- Co-authored-by: Alexandru Rotaru <rotarualexandruandrei94@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: GuillaumePELLUET <guillaume@graybx.com>
…d csv (#183) * Implement comprehensive audit logging for all gRPC user interactions with before/after tracking - Create AuditLogger class in backend/audit_logger.py with thread-safe JSON/CSV writing - Initialize audit loggers in ExperimentService and DataService with root_log_dir - Log all 8+ gRPC handlers with detailed before/after values: * ExperimentCommand: hp_change, mode_switch, pause, resume * GetLatestLoggerData: metrics_fetch * RestoreCheckpoint: checkpoint_restore * TriggerEvaluation: evaluation_start * EditDataSample: tag_add, tag_remove, sample_discard, sample_restore * ApplyDataQuery: query_execute * GetDataSamples: data_fetch - Append-only audit_log.json and audit_log.csv files in root_log_dir - ISO 8601 timestamps with microseconds in JSON; CSV with escaped JSON details - Thread-safe file operations Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add comprehensive unit tests and documentation for audit logging Tests: - 26 unit tests covering all AuditLogger functionality - Tests for JSON and CSV output formats - Thread-safe concurrent logging with 10+ threads - Error handling and edge cases - Real-world scenario tests (hyperparameter changes, data edits, training control) - Special characters and Unicode handling - All tests passing Documentation: - Comprehensive audit_logging.rst guide with examples - Overview of what's logged (7+ action types) - JSON and CSV format specifications - Configuration and file locations - Real-world scenarios and troubleshooting - API reference and best practices - Added to docs index for discoverability Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Reorganize documentation: separate gRPC functions and audit logging Documentation restructuring: - Created new gRPC section with subsections - docs/grpc/index.rst: Overview and architecture of gRPC communication - docs/grpc/grpc_functions.rst: Complete reference of all RPC handlers (13 methods) * ExperimentCommand (HP changes, pause/resume, mode switching) * GetLatestLoggerData (metrics and signals) * RestoreCheckpoint, TriggerEvaluation, GetEvaluationStatus, CancelEvaluation * GetDataSamples, ApplyDataQuery, EditDataSample, GetDataSplits * GetWeights, GetActivations, GetSamples * Includes: request/response types, parameters, behavior, audit logging status * Covers: error handling, performance considerations, debugging, common patterns - docs/grpc/audit_logger.rst: Comprehensive audit logging documentation * Moved from docs/audit_logging.rst with updated cross-references * Explains what gets logged, file formats (JSON/CSV), configuration * Real-world scenarios, troubleshooting, API reference, best practices - Updated docs/index.rst to reference new gRPC section Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add configurable audit log output format via AUDIT_LOG_FORMAT environment variable - Modified AuditLogger to write only one format (json OR csv), not both - Added format parameter to AuditLogger.__init__() with environment variable support - AUDIT_LOG_FORMAT=json (default) or AUDIT_LOG_FORMAT=csv - Explicit format parameter takes precedence over environment variable - Updated all 33 tests to work with format selection: - Fixed TestAuditLoggerCSV, TestAuditLoggerErrorHandling, TestAuditLoggerThreadSafety - Added TestAuditLoggerFormat class with 4 new tests for format configuration - Updated docs/grpc/audit_logger.rst Configuration section with AUDIT_LOG_FORMAT details and precedence rules - All tests passing: 33/33 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add ability to disable audit logging with AUDIT_LOG_FORMAT=none - Added "none" format option to AuditLogger to disable audit logging entirely - When format="none", log_event() returns early without creating files - Added AUDIT_LOG_FORMAT=none to docs/configuration.rst environment variables - Updated docs/grpc/audit_logger.rst Configuration section with disable feature - Added 3 new tests for disable functionality (36 total tests, all passing): - test_none_format_disables_logging() - test_none_format_from_environment_variable() - test_explicit_format_none_overrides_json_default() - Precedence unchanged: explicit format > environment variable > default Use cases for disabling: - Reduce disk I/O overhead in high-performance scenarios - Disable audit history for development/debugging sessions - Focus on other logging without audit pollution Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix FutureWarning: Set incompatible dtype column to object before assignment When upserting data with mixed dtypes (e.g., initializing column with bool False, then assigning string/array values), pandas raises a FutureWarning about incompatible dtypes. Fix by casting both the existing column and incoming values to object dtype before assignment to prevent dtype conflicts during merge operations. This resolves: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Tests: test_h5_dataframe_store.py passes, FutureWarning no longer raised Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix AttributeError: Use correct EDIT_ACCUMULATE instead of non-existent EDIT_ADD enum The SampleEditType enum only defines: - EDIT_OVERRIDE: Replace all tags - EDIT_ACCUMULATE: Add/accumulate tags - EDIT_REMOVE: Remove tags The audit logging code was trying to use the non-existent EDIT_ADD enum value. Fixed by using EDIT_ACCUMULATE for tag_add operations, which is the correct enum value for adding/accumulating tags based on _calculate_tag_column_updates docstring. Error was: AttributeError: Enum SampleEditType has no value defined for name 'EDIT_ADD' Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix subscribe function to allow user to compute history-based samples * remove caching * fix warning issue with h5 * add sanity check on modeling feature and fix datasampler issues * clean custom signals decorator feature in readme and doc * Refactor query_per_sample to return dict of samples instead of list of tuples Changed return format from: List of (sample_id, step, value) tuples To: Dict mapping sample_id → list of dicts with 'model_age' and 'signal_value' keys Example: {'0': [{'model_age': x, 'signal_value': y}, ...], '1': [...]} Benefits: - More structured and readable format - Keys are labeled, not positional - Easier to work with in custom signals (e.g., computing loss variance) - Matches the format expected in SignalContext.subscribed_history Both get_current_signaL_history_per_sample and query_per_sample now return the new format. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix registered subscribed signals * Remove data fetch operations from audit logging Audit logging should only track user actions (write operations), not read-only operations like: - GetLatestLoggerData (metrics_fetch) - GetDataSamples (data_fetch) These are passive retrieval operations, not modifications to experiment state. Changes: - Removed data_fetch and metrics_fetch from audit logging documentation - Updated audit_logger.rst to list only user action types - Changed GetDataSamples and GetLatestLoggerData to 'Audit Logged: No (read-only operation)' - Updated reproducing experiment scenario to focus on user actions only Audit logging now logs only: - Model Control: hp_change, pause, resume, mode_switch - Data Operations: tag_add, tag_remove, sample_discard, sample_restore, query_execute - Checkpoint & Evaluation: checkpoint_restore, evaluation_start Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix audit logger implementation: reverse chronological ordering and test fixes - Fix _flush_to_json to reverse event order within batches for strict reverse chronological ordering (newest first) - Add buffer_size=1 to all test instances to ensure events flush immediately during testing - Update test expectations for reverse chronological order (newest events appear first in JSON) - Fix timestamp assertions in training control scenario to expect decreasing order (ts1 > ts2 > ts3) - All 41 audit logger tests now pass Features verified: 1. Persistence: audit logs append when restarting experiments from existing root_log_dir 2. Reverse chronological: newest events appear first in JSON output 3. Buffering: events batch in memory before writing to disk, with configurable buffer_size Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add audit logging for plot note operations - Log note_write action when users save or clear notes on plot points - Capture metric_name, model_age, note_text, and note_action (saved/cleared) - Update audit logger documentation to include note_write in actions list This allows compliance tracking of all user annotations and notes. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Remove buffering approach from audit logger - use immediate writes instead - Remove buffer_size parameter and _event_buffer from AuditLogger - Change to immediate writes on each log_event() call - Rename _flush_to_json/_flush_to_csv to _write_json/_write_csv for single event writes - Remove flush() and _flush_buffer() methods - Remove all buffering-related tests - Update documentation to reflect immediate write approach Benefits: - No data loss on process crash or sudden termination - All audit events are persisted immediately to disk - Simpler implementation with same persistence guarantees - Still maintains reverse chronological ordering (newest first) Tests: 38 passed (3 buffering tests removed) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Expand signal decorator and SignalContext documentation - Add comprehensive parameter reference table for @wl.signal decorator * name, subscribe_to, compute_every_n_steps, include_history, include_history_metadata * Include performance considerations and use cases - Add advanced example from weightslab_kitchen: loss coefficient of variation * Shows how to access subscribed_history for multi-step analysis * Demonstrates history entry structure (signal_value, model_age) * Real-world use case: detect training instability - Expand SignalContext documentation with detailed attribute reference * Separate sections for dynamic signals vs. static signals * Document subscribed_history structure and access patterns * Add convenience properties (image, points, is_static, is_dynamic) * Include usage patterns and code examples This makes it clearer how to write effective custom signals with full history access. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add sanity check with iterator * Fix utests bug with AuditLogs * update hard coded signals desc * set needs btw utests and pip publish packages and test * fix code quality issues * remove data fetching from audit and slow useless part of checkpoint manager loading (rng and data iterator state) as we do not manage data state reproducibility for now --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
* Upgrade database to handle multi-indexing samples_id // instance_id, and subsequent fuctions
* Fix certificate generation prompts in Windows tests
Remove certificate generation and validation prompts that appear when running
test_ui_docker_bridge tests on Windows. The test_complete_onboarding_workflow
was calling actual certificate generation code without mocking it, which would
trigger Windows certificate store installation prompts.
Changes:
- Added proper mocking of _generate_certs_with_fallback() function
- Added mocking of _run_shell_script() to prevent bootstrap script execution
- Properly configured CertAuthManager mocks with check_and_apply() return value
- Added from_env_or_default() mock configuration for nested calls
All 40 tests in test_ui_docker_bridge.py now pass without prompting for
certificate validation on Windows.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Update ws-detection to use per-instance metrics and losses
Switch detection example from per-sample to per-instance metrics:
- Replace PerSampleDetectionLoss with PerInstanceDetectionLoss
- Replace PerSampleIoU with PerInstanceIoU
- Log hierarchical loss levels: instance, sample, and batch
- Enable per-instance loss tracking for multi-instance dataframe support
Changes:
- Import PerInstanceDetectionLoss and PerInstanceIoU from criterions
- Configure losses with return_levels=True to get instance/sample/batch breakdown
- Manually log per-instance and per-sample metrics via wl.log_sample_signals()
- Use 'batch' level loss for backward pass to ensure proper gradient flow
- Maintain per-instance IoU computation for bounding box evaluation
This enables comprehensive per-annotation analysis in the UI while maintaining
per-sample aggregation for backward pass compatibility.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Add both per-sample AND per-instance metrics to ws-detection
Track metrics at both granularity levels:
Per-Sample Metrics (aggregated):
- PerSampleDetectionLoss: Bounding box, classification, DFL losses averaged per sample
- PerSampleIoU: IoU averaged per sample
- Auto-logged via per_sample=True
- Signal names: train/bbxs_sample, train/clsf_sample, train/dfl_sample, iou/train_sample
Per-Instance Metrics (per annotation):
- PerInstanceDetectionLoss: Individual bbox losses for each annotation
- PerInstanceIoU: IoU for each bounding box
- Manually logged via wl.log_sample_signals()
- Signal names: train/bbxs_instance, train/clsf_instance, train/dfl_instance, iou_instance
This enables:
- Aggregate per-sample analysis for model evaluation
- Fine-grained per-annotation debugging
- Identification of problematic detections at the instance level
- Proper gradient flow through batch-level loss for training
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Auto-save per-instance signals via per_instance flag on watch_or_edit
Add framework-side support for per-instance signal logging, mirroring the
existing per_sample flow. Users can now wrap a per-instance loss/metric with
`per_instance=True` and WeightsLab will:
1. Extract instance values from dict outputs (`{'instance','sample','batch'}`
from PerInstanceDetectionLoss) or flat tensors (PerInstanceIoU).
2. Look up `batch_idx` from the second positional argument (the standard
detection `batch` dict) or from kwargs, mapping each instance to its
sample position.
3. Assign annotation_ids 0,1,2,... within each sample.
4. Save per-instance values to the dataframe at `(sample_id, annotation_id)`
via the new `DATAFRAME_M.enqueue_instance_batch`.
5. Still log the per-sample aggregated mean for the dashboard and return the
original dict to the caller so `out['batch']` works for backward.
Changes:
- `dataframe_manager.enqueue_instance_batch`: writes per-annotation rows
using `update_values` (handles multi-index natively).
- `src.save_instance_signals`: helper that maps instance values to
(sample_id, annotation_id) via batch_idx and routes to the dataframe.
- `wrappered_fwd`: detects `per_instance=True`, unwraps dict outputs,
invokes `save_instance_signals`, and returns the original dict.
- `ws-detection/main.py`: replaces manual `wl.log_sample_signals` calls
with `per_instance=True` on the watch_or_edit registrations.
- New unit test `test_enqueue_instance_batch_writes_per_annotation`
validates the end-to-end write path.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Simplify PerInstanceDetectionLoss to return flat instance tensor
Drop the dict-with-levels return type from PerInstanceDetectionLoss — only
the per-instance values are needed, since PerSampleDetectionLoss already
provides the per-sample gradient path for backward.
PerInstanceDetectionLoss now returns a flat `(num_instances,)` tensor,
ordered as in `batch['batch_idx']`. With `per_instance=True` on
watch_or_edit, the framework auto-saves these values at
`(sample_id, annotation_id)` in the dataframe.
Changes:
- `criterions.py`: remove `return_levels` param; forward returns a flat
instance tensor and only that.
- `main.py`: backward now uses `per_sample.mean()` from
PerSampleDetectionLoss; per-instance criterions are called only for their
side-effect of auto-saving annotation-level signals.
- `src.py`: skip the per-sample save_signals path when `per_instance=True`
(instance-length tensors don't map 1:1 to batch_ids).
- New test `test_save_instance_signals_maps_batch_idx_to_annotation_ids`
verifies the (sample_id, annotation_id) mapping from batch_idx.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Fix duplicate-label error when per-sample buffer flushes into multi-index df
enqueue_batch produces single-level (sample_id) records, but when the global
dataframe has a MultiIndex, concatenating them directly creates a hybrid index
with both tuples and ints. The next flush then crashes with:
ValueError: cannot reindex on an axis with duplicate labels
at _apply_buffer_records_nonblocking.
Root cause: _apply_buffer_records and _apply_buffer_records_nonblocking didn't
bridge between the single-level buffer and the multi-index global dataframe.
Fix: add _broadcast_to_multi_index which expands each single-level (sample_id)
buffer record into one row per existing (sample_id, annotation_id) pair. Both
apply paths now invoke it before merging, so the global dataframe stays a
proper MultiIndex and per-sample signals are broadcast to every annotation of
the sample.
Adds regression test test_per_sample_buffer_into_multi_index_does_not_corrupt
that asserts index integrity through two consecutive flushes and that
per-sample values are correctly broadcast to all annotations.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Fix _normalize_arrays_for_storage on multi-index rows
When the dataframe is multi-indexed, `row.name` is a
`(sample_id, annotation_id)` tuple. The code passed that tuple directly to
`dataset.get_index_from_sample_id` (which expects a plain sample_id string),
causing every array-column normalization to fail with a KeyError. The error
was caught and only logged at DEBUG level, but it flooded the log and
disabled the target/prediction normalization on every flush.
Extract `sample_id = row.name[0]` when `row.name` is a tuple; otherwise
fall back to the original row.name.
Adds regression test test_normalize_arrays_for_storage_handles_multi_index_row
that injects a fake dataset and asserts the sample_id (not the tuple) is
passed to `get_index_from_sample_id`.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* cleaning branch
* fix usecases examples configs and python file
* fix grpc and agent interface with multi-indexing
* fix multi-indexing issues
* fix ext files
* Add cat. tag management and fix tests
* Fix documentation
* Fix h5 compat. with multi-indexing
* Fix utests and add new ones
* Fix multi-index issues with trainer gRPC functions; h5 array issues for sync batch idx; and detection tasktype bb; and finally update the documentation
* Fix broken utests
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
* DDP support (WIP): SPMD primitives + 4-plane model + workers-correctness fixes
Wraps the existing single-process YOLO ws-detection example for DDP via
mp.spawn — no train.py edits needed past the spawn shim. Surfaces:
- weightslab/components/ddp_basic_building_blocks.py — SPMD primitives:
register_consistent_state, reconcile_all (bundled DOWN broadcast),
register_outbox + flush_outbox (bundled UP gather), sync_step (per-step
anchor + collective pause-spin). One broadcast + one gather per step.
- weightslab/components/ddp_planes.py — the 4-plane model (CONFIG /
CONTROL / DATAFRAME / LOGGER) + 5 dtype-keyed reducers (MAX / LATEST /
UNION / RANK_0_ONLY / IGNORE) + DOWN_ONLY whitelist for cross-rank DOWN-
flowing per-sample columns.
- components/global_monitoring.GuardContext — guard_training_context now
auto-registers the core states + invokes sync_step on first DDP entry.
- data/dataframe_manager.py — shm mirror of DOWN_ONLY columns visible to
DataLoader subprocess workers via fork; per-cell value-change gate so
rank-N's idempotent reconcile applies don't thrash worker resets; iter
invalidation triggers on real DOWN_ONLY mutations only.
- backend/dataloader_interface.py — WeightsLabDataSampler composes
DistributedSampler under DDP; sampler reads the shm mirror at yield time
(fork-safe); DataLoaderInterface gains _invalidate_iter to drop prefetched
stale batches on the trainer thread (avoids the std::terminate crash from
worker shutdown on a non-owning thread).
- trainer/services/experiment_service.py — RestoreCheckpoint passes
force=True (data snapshot was silently skipped when hashes appeared
equal) and re-pauses post-load (saved hp had is_training=True, which
load_state's register_hyperparams would otherwise re-apply).
- components/checkpoint_manager.py — three reset_iterator sites route
through the lazy _invalidate_iter path under DDP+workers.
- examples/PyTorch/ws-detection/src/main_ddp.py — spawn shim worker.
- examples/PyTorch/ws-detection/src/yolo_pipeline.py — extracted YOLO
pipeline (replaces the older ddp_smoke._build_pipeline / decode helpers).
- examples/PyTorch/ws-detection/src/ddp_test_suite.py — 8-scenario gRPC
integration suite: epoch_then_pause, discard_subset_freezes,
break_by_slice, lr_batch_propagate, checkpoint_data_roundtrip,
signal_coverage_all_graphs, resume_continues_curve, process_topology.
- tests/test_ddp_primitives.py — trivial 3-rank gloo verification of
reconcile_all (convergence + idempotency + change propagation).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add 4 gap-coverage scenarios + collective-budget instrumentation
12 scenarios green end-to-end (8 original + 4 new). The new ones cover
gaps that were missing direct verification:
- scenario_multi_epoch_stability — 3 epochs back-to-back, asserts
(sid, model_age) entries are unique per graph (idempotent dedup at the
outbox merge) + age is strictly monotonic across epochs. Catches the
regression where outbox flushes would append rather than upsert per-step.
- scenario_empty_shard_starvation — discards ~95% of populated samples;
asserts the trainer does NOT silently hang at the next grad all-reduce
when one rank's shard ends up empty. Verifies loader cycle-and-skip
semantics under heavy filtering.
- scenario_seed_determinism — two consecutive break_by_slice pulls of the
per-sample loss history return byte-identical (sid, age, val) triples.
Catches stochasticity leaks in the read path that would silently break
the loss-shape descriptor downstream.
- scenario_collective_budget — programmatically asserts that every
training step uses EXACTLY 2 collectives (1 reconcile_all broadcast +
1 flush_outbox gather). Hard perf gate against future regressions that
add a stray dist.broadcast / dist.all_reduce in a hot path. Requires a
small SDK hook: WL_DDP_COLLECTIVE_LOG=<path> appends the prior step's
count to a file from inside reset_collectives() — opt-in, no overhead
when unset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add scenario_curate_lifecycle — end-to-end UI curation flow
Tests the realistic multi-edit workflow under DDP:
epoch 1 → tag 3 suspects → discard them → epoch 2 → un-discard + tag
'verified' → epoch 3 → assert loss trajectory shows the gap.
Assertions:
[1] LIFECYCLE — for each suspect: pre-discard entries exist, NO entry
in the (discard_age, undiscard_age] window for any of them (this
is the proof that discard reached the workers' shm + sampler
fast-path), AND ≥1 suspect resumes post-undiscard.
[2] TAG COMPOSE — break_by_slice('verified') returns all 3 suspects
(proves multi-tag stacking on the same sample).
[3] PLOT METRICS — scalar plot has ≥3 epochs worth of points.
Side change: Client.discard now accepts discarded=False to un-discard via
the same EditDataSample RPC.
This brings the suite to 13 scenarios, all green at WL_DDP_BATCH=4,
WL_DDP_WORKERS=0, WL_DDP_WORLD_SIZE=2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Cleanup: drop unused primitives + document drop_last=False rationale
Removes ~155 LOC of dead-code surface that accumulated during the WIP push:
- weightslab/components/ddp_basic_building_blocks.py: drop aggregate_up
decorator, replicate_down decorator, reconcile_down (single-state hook),
plus the combine-helpers (_concat, combine_rank0, make_concat_combine)
that only existed to serve aggregate_up. The outbox/flush pattern
superseded ① aggregate_up for per-sample hot writes (one gather/step
instead of one per call); reconcile_all replaced reconcile_down (bundled
broadcast); replicate_down was never invoked. Zero external references
to any of them. Net -93 LOC.
- weightslab/utils/tools.py + utils/__init__.py: drop DistributedCounter
(CUT-tagged in the design notes, never adopted). Net -62 LOC.
- weightslab/backend/dataloader_interface.py: keep drop_last=False on the
DistributedSampler, but document why. Padded yields are real training
events that land in the loss trajectory as real (sid, model_age, value)
encounters with distinct ages from the sample's earlier yield; the
trajectory is encounter-keyed, not per-epoch-unique-keyed, so padding is
honest rather than pollution. drop_last=True was considered but rejected
as too trivial — it'd silently drop the trailing (world-1) samples each
epoch and bias coverage downward.
Verified: scenario_discard_subset_freezes PASS at WL_DDP_BATCH=4,
WL_DDP_WORKERS=0, WL_DDP_WORLD_SIZE=2, WL_DDP_TEST_STEPS=20 — 156
populated samples, 5 discards held frozen across epoch 2, ~80% advance
on non-discarded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Rename ddp_planes → parallel_state, ddp_basic_building_blocks → parallel_primitives
Generalizes naming away from "DDP" since the primitives don't assume the
specific torch.distributed-DDP topology — they hold for FSDP / ZeRO / any
SPMD setup with a rendezvous-on-collective contract.
- weightslab/components/parallel_primitives.py (was ddp_basic_building_blocks)
- weightslab/components/parallel_state.py (was ddp_planes)
- docs/ddp_design.md (was components/ddp_design_notes.md)
Updated import sites (4): global_monitoring.py, dataframe_manager.py,
parallel_primitives.py (self-ref), tests/test_ddp_primitives.py.
Also: scenario_lr_batch_propagate threshold loosened from
`(expected + rank0_only) / 2` to `rank0_only + 1`. The old midpoint sat
right at the noise band; under drop_last=False with mid-iter batch-size
transitions the observed rate floats 13–14 samples/step, occasionally
tripping at exactly 13.75 < 14. New threshold cleanly distinguishes
"both ranks doubled" from "only rank-0 doubled" (rate ~12).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix load_state to preserve model identity on same-arch restore
ledgers.register_model(new_model) replaced the registered object, orphaning
any captured reference (e.g. `model = trainer.model` in a training loop, or
DataLoaderInterface.self.model). Post-restore the trainer trained a stale
model while pause-checks read the fresh one, so pause_at_step never fired —
caught by scenario_resume_continues_curve in the DDP suite.
Skip register-replace + guard updates when existing model has same keys
AND shapes as the saved weights; let apply-weights load in-place. Add a
regression assertion in test_06 that captures the wrapped model identity
across load_state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Rewrite docs/ddp_design.md as a concise design overview
Cut 159 → 66 lines (~58% smaller, 1426 → 588 words). Drops decision-tag
scratchpad ([DECIDED]/[OPEN]/[DEFER]/[CUT]), per-state placement debate,
wrapper-prologue code sketches, and the open-questions section — all
process residue. Keeps:
- Two-space framing (train-space vs sdk-space, kernel/user analogy).
- SPMD-with-one-privileged-rank constraint.
- Two-kinds-of-sync framing (grad reduction = off-the-shelf; async UI = WL's job).
- The loop-iteration-as-transaction insight and the train→sdk transition
as the consistency boundary.
- State × direction table.
- DOWN broadcast / UP outbox / shm mirror mechanisms with API entry points
(register_consistent_state / register_outbox + the anchor functions).
- Collective budget (~2 rendezvous/step) + instrumentation env vars.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Outbox ships per-step deltas, not full snapshots
local_df_writes / local_signal_triples emitted the WHOLE dataframe and the
WHOLE per-sample signal history every step, gathered to rank-0 each step. The
~2-collectives/step budget bounds the COUNT of rendezvous but not their bytes,
so payload scaled with N_samples x world (df) and grew unboundedly (signals) —
the real scaling wall.
Each rank now dumps only what changed since its last flush: changed dataframe
rows (vs a process-local _LAST_SENT_DF signature cache) and signal triples past
a per-(graph, exp_hash) cursor read straight off the append-only buffers. On
respawn/restore the cache resets to a one-time full resend, safe because every
merge is idempotent.
merge_df_writes seeds rank-0's current values first (existing-first) before the
per-column reducer, so a delta that omits a sample rank-0 already holds a higher
value for cannot regress MAX/UNION, while LATEST still resolves to the newest
delta. clear_registry resets the caches.
Docs + outbox comment updated to describe the delta transport and clarify the
budget governs collective count, not payload; shm section corrected to note only
the bool deny-list (`discarded`) is read at __getitem__, not user_tags.
Validated: scenario_signal_coverage_all_graphs PASS (per-sample 940/940 across
both ranks; test_ddp_primitives 3-rank reconcile PASS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Align DDP module docstrings with ddp_design (remove stale primitives)
The parallel_primitives module docstring advertised three decorator-form
primitives — AGGREGATE_UP, REPLICATE_DOWN — and a reconcile_down single-state
hook, none of which exist in the code (grep finds them only in the docstring).
The implemented + design-doc'd surface is two mechanisms: register_consistent_
state/reconcile_all (DOWN) and register_outbox/flush_outbox (UP), plus the shm
deny-list mirror and the sync_step anchor. Rewrote the docstring to match that
and dropped the ①/②/③ numbering everywhere (parallel_state plane table and the
inline anchor comments now use plain DOWN/UP, matching docs/ddp_design.md).
Comment/docstring-only; no logic change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add fast unit tests for the outbox delta optimization
Proves the delta change's correctness in ~30ms instead of via the multi-minute
YOLO scenarios: df_writes emits only changed rows; merge_df_writes seeds rank-0's
current value so a stale/lower delta can't regress a MAX column (last_seen) while
a higher one still raises it and LATEST picks the newest delta; local_signal_
triples advances a per-(graph, exp_hash) cursor and resends from 0 if the buffer
shrank under it (restore safety). No DDP spawn — ledger getters are monkeypatched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Document DDP sampler sharding is training-only (eval is latent-unsupported)
The sampler shards EVERY loader, but the per-step anchor runs only in training
context — so a sharded eval under DDP would have each rank score 1/world with no
scalar-metric aggregation (undercount). No eval runs under DDP today, so this is
latent; added a TODO(ddp-eval) guard at _get_dist_sampler so nobody adds a DDP
eval loop without first resolving the eval sharding/aggregation policy.
Comment-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cap shm deny-list index so large/sparse uids don't blow allocation
The DOWN-only shm vec is indexed directly by int(sample_id), so a single large
sparse uid (e.g. an inode-based id ~1e8) allocated a ~100MB bool array. Cap the
index at 1<<22 (~4M → ~4MB max per origin/col): ids at/above the cap skip the
shm fast-path and fall back to the sampler's pandas deny-list check, which is
the actual read site (the shm is read in the main-process sampler, not workers —
docstring corrected to say so). Dense 0..N id schemes keep the fast-path. Warns
once per origin when the cap is hit.
Adds test_ddp_shm_cap: huge id neither allocates a giant array nor breaks the
small-id fast-path; undiscard clears the cell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Restrict shm mirror to boolean deny-list columns (skip user_tags)
_propagate_to_shm ran bool(val) over every DOWN_ONLY column into the bool shm
vec, so user_tags (a list column) stored a meaningless bit that nothing reads
(the sampler only queries 'discarded'). Filter the mirror to genuinely boolean
columns via _shm_bool_eligible: bool dtype, or an object column whose non-null
cells are all bool / 0-1 scalars. user_tags still reconciles to children via the
DOWN broadcast; it just no longer gets a bogus shm array.
Test: a user_tags list column allocates no shm array while discarded still does.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Decouple DDP shard reshuffle from iterator reset (epoch → reshuffle_seq)
_generate_indices auto-advanced the DDP epoch on every fresh iterator, so a
mid-loop discard/tag — which forces an iterator reset — reshuffled the entire
per-rank shard. In a curation workflow (discard-heavy by design) the shard order
churned on every discard, and "epoch" counted iterator resets, not dataset
passes — a meaningless abstraction here.
Rename _epoch → _reshuffle_seq (it's a reshuffle generation, not a pass count)
and stop auto-advancing it. The reshuffle generation now advances ONLY on a
genuine pass-end reset (loader calls sampler.advance_reshuffle() on the
_epoch_exhausted path); the discard-invalidation reset path leaves it untouched,
so a discard re-filters the SAME permutation instead of reshuffling.
Reproducibility across resets is preserved but composed correctly: the per-rank
permutation is a pure fn of (ddp_seed, reshuffle_seq, rank, world), so
capture/restore_iteration_state now save/restore reshuffle_seq + seed; combined
with samples_yielded (offset) and the deny-list (checkpointed as a DOWN_ONLY df
column) this reproduces the exact filtered stream. Warns on a seed mismatch.
Side effect: also fixes the __len__-vs-iteration epoch off-by-one — neither
_generate_indices nor _rank_indices_snapshot advances during iteration now, so
they read the same generation.
Adds test_ddp_reshuffle_seq: re-gen without advance is stable; advance reshuffles;
restore reproduces; ranks partition disjoint+cover; seed mismatch warns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update README.md
* Guard user callbacks at collective boundaries so one can't hang the group
reconcile_all built its snapshot ({name: snap()}) and applied children states
unguarded, and flush_outbox built its payload ({name: dump()}) unguarded. Under
SPMD a callback that raises on one rank but not others crashes that rank BEFORE
(snapshot/dump) or AFTER (apply) a collective, leaving every other rank blocked
forever on the broadcast/gather.
Wrap each snap()/apply()/dump() in try/except (merge() already was), so the
collective itself is ALWAYS reached: a failed state/channel ships as None
(apply/merge already tolerate None) and the rest sync normally. Matches the DDP
module's existing swallowed-exception style (logger.debug("[tag] ... failed: %s",
exc)). Also switch parallel_primitives' logger from getLogger("weightslab.ddp")
to getLogger(__name__) to match the rest of the codebase.
Adds test_ddp_collective_resilience: a "bad" state (snapshot raises on rank 0,
apply raises on children) + a "bad" outbox (dump/merge raise) run two anchor
rounds + a barrier without hanging, and the healthy state still syncs on all
ranks. Original test_ddp_primitives still passes (no reconcile regression).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add world>2 uneven-shard coverage to the reshuffle test
n=25, world=3 exercises DistributedSampler's drop_last=False padding: each shard
pads to length 9 and the union still covers the whole universe. Complements the
even world=2 disjoint+cover case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove the redundant shm deny-list mirror
The shared-memory DOWN_ONLY mirror was never load-bearing: the deny-list is
enforced sampler-side (a discarded sample is never yielded), the sampler's pandas
cache already refreshes on the deny-list revision bump (so a live discard is
reflected within one index), and a sample already in a worker's prefetch queue is
dropped by iterator invalidation tearing the workers down — NOT by shm (shm
filtered at yield time, before the discard existed, so it never had power over
queued samples). No test depended on it; nothing read it except the main-process
sampler.
Remove _propagate_to_shm / _ensure_shm_capacity / _shm_bool_eligible /
is_in_down_only_shm + the shm fields and the sampler's shm fast-path (~190 LOC,
plus the ctypes/multiprocessing/os imports). Replace the invalidation gate — which
must stay gated on an ACTUAL value change so rank-1+ don't respawn workers every
step under DDP — with a pandas before/after diff (_down_only_changed) computed
before the upsert merges. Update docs (design + comments) to describe sampler-side
enforcement + invalidation, dropping the inaccurate "workers fork-read the shm"
story. Supersedes the shm-cap (task #3) and user_tags-shm (task #4) fixes.
Drops test_ddp_shm_cap; adds test_ddp_down_only_change covering the gate,
including the DDP no-respawn invariant (re-applying the same snapshot → no change).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* DDP: rebalance-on-discard sharding, drop per-flush image decode, anchor/delta cleanups
- Sampler: replace shard-then-filter with a filter->pad->stripe REBALANCE so live
shards are always equal length across ranks — fixes the empty-shard deadlock
(scenario_empty_shard_starvation) by construction; order-preserving + deterministic
(not a reshuffle). persistent_workers=True makes a discard/undiscard rebuild a
cheap reset (reuse workers, drop stale prefetch) instead of a fork+reinit.
drop_last=False under DDP keeps a tiny live set training so age still advances.
- dataframe_manager: stop the storage-time bbox->seg get_mask conversion — it
re-decoded a full image per signal flush (~13% of rank-0 wall) and silently
corrupted detection prediction_raw; mask rendering stays available on-demand via
get_prediction_mask. Remove the now-dead normalize/_is_array_column/_get_loader.
- Anchor split DOWN(__enter__)/UP(__exit__); outbox ships per-rank deltas via a
writer dirty-set; remove post-hoc active-sample masking from the model wrapper.
- Test suite: compute epoch_steps from WL_DDP_BATCH (was config's mono batch=4 while
the loader trained at 16 -> every "epoch" silently covered ~4 passes); add
scenario_progressive_resample (shrink->grow) + per-phase timing + a WL_DDP_SELFSPY
self-profiling hook.
- docs/ddp_design.md: document rebalance-not-reshuffle + persistent-worker reset.
Note: 2 coverage assertions (epoch_then_pause populated~=shard, progressive_resample
advance%) still assume the old over-training and read the per-sample gather before it
fully settles for a now-correct single epoch; recalibration deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* DDP: make WL_DDP_BATCH effective, event-based pause (no sleep), resumable suite
- yolo_pipeline: push WL_DDP_BATCH into the in-memory cfg, not just the DataLoader
ctor. _sync_batch_size_from_ledger re-applies the ledger's batch every iteration,
so without this the loader silently reverted to config's mono batch=4 and the env
was dead — the suite trained at 4 while epoch_steps assumed 16 (¼-epoch coverage,
which failed epoch_then_pause / progressive_resample after the epoch-math change).
Now img=16 for real: full single-epoch coverage + genuine ~23% speedup (4× fewer
steps, same work). config.yaml on disk untouched (mono unaffected).
- parallel_primitives / global_monitoring: kill the 20ms busy-sleep pause-spin in
sync_step. Rank 0 blocks on the pause_controller resume Event (new wait_for_resume);
rank-1+ block inside the next reconcile_all broadcast. Neither spins (gloo socket-
waits; NCCL would busy-spin — noted). Bounded timeout kept for SIGINT/SIGTERM
responsiveness, not polling.
- ddp_test_suite: WL_DDP_SKIP (comma-sep substrings) so a killed run resumes by
skipping already-passed scenarios.
Full 14-scenario suite green at batch-16 (incl. empty_shard_starvation,
progressive_resample, and the event-based pause via epoch_then_pause).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* DDP: trim verbose comments (no behavior change)
Condense the "novel"-length comments added during the DDP work to 1-2 tight
sentences each — worst offender was dataloader_interface (rebalance/reshuffle,
__iter__, __len__, persistent_workers, _reset_iterator). Key invariants kept;
redundant restatements dropped. ~80 fewer comment lines. Code unchanged
(reshuffle unit test still green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* DDP: delta the DOWN reconcile + vectorize the UP merge (anchor 2x faster, no O(N)/step)
Two hidden O(N)-per-step costs in the cross-rank anchor, both removed:
- DOWN reconcile (rank0_df_down_state) rebuilt+pickled+broadcast {col:{sid:val}} for
ALL non-null DOWN_ONLY cells every step (discarded defaults False=non-null, so all
touched samples rode) — O(N). Now a DELTA: rank-0 ships only the sample-ids whose
DOWN_ONLY changed since the last reconcile (drain_down_delta dirty-set, populated in
upsert_df on a real DOWN diff), with one full snapshot on first reconcile / post-
restore (mark_down_full_resend, hooked in _load_existing_data) so children converge.
N-sweep: full build+pickle was 1.5ms@1k / 119ms@100k / 619ms@500k; delta ~0 when
unchanged, O(discards) otherwise.
- DOWN_ONLY trimmed to {"discarded"} — "user_tags" was never a real column (it's
"tag"), tags are rank-0 UI state (tag->label override is vestigial), and tag queries
gather signals UP + filter on rank-0, so nothing tag-shaped needs to reach a sampler.
- UP merge (merge_df_writes): replaced per-column groupby.apply(python reducer) with one
vectorized groupby.agg({col:'max'/'last'}) — _r_max/_r_latest map exactly to skipna
max/last; policy_for only yields MAX/LATEST here (UNION is tags, DOWN-filtered). And
_rank0_existing_seed stopped copying the WHOLE dataframe every flush — it now indexes
just the delta's ~batch rows.
Anchor 168 -> 153 (DOWN delta) -> 78.6 ms/step (merge), ~2x. Validated:
discard_subset_freezes, progressive_resample, break_by_slice, curate_lifecycle,
signal_coverage all PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add ddp_ablation.py — 3-mode WL SDK overhead harness (time/mem/IO/bytes per rank)
WL_ABLATE=ul|wlimport|wl on 2 gloo ranks, WL_ABLATE_STEPS configurable. Per mode:
per-section ms/step, grad bytes/step, per-rank RSS + /proc/self/io (rchar/wchar =
syscall bytes incl. gloo sockets; read_bytes/write_bytes = actual disk), and the WL
df RAM + H5 store sizes + H5 flush config.
256-step result: WL time tax +247ms/+17.6% (criterions+log +148ms = save_signals +
NMS decode-for-logging is the biggest, anchor +89ms, loader/wrapper +28ms); RAM +108MB
import-idle + ~40MB active; df/H5 tiny. I/O surfaced that rank-1 redundantly persists
~6MB to H5 (should be outbox-only — rank-0 is authoritative).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Relocate DDP integration suite + perf/ablation into tests/
PR #185 review: tests don't belong in the usecase dir. Move ddp_test_suite.py,
ddp_ablation.py, aggregate_wl_ownership.py and the report driver out of
examples/PyTorch/ws-detection/src into tests/integrations/ultralytics/ddp/.
One god-script (run_ddp_report.sh) with modes info/scenarios/ablation/profile
emits a single report: perf counters, per-scenario times, and the wl-ulmanual
ablation delta. Scripts path-bootstrap back to the usecase src so
yolo_pipeline / utils.* / config.yaml resolve. Added README + .gitignore.
Locally-run (needs GPU + dataset) — not a CI unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* PR #185: all_reduce_scalar(avg) + mono/DDP usecase README
- utils.tools: add all_reduce_scalar(value, reduction="sum"|"avg"); avg = sum/world
since gloo has no ReduceOp.AVG. Keep all_reduce_sum_scalar as a back-compat wrapper.
- examples/.../ws-detection/README.md: document the mono (main.py) and DDP (main_ddp.py)
usecases, how to run, and the single-GPU gloo DDP simulation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* PR #185: exclude is_training/pause_at_step/root_log_dir from the saved HP snapshot
get_HP_snapshot dumped the whole hp dict, so a restore's register_hyperparams(
saved_config) resurrected experiment STATE (notably is_training=True) — the bug the
post-restore force in experiment_service worked around. Strip the same state-only keys
already excluded from the experiment hash, on a copy (never mutate the ledger). The
post-restore pause stays, now as the intentional "user drives the next cycle", not a
workaround.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix ruff F841: drop unused group_ids / pre_restore_max_plot_age / ctx
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Alexandru Rotaru <rotarualexandruandrei94@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…files, heading levels
- PRs now formatted as [#N](url) title — date with author GitHub profile links
- Contributors section links to github.com/{login} (from PR authors)
- Commits capped at 25 most recent non-merge commits
- Title: ## **Weightslab** (no version, ## level)
- Sections demoted to ### level
- Removed separator between title and LinkedIn/Graybx links
- Dev release routes PRs from --base dev, main from --base main
- Doc build gated on main-branch check (not just tag pattern)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…, bb_B], batch_idx=[batch_idA, B, ..etc]
* Add to detection usecase dump history and custom signal labelling * upgrade documentation with new functions and examples * add and fix utests * refactor the logger and add instances history and queries functions, with a user wl.write_history function * add df writing for user during exp * fix code quality issues
The `weightslab` CLI resolves its assets relative to the installed package (ui_docker_bridge.py). With docker/ and examples/ living at the repo root as non-package siblings, a wheel install shipped only the weightslab package, so `weightslab ui launch` and `weightslab start example` failed to find their files after `pip install` (they only worked in an editable checkout). - Move docker/ -> weightslab/docker/ and examples/ -> weightslab/examples/. - Move baseline_models/ -> weightslab/examples/utils/baseline_models/ and update its import path everywhere (package __main__ demos, examples, tests). - ui_docker_bridge: resolve paths via `Path(__file__).parent` (siblings now live inside the package) instead of `.parent.parent`. - pyproject: include-package-data + package-data globs for docker/** and examples/**, excluding the gitignored .env and pycache. Verified the built wheel bundles the compose stack, envoy/nginx configs, shell scripts, example projects and baseline_models, and omits .env. All unit tests pass (1104 passed, 95 skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`num_classes < label_raw.max()` raised "zero-size array to reduction operation maximum which has no identity" for detection samples with no objects (empty label array), aborting every sample row and leaving the grid empty (seen in the yolo-detection playwright workflow). Guard the sanity-check warning with to_numpy_safe + a size check, so empty arrays, dicts and None labels are skipped instead of crashing — consistent with how label_arr is already guarded a few lines above. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a weightslab_version field to GetLatestLoggerDataResponse and populate it (once, cached) in the ExperimentServiceServicer facade. The studio polls GetLatestLoggerData continuously, so the backend version is available to the UI without a dedicated RPC. Best-effort: a version lookup failure never breaks the poll. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Your Name <you@example.com>
Rename all bundled example directories from the ws- prefix to wl- (10 examples across Lightning/PyTorch/Ultralytics/Usecases) and update every reference in docs (.rst), READMEs, the ui_docker_bridge _EXAMPLES map, and test harnesses. The ws-ultralytics_yolo dataset path (external kitchen repo) and unrelated tokens are left untouched. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add `weightslab tunnel` command + Colab classification notebook Enable training on a remote machine (e.g. Google Colab, no Docker) while watching it live in Weights Studio running locally. - weightslab/tunnel.py: raw-TCP forwarder that exposes a remote gRPC backend at a local port (default 50051, the port the bundled Envoy upstream dials), so the UI reaches it unchanged. No protocol parsing so HTTP/2 gRPC frames pass through untouched; requires a raw-TCP remote tunnel + plaintext backend. - ui_docker_bridge.py: wire `weightslab tunnel [ENDPOINT]` subcommand. ENDPOINT optional, falls back to $WEIGHTSLAB_TUNNEL_ENDPOINT; configurable --listen-port / --listen-host (auto per-OS) / --remote-port. - examples/Notebooks/PyTorch/ws-classification.ipynb: Colab notebook (Ultralytics style) — train MNIST with WeightsLab instrumentation, inspect highest-loss samples, and optionally expose the backend via ngrok for the local UI. - docs/user_commands.rst: document the tunnel command. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Use bore (no-signup TCP tunnel) instead of ngrok in Colab flow ngrok now requires a credit card for TCP endpoints on the free tier (ERR_NGROK_8013), which is friction for a "try it now" notebook. Switch the Colab example and docs to `bore` (github.com/ekzhang/bore) — a raw-TCP tunnel with a free public relay (bore.pub), no signup/account/card. The `weightslab tunnel` command is unchanged (still takes host:port); only the recommended exposer switches. Notebook §6 downloads the bore binary and prints the exact `weightslab tunnel bore.pub:<port>` to run locally. Docs, CLI help, and tunnel.py examples updated to match; ngrok kept as a noted alternative. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix imports cls usecase * Add serving_bore to wl.serve; verbose flag for exports; quieter flush logs - wl.serve(serving_bore=True[, bore_port]): expose the gRPC backend over a bore raw-TCP relay from inside the training script (downloads bore, opens the tunnel, prints the `weightslab tunnel <endpoint>` command). serve() now returns the public endpoint. Implemented via tunnel.serve_bore + platform-aware bore download helpers. - write_history/write_dataframe: new verbose=False param — their per-call progress lines are now DEBUG by default, INFO only when verbose=True, so periodic exports don't spam the training log. - dataframe_manager flush/flush_if_needed_nonblocking: demote per-flush INFO lines to DEBUG (they fired on every buffer flush). - Notebook: use wl.serve(..., serving_bore=EXPOSE_UI) instead of a manual bore cell; §6 is now just the EXPOSE_UI toggle. Docs note the serving_bore shortcut. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Warn in notebook/Colab when serving_grpc without serving_bore The UI runs on the user's own machine; in a notebook/Colab a bare serving_grpc backend is unreachable without a tunnel. Detect notebook/Colab (_running_in_notebook) and, when serving_grpc=True and serving_bore=False, warn the user to pass serving_bore=True. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Final fix * rm max samples * Notebooks: config-dict refactor + example/usecase notebooks; wl.signal min_step - Add min_step to @wl.signal: dynamic (subscribed) signals are skipped while current_step < min_step (default 0). Gates the subscribe execution path; documented in user_functions.rst. - Classification notebook: remove all emoji, hoist every knob into one commented `config` dict (config.yaml style), add PyPI version/downloads/python-version badges under Setup. - New Colab notebooks for the remaining PyTorch examples (segmentation, detection, clustering, generation) via a clone-and-run pattern (they need the repo's local utils/ + bundled/downloaded data), same style: no emoji, config from config.yaml, badges, bore live-UI tunnel. - New usecase notebook Notebooks/Usecases/ws-segmentation-loss-shapes.ipynb: segmentation trained in-kernel with a user-defined @wl.signal loss-shape classifier subscribing to the per-sample loss, demonstrating min_step=505. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add bottom-right Open-in-Colab badge to example pages Each example page with a notebook (5 PyTorch examples + the loss-shape usecase) now shows an "Open in Colab" badge at the bottom-right, linking to its notebook under weightslab/examples/Notebooks (on main). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add Open-in-Colab badge to the examples gallery cards The examples gallery (examples-gallery.js) now shows a bottom-right "Open in Colab" badge on each card that has a notebook (5 PyTorch examples + the loss-shape usecase). Cards are wrapped so the badge (a sibling <a>, since it can't nest in the card link) and the filter show/hide stay in sync. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: gallery Colab badge -> top-right, logo only Per feedback: on the examples gallery cards, show just the Colab logo icon (no text) pinned top-right, instead of the full bottom-right badge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * notebooks: use dark WeightsLab banner Swap the banner image in all example/usecase notebooks from weightslab-banner-light.png to weightslab-banner-dark.png. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Proxy.__getitem__ returned the raw value while Proxy.get() returned a live
_ValueProxy. Mirror .get() for mapping subscript access so both
`b.get('test')` and `b['test']` return a tracking proxy for simple values.
Scoped to mapping access (object has .get + keys); list/slice indexing and
list/dict/callable values are returned raw as before. str/torch.device stay
plain via _plain_get_value.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Extend agent: model introspection/management, column-write safety, chained intents [force ci]
Data-manipulation agent improvements:
- Chained multi-step intents now resolve correctly. A later step's
conditions can reference a tag/column an earlier transform step in the
same intent is about to create (e.g. "Tag as 'Disabled' samples with
loss > 0.3 and loss_shape 'plateaued'. Then discard these data.").
Pending target columns are registered during resolution so column
matching succeeds instead of silently dropping the filter.
- Column-write safety invariant: the agent may create NEW derived columns
and update its own control columns (discarded, tag:*), but may NEVER
overwrite values in an existing data column (signals/labels/sample_id/
origin/...). Enforced in the planner (_coerce_protected_transform_intent)
and again at the executor (df.modify safety gate). Overwrite requests are
redirected to a new derived column.
- Model introspection: new model_info kind answers architecture questions
from a live layer table (layer id/type, neuron counts, frozen state),
e.g. "which layer has more than 2000 neurons?", "which layers are
frozen?", "show me the complete model details".
- Model management: new model_action kind freezes/resets layers or specific
neurons, reusing the same ManipulateWeights architecture path as the
Weights Studio grid controls. (No unfreeze verb; reset undoes a freeze.)
- Prompt (intent_prompt.py): added MODEL CONTEXT section, COLUMN WRITE
SAFETY rule, model_info/model_action strategy rows and schema, plus
examples Ex24-Ex31 (chained tag+discard, derive column, refuse overwrite,
model queries, freeze/reset).
- Wiring: ExperimentService injects model_service into DataService so the
agent can drive architecture ops in-process; ModelService.ManipulateWeights
tolerates context=None for in-process callers.
Tests: new test_agent_model_and_safety_unit.py (column safety, chained
tag+discard, model schema/handlers, executor dispatch) and prompt-format
guards in test_agent_prompt_unit.py.
Docs: new docs/agent.rst (capabilities, safety invariants, UI/CLI/config
initialization, usage) linked from index and data_exploration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add temp-column cleanup, unfreeze support, and prompt illustrations [force ci]
- Temporary/scratch columns: a transform step can be marked is_temporary
when it only exists to help compute a LATER step's result within the
same request (e.g. two helper tags combined into a final one, as in the
goldset hard/easy example). These are auto-dropped (df.drop_column, both
from the in-memory view and the ledger) right after the request finishes,
so only the column the user actually asked for persists.
- Unfreeze: model_action_name now accepts "unfreeze" alongside freeze/reset.
Freezing a neuron TOGGLES its learning rate (new_lr = 1.0 - current_lr),
so no new backend primitive was needed — unfreeze re-applies the same
freeze op, but is always constrained to layers/neurons that are ALREADY
frozen (via the new per-neuron model_neurons_df), so it can never
accidentally freeze something that wasn't frozen yet. Supports whole-model
("unfreeze everything"), layer-scoped, and neuron-scoped requests.
- intent_prompt.py: documented is_temporary and unfreeze, updated Ex22 to
mark its two helper tags temporary, added Ex22b (generic temp-column
derivation) and Ex32-34 (unfreeze layer/neurons/everything).
- docs/agent.rst: added a full "Example prompts by task" section (sorting/
filtering, tagging/discarding, deriving columns, data questions, model
introspection, model management) and updated the freeze/reset/unfreeze
and column-safety explanations accordingly.
- Tests: temp-column cleanup ordering, executor df.drop_column dispatch,
and unfreeze safety (frozen-only layer/neuron selection, no-op on
already-unfrozen targets) in test_agent_model_and_safety_unit.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix split-value guessing and same-column OR-as-AND bug in agent [force ci]
- Origin/split value resolution is now deterministic instead of prompt-only
regex heuristics: _resolve_categorical_value() maps a user's natural word
("test", "inference", "test data") to whatever the dataset's ACTUAL origin
value is (e.g. "test_split", "test_loader", "inf_split", "holdout") via
exact match -> substring -> train/val/test/inference/holdout family
matching -> generic token overlap, hooked into _build_python_mask's literal
casting for any categorical column. The prompt now tells the LLM to just
write the user's own wording and trust this resolution, instead of trying
to guess the exact stored spelling itself.
- Fixed a real reported bug: "keep validation or test samples" was planned
as two separate `origin == 'val'` / `origin == 'test'` conditions, which
_build_python_mask always ANDs together into an impossible, always-empty
filter (a column can't equal two different literals at once). Added
_coalesce_same_column_equality(), called at the top of _build_python_mask,
which deterministically merges multiple equality conditions on the same
column into a single `in` (OR) condition — independent of how the LLM (or
a multi-turn correction) phrased it. Also strengthened the prompt with an
explicit AND-only condition-combination rule, guidance on using `in` for
same-column OR and `analysis_expression` with `|` for cross-column OR, and
worked examples (Ex35/Ex36) matching the exact reported scenario.
- docs/agent.rst: updated "using the agent effectively" and the example
prompt table to reflect both fixes.
- Tests: TestSplitValueResolution and TestSameColumnEqualityCoalescing in
test_agent_model_and_safety_unit.py, including an end-to-end reproduction
of the reported bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix numeric coercion, and/or crashes, origin/tag resolution gaps, UI refresh [force ci]
Fixes for real bugs surfaced by live testing against a running experiment
("weightslab start example --seg") and a real OpenRouter model:
- Ordering comparisons (>,<,>=,<=,between) against a numeric literal (incl.
scientific notation like "2e-4") now coerce correctly even when a column's
dtype was misclassified as categorical/object (common for signal columns),
instead of crashing with "'>' not supported between float and str".
- Two equality conditions on the same column ("keep validation or test
samples") are deterministically coalesced into a single `in` (OR) instead
of silently AND-ing into an impossible, always-empty filter.
- Generated `and`/`or` between pandas boolean Series (a very common LLM
mistake) is rewritten via AST to bitwise `&`/`|` before evaluation, fixing
"truth value of a Series is ambiguous" crashes in df.apply_mask/df.modify/
df.analyze.
- Origin/split literal resolution ("train" -> "train_loader") now also
applies to free-form generated code (transform_code/analysis_expression),
not just structured Condition objects -- this was the reason "Tag train
samples..." silently tagged nothing. The origin schema line now shows the
actual stored values with an explicit substring-matching rule instead of
either hiding them or leaving the LLM to guess blindly.
- Generic words (e.g. "loss") no longer fuzzy-resolve to a `tag:*` control
column just because the tag's name happens to contain that word as a
substring (e.g. tag:high_train_loss) -- this caused a "numpy boolean
subtract" crash when arithmetic was attempted on the wrong (boolean)
column. Column tokenization now also splits on ':' so an explicit "tag ..."
mention still resolves correctly.
- df.analyze now shares the same eval context as df.modify (exposing
origin/sample_id when they live in the index, plus backward-compat for
`df['origin']`), fixing "Analysis Error: 'origin'" on datasets where the
split is an index level. df.apply_mask gets the same treatment.
- A response the LLM can't be parsed into a plan (garbled JSON, or long
prose with none at all) is now always wrapped as a clarify-style
out_of_scope reply using the model's own text, instead of a hard
"Internal Agent Error: Failed to generate a plan."
- A request that both mutates the schema (creates/drops a column) AND asks
an analysis question in the same turn is now always reported as a
data-changing (FILTER) intent, never downgraded to ANALYSIS -- Weights
Studio only refreshes the grid/column list on FILTER intent, so a new
column could otherwise silently never appear in the UI.
- Fixed two `_pull_into_all_data_view_df() or pd.DataFrame()` spots that
crashed with "truth value of a DataFrame is ambiguous" whenever the pull
returned a real (non-empty) DataFrame -- hit on every "reset the view"
request against live data.
Also adds an opt-in live-LLM evaluation harness
(test_agent_live_prompt_evaluation.py) that runs a battery of realistic
prompts -- including every scenario above -- against a real OpenRouter model
when UTEST_AGENT_PROMPT_EVALUATION is set, verifying the resulting dataframe
state; skipped (with a log note) otherwise. A small always-on sanity check
for the harness's own fixture/runner logic runs regardless.
Tests: ~50 new unit tests across test_agent_model_and_safety_unit.py
covering every fix above with direct reproductions of the reported bugs.
Full trainer/gRPC/data/modules suite: 362 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Diagnose silent model-lookup failures, test conversation memory [force ci]
- _setup_model_schema() had two silent give-up paths with zero logging:
when components.get('model') returned None, and a broad except that only
logged the exception message (no traceback). Both now log at INFO/WARNING
with enough detail to diagnose "model is registered but the agent still
says it isn't" reports -- the model-not-found case now logs the actual
ledger-registered model names, and the broad except now includes a full
traceback.
- Added TestConversationHistory (test_agent_model_and_safety_unit.py):
pins down the exact cross-turn memory contract -- self.history
accumulates "User: <text>"/"Action: N ops executed" pairs, only the last
5 entries are fed into the next prompt, and a failed query never
corrupts history with a partial entry.
- Added a live two-turn memory test
(test_cross_turn_memory_followup_references_prior_tag): tags samples in
one query() call, then asks a vague follow-up ("now discard those
samples") in a separate call on the same agent, verifying the model
correctly resolves it from the trimmed raw-text history rather than
needing the tag re-stated.
- docs/agent.rst: documents the actual (limited) conversation-memory
contract, how to run the non-live agent test suite, and points to the
new tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add one live test per documented agent.rst example prompt [force ci]
Adds TestAgentRstDocumentedPrompts (test_agent_live_prompt_evaluation.py):
one independently-runnable test per example prompt across every table in
docs/agent.rst's "Example prompts by task" section (28 prompts: sorting/
filtering, tagging/discarding, deriving columns, data questions, model
introspection, model management), using the same wording as the docs so
each documented promise has its own regression test that can be re-run in
isolation (pytest --lf, or -k test_doc_<name>) instead of the whole suite.
- Extended the shared fixture with val_loss/confidence columns so the docs'
"loss_ratio"/"combined_score" derived-column examples reference columns
that actually exist (previously only train_loss/signals existed).
- "Untag 'goldset' on validation samples" needed pre-existing tagged data
to meaningfully untag; gets its own isolated fixture+agent (seeded with
tag:goldset across all three splits) rather than polluting the shared
fixture, which would have collided with the separate "add tag 'goldset'
to 50%" example.
- Model introspection/management prompts assert the correct graceful
"no model registered" behavior (this harness has no live model), which
still verifies the LLM classifies them as model_info/model_action rather
than misrouting to a dataframe operation.
- "average loss of the 10 hardest samples" directly guards against a
previously reported "nan" result.
docs/agent.rst: documents the two live test classes and the re-run-only-
failures workflow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix stale agent-availability reporting vs real 401 rejection [force ci]
Reported bug: CheckAgentHealth said the agent was available ("Ready to
help you"), but a real query immediately failed with "Agent not connected:
... (401 Unauthorized)". Root cause: is_available() only checks that a
provider CLIENT OBJECT was constructed, not that its credentials were ever
confirmed to work. That's fine for keys entered via the /init UI flow
(which already runs a live connectivity probe before reporting success),
but any key loaded from agent_config.yaml/OPENROUTER_API_KEY at backend
startup skipped verification entirely -- _setup_providers() just builds a
ChatOpenAI client from whatever key string it's given, so is_available()
would report True indefinitely regardless of whether the key actually
works, until a real query happened to fail.
Two fixes:
- _verify_startup_providers() (called once, right after _setup_providers()
in __init__) probes a construction-time OpenRouter chain exactly like
/init already does, and clears it if the probe fails -- a bad startup key
is now caught immediately instead of silently reporting "available".
- When a live query's failure is detected as an auth error (401), the
cached chain_openrouter is now invalidated (set to None) so the very next
CheckAgentHealth call reflects reality, instead of staying stale until
the process restarts. Non-auth failures (timeouts, etc.) leave the
connection untouched, since those don't indicate broken credentials.
Deliberately does NOT touch _setup_providers() itself or the
initialize_with_cloud_key()/change_model() call paths, which already run
their own explicit post-setup connectivity check -- avoids double-probing
and keeps their existing error messages/tests intact.
Tests: TestStartupProviderVerification (bad/good startup key, no-chain
no-op, 401-invalidates-connection, non-auth-failure-leaves-connection-
alone). Full trainer/gRPC/data/modules suite: 398 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix default path in seg example
* Fix checkhealthAgent fct and model wise agent interactions
* Strengthen weak live-test assertions with real expected values [force ci]
Three analysis-question tests and one derived-column test only checked
"a result was produced" rather than that the result was CORRECT:
- test_doc_average_loss / test_doc_average_loss_of_10_hardest_samples /
test_doc_samples_per_origin only asserted "Analysis Result:" appeared in
the message, never parsing/checking the actual returned number.
- test_doc_create_is_outlier_column only checked the count of flagged rows
was small, not which specific rows were flagged.
Fixed:
- Added _extract_analysis_number(), a regex helper that pulls the numeric
value out of an "Analysis Result: ..." message (with a negative lookbehind
so it can't mistake digits embedded in a type name, e.g. the "64" in
"np.float64(...)", for the actual value). average_loss and
average_loss_of_10_hardest_samples now compare the parsed value against
the real computed mean (accepting either train_loss/val_loss and either
top/bottom-10 interpretation, since "loss"/"hardest" are genuinely
ambiguous wording, but the returned number must match one of the valid
interpretations, not just be "some number").
- samples_per_origin now checks the real counts (8 train_loader, 4 each for
val/test_loader) appear as whole numbers in the message.
- create_is_outlier_column now does an exact per-row boolean comparison
against the real mean+2*std mask. This is robust to the train_loss/
val_loss ambiguity because val_loss is an affine transform of train_loss,
which preserves z-scores exactly -- the outlier mask is identical either
way.
- Added an always-on (no API key needed) unit test for
_extract_analysis_number covering plain floats, numpy-repr-wrapped
values, scientific notation, and non-matches.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reconcile doc/test prompt wording, add compound CLI example test [force ci]
Audited every quoted example prompt across docs/agent.rst against the live
test suite and found wording drift in several places outside the canonical
"Example prompts by task" tables (which were already 100% covered
verbatim):
- Top summary table used different phrasing than the tested versions for
three prompts ("Sort by train loss descending" vs tested "...highest
first"; a different tag/threshold example; "Which layers are frozen?"
missing "currently"). Reconciled to match the tested wording exactly.
- The /init walkthrough example used "above 1.5" instead of the tested
"greater than 1.5". Reconciled.
- The is_outlier test used "2 std" while both doc occurrences say "2·std"
(middle dot). Fixed the test to match.
- A prose reference to the AND/OR bug fix said "Keep validation or test
samples" (missing "only") vs the tested "Keep only...". Reconciled.
- The CLI section's compound example ("discard all samples with loss > 5
and tag them as hard_examples") combines two different action types in
one instruction and was never tested as a single combined prompt (only
each half separately) -- added test_doc_cli_discard_and_tag_combined.
Every prompt string that appears in docs/agent.rst now has a corresponding
test using the identical wording.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix default agent to gemini flash
* Fix banner offset bug
* Add bundled CLI command with Weightslab
* Optimize agent time, add modeling queries (freeze, reset, model info); and related utests
* Fix default agent to gemini flash
* Fix banner offset bug
* Add bundled CLI command with Weightslab
* [force ci] Optimize agent time, change default OpenR model, add modeling queries (freeze, reset, model info); and related utests
* Update WeightsLab banner
* Update agent prompts examples
* Include save data state and save model state and archi (optionnal) actions
* Allow agent to load exp. state / model weights at step x
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, example (#244) * examples: add idiomatic per-sample signals MNIST example A single self-contained PyTorch example showing the three ways to declare per-sample signals in WeightsLab, plus an honest overhead breakdown: - base signals (from logits) pushed with wl.save_signals - a live derived @wl.signal (loss normalized by a running mean) - end-of-run derived signals from each sample's full loss trajectory: a six-class shape as a text categorical tag (100% coverage) + loss_cv / loss_drop, written back by sample_id The run reports a per-step cost breakdown (baseline / loss-logging / signal compute / persist) vs plain PyTorch, CUDA-synced, showing the signal math is cheap and persistence dominates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * examples: tune ledger_flush_max_rows to cut per-step signal overhead The ledger default (flush_max_rows=100) flushes every ~1.5 steps at batch 64, which dominates the per-step cost. A sweep (batch 64, MNIST, single GPU): flush=4 +2086% flush=512 +440% flush=64 +1577% flush=4096 +210% flush=100 +631% flush=8192 +200% (H5 on/off no longer matters) Setting flush_max_rows well above the batch size (here 8192) flushes only a few times per epoch and cuts the measured overhead ~6x (185 -> 32 ms/step) with the report unchanged (100% coverage). A value <= batch size is a landmine (it flushes mid-batch). Flush frequency governs both the persist and the watched-loss logging cost; H5-persistence on/off is irrelevant once flush is high. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(signals): vectorized @wl.signal(batched=True) path The dynamic signal executor calls func(ctx) once per sample in a Python loop, allocating a SignalContext per sample. For signals that read the ledger this is catastrophic: each of the B calls does its own query_sample_history (lock + flush + scan). Add an opt-in batched contract: - BatchSignalContext carries the whole batch as arrays (sample_ids, subscribed_values) and exposes .history(name) as ONE query_per_sample for the batch (not one per sample). - executor branches on meta['batched']: build one context, call func once, expect a length-B array back. - fully backward compatible; the per-sample path is unchanged. Measured (batch 64, MNIST, single GPU, flush 8192): L1 compute-only (subscribed_value): 31 -> 18 ms/step (-43%) L2 history read (per-sample query): 254 -> 63 ms/step (-75%, 4x) subscriber invocations: per-sample 1728 -> batched 27 (one per step) values identical to the per-sample path (max abs diff 0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(signals): multi-input signals via ctx.latest() / bctx.latest() A dynamic signal subscribes to ONE trigger metric, but can now ingest any number of OTHER signals by reading their current value from the ledger: @wl.signal(name="sig/hardness", subscribe_to="train/loss_sample", batched=True) def hardness(bctx): return bctx.subscribed_values * bctx.latest("sig/entropy") \ * (1.0 - bctx.latest("sig/confidence")) - BatchSignalContext.latest(name): most recent value of another signal for the whole batch in ONE query -> (B,) array. - SignalContext.latest(name): same for a single sample. Requirements: the ingested signals must be logged (a watched metric, another @wl.signal, or save_signals(..., log=True)) and written earlier in the step than the trigger, so latest() sees this step's values. Verified: hardness == loss*entropy*(1-confidence) to 2e-7 over 640 rows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(signals): enforce fresh ingests (StaleSignalError), stop swallowing errors A signal that ingests other signals (subscribes to one trigger, reads others via ctx.latest) could silently read STALE or missing inputs if they weren't logged, or were written after the trigger. Make that a loud error instead: - ctx.step / bctx.step carry the trigger's step. - ctx.latest(name, require_fresh=True) / bctx.latest(...) raise StaleSignalError unless the ingested signal has a value at the current step. - @wl.signal(..., ingests=["sig/a","sig/b"]) declares inputs; the executor validates their freshness for the whole batch BEFORE calling the signal. - StaleSignalError now propagates out of the subscriber dispatch (previously ALL subscriber exceptions were debug-logged and swallowed — silent failures). Verified: correct order (inputs logged before trigger) runs; wrong order raises StaleSignalError naming the signal, step, and affected samples. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(signals): reactive inputs=[...] — order-independent multi-input + chaining Unify subscribe_to and ingests into one dependency declaration. A signal with inputs=[a, b, c] fires when ALL its inputs are present at a step, regardless of the order they were logged — eliminating the ordering footgun instead of just detecting it. Fired signals are themselves inputs, so signals chain. - _react_dependents(): iterative, dedup'd (fires each signal at most once/step), reads inputs fresh from the logger, persists results (log=True), and cascades via a work-queue (no re-entrant dispatch). - wired into wrappered_fwd (watched metric) and save_signals(log=True) so ANY logged signal can satisfy a dependent — that's what makes it order-independent. - subscribe_to keeps the legacy dispatch (handles non-per-sample metrics); the two paths coexist without double-firing (inputs= vs subscribe_to). Verified (batch 64): order-independent (entropy logged AFTER loss still fires the loss*entropy signal), exactly-once/step, chaining (B<-A<-entropy), values correct, and the existing subscribe_to signal still fires. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(signals): in-core signal-worker thread (ledger_signal_worker) Move reactive signal computation off the training thread. When hyperparam ledger_signal_worker=True, the reactive dispatch is handed to a background worker (WL-Signal-Worker) instead of running inline; the train thread just detaches + enqueues (seed_names, ids, step). The seed signals are logged synchronously, so the worker only defers the derived compute + persistence. - _dispatch_or_enqueue() at both injection sites; drain_signals() (exported) joins the queue; write_dataframe() auto-drains so the report is complete. - _gather_inputs_fresh selects the value AT the job's step (not "latest"), so a lagging worker still gathers that step's inputs. - save_signals now honors an explicit step= (as its docstring says) instead of letting the live model age override it — required so reactive backfill logs derived values at the job's step; otherwise chained signals can't find them. Verified worker on vs off: exactly-once/step, order-independent, chaining (B<-A<-entropy) all identical; drain makes the report complete. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(signals): step-scoped query cache + value-at-step gather; cycle warning Reactive-signal overhead was dominated by redundant, growing ledger reads. Logger query cache (backend/logger.py): - Memoize query_per_sample via functools.lru_cache keyed (signal, ids, hash, version[signal]); _stage_sample_row bumps the per-signal version (per-signal, not global, so persisting a derived signal doesn't invalidate the loss its siblings are still reading). - Step-scoped: clear both caches when the training step advances. The loader reshuffles ids and the version bumps every step, so a key never recurs across steps — cross-step entries are pure dead weight. Bounds the cache to one step (currsize ~6 vs ~1600 accumulating), killing the churn while keeping the intra-step reuse (10 signals sharing the loss -> 1 query). - query_per_sample_at_step(name, ids, step): O(batch) value-at-step read (WHERE step=? in-engine) behind its own step-cache. Core (src.py): - _gather_inputs_fresh uses the value-at-step read -> flat as history grows (single query 4.7->13ms/2.8x before; 2.1->2.4ms/1.1x after). - Circular-dependency detection: _detect_signal_cycles (grey/black DFS over the inputs graph) + _warn_on_signal_cycles, lazy+idempotent, fired from start_training AND first reactive dispatch so it works headless (no wl.serve). Warns, does not raise; cyclic signals just stay dark. Measured (100-epoch MNIST headless, no wl.serve): per-step time flat (~157ms) vs rising 170->248ms; total wall 430s->286s (-33%). Reactive verify, multi-level/multi-parent chain, and cycle tests all pass. * perf(signals): staging-buffer value-at-step read + vectorized id->scalar dict Two profile-driven per-step wins (−30% wall at N=60k, full reactive config; 124.8 → 87.2 ms median), correctness unchanged (signal stress verify all pass). #2 (logger.py): serve query_per_sample_at_step from the in-memory staging buffer. The reactive gather reads the CURRENT step's value, which was just staged and isn't in DuckDB yet — so a reverse scan of the staging list (early break once all ids found) skips the flush -> register(pandas) -> INSERT -> unregister -> SELECT round-trip that profiling showed was ~a third of per-step cost. Falls through to DuckDB only when an id isn't in the buffer (mid-step flush moved it / older step). Also flattens the epoch-over-epoch drift: current -step reads no longer scan the growing per_sample table (measured flat ~75 ms over 98 epochs at 70k, vs the old code rising 110→137 in 7). #3 (src.py): vectorize the id->scalar dict build — one .tolist() (a single device sync) instead of a per-element .item() comprehension (B device syncs). Note: a row-wise executemany flush was tried and REVERTED — it was ~6x slower; register(pandas)+INSERT SELECT is DuckDB's fast bulk path (comment left in _flush_stage to prevent re-attempts). * feat(signals): logit-derived signals + batched reactive persist + preds_raw gate Makes @wl.signal the efficient path (no user save_signals needed), verified by the signal stress test (all checks pass). - ctx.logits/preds/targets on SignalContext + BatchSignalContext: a subscribe_to signal can compute from the model output (entropy, confidence, ...) — logit-derived signals become @wl.signal, no save_signals push. - Subscriber results logged id-keyed (dict), so reactive dependents can read them; reactive dispatch seeded with subscriber results so chains like conf_r<-entropy<-logits fire. - Batched reactive persist: _react_dependents accumulates all fired signals in-memory (chains read from there) and persists once, not per signal. DuckDB execute 1404->936, register 702->468 (-33%); the @wl.signal path is now faster than batched save_signals. - ledger_store_preds_raw hyperparam (default on): gate off to skip persisting the (B,C) logits every step (~7% saving); ctx.logits still works. - Trim verbose comments ~70% throughout. * examples: rewrite ws-signals-mnist to zero save_signals (ctx.logits + reactive) 460->180 lines. Per-step user code is just the watched loss; entropy is a @wl.signal from ctx.logits, loss_norm/hardness are reactive, shapes classified at the end. No save_signals. * perf(signals): don't query the ledger for reactive-derived inputs mid-pass A reactive-derived input only lives in the in-memory fresh_cache during the pass; querying the ledger for it (when a dependent is attempted before it fires) was a guaranteed-miss flush every step. Skip cheaply instead. Per-step 107->55ms (-48%), 1 flush/step -> 0; verify all pass. * examples: unify signals-mnist with the stress harness (train+eval, live loss_shape signal, overhead readout) * tests: unit tests for signal-refinements (query cache, value-at-step, cycle detection, ctx.logits, freshness, reactive-derived gather skip) * update gitignore - remove coverage files * remove last merged conflicts * logger: make query-cache maxsize env-configurable (WL_QUERY_CACHE_MAXSIZE, default 2048) Addresses PR #244 review (Guillaume): let users size the dedicated query cache. * feat(signals): default loss-shape classifier + report summary (PR #244 review #3) Two-layer, reusable over any per-sample signal: - classify_loss_shape(values, ...): default shape classifier, all thresholds named + configurable (natural defaults; tune without rewriting). - write_signal_shapes(signal, tag, classifier): engine — classify every sample's trajectory of ANY signal (loss, accuracy, a second loss, any metric) into a categorical tag; returns the {label: count} distribution. - write_loss_shapes(): convenience wrapper for the loss signal. - write_dataframe(loss_shape_signal=...): runs the summary on report write (tags + logs the distribution), so a report written every N steps stays current. Exported: classify_loss_shape, write_loss_shapes, write_signal_shapes, LOSS_SHAPES. Tests added; existing signal/logger suites pass. * refactor(signals): clean, layered, overridable shape API - trajectory_stats(values): generic reusable feature layer (renamed from loss_trajectory_stats — works on any trajectory). - classify_loss_shape: built on trajectory_stats; every threshold a named param. - write_signal_shapes: engine over any per-sample signal; write_loss_shapes: wrapper. - enable_loss_shape_signal(loss_signal, every=N): live per-step @wl.signal toggle (the heavier live counterpart to the report-time write_loss_shapes). - example ws-signals-mnist: drop the local classifier, use the core hook wl.write_dataframe(loss_shape_signal=...). Exports + tests updated. --------- Co-authored-by: Alexandru-Andrei Rotaru <rotarualexandruandrei94@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: GuillaumePELLUET <guillaume@graybx.com>
guillaume-byte
temporarily deployed
to
github-pages
GitHub Actions
Inactive
The main->dev merge kept dev's old detection code alongside main's updated comment and the new guarding tests, so origin/dev failed test_all_nan_float_column_stays_numeric. A genuine numeric-dtype (float/int) column that is entirely NaN now stays numeric (empty bins) instead of surfacing a spurious "unset" categorical bar. Detection keys off the pandas dtype: genuine numeric dtypes are always numeric; object columns are numeric if any value coerces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
guillaume-byte
temporarily deployed
to
github-pages
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v1.3.3 Release
Features & Improvements
Bug Fixes
Documentation
Review