Retrain ML model on real feedback (replace synthetic bootstrap)

[ringo380]

Context

The currently-ACTIVE model (query_grader, version 20260521_021306) was trained as a bootstrap on synthetic/seed TrainingData to give the issue-#5 monitoring pipeline an ACTIVE target. Its metrics reflect that:

• Training accuracy: 0.804
• Validation accuracy: 0.712 (just over the 0.7 auto-deploy threshold)
• Test accuracy: 0.095 — effectively no real predictive power.

It satisfies "an ACTIVE model exists so monitoring evaluates," but it is not a quality predictor. The low-confidence / low-user-agreement alerts it generates reflect exactly this.

Goal

Replace the synthetic bootstrap with a model trained on accumulated real user feedback once enough has been collected.

Tasks

• Define a minimum real-feedback sample threshold before retraining (current ML_MIN_TRAINING_SAMPLES=50 counts synthetic seed rows — may want a separate "validated real feedback" gate).
• Confirm process_ml_feedback → TrainingData flow is converting real QueryFeedback into training samples.
• Establish a retrain cadence (the ML_AUTO_RETRAIN / beat machinery exists) and a quality gate higher than the bootstrap's 0.7 val / 0.095 test.
• Re-evaluate model_type: bootstrap is QUERY_GRADER; the grading path expects HYBRID_SCORER (per CLAUDE.md) once HybridQueryGrader is wired in.

Notes

• Training runs in-container: railway ssh -s querygrade-worker "python manage.py train_ml_model …" (local railway run can't reach postgres.railway.internal).
• Related: model-artifact durability (Move ML model artifacts to object storage (model registry) #91) becomes relevant once predictions actually load the model.

[ringo380]

Worked the second task ("confirm process_ml_feedback -> TrainingData is converting real QueryFeedback into training samples"). It was not converting anything: the command could not run at all. Fixed in b7ba6a6. The retrain itself stays blocked - see the data below.

The flow was broken end to end

process_ml_feedback was written against a schema QueryFeedback never had. It queried is_helpful, score_agreement, .user, and Query.queryfeedback_set - none are real fields. Every mode died on the first ORM call, with or without data:

$ python manage.py process_ml_feedback --stats-only
CommandError: Cannot resolve keyword 'is_helpful' into field.
$ python manage.py process_ml_feedback --dry-run
CommandError: Cannot resolve keyword 'queryfeedback' into field.

Confirmed against production, not just locally. So even if real feedback had been arriving this whole time, none of it could ever have become TrainingData. It is not on the beat schedule, so it was not failing silently in prod - just unusable at the moment it was needed.

The ML core is fine. FeedbackCollector.collect_feedback_for_query uses the real schema and works; only the CLI wrapper had rotted. Root cause is plain: the command had zero test coverage.

Three further bugs found while fixing it:

• Feedback reaches a Query only through UserQueryHistory, in two forms - a detailed QueryFeedback, or a simple was_helpful flag. The collector accepts both; the command counted only the detailed rows, so it under-reported.
• --min-feedback defaulted to 2 while the collector enforces 3, so it would report queries as ready that processing then silently dropped.
• --stats-only divided by total feedback, so it raised ZeroDivisionError on zero feedback - exactly the state you would run it to diagnose.

Added the command's first tests (both feedback forms, empty dataset, threshold agreement, end-to-end TrainingData creation). All 8 fail against the previous version, so they catch the real bug rather than a hypothetical one.

The retrain is blocked, and not by code

Production, via the now-working command:

QueryFeedback (real feedback)	0
UserQueryHistory	3
TrainingData	86 - all synthetic_seed, 0 real
Queries ready to process	0

No user has ever submitted feedback. That is consistent with #106, which established there has never been an authenticated session and /register/ has never been viewed. So "retrain once enough real feedback has been collected" has a sample size of zero and no near-term path to a non-zero one. This issue is gated on traffic, not engineering.

A trap for the first task

The plan suggests a "validated real feedback" gate. is_validated cannot be that gate - all 86 synthetic seed rows have is_validated=True, so it does not separate real from bootstrap at all. The only honest discriminator today is validation_source (synthetic_seed vs anything else), which is what the fixed stats now split on. Worth settling before the threshold in task 1 gets written against the wrong field.

Suggested state

Leaving this open: the goal is unmet and the bootstrap is still ACTIVE. But the pipeline is now genuinely ready to receive data, and the retrain trigger is a product/traffic question. The remaining two tasks (retrain cadence + quality gate, and the QUERY_GRADER -> HYBRID_SCORER re-evaluation) are both independent of feedback volume and could be done now if worth it.

[ringo380]

Took the two remaining tasks, since both are independent of feedback volume. Both turned out to be broken rather than merely unfinished (22118af). Verified live in prod.

The ML grading path has never run

analyze_query imported ..ml.hybrid_grader. The module is at ..ml.core.hybrid_grader. The resulting ModuleNotFoundError was caught by a bare except Exception and logged as "ML grading failed, falling back to rule-based":

>>> analyze_query("SELECT * FROM users WHERE id = 1", "MySQL", use_ml=True)
WARNING ML grading failed, falling back to rule-based: No module named 'analyzer.ml.hybrid_grader'

With ML_ENABLED=True and ML_HYBRID_GRADING=True in production, every grade has been rule-based, including when use_ml=True was explicitly requested. Nothing failed, so nothing surfaced - the only trace was a warning line.

This reframes the model_type task. The QUERY_GRADER vs HYBRID_SCORER mismatch was never reachable: the grader that wants HYBRID_SCORER was never constructed. It also means the bootstrap's 0.095 test accuracy never touched a user - predictions never loaded the model, which is also why #91 (artifact durability) has stayed quiet.

Fixed the import, and let ImportError propagate rather than degrade - a missing module is a packaging bug, not a runtime condition. Genuine runtime failures (missing artifact, bad model) still fall back as before.

The fix is behaviour-neutral today. HybridQueryGrader looks for an ACTIVE HYBRID_SCORER, prod has zero, so it degrades to rule-based. Grades are identical across both paths:

Query	Rule	Hybrid
SELECT * FROM users WHERE id = 1	A/90.0	A/90.0
good join+group-by	A/100.0	A/100.0
UPPER() + NOT IN + leading wildcard	F/0.0	F/0.0
cartesian product	F/50.0	F/50.0

So this only takes effect once a hybrid model is actually trained - it does not switch a bad model on.

Why the bootstrap deployed: the gate never looked at test accuracy

The gate was validation_accuracy >= performance_threshold, nothing else. The bootstrap cleared it at val=0.712 against a 0.7 bar while scoring test=0.095. Validation alone cannot catch that; the held-out score is the entire point of holding it out.

The gate now requires both bars plus a bounded validation-test gap (a 0.6 spread means it did not generalize even if both numbers passed), and refusals state a reason instead of silently not deploying. It lives on TrainingConfig, so the pipeline and train_ml_model cannot drift apart. Confirmed in prod:

gate: val>=0.7 test>=0.7 gap<=0.15
bootstrap(0.712/0.095) deploys? (False, 'test accuracy 0.095 < 0.700')

The model that is ACTIVE today would not be deployable by the current gate.

The retrain cadence does not exist

Worth correcting the issue text: there is no ML_AUTO_RETRAIN machinery. ML_AUTO_RETRAIN, ML_PERFORMANCE_THRESHOLD and ML_MIN_TRAINING_SAMPLES were defined in settings and read by nothing - tuning them did nothing at all. The beat schedule contains only monitor-ml-models; nothing retrains.

I wired ML_PERFORMANCE_THRESHOLD and ML_MIN_TRAINING_SAMPLES into TrainingConfig and added ML_TEST_PERFORMANCE_THRESHOLD / ML_MAX_VALIDATION_TEST_GAP alongside them, so the gate is now env-tunable. ML_AUTO_RETRAIN is still inert and I left it that way deliberately - scheduling automatic retrains while real feedback is 0 would only re-derive the bootstrap on synthetic data. The cadence is worth defining when there is something to train on.

Where this leaves the issue

Still open, and still gated on traffic rather than code. What changed is that the blockers are now mechanical rather than hidden:

• feedback -> TrainingData can run (previous comment)
• the ML grading path can reach the grader
• a bootstrap-quality model can no longer auto-deploy

Remaining, in dependency order: get real feedback (needs traffic - see #106), train a HYBRID_SCORER on it, and only then decide the retrain cadence. Suite is 760 tests, same 3 pre-existing errors.

[ringo380]

Reconciliation: HybridQueryGrader is already wired; the real gates are model_type + artifact durability

The task note "the grading path expects HYBRID_SCORER ... once HybridQueryGrader is wired in" reads as if the grader still needs wiring. It does not - it is already invoked on every grade.

Trace:

• analyzer/views/query_grading_views.py:93,382,683 -> analyze_query() in analyzer/analyzers/base.py:376.
• With ML_HYBRID_GRADING + ML_ENABLED both defaulting True, base.py:399-404 builds a HybridQueryGrader and calls .analyze_query(use_ml=True) on every grade.

So the only things standing between today and live ML predictions are:

1. Model type. _load_current_model accepts only model_type="HYBRID_SCORER" (analyzer/ml/core/hybrid_grader.py:243). The ACTIVE bootstrap is QUERY_GRADER, so it is never loaded. Marking a HYBRID_SCORER model ACTIVE is what actually turns prediction on - the wiring is already there.
2. Artifact durability (Move ML model artifacts to object storage (model registry) #91). The moment a HYBRID_SCORER goes ACTIVE, the web container tries to load the .pkl from its own ephemeral disk and fails (hybrid_grader.py:252), logging an error every grade and falling back to rule-based. Move ML model artifacts to object storage (model registry) #91 must land first.

Net: this issue (retrain on real feedback) is blocked by #91, and its retrained model must be saved as HYBRID_SCORER for the existing request-path wiring to pick it up. The "wire in HybridQueryGrader" framing can be dropped.

[ringo380]

Update: the artifact-durability blocker (#91) is resolved in c065848 - trained models now persist in the shared DB (MLModelArtifact) and load on any service. What remains for this issue is unchanged: accumulate real QueryFeedback, retrain a HYBRID_SCORER model that clears a real quality gate (not the 0.095-test bootstrap), and mark it ACTIVE. Once that model is ACTIVE, the grade-path wiring already loads it durably.

[ringo380]

Status audit + progress (46785e6)

Checked the four tasks against the current tree and prod. Three were already done or nearly so; one had a real bug, now fixed. The core goal (train on real feedback) remains blocked on data that does not exist yet.

Prod ground truth: 1 MLModel (the QUERY_GRADER bootstrap, ACTIVE), 86 TrainingData rows (all validation_source="synthetic_seed"), 0 QueryFeedback. So there is currently no real feedback to train on.

Task-by-task

1. Minimum real-feedback threshold - DONE in 46785e6. Added ML_MIN_REAL_FEEDBACK_SAMPLES (default 25) and a training_gates helper that counts only non-synthetic rows. Both retrain entry points (hybrid_grader.train_model, training_pipeline.run_training_pipeline) now refuse to train on synthetic-only data; a forced run bypasses it. This closes a real bug: ML_AUTO_RETRAIN is on and beat evaluates every 15 min, and the old gate counted the 86 seed rows as "≥50, enough", so it could have retrained on pure synthetic and shipped another non-predictive model (the hybrid path deploys on a weak r2 threshold alone).

2. process_ml_feedback → TrainingData flow - already built. process_ml_feedback handles both detailed QueryFeedback and simple was_helpful history, splits real vs synthetic by validation_source, and FeedbackCollector.collect_feedback_for_query creates the rows. Nothing to add; it just has no real feedback to convert yet.

3. Quality gate above the bootstrap - already done. TrainingConfig.meets_quality_gate enforces val ≥ 0.7, test ≥ 0.7, and a ≤ 0.15 val-test gap (ML_TEST_PERFORMANCE_THRESHOLD / ML_MAX_VALIDATION_TEST_GAP). The 0.712-val / 0.095-test bootstrap would be rejected today. Cadence machinery (ML_AUTO_RETRAIN + beat) already exists.

4. model_type QUERY_GRADER vs HYBRID_SCORER - resolved for future retrains. TrainingConfig.model_type now defaults to HYBRID_SCORER, which is exactly what the grade-path loader (hybrid_grader._load_current_model) filters on. The ACTIVE bootstrap is QUERY_GRADER only because it predates that default; a new retrain produces HYBRID_SCORER and is loadable. (The stale bootstrap is doubly inert right now: ML_HYBRID_GRADING defaults off, and the loader ignores its type anyway. It can be archived whenever a real model replaces it.)

What remains (genuinely blocked)

Only the actual retraining, which needs real QueryFeedback to accumulate past the gate - and that needs authenticated usage. The preconditions are now correct, so the moment real feedback crosses the threshold the system will retrain, gate on the tightened quality bar, and produce a HYBRID_SCORER the grade path can load durably (via #91). Keeping open and blocked-on-real-usage.