ExplainChess

Per-player chess puzzle difficulty prediction, with explainability and a quality-audit agent.

MSc dissertation project. Instead of predicting a single difficulty rating, ExplainChess predicts the full per-player difficulty curve — how likely a player at each skill band (1050–2050, rapid & blitz) is to solve a puzzle — from features derivable from the position. On top of that it provides per-band SHAP explanations, a user-skill calibration method, and an agentic puzzle-quality auditor.

---

Three pillars

Pillar	What it does	Code
Per-player difficulty	Predicts the 22-value solve-probability curve (11 rating bands × rapid/blitz) from 38 position features. One MultiOutputRegressor(LightGBM).	src/prediction_model/
Explainability + calibration	Exact per-band TreeSHAP (why a puzzle is hard for each skill level) and a max-likelihood user-rating estimator.	src/explainability/, src/prediction_model/calibration.py
Quality assessment	A negative-result study (engine-quality vs human Popularity) + an agentic Stockfish flaw auditor with synthetic-flaw validation.	src/quality_pillar/

---

Project structure

ExplainChess/
├── src/
│   ├── stockfish/                       # shared feature foundation
│   │   ├── stockfish_features.py        #   Stockfish extraction, decode_eval_columns, _pov_cp
│   │   ├── feature_engineering.py       #   move-derived features
│   │   ├── dataset.py                   #   FEATURE_COLUMNS (engine feature names)
│   │   └── cup_data.py                  #   data loader, feature groups, fixed holdout, SUCCESS_PROB_COLS
│   ├── tactical/
│   │   └── tactical_features.py         #   TACTICAL_COLUMNS (python-chess derived)
│   ├── prediction_model/                # PILLAR 1 — per-player core
│   │   ├── verify_multioutput_treeshap.py  # trains + saves the curve model
│   │   └── calibration.py              #   user-skill calibration
│   ├── explainability/                  # PILLAR 2
│   │   ├── explain_curve.py            #   per-band SHAP + skill-gap driver
│   │   └── feature_descriptions.py     #   human-readable feature names
│   ├── quality_pillar/                  # PILLAR 3 (live system)
│   │   └── flaw_agent.py              #   agentic Stockfish flaw auditor (live, any FEN)
│   ├── evaluation/                      # evaluation & experiment scripts (see its README)
│   │   ├── evaluate_model.py           #   per-band MAE / ranking AUC / slope-corr (no retrain)
│   │   ├── baseline_comparison.py      #   model vs global-mean / rating-only / linear
│   │   ├── tactical_ablation.py        #   tactical-feature block + per-feature SHAP
│   │   ├── tune_perplayer.py           #   Optuna hyperparameter study (future work)
│   │   ├── quality_study.py            #   engine-quality vs Popularity correlation study
│   │   ├── validate_flaw_agent.py      #   synthetic-flaw recall/specificity validation
│   │   └── threshold_sweep.py          #   flaw-agent threshold recall-vs-margin sweep
│   └── pipeline.py                      # single-FEN inference: all outputs in one call
├── configs/config.yaml                  # paths + Stockfish + bigdata_cup settings
├── results/models/perplayer_curve_lgbm.pkl   # the trained per-player model (shipped)
├── requirements.txt
└── README.md  ← you are here

Data availability & reproducibility

The datasets are NOT included in this repository (they are large, and the competition data is not ours to redistribute). The code expects, relative to the repo root:

data/training_data_02_01.csv      # IEEE BigData 2024 Cup training data: FEN, Moves, Rating,
                                  #   Popularity, NbPlays, and 22 success_prob_* columns (~2 GB)
data/stockfish_features_700k/     # pre-extracted Stockfish engine features (Parquet batches,
                                  #   ~700k puzzles) — produced by the extraction step below

How to obtain each:

• training_data_02_01.csv — the official IEEE BigData 2024 Cup / FedCSIS 2025 "Predicting Chess Puzzle Difficulty" dataset, available from the competition organisers (knowledgepit.ai). It is not redistributable here.
• stockfish_features_700k/ — regenerated locally from the CSV by running the engine feature extraction (src/stockfish/stockfish_features.py); this is the multi-hour, resume-safe step. There is no separate download — it is derived from the CSV above.

success_prob_* are Maia-2-derived human solve-rates per rating band (the target of the per-player model); the Stockfish features are the inputs.

What you CAN run without the data: the trained per-player model ships in the repo at results/models/perplayer_curve_lgbm.pkl, so the inference pipeline (src/pipeline.py) and the live quality auditor run on any single FEN without the dataset. The unit tests (pytest tests/) and the CI are also fully data-free. Only the dataset-wide evaluation and training scripts require the files above.

Honest note: because the official test labels are withheld (secret test set), all reported metrics are on a fixed, seeded, rating-stratified hold-out of the training set — not the competition's secret test set, so no leaderboard rank is claimed.

---

Setup

pip install -r requirements.txt
brew install stockfish          # macOS — the flaw_agent needs the Stockfish binary
which stockfish                 # confirm path matches configs/config.yaml -> stockfish.path

---

Usage

All commands run from the repo root. --sample N subsamples for a quick run; omit for the full set.

Pillar 1 — per-player difficulty model

# Train + verify + save the per-player curve model (one MultiOutputRegressor, 22 boosters).
# Reports mean per-band MAE, slope correlation, and that TreeSHAP is exact + deterministic.
python -m src.prediction_model.verify_multioutput_treeshap --sample 150000 --save

Saves results/models/perplayer_curve_lgbm.pkl = {model, input_cols, target_cols, bands}. (The repo ships a pre-trained copy, so you can skip straight to the steps below.)

Pillar 2 — explainability + calibration

# WHY a puzzle is hard for each skill band (per-band TreeSHAP) + the skill-gap driver:
python -m src.explainability.explain_curve --sample 5000 --n 2

# Estimate a user's rating from their solve/fail pattern. Sweeps estimator (mean/mode/median)
# × #calibration-puzzles and prints the per-band recovery table for the best config.
python -m src.prediction_model.calibration --sample 8000

Pillar 3 — quality assessment

# Negative-result study: engine-quality criteria vs human Popularity vs difficulty (Spearman).
python -m src.evaluation.quality_study

# Agentic flaw auditor — flags AMBIGUOUS / WRONG_KEY / WEAK puzzles (live Stockfish, any FEN):
python -m src.quality_pillar.flaw_agent --sample 200
python -m src.quality_pillar.flaw_agent --fen "<FEN>" --moves "e2e4 e7e5 ..."   # single puzzle

# Validate the auditor by injecting known flaws into clean puzzles (recall / specificity):
python -m src.evaluation.validate_flaw_agent --n 120

Evaluation & reproducibility scripts (measure the model, don't change it)

# Evaluate the SHIPPED model with no retraining — per-band MAE, ranking AUC, slope-corr, TreeSHAP:
python -m src.evaluation.evaluate_model --sample 150000

# Prove the per-player model beats obvious baselines (global mean / rating-only / linear):
python -m src.evaluation.baseline_comparison --sample 150000 [--all22]

# Ablate the 14 tactical features (block contribution + per-feature SHAP ranking):
python -m src.evaluation.tactical_ablation --sample 150000

# Optuna hyperparameter study for the per-player model (vs the hardcoded defaults):
python -m src.evaluation.tune_perplayer --sample 120000 --trials 25

# Flaw-agent threshold sweep (recall vs cp margin — justifies the thresholds):
python -m src.evaluation.threshold_sweep --n 300

# Data-free unit tests (mate-score decoding, calibration, tactical extraction):
pytest tests/ -q

---

Key results (held-out slice of the official train set)

Pillar	Metric	Result
Per-player curve	mean per-band ranking AUC	0.86 (0.854–0.860 across bands)
Per-player curve	mean per-band MAE / slope corr	~0.18 / ~0.35–0.45
Per-player curve	vs baselines (global mean / rating-only)	beats both; +0.12 slope-corr over rating-only
Calibration	recovery error (mode estimator, 60 puzzles)	~134 rating points (mid-bands best)
Quality study	engine-quality vs Popularity / vs Rating (Spearman)	−0.02 (orthogonal) / +0.39 (predicts difficulty)
Flaw auditor	recall on injected flaws (threshold-robust 25–200cp)	~1.00

Quality finding: objective (engine-derived) puzzle quality is uncorrelated with human Popularity but strongly predicts difficulty — the two are orthogonal on Lichess. The useful, objective quality task is therefore flaw detection, which an engine agent does reliably.

---

Notes

• Reproducibility: all randomness is seeded via training.seed in configs/config.yaml; the holdout split is fixed (make_holdout), so numbers are comparable across runs.
• Feature directory: several scripts default to ../data/stockfish_features_600k. Change there or via configs/config.yaml -> paths.stockfish_features_dir if your path differs.
• Stockfish binary is required only by the quality_pillar flaw agent (live analysis).