A high-performance, zero-cost pipeline for normalizing German industrial product features from unstructured text into standardized taxonomy values. Achieves 87.01% exact-match accuracy on validation with $0 cost (no LLM/API calls).
Built in 2.5 days for the Unite Feature Normalization Hackathon challenge.
- Problem
- Results
- Architecture Overview
- Pipeline Tiers
- Project Structure
- Module Details
- run.py — Entrypoint
- src/pipeline.py — Orchestrator
- src/extractors/numeric.py — Numeric Extractor
- src/extractors/categorical.py — Categorical Extractor
- src/extractors/learned_patterns.py — Pattern Learner
- src/extractors/fallback.py — Fallback Extractor
- src/auto_learn/ — Auto-Learning Modules
- src/taxonomy_parser.py — Taxonomy Parser
- src/data_loader.py — Data Loader
- src/text_cleaner.py — Text Cleaner
- Quick Start
- Design Decisions
- Limitations & Next Steps
Unite manages ~200 million industrial products whose attributes are buried in unstructured German titles and descriptions. The challenge: given a product's title, description, and category, predict normalized feature_value entries for every feature defined in the taxonomy — matching the exact format, units, and allowed values.
Input: Product text (title + description) + category
Output: Standardized (uid, feature_name, feature_value, feature_type) rows
Metric: Exact Value Match Accuracy per (uid, feature_name) pair
| Split | Accuracy | Throughput | Cost |
|---|---|---|---|
| Validation | 87.01% | ~50,000 rows/sec | $0 |
| Test | Submitted | — | $0 |
- Categorical accuracy: High (exact substring + token matching + auto-learned rules)
- Numeric accuracy: High (specialized regex extractors for 30+ feature types)
- Zero external dependencies: No LLMs, APIs, or network calls
┌─────────────────────────────────────────────────────────┐
│ run.py │
│ (CLI entrypoint & evaluator) │
└──────────────────────┬──────────────────────────────────┘
│
┌──────────────────────▼──────────────────────────────────┐
│ ExtractionPipeline │
│ (src/pipeline.py) │
│ │
│ ┌─── Train Phase ───┐ ┌─── Predict Phase ──────┐ │
│ │ LearnedPatterns │ │ Tier 1: Direct regex │ │
│ │ CategoricalLearner │ │ Tier 1.5: Auto-learned │ │
│ │ NumericLearner │ │ Tier 2: Training-enh. │ │
│ │ DimensionModel │ │ Tier 3: Fallback │ │
│ └────────────────────┘ │ Tier 4: Defaults │ │
│ │ Tier 5: Value priors │ │
│ └────────────────────────┘ │
└──┬──────────┬─────────┬─────────┬───────────────────────┘
│ │ │ │
┌──▼──┐ ┌───▼──┐ ┌───▼───┐ ┌─▼────────────┐
│Num. │ │Cat. │ │Fallbk │ │ Auto-Learn │
│Extr.│ │Extr. │ │Extr. │ │ (3 modules) │
└─────┘ └──────┘ └───────┘ └──────────────┘
The pipeline processes each (uid, feature_name) pair through up to 6 tiers, stopping at the first tier that produces a confident result.
| Tier | Name | Strategy | When It Fires |
|---|---|---|---|
| 1 | Direct Extraction | Hand-crafted regex + exact/substring match against taxonomy values | First attempt for every row |
| 1.5 | Auto-Learned Rules | Discriminative token rules + dimension model + numeric context scoring | When Tier 1 misses |
| 2 | Training-Enhanced | Learned unit patterns + text-variant normalization maps | When auto-learn misses |
| 3 | Fallback | Single-value categoricals, text-scan, training-data priors | When extraction fails |
| 4 | Feature Defaults | Domain-specific defaults (e.g., "Säulentyp" → "Analytisch") | For known feature patterns |
| 5 | Value Priors | Most-common training value validated against allowed values | Last resort |
feature-normalization-hackathon/
├── run.py # CLI entrypoint
├── Feature_Normalization_Challenge.md # Challenge description
├── data/
│ ├── taxonomy/taxonomy.parquet # Feature definitions per category
│ ├── train/ # Training products + ground truth
│ ├── val/ # Validation products + ground truth
│ └── test/ # Test products + submission template
└── src/
├── __init__.py
├── pipeline.py # Main orchestrator (6-tier extraction)
├── data_loader.py # Parquet file loading
├── text_cleaner.py # HTML/whitespace normalization
├── taxonomy_parser.py # Taxonomy → lookup structures
├── extractors/
│ ├── __init__.py
│ ├── numeric.py # 1646-line numeric feature extractor
│ ├── categorical.py # 870-line categorical feature extractor
│ ├── learned_patterns.py # Training-data pattern learner
│ └── fallback.py # Last-resort value guessing
└── auto_learn/
├── __init__.py
├── categorical_learner.py # Discriminative token rule learner
├── numeric_learner.py # Context-aware number selector
└── dimension_model.py # HxBxT dimension position mapper
Purpose: CLI interface that orchestrates the full train → predict → evaluate workflow.
Key functions:
main()— Parses args (--split val|test), loads data, trains pipeline, runs prediction, savessubmission_<split>.parquet, evaluates if ground truth exists_build_template()— Constructs submission template from taxonomy × products (for val split)evaluate_accuracy()— Computes exact-match accuracy with per-type and per-feature breakdowns (top/bottom 10 features)
Usage:
python run.py --split val # Train + predict + evaluate on validation
python run.py --split test # Train + predict on test (generates submission)Purpose: The central ExtractionPipeline class that routes each (uid, feature_name) through the 6-tier extraction cascade.
Key class: ExtractionPipeline
__init__(taxonomy_df)— Builds taxonomy lookup, initializes all sub-modules (LearnedPatterns, CategoricalLearner, NumericLearner, DimensionModel)train(train_products, train_features)— Calls.learn_from_training()and.fit()on all learner modulespredict(products, submission_template)— Builds text lookup, processes in batches of 50k, returns filled DataFrame_extract_single(text, text_lower, category, feature_name, feature_type)— The core routing function: tries each tier in order
Notable design patterns:
- 20+ feature-specific special cases in
_extract_single()for German industrial domain quirks:Übersetzung— gear ratio "N:1" extractionBewegungsauflösung— DPI range bin matchingOberfläche— "gvz" abbreviation → "galvanisch verzinkt"Körnung— grit values (categorical type but numeric extraction)Schleifstoff— priority ordering (edelkorund > zirkonkorund > normalkorund)Antrieb— screw drive type detection (Torx/TX/Pozidriv/PZ)Sitz verstellbar— boolean detection from keyword presence/absence- ... and many more
- Post-processing: "ab" prefix detection for numeric ranges, comma-quote artifact cleanup
_maybe_add_ab_prefix()— Handles "ab 720 mm" vs "720 mm" ambiguity by checking which variant exists in allowed values
Purpose: Extract numeric feature values from German product text. The largest module (1646 lines) with 32 functions handling 30+ feature types.
Main entry point:
extract_numeric_value(text, text_lower, feature_name, unit, allowed_values)— Routes to specialized extractors or falls through to general extraction
Specialized extractors (selected):
| Function | Feature | Algorithm |
|---|---|---|
_extract_hxbxt_dimension() |
Höhe/Breite/Tiefe | Parses "HxBxT 800x600x400 mm" with prefix-aware dimension ordering |
_extract_faecheranzahl() |
Fächeranzahl | Handles multiplication: "2x5 Fächer" → "10" |
_extract_einzelzeichen() |
Einzelzeichen | Extracts single letters: Buchstabe "A" → "A" |
_extract_schutzart() |
Schutzart | IP rating: "IP65" → "IP65" |
_extract_koernung() |
Körnung | Grit values: "K120", "P80", "Körnung 120" |
_extract_verpackungseinheit() |
Verpackungseinheit | Packaging qty with German thousands: "1.000 Stk" → "1000" |
_extract_gewinde() |
Gewinde-Ø | M-thread + imperial: "M8", "1/4"" |
_extract_nenn_oe() |
Nenn-Ø | Drill diameter: "D.8.5mm" |
_extract_innen_aussen_oe() |
Innen-Ø / Außen-Ø | From NxNxN gasket patterns |
_extract_spannbereich() |
Spannbereich von/bis | Range extraction: "Spannbereich 3...12 mm" |
_extract_messbereich() |
Messbereich von/bis | Density ranges for areometers |
_extract_format() |
Format | "DIN A4", "75x533 mm" → matched against allowed values |
_extract_geraeuschpegel() |
Geräuschpegel | dB extraction + range bin matching ("25 - 30 dBA") |
_extract_luefterdrehzahl() |
Lüfterdrehzahl | RPM: "1500 rpm", "1500 U/min" |
_extract_nennweite() |
Nennweite | "DN 50" pipe nominal diameter |
_extract_groesse() |
Größe | Size values: "Größe 44" |
General extraction pipeline (when no specialized extractor matches):
_match_allowed_numeric_values()— Try to find each allowed value's number in text near matching unit_spec_line_extraction()— Parse "Key: Value" specification lines_keyword_anchored_extraction()— Find numbers near feature-name keywords_unit_extraction()— Find any number followed by the expected unit_match_ab_prefix_bins()— Match "ab N unit" prefix bin patterns_normalize_to_allowed()— Snap extracted value to closest allowed value (handles unit normalization)
Key regex patterns: Inline dimension (B1500xT500xH1950mm), HxBxT with prefix, 2D/3D bare dimensions, screw dimensions, RAL codes, German decimal/thousand separators.
Purpose: Match categorical feature values from taxonomy against product text. 870 lines with multi-strategy matching.
Main entry points:
extract_categorical_value(text, text_lower, feature_name, allowed_values, allowed_values_lower)— Primary extractionextract_categorical_with_fuzzy(...)— Extended matching with fuzzy/Levenshtein fallback
Extraction strategies (in order):
- Exact substring match (case-insensitive) with word-boundary checking
- Parenthesis removal — matches "Edelstahl (A2)" when text says "Edelstahl A2"
- Abbreviation matching — "NBR (Nitrilkautschuk)" matches on just "NBR"
- Token-based matching — catches non-contiguous matches like "Stahl" + "4.8" → "Stahl 4.8"
- Disambiguation — when multiple values match, picks by specificity (longest match wins), keyword proximity, and feature-specific priority rules
Specialized extractors:
| Function | Feature | Algorithm |
|---|---|---|
_extract_ral_color() |
Frontfarbe / Korpusfarbe | RAL code lookup with 200+ color mappings (RAL 7035 → "Lichtgrau") |
_extract_ral_farbe() |
Farbe | Generic RAL color extraction |
_extract_antrieb() |
Antrieb | Screw drive type: Torx/TX, Pozidriv/PZ, Phillips/PH, Kreuzschlitz, Innensechskant, Schlitz |
_extract_material() |
Material | German material aliases + DIN grades (A2, A4, 8.8, 10.9) |
_extract_waermequelle() |
Wärmequelle | Heat source classification |
Notable patterns:
_check_word_boundary()— Stricter for short values (≤3 chars, prevents "rot" matching inside "Schrott"), lenient for long values (allows German compound words)_disambiguate_matches()— Multi-factor scoring: match length, keyword proximity, specificity preference- Full RAL color code dictionary (
_RAL_CODES) with ~200 entries for German color names
Purpose: Learn extraction patterns from training data: value frequencies, text normalization maps, and unit patterns.
Key class: LearnedPatterns
learn_from_training(train_products, train_features)— Builds all lookup structures from training dataget_prior_value(feature_name)— Returns most common value (for fallback)get_learned_unit(feature_name)— Returns learned unit string (when taxonomy doesn't specify one)get_normalization_map(feature_name)— Returns{text_variant: canonical_value}mappingget_top_values(feature_name, n)— Returns N most common values
What it learns:
- Value priors:
Counterof value frequencies per feature (top 500 values kept) - Most common value: Per-feature default for fallback
- Normalization maps: Maps text-as-found → canonical value (handles abbreviations, typos)
- Learned units: Extracted from numeric value patterns (e.g., "250 mm" → unit is "mm")
- Position bias: Average position ratio where values appear in text (0=start, 1=end)
Algorithm: For categorical patterns, it finds how each ground-truth value appears in training text and builds variant→canonical mappings. Uses edit-distance-1 similarity for short-string fuzzy matching (_similar()).
Purpose: Last-resort value guessing when primary extractors fail.
Key function: fallback_value(feature_name, feature_type, allowed_values, learned, text_lower)
Priority cascade:
- If categorical with exactly 1 allowed value → return it (no ambiguity)
- Text-scan: lightweight token search for any allowed value in text (
_text_scan_categorical) - Training-data prior: most common value from training, validated against allowed values
- If categorical with exactly 2 allowed values → return first
None(give up)
Design decision: The text-scan fallback (_text_scan_categorical) only fires when exactly one allowed value has a significant token match, avoiding false positives from ambiguous partial matches. For multiple matches, requires a "clear winner" (longest match by 2+ characters).
Three modules that learn extraction strategies from training data:
Purpose: Learns "if token X in text → value Y" rules with precision/recall statistics.
Key classes:
CategoricalRule— A single rule:token → valuewith precision, recall, supportAbsenceRule— Fires when discriminative tokens are absent (e.g., no "verstellbar" → "nein")CategoricalLearner— Main learner class
Algorithm:
- Tokenize all training texts with unigrams + bigrams (filtering German stopwords)
- For each
(token, feature_value)pair, compute:- Precision = P(value | token in text) — must be ≥ 0.75
- Recall = P(token in text | value) — must be ≥ 0.03
- Support — minimum 5 occurrences
- Lift = precision / baseline_rate — must be ≥ 1.5
- Keep top 300 rules per feature, sorted by precision
- Also learn absence rules for binary/small-value features (≤5 values)
Prediction: Score each allowed value by summing matching token precisions. Fire if score ≥ 0.5.
Purpose: When multiple numbers appear in text, selects the correct one for a given feature using learned context keywords and value ranges.
Key classes:
NumberCandidate— A number found in text with position, unit, and context wordsFeatureNumericModel— Learned model: keyword weights, value range (5th–95th percentile), common unitsNumericLearner— Main learner class
Algorithm:
- For each numeric feature in training, find the ground-truth number in the text
- Record context words within a 7-word window around the match
- Compute keyword weights (TF-normalized) and value range bounds
- At prediction: extract all candidate numbers, score each by keyword overlap + range plausibility + unit match
Purpose: Learns which position in "AxBxC" dimension patterns corresponds to which feature (Höhe/Breite/Tiefe/Länge).
Key insight: German industrial products often specify dimensions as "HxBxT 800x600x400 mm" but the ordering varies by product category.
Algorithm:
- For each dimension feature, find AxBxC patterns in training text
- Record which position (1st/2nd/3rd) contains the ground-truth value
- Learn per-feature position preferences (e.g., "Höhe is usually position 0 in 3D patterns")
- Also learn size preferences (largest/middle/smallest heuristic)
- Detect explicit label patterns ("HxBxT", "LxBxH", "BxHxT") to override position defaults
Supported label patterns: HxBxT, LxBxH, BxHxT, BxTxH, LxB, BxH, HxB, BxT (with German word variants)
Purpose: Parses taxonomy.parquet into fast lookup structures used by all extractors.
Key functions:
build_taxonomy_lookup(taxonomy_df)— Returns a dict containing:category_features—{category: [(feature_name, feature_type), ...]}feature_allowed_values—{(category, feature_name): [allowed_values]}feature_units—{(category, feature_name): unit_str}(for numeric features)feature_type—{(category, feature_name): 'categorical'|'numeric'}cat_feature_values_lower—{(category, feature_name): {lower_val: original_val}}(for fast case-insensitive lookup)
parse_aggregated_values(agg_str)— Parses{'[val1]','[val2]'}format, handles comma-quote artifactsextract_unit_from_values(values)— Infers unit from example values (e.g.,['250 mm', '500 mm']→'mm')get_feature_keywords(feature_name)— Returns search keyword variants for 40+ feature names (e.g., "Gewinde-Ø" →["gewinde-ø", "gewinde", "m ", "m="])
Design decision: Pre-computes lowercased value maps at build time to avoid repeated .lower() calls during extraction (significant speedup at 50k+ rows/sec throughput).
Purpose: Loads all Parquet files into Pandas DataFrames.
Key functions:
load_all(data_dir)— Returns dict with all 7 DataFrames: taxonomy, train_products, train_features, val_products, val_features, test_products, test_submissionload_split(data_dir, split)— Loads products + features + submission for a single split
Purpose: Lightweight text preprocessing for product titles and descriptions.
Key functions:
clean_text(text)— Strips HTML tags, decodes HTML entities (including German umlauts:ä→ä), normalizes whitespacecombine_text(title, description)— Joins title and description with|separator
Design decision: Minimal cleaning — preserves original casing and punctuation since the extractors need both case-sensitive and case-insensitive matching. HTML entity decoding covers German-specific entities (ä, ö, ü, ß).
- Python 3.8+
- pandas
- pyarrow (for Parquet I/O)
pip install pandas pyarrowpython run.py --split valThis will:
- Load all data from
data/ - Train patterns from training data (~10s)
- Predict on validation split
- Save
submission_val.parquet - Evaluate and print accuracy breakdown
python run.py --split testGenerates submission_test.parquet for final submission.
python3 -c "
import pandas as pd
val = pd.read_parquet('data/val/product_features.parquet')
sub = pd.read_parquet('submission_val.parquet')
m = val.merge(sub, on=['uid','feature_name'], suffixes=('_true','_pred'))
correct = (m.feature_value_true == m.feature_value_pred).sum()
print(f'{correct}/{len(m)} = {correct/len(m)*100:.2f}%')
"-
Regex-first, no LLMs: The 40% cost criterion in the rubric incentivizes zero recurring cost. Pure regex/rule extraction achieves $0 cost and high throughput.
-
6-tier cascade: Each tier has increasing generality but decreasing precision. Early tiers use hand-crafted domain rules; later tiers use statistical priors. This maximizes accuracy while minimizing false positives.
-
Feature-specific special cases (20+ in pipeline.py): German industrial naming conventions require domain-specific logic. For example, "gvz" is a ubiquitous abbreviation for "galvanisch verzinkt" (galvanically zinc-plated) that no general-purpose extractor would catch.
-
Auto-learning from training data: Three learned modules (CategoricalLearner, NumericLearner, DimensionModel) automatically discover patterns that hand-crafted rules miss, without requiring manual feature engineering for every feature name.
-
Taxonomy-constrained output: All predictions are validated against allowed values from the taxonomy before output. Numeric values are snapped to the closest allowed value via
_normalize_to_allowed(). -
Pre-computed lookups: Taxonomy is parsed into optimized dict structures (including pre-lowercased value maps) at initialization, enabling ~50k rows/sec throughput.
-
Batch processing: Predictions are processed in configurable batches (default 50k) for memory efficiency on the full 200M product scale.
- Long-tail features: Some rare feature names have no training examples and rely solely on generic extraction or fallback heuristics
- Compound German words: German compound nouns (e.g., "Drehmomentschlüssel") can embed feature values in ways that substring matching misses
- Multi-value features: Some features accept comma-separated values which are harder to extract reliably
- Unit conversion: Limited handling of imperial↔metric conversion (mainly for thread sizes)
- Potential improvements:
- Fuzzy matching with Levenshtein distance for near-misses
- Category-specific extraction rules learned from training data
- Ensemble with a lightweight local LLM for the ~13% that rules miss