Live App: https://signaltext.streamlit.app/
A sentiment pipeline over 250,000 Amazon product reviews that does one thing most sentiment projects don't: instead of reporting a single accuracy number, it runs two different model architectures against each other and surfaces the cases where they disagree — because those disagreements are exactly where automated sentiment scoring is least trustworthy.
A fine-tuned transformer (DistilBERT) and a zero-shot LLM (Llama-3.2-3B via Ollama) score the same reviews. Where they agree, the label is safe. Where they don't, a human should look — so the pipeline ranks those disagreements by joint confidence and exports them for review.
Data: Amazon Reviews for Sentiment Analysis
(Kaggle), derived from the Stanford SNAP Amazon corpus. Uses the test.ft.txt split (400k reviews), sampled to a balanced 250k. Research use.
On the 2,990 reviews scored by both models:
| Model | Accuracy | F1 (pos) | Mean confidence |
|---|---|---|---|
| DistilBERT (fine-tuned SST-2) | 90.4% | 0.902 | 0.983 |
| Llama-3.2-3B (zero-shot) | 95.6% | 0.955 | 0.906 |
Two things stand out.
The zero-shot LLM beats the fine-tuned classifier — 95.6% vs 90.4%. A model never trained on this task outperforms one fine-tuned for sentiment, on that task's home turf.
The weaker model is the more confident one. DistilBERT averages 0.983 confidence while being wrong more often; Llama averages 0.906. Confidence and correctness point in opposite directions, which is a problem if you were trusting the confidence score to tell you when to worry.
The models agree on 91.4% of reviews. On the 257 disagreements, the LLM is right 80.2% of the time (measured against the corpus star-rating labels). The disagreement direction is nearly symmetric — 134 cases where DistilBERT says negative and Llama says positive, 123 the other way — so this isn't one model being biased toward a label. It's a difference in how they read.
Status: The quantitative error analysis below is a hypothesis drawn from inspecting disagreement cases, not yet a labeled result. A systematic hand-labeling pass over the 150 highest-confidence disagreements is underway; this section will be updated with counts and a confusion breakdown once it's complete.
Reading through the ranked disagreements, DistilBERT's failures cluster into a recognizable pattern: it scores the emotional words without tracking whose opinion they are or what they're about. For example —
- A candy review titled "Love the Candy ~ Hate the Shipping Cost" — glowing about the product, annoyed at shipping. DistilBERT reads the complaint words and calls it negative at 99.8% confidence. The sentiment toward the product is positive.
- A tutorial that quotes someone else's negative review ("the songs would sound very choppy…") before solving the problem. DistilBERT attributes the quoted complaint to the reviewer.
- A Lord of the Flies review that opens "God this was terrible," spends two paragraphs praising the book's social commentary, then rejects it. DistilBERT counts the praise and flips positive.
- Sarcasm — "Great, another product that lasted exactly as long as the warranty. Truly impressive engineering." DistilBERT: positive, 100%.
- Llama fails in a different, narrower way: short, dry, factual complaints with no emotional vocabulary — e.g. a ticking-clock review ("It ticks!") or "the lamp… was inferior… I returned it." No sentiment words, so the zero-shot model sometimes defaults positive.
The working thesis is that DistilBERT — fine-tuned on movie reviews, where sentiment words map cleanly to verdicts — fails on attribution when moved to product reviews, where the negative words often attach to shipping, price, or a quoted review rather than the product. The hand-labeling pass will test whether the accuracy gap is in fact explained by attribution errors, or whether something else is going on.
test.ft.txt (400k reviews)
│
▼
01_ingest ──► corpus.parquet (250k, balanced 125k/125k, seed=42)
│
├──────────────────────┬──────────────────────┐
▼ ▼ ▼
02a DistilBERT 02b Llama-3.2 03a MiniLM
full corpus zero-shot, 3k embeddings → FAISS
│ sample, resumable │
└──────────┬──────────┘ │
▼ │
02c compare ──► leaderboard │
──► manual_review.csv │
│ │
└───────────────┬───────────────────┘
▼
Streamlit app
(live transformer + semantic search
over the corpus, disagreements flagged)
| Layer | Choice | Why |
|---|---|---|
| Corpus | Amazon Reviews Polarity, 250k slice | Balanced labels; paragraph-length text that embeds well |
| Transformer | DistilBERT SST-2 | Fast enough to score the full corpus |
| LLM | Llama-3.2-3B via Ollama | Free, local, no API key — the repo runs without credentials |
| Embeddings | all-MiniLM-L6-v2 (384-d) | Small, strong, CPU-viable |
| Vector store | FAISS IndexFlatIP |
Exact search, no service dependency, corpus exceeds Pinecone's free tier |
| App | Streamlit | Fast path to a shareable URL |
The corpus is fully regenerable from one script with a fixed seed — the same command produces the identical 250k sample on any machine (verified: numbers matched to the decimal across two machines). Raw data and large index artifacts are gitignored; a fixed-seed ingestion script rebuilds them.
pip install -r requirements-local.txt
python pipelines/01_ingest.py --input data/raw/test.ft.txt --output data/corpus.parquet
python pipelines/02a_score_transformer.py --input data/corpus.parquet --output data/scored_transformer.parquet
ollama pull llama3.2
python pipelines/02b_score_llm.py --input data/corpus.parquet --output data/llm_scores.jsonl --sample 3000
python pipelines/02c_compare.py --scored data/scored_transformer.parquet --llm data/llm_scores.jsonl --outdir data/
python pipelines/03a_embed.py --input data/scored_transformer.parquet --outdir models/ --limit 50000
python pipelines/03b_search.py --outdir models/ --cluster 12
streamlit run app/streamlit_app.py- The LLM scores a 3,000-row stratified sample, not all 250k. Llama-3.2 on CPU runs ~4 sec/review; the full corpus would take days. 3k is enough to characterize the disagreement distribution, and the transformer still covers all 250k. Stated choice, not an unfinished job.
02bis resumable (append-only JSONL, skips scored IDs) so the run survives interruption. - LLM scores are precomputed, not live in the deployed app. Ollama can't run on hosted Streamlit, so the app serves LLM scores for corpus passages from a lookup and runs live inference only for the transformer on user-pasted text.
- Star ratings are a proxy for sentiment, not ground truth. A sarcastic 5★ review is labeled positive. When a model contradicts the label, the model may be right — which is the entire premise of the manual-review step.
- No neutral class. The corpus authors dropped all 3★ reviews when building the polarity labels, so there's no genuinely ambivalent text. The LLM occasionally tries to output "mixed" anyway (~0.3% of the sample) — noted because it's the model flagging exactly the cases the label space can't represent.
See
data/DATA_DICTIONARY.mdfor the full schema and limitations.
Beyond sentiment, the 50k-passage FAISS index supports meaning-based retrieval: a query like "battery died after a month" returns reviews about battery failure that share no keywords with the query. K-means over the embeddings recovers product categories from text alone — books, music, film, electronics, kitchen, toys, games — despite the corpus carrying no category metadata (it was dropped at ingestion). Notably, book reviews split into two clusters at 67% vs 40% positive: casual-reader language ("love, reading") versus critical-register language ("author, characters, novel"), separated purely by the embedding. The deployed demo uses a 5,000-passage index to stay within GitHub's file-size limits; the full pipeline builds a 50,000-passage index locally.
- The dropped 3★ reviews remove exactly the ambiguous cases where the two models would be most interestingly split. A corpus retaining neutrals would sharpen the comparison.
IndexFlatIPis exact but scans linearly; past ~1M vectors this needsIndexIVFFlat.- Embeddings are of full review text, which for long reviews averages several opinions into one vector. Sentence-level chunking would make retrieval more precise.