How this system is built

What this is

One person, one pipeline, every layer: data ingestion, feature engineering, model training, evaluation, and live serving — for predicting Counter-Strike 2 match outcomes.

• Ingestion: match pages, per-map results, round histories, player scoreboards, and full demo files (a multi-terabyte archive parsed down to per-kill positions and per-round economy), plus an hourly odds snapshot tape and prediction-market quotes.
• Feature pipeline with strict leak prevention: every feature is computed strictly before the match it describes (< cutoff, never <=); rosters are attributed from per-match lineups, not a player's current team; the model is trained with team-swap augmentation so it has no side bias.
• Model lifecycle: a map-level gradient-boosting model (the map is the atomic unit; series probability is composed analytically by format). Twelve versions have been promoted, each through the same evaluation gates: beat the incumbent out-of-sample, hold up on weak-slice checks (missing-rank and deep-tier matches), stay monotone across walk-forward folds — with a rollback snapshot kept every time. Candidates that don't clear the gates don't ship, no exceptions.
• Live, self-graded output: predictions are stamped to an append-only log before matches start, graded automatically against results, and published here — pending picks visible before outcomes are known.

Case study — my backtest was lying to me by 3.5x

Early versions of this project produced backtests with great ROI. The numbers were exciting, repeatable, and wrong.

The bugs

An audit of every historical profitability harness found that none of them could be trusted:

• The extended backtest joined predictions to a historical odds table with no deduplication after the join — matches with multiple odds rows became multiple "bets", inflating the bet count about 3.5x and compounding the winners. That one bug was the source of the entire "profitable in 2025" narrative.
• Two other harnesses evaluated the model against synthetic odds derived from its own Elo ratings — so their "ROI" measured disagreement between the model and Elo, not profit against a market. One Monte Carlo layer sampled outcomes from the model's own probabilities, which is circular by construction.
• Retro-scraped historical odds carried scrape timestamps, not capture timestamps — "closing line" semantics that could never be verified.

The rebuild

• Predictions stamped before matches. Every flag is written to an append-only log with the odds at flag time, before the match starts. No retroactive entries, ever.
• Automatic grading. Results are joined by pipeline, not by hand; walkovers void; flat 1-unit stakes at the locked price.
• Only self-scraped closing odds are decision-grade. The hourly snapshot tape (Pinnacle, bet365) is the only closing-line source trusted for verdicts. Retro-scraped odds are labeled proxies on every report that uses them and are never allowed to drive a promotion or profitability claim.
• Per-bet CLV. Each flag is compared to the captured closing line of the same book — closing-line value converges 5–10x faster than ROI, so it is the first honest read on edge. Eras with different close quality are never pooled.
• Sample-size floors before any verdict. No ROI judgment under 100 settled bets; no CLV verdict under 150 valid pairs. The report generator prints the gate status instead of a conclusion.

What the honest numbers say right now

As of 2026-07-17, the forward ledger (simulation, flat 1u, flag-time entry) reads:

A separate walk-forward historical backtest (per-fold models trained strictly before their test windows, ~4,300 out-of-sample series, the ~280 joinable to usable historical odds) lands at roughly −5% ROI with a confidence interval spanning zero — clearly labeled as resting on proxy odds.

How ideas earn their way in

Every feature idea goes through a measure-before-build protocol: before writing pipeline code, run a cheap diagnostic asking whether the current model's out-of-sample residuals contain the proposed signal at all. If the live model already prices it, the idea dies in an afternoon instead of costing a feature cycle. Three real outcomes:

Schedule strength shippedpassed diagnostic

Hypothesis: recent-form ratings don't ask who the form was earned against. The diagnostic found the effect in the live model's residuals at 4.6–5.4 standard errors, stable across all four chronological folds, and it survived six adversarial refutation attempts (placebo permutations, kitchen-sink controls, block bootstraps). Interestingly, the original interaction hypothesis was wrong — the data said the fix was a simple additive feature. Built, selected by the trainer on its own merit, shipped in model v12.

Editorial star ratings killed before buildscrape never paid for

Match-importance stars would have cost a scraping fight to collect. The diagnostic went first: importance is a property of the match, identical for both teams, so by symmetry it can't carry a winner signal directly — and all twelve interaction probes plus the recalibration channel came back null (≤0.0005 log-loss ceiling). The scrape was never built. Cost of the answer: one afternoon, zero code in production.

Lineup-continuity features built, then killed with evidence

Assumption: the model must be miscalibrated on teams with roster churn, since their ratings were earned by a different five. Tested on 3,588 out-of-sample predictions: the lowest-continuity bucket was the best calibrated, and the correlation between residuals and continuity was ≈0. The modeling thread was closed with three independent nulls. The real value lived at a different layer — a serve-time guardrail that flags when tonight's announced lineup differs from the roster the ratings were earned by — and that part stayed.

Stack

• Core: Python, pandas, DuckDB (single-file analytical DB), scikit-learn gradient-boosted trees with isotonic/sigmoid calibration; feature store as Parquet.
• Data collection: Selenium with Cloudflare-aware scraping (undetected-chromedriver, persistent profiles, rate limiting), REST polling for odds and prediction markets, and bulk demo parsing (demoparser2/awpy) tuned for CS2's 64-tick GOTV format.
• Serving: this site — plain static files, no build step, everything inlined; pages are regenerated by the daily pipeline run, and the veto predictor runs entirely in your browser.
• Automation: Windows Task Scheduler jobs — an hourly odds/lineup capture cycle and a daily 15-step update (ingest, derived tables, grading, page publishing) with lock files to keep the single-writer database honest.
• Testing: ~1,000 pytest tests, including tests that pin data conventions (units, sign conventions, era boundaries) so past data bugs can't silently return.