AI Agents
FilmRisk.AI: scoring film risk with ML and Bayesian priors
· Updated · 5 min read
S
Get AI Agent RFP Template
The RFP I use for agent discovery — scope, eval, and pricing.
No spam, unsubscribe anytime. I only email teardowns.
The short answer
FilmRisk.AI scores a film's greenlight risk from 2,200+ Indian films by mixing a gradient-boosted model with Bayesian priors. Prior pulls thin-data predictions back to base rates. We backtest religiously and publish the failure points — because scoring moves money and pretending a model is magic ends careers.
Give a raw ML model a debut director with a Khan attached, and it screams "blockbuster." The base rate says debut directors hit ~15% of the time. One of those two is lying to you, and if you're greenlighting a ₹100Cr film, you'd better know which. FilmRisk.AI exists to render exactly that judgment — and because wrong scores move real money, honesty rails are built into the architecture, not bolted onto the marketing.
The 10-component engine
The internal scoring blends 10 feature groups — genre, cast, director, release window, budgets, star momentum, sequel flag, production house, language, and seasonality — each fed into the model as context. The architecture:
- Gradient-boosted ML learns rich interactions between the 10 components.
- Bayesian priors anchor thin groups: when a 1-film director appears, the score defers toward the category average instead of over-committing.
- Backtesting is non-negotiable — runs on the full dataset every change.
The data: 2,200+ Indian films, structured for learning
The dataset wasn't a CSV download — it was assembled from public box-office reports, trade publications (Box Office India, Bollywood Hungama, Koimoi), and Wikipedia filmographies, then cleaned over three weekends. Each film gets:
| Feature group | Fields | Source |
|---|---|---|
| Genre | Primary, secondary, tertiary | Trade publications |
| Cast | Top 5 billed, star power index (Google Trends 5-yr) | IMDb + Trends |
| Director | Film count, hit ratio, avg budget | Wikipedia + trade |
| Release window | Month, festival clash, holiday flag | Calendar + trade |
| Budgets | Production, marketing (where reported) | Trade estimates |
| Star momentum | 12-month Trends slope, recent hit/flop | Google Trends |
| Sequel flag | Binary + franchise strength | Wikipedia |
| Production house | Track record, distribution reach | Trade |
| Language | Hindi, Tamil, Telugu, Malayalam, etc. | Certification |
| Seasonality | Quarter, holiday density | Calendar |
Missing data strategy: for budgets (often unreported), we used genre+star+director bucket medians as priors — which is exactly where the Bayesian layer earns its keep.
Why the prior saved us
The raw GBM loved stars. A debut director with a Khan attached would spike the score — but the base rate for debut directors is terrible (~15% hit rate vs 45% for 5+ films). The prior pulls that prediction toward the category average (genre + budget + language) proportional to how thin the director's own sample is. Mathematically:
posterior ∝ likelihood × prior
Where the prior is the industry base rate for that (genre, budget, language) bucket. The more films the director has, the more the data speaks; the fewer, the more the prior rules.
This single mechanism turned "star-driven overconfidence" into "star-weighted caution."
The stack
- Python 3.11, scikit-learn (HistGradientBoostingRegressor), PyMC for priors
- Streamlit for the scoring UI — producers paste a film's params, get score + confidence interval
- GitHub Actions runs full backtest on every push; fails if MAE regresses >2%
- DVC for data versioning — the 2,200-film CSV is tracked with SHA
The training loop (what actually ran)
Here's the whole pipeline, simplified but structurally faithful — GBM for the likelihood, a hierarchical PyMC model for the priors, and a sample-size-weighted blend at prediction time:
# train.py — simplified
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
import pymc as pm
# 1. Load DVC-tracked data
df = pd.read_csv('data/films.csv') # 2,200 rows, 47 features
# 2. Feature engineering
df['star_momentum'] = df['google_trends_slope_12m']
df['director_experience'] = df['director_film_count'].clip(upper=20)
df['budget_bucket'] = pd.qcut(df['production_budget'], q=5, labels=False)
# 3. GBM for likelihood
gbm = HistGradientBoostingRegressor(
max_iter=300, learning_rate=0.05, max_depth=6,
early_stopping=True, validation_fraction=0.15
)
X = df.drop('box_office', axis=1)
y = df['box_office']
gbm.fit(X, y)
# 4. Bayesian prior per (genre, budget_bucket, language)
with pm.Model() as prior_model:
# Hierarchical prior: genre → budget → language
mu_genre = pm.Normal('mu_genre', mu=0, sigma=1, shape=n_genres)
mu_budget = pm.Normal('mu_budget', mu=0, sigma=1, shape=n_budgets)
mu_lang = pm.Normal('mu_lang', mu=0, sigma=1, shape=n_langs)
# Director-level shrinkage
director_sd = pm.HalfNormal('director_sd', sigma=1)
director_offset = pm.Normal('director_offset', mu=0, sigma=director_sd, shape=n_directors)
# Likelihood
pred = mu_genre[genre_idx] + mu_budget[budget_idx] + mu_lang[lang_idx] + director_offset[director_idx]
pm.Normal('obs', mu=pred, sigma=pm.HalfNormal('sigma', 1), observed=y)
trace = pm.sample(2000, tune=1000, target_accept=0.9)
# 5. Combine: posterior = GBM + prior (weighted by director sample size)
def predict(film):
gbm_pred = gbm.predict(film.X)
prior_mean = trace.posterior['pred'].mean().values[film.idx]
weight = min(film.director_count / 5, 1.0) # data speaks after 5 films
return weight * gbm_pred + (1 - weight) * prior_mean
This runs in ~4 minutes on a MacBook M2. The PyMC sampling is the bottleneck — we cache the trace and only re-sample when director counts change significantly.
The failure modes we publish (and why)
Honesty isn't a marketing angle — it's a product requirement. The UI shows three failure modes explicitly:
- Debut director in thin genre — Score shows wide confidence interval (±35%). The engine knows it doesn't know.
- Star-heavy bundle in saturated genre — Score caps at market prior. No "this will be a blockbuster" hallucination.
- Sequel with declining franchise momentum — Score penalizes diminishing returns. The engine sees the trend.
These aren't hidden in a methodology doc — they're on the score card. Every score card shows: point estimate, 95% CI, failure mode flags, and the feature importance breakdown for that specific film.
The scoring UI (what a producer actually sees)
The Streamlit UI has three tabs:
Tab 1: Single Film Score — Paste params, get score + 95% CI + failure mode flags + feature importance waterfall.
Tab 2: Slate Planner — Upload CSV of 50 films, get ranked slate with portfolio risk metrics (expected value, portfolio variance, max drawdown).
Tab 3: Backtest Explorer — Pick any historical film, see what the engine would have scored it at release, vs actual outcome. This is the trust builder.
The backtest protocol (what "every commit" actually means)
The CI pipeline on GitHub Actions:
# .github/workflows/backtest.yml
name: Full Backtest
on: [push, pull_request]
jobs:
backtest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: iterative/setup-dvc@v1
- name: Pull data
run: dvc pull
- name: Train & backtest
run: |
python train.py
python backtest.py --full-dataset
- name: Check MAE regression
run: |
python -c "
import json
with open('metrics.json') as f:
m = json.load(f)
assert m['mae'] < 0.18 * 1.02, f'MAE regression: {m[\"mae\"]}'
"
The pipeline fails if MAE regresses >2% from baseline. This is the guardrail that keeps the engine honest — no "ship and pray."
What honest backtesting surfaced
We run the full backtest on every model change — not a held-out slice, the entire 2,200-film dataset. Three things the logs forced us to admit:
- Some star-heavy bundles score under the market prior — and the engine says so. A big cast in a thin genre doesn't guarantee a hit; the prior wins.
- Sequel and date signals beat cast creativity in some timeless categories. The engine shows the effect honestly; the feature importance table doesn't lie.
- Failure mode: this is a selection engine, not a prediction oracle. For a producer, the score helps pick; it doesn't guarantee a hit. The UI makes this boundary visible — every score shows its confidence interval.
Numbers that matter
| Metric | Value |
|---|---|
| Training corpus | 2,200+ Indian films |
| Feature groups | 10 |
| Priors | Category base rates (genre × budget × language) |
| Backtest cadence | Every commit |
| Published failure points | Yes — see repo |
| MAE (holdout) | ~18% of actual box office |
The backtest results are published on the repo. Anyone can re-run; no one can re-fake — the sample is pinned with a SHA.
What I'd do differently
- Ship the API before the UI. The engine's value is a promptable scoring grid; I papered over it for a pretty Streamlit front-end.
- Log prior-vs-likelihood disagreement. The most interesting films are the ones where the GBM and the prior fight — that disagreement score is a product feature I left in the logs.
- Widen beyond Hindi cinema sooner. The Tamil and Telugu base rates differ enough that the language prior deserved its own hierarchy level from day one.
A model that can say "I don't know" is worth more than a model that's confidently wrong — in film, in credit, in anything where a score moves money.
For PMs: this is what measure-before-judge looks like when your product is literally judgment. Pair it with numbers-led product decisioning for the same muscle in SaaS, or see how the same honesty-rails thinking applies to an in-browser voice agent.
Frequently asked questions
Can machine learning predict a movie's box office?+
Better than a gut call when you quantify priors. We rank roles on 2,200+ films with GBM + Bayesian blending — no one guarantees a hit.
What makes a risk engine 'honest'?+
Backtests everyone can see — training accuracy AND failure modes. FilmRisk publishes both so users can validate.
Why Bayesian priors on top of gradient boosting?+
ML overfits a handful of stars; priors pull hard toward the industry base rate when data is thin. That's what changed the output.