How well does WIN work?
Live A/B test. Each new visitor is put in one group at random: most get AI re-ranking, some get the engine only. Rates are per card shown; the range is a 95% confidence interval, so groups only really differ when their ranges don't overlap.
Offline results (MovieLens)
Before any real users: MovieLens users each gave their last 5 liked movies, and we checked whether the picks matched what they liked next (581 users for search, 100 for the AI tests to keep costs low).
| Change | Recall | Result |
|---|---|---|
| Hybrid search (keywords + vectors) | recall@30 0.090 → 0.111 | +24%, significant |
| AI re-ranking of the top 30 | recall@10 0.032 → 0.046 | +43%, significant (100 users) |
| AI alone, no engine | recall@10 0.032 → 0.046 | not significant; 11.5% of titles not in the catalog |