LLM Evaluation Checklist for Recommendation Teams in 2026

Measure precision@K, recall@K, and NDCG@K for the top recommended items at K=5, 10, and 20. Use held-out user interaction data as ground truth. NDCG@10 is the most commonly used metric for ranking quality comparison across system versions.

Evaluate the correlation between predicted relevance scores and actual click-through rates. If your model ranks items highly but users do not click them, relevance scoring is miscalibrated. Track prediction accuracy by item category and user segment.

Quantify how different recommendations are across users. If all users see similar recommendations, personalization is failing. Calculate pairwise recommendation overlap across user pairs and track average overlap. Target less than 30% overlap for non-trending content.

Measure what percentage of your catalog appears in recommendations across all users. Popularity bias concentrates recommendations on a small percentage of items. Track catalog coverage weekly and set minimum thresholds per category.

Test whether recommendations adapt to contextual signals: time of day, day of week, season, device type, and user location. An LLM-powered recommender should leverage context that collaborative filtering misses. Compare contextual vs. non-contextual recommendation quality.

Evaluate whether the system correctly accounts for the user's recent interaction sequence, not just aggregate preferences. The most relevant next item depends on what the user just viewed. Test with session-aware evaluation protocols.

If the system generates natural language explanations for recommendations, evaluate their accuracy, helpfulness, and alignment with the actual recommendation reasoning. Misleading explanations erode trust. Human-evaluate explanations on a 5-point scale.

If recommending across domains (e.g., products and content), test whether user preferences transfer correctly between domains. Incorrect cross-domain inference leads to irrelevant suggestions. Evaluate domain transfer quality per domain pair.

Test whether the system correctly adjusts recommendations after explicit negative signals: 'not interested' clicks, returns, or low ratings. Ignoring negative signals frustrates users. Measure recommendation adjustment speed after negative feedback.

Evaluate how quickly new items enter the recommendation pool and how effectively the system balances new content with proven performers. Stale recommendations reduce engagement; too much novelty reduces relevance. Track new item recommendation rates.

Calculate the diversity within each recommendation list using category diversity, embedding distance, or content similarity metrics. A list of 10 nearly identical items is less useful than a diverse list of 10. Set minimum intra-list diversity thresholds.

Measure how recommendation diversity changes over time for individual users. Decreasing diversity over weeks indicates filter bubble formation. Track diversity trends and implement diversity floors that prevent recommendations from narrowing indefinitely.

Evaluate whether recommendations maintain balanced representation across content categories rather than over-indexing on the user's primary interest area. Users who buy electronics also eat food. Monitor category distribution per user over time.

Track the rate at which recommendations surface items users would not have discovered themselves but end up engaging with positively. Serendipitous recommendations are the unique value of recommender systems. Measure serendipity separately from relevance.

Implement automated detection of echo chambers where users receive increasingly narrow, self-reinforcing recommendations. Track opinion diversity in content recommendations and political/viewpoint balance. Alert when echo chamber indicators exceed thresholds.

Quantify the extent to which recommendations favor popular items over niche items disproportionately. Calculate the Gini coefficient of recommendation frequency across items. High Gini indicates excessive popularity bias that hurts long-tail discovery.

Map the Pareto frontier between diversity and relevance metrics. Increasing diversity typically reduces short-term relevance. Identify the optimal operating point for your business objectives. A/B test diversity levels to measure engagement impact.

Audit whether recommendations reflect appropriate demographic diversity in content creators, viewpoints, and representation. Biased recommendations can amplify societal biases at scale. Track diversity metrics across protected categories.

Measure what fraction of recommended items each user has not previously encountered. Extremely low novelty indicates the system is re-recommending known items. Set minimum novelty thresholds per recommendation position.

Collect explicit user feedback on recommendation diversity through surveys or UI controls. Users' desired diversity level varies by context and individual. Use feedback to calibrate diversity parameters per user segment.

Evaluate recommendation quality for users with fewer than 5 interactions. Cold start quality determines first-session retention. Compare new user engagement rates against established user baselines. LLM-powered systems should leverage demographic and contextual signals to compensate.

Measure how quickly new items in the catalog receive their first recommendations and interactions. Items that never get recommended never collect engagement signals, creating a feedback loop. Set maximum time-to-first-recommendation SLAs.

If using an onboarding questionnaire or preference selection flow, measure how much it improves recommendation quality versus zero-signal recommendations. Effective onboarding should measurably improve first-session engagement. A/B test onboarding variations.

Separately evaluate recommendation quality for users in the 10th, 25th, and 50th percentile of interaction volume. Aggregated metrics can hide poor performance for less active users. Set minimum quality thresholds per activity tier.

When collaborative filtering signals are insufficient, evaluate the quality of content-based or LLM-based fallback recommendations. The fallback system should provide reasonable recommendations, not random or purely popular items. Test fallback quality in isolation.

If using cross-platform user signals or pre-trained embeddings for cold start, measure their contribution to recommendation quality. Transfer signals should measurably improve cold start over baseline. A/B test with and without transfer.

Evaluate recommendations generated from within-session behavior only, without historical user data. Session-based recommendations are critical for anonymous users and first-time visitors. Benchmark against a random baseline.

Test how effectively the system uses item metadata (descriptions, categories, attributes) to recommend items with zero interaction history. Rich metadata should enable reasonable recommendations even for brand-new items. Compare metadata-only vs. interaction-based quality.

Track how many interactions a new user or item needs before recommendation quality reaches established user or item baselines. Faster warm-up means better early experience. Plot quality curves by interaction count.

Design and validate A/B testing methodology specifically for cold start scenarios, accounting for high variance and small sample sizes. Standard A/B testing assumptions often do not hold for new users. Use appropriate statistical methods.

Measure end-to-end recommendation latency from request to response at P50, P95, and P99. Most recommendation UIs require responses within 100-200ms. Track latency by request type (homepage, item page, search) and user segment.

Separately measure the latency contribution of LLM components (embedding generation, reranking, explanation generation) versus traditional components. LLM calls are often the latency bottleneck. Profile each LLM call independently.

Measure the staleness of user and item features used in recommendation generation. Stale features reduce personalization accuracy. Track feature computation lag and set freshness SLAs for real-time versus batch features.

Profile the candidate generation stage for throughput and latency. If using LLM-based candidate generation, compare its efficiency against traditional approximate nearest neighbor search. Target candidate generation under 50ms.

Track recommendation cache hit rates and measure the freshness-latency tradeoff. Cached recommendations are faster but may be stale. Set maximum cache TTL based on how quickly user preferences and item availability change.

Calculate the cost per recommendation request including embedding generation, LLM inference, and feature retrieval. Compare against the revenue per recommendation to ensure positive ROI. Track cost trends as usage scales.

Test that infrastructure auto-scales correctly under traffic spikes (flash sales, viral content, peak hours). Measure time-to-scale and latency during scaling events. Verify that quality does not degrade during scale-up.

Measure throughput for batch recommendation precomputation jobs that prepare offline recommendations. Batch jobs must complete within their scheduling window. Profile bottlenecks and plan capacity for catalog growth.

Benchmark feature store read latency for user and item features under concurrent load. Feature store performance directly impacts recommendation serving latency. Profile by feature type and query pattern.

Test system behavior when components fail: feature store unavailable, LLM service down, or embedding service degraded. The system should fall back to simpler recommendation strategies, not return errors. Verify fallback quality.

Verify that user randomization into treatment and control groups is truly random and consistent across sessions. Incorrect randomization invalidates all experiment results. Test randomization uniformity and verify no systematic biases.

Validate that experiments run with sufficient sample sizes to detect meaningful effect sizes with adequate statistical power (typically 80%+). Underpowered experiments lead to false negatives. Calculate required sample sizes before launching experiments.

Verify that primary and guardrail metrics for A/B tests capture both short-term engagement and long-term user value. Optimizing click-through rate alone can increase clickbait and reduce satisfaction. Include diversity, satisfaction, and retention metrics.

Test for spillover effects between treatment and control groups, especially in social or marketplace contexts. If treatment group behavior affects control group outcomes, standard A/B test analysis is invalid. Use cluster randomization if needed.

Monitor for novelty effects where users engage more with new recommendations simply because they are different, not because they are better. Run experiments for sufficient duration (minimum 2 weeks) to distinguish novelty from genuine improvement.

Implement automated checks that halt experiments if guardrail metrics (revenue, user retention, error rates) deteriorate beyond acceptable thresholds. Experiments should not cause measurable harm while running. Set and enforce guardrail thresholds.

Extend key experiments beyond the standard test period to measure long-term effects on user retention, lifetime value, and satisfaction. Short-term engagement gains sometimes reverse over longer periods. Plan holdout groups for longitudinal analysis.

Test for interactions between concurrent experiments that could amplify or cancel each other's effects. Multiple simultaneous experiments are common but their interactions are rarely tested. Implement interaction detection in your experimentation platform.

Account for position bias in recommendation evaluation: users are more likely to interact with items in prominent positions regardless of quality. Use inverse propensity weighting or position-debiased evaluation metrics. Validate correction effectiveness.

Maintain detailed documentation for every experiment: hypothesis, configuration, duration, results, and decisions. Enable experiment reproduction for validation. Track the cumulative impact of shipped experiments on key metrics.