knowledge-base/wiki/concepts/llm/semantic-similarity-rating-ssr.md

created	updated	sources	tags
2026-06-12	2026-06-12	xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md	llm market-research synthetic-consumers semantic-similarity-rating

Semantic Similarity Rating (SSR) — Synthetic Consumer Research

Overview

Semantic Similarity Rating (SSR) is a method developed by Colgate-Palmolive and PyMC Labs (2025) that uses LLMs to predict consumer purchase intent with ~90% correlation attainment. Instead of asking an LLM for a direct numerical rating (which produces mediocre results), SSR uses role-play and semantic analysis to extract realistic consumer preferences.

The Problem with Direct LLM Ratings

Asking an LLM "Rate this product from 1 to 5" produces safe, middle-of-the-road responses that don't reflect real consumer behavior. LLMs tend to hedge and produce average scores when asked for direct numerical ratings.

The SSR Method

1. Demographic profiling — The LLM is given a specific demographic profile (age, income bracket, lifestyle)
2. Product concept exposure — The LLM is shown a product concept or description
3. Role-played free-text response — The LLM is asked to role-play as that consumer and write down raw, unfiltered thoughts about the product
4. Semantic translation — A semantic model converts the free-text thoughts into a numerical score

Key Results

• Tested against 57 real corporate surveys and 9,300 actual human responses
• Synthetic AI consumers matched real human buying behavior with ~90% correlation attainment
• Accurately mirrored how different age brackets and income levels react to price changes
• Provided qualitative feedback that was deeper and more critical than human-written responses

Important Caveats

• The metric is correlation attainment (not raw accuracy): the AI panel hits ~90% of human test-retest reliability
• Measures survey purchase intent, not actual purchase behavior
• Works best in categories the model already knows well during training
• Independent replication attempts have struggled to match the paper's results (per @miromusing)

Implications for Market Research

• Speed: Can simulate 1,000 hyper-targeted customer interviews overnight
• Cost: Eliminates the need for expensive survey panels
• Granularity: Can A/B test pricing across every demographic instantly
• Depth: Provides richer qualitative feedback than typical survey responses

Risks & Limitations

• Data feedback loop: LLMs are trained on data from the very market research they would replace — if that data source dries up, future models may lose accuracy
• Replicability: Early independent attempts suggest the method is not yet consistent enough to fully replace traditional research
• Category dependency: Performance degrades for niche or novel product categories
• Survey vs. reality: Measures stated purchase intent, not actual purchase behavior (a well-known gap in traditional research too)

Related Concepts

• LLM Knowledge Base — foundational pattern for LLM-curated knowledge
• Synthetic data generation
• AI-powered market research
• Role-play prompting techniques

Sources

• X Post by @HowToAI_ (2026-06-11)
• arXiv Paper — Colgate-Palmolive + PyMC Labs, 2025