1c801c5a39
- raw: raw/xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md (created) - wiki: wiki/concepts/semantic-similarity-rating-ssr.md (created) - index: updated - log: updated
63 lines
3.3 KiB
Markdown
63 lines
3.3 KiB
Markdown
---
|
|
created: 2026-06-12
|
|
updated: 2026-06-12
|
|
sources: [xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md]
|
|
tags: [llm, market-research, synthetic-consumers, semantic-similarity-rating]
|
|
---
|
|
|
|
# Semantic Similarity Rating (SSR) — Synthetic Consumer Research
|
|
|
|
## Overview
|
|
|
|
Semantic Similarity Rating (SSR) is a method developed by Colgate-Palmolive and PyMC Labs (2025) that uses LLMs to predict consumer purchase intent with ~90% correlation attainment. Instead of asking an LLM for a direct numerical rating (which produces mediocre results), SSR uses role-play and semantic analysis to extract realistic consumer preferences.
|
|
|
|
## The Problem with Direct LLM Ratings
|
|
|
|
Asking an LLM "Rate this product from 1 to 5" produces safe, middle-of-the-road responses that don't reflect real consumer behavior. LLMs tend to hedge and produce average scores when asked for direct numerical ratings.
|
|
|
|
## The SSR Method
|
|
|
|
1. **Demographic profiling** — The LLM is given a specific demographic profile (age, income bracket, lifestyle)
|
|
2. **Product concept exposure** — The LLM is shown a product concept or description
|
|
3. **Role-played free-text response** — The LLM is asked to role-play as that consumer and write down raw, unfiltered thoughts about the product
|
|
4. **Semantic translation** — A semantic model converts the free-text thoughts into a numerical score
|
|
|
|
## Key Results
|
|
|
|
- Tested against **57 real corporate surveys** and **9,300 actual human responses**
|
|
- Synthetic AI consumers matched real human buying behavior with **~90% correlation attainment**
|
|
- Accurately mirrored how different age brackets and income levels react to price changes
|
|
- Provided qualitative feedback that was deeper and more critical than human-written responses
|
|
|
|
## Important Caveats
|
|
|
|
- The metric is **correlation attainment** (not raw accuracy): the AI panel hits ~90% of human test-retest reliability
|
|
- Measures **survey purchase intent**, not actual purchase behavior
|
|
- Works best in **categories the model already knows well** during training
|
|
- Independent replication attempts have struggled to match the paper's results (per @miromusing)
|
|
|
|
## Implications for Market Research
|
|
|
|
- **Speed:** Can simulate 1,000 hyper-targeted customer interviews overnight
|
|
- **Cost:** Eliminates the need for expensive survey panels
|
|
- **Granularity:** Can A/B test pricing across every demographic instantly
|
|
- **Depth:** Provides richer qualitative feedback than typical survey responses
|
|
|
|
## Risks & Limitations
|
|
|
|
- **Data feedback loop:** LLMs are trained on data from the very market research they would replace — if that data source dries up, future models may lose accuracy
|
|
- **Replicability:** Early independent attempts suggest the method is not yet consistent enough to fully replace traditional research
|
|
- **Category dependency:** Performance degrades for niche or novel product categories
|
|
- **Survey vs. reality:** Measures stated purchase intent, not actual purchase behavior (a well-known gap in traditional research too)
|
|
|
|
## Related Concepts
|
|
|
|
- [LLM Knowledge Base](concepts/llm-knowledge-base.md) — foundational pattern for LLM-curated knowledge
|
|
- Synthetic data generation
|
|
- AI-powered market research
|
|
- Role-play prompting techniques
|
|
|
|
## Sources
|
|
|
|
- [X Post by @HowToAI_](https://x.com/HowToAI_/status/2065118982659883350) (2026-06-11)
|
|
- [arXiv Paper](https://arxiv.org/abs/2510.08338) — Colgate-Palmolive + PyMC Labs, 2025
|