65 lines
2.8 KiB
Markdown
65 lines
2.8 KiB
Markdown
|
---
|
||
|
|
type: xpost
|
||
|
|
source_url: https://x.com/HowToAI_/status/2065118982659883350
|
||
|
|
retrieved: 2026-06-12
|
||
|
|
author: "@HowToAI_"
|
||
|
|
is_thread: true
|
||
|
|
quote_count: 872
|
||
|
|
retweet_count: 139
|
||
|
|
paper_url: https://arxiv.org/abs/2510.08338
|
||
|
|
tags: [llm, market-research, semantic-similarity-rating, colgate, synthetic-consumers]
|
||
|
|
---
|
||
|
|
|
||
|
|
# Colgate: LLMs Predict Purchase Intent at 90% Accuracy via Semantic Similarity Rating (SSR)
|
||
|
|
|
||
|
|
**Author:** How To AI (@HowToAI_)
|
||
|
|
**Posted:** 2026-06-11
|
||
|
|
**Source:** https://x.com/HowToAI_/status/2065118982659883350
|
||
|
|
|
||
|
|
## Summary
|
||
|
|
|
||
|
|
Colgate-Palmolive (in collaboration with PyMC Labs, 2025) published a paper demonstrating that LLMs can predict real purchase intent with ~90% correlation attainment using a method called **Semantic Similarity Rating (SSR)**.
|
||
|
|
|
||
|
|
### The Problem
|
||
|
|
|
||
|
|
Asking an LLM directly "Rate this product from 1 to 5" produces safe, middle-of-the-road garbage — mediocre numerical ratings that don't reflect real consumer behavior.
|
||
|
|
|
||
|
|
### The SSR Method
|
||
|
|
|
||
|
|
Instead of asking for a number, researchers:
|
||
|
|
1. Gave the LLM a **demographic profile** (age, income bracket, etc.)
|
||
|
|
2. Showed it a **product concept**
|
||
|
|
3. Asked it to **role-play as a consumer** and write down raw, unfiltered thoughts
|
||
|
|
4. Used a **semantic model** to translate those written thoughts into a numerical score
|
||
|
|
|
||
|
|
### Results
|
||
|
|
|
||
|
|
- Tested against **57 real corporate surveys** and **9,300 actual human responses**
|
||
|
|
- Synthetic AI consumers matched real human buying behavior with **~90% reliability**
|
||
|
|
- Perfectly mirrored how different age brackets and income levels react to price changes
|
||
|
|
- Provided detailed, qualitative feedback that was deeper and more critical than what actual humans wrote
|
||
|
|
|
||
|
|
### Key Clarifications (from thread)
|
||
|
|
|
||
|
|
- The metric is **correlation attainment**: the AI panel hits ~90% of human test-retest reliability (i.e., ~90% of the way to how consistent real people are with their own answers on a retest)
|
||
|
|
- It measures **survey purchase intent**, not actual purchase behavior
|
||
|
|
- Works best in **categories the model already knows well**
|
||
|
|
|
||
|
|
### Implications
|
||
|
|
|
||
|
|
- Destroys the economics of traditional market research
|
||
|
|
- Can simulate 1,000 hyper-targeted customer interviews overnight
|
||
|
|
- Can A/B test pricing across every demographic instantly
|
||
|
|
- No need to wait a month to see if a product will sell
|
||
|
|
|
||
|
|
### Critical Perspectives (from comments)
|
||
|
|
|
||
|
|
- @miromusing: Attempted replication struggled to get the same results — approach is useful but not yet consistent/reliable enough to replace real research
|
||
|
|
- @cenkercakin: The LLM is trained on data from the very resource it replaces — what happens when that data source dries up?
|
||
|
|
- @chris_byrne: Prof. Juster invented the Buyer Intention Scale (0-10) in 1966; Bain later rebadged it as NPS
|
||
|
|
|
||
|
|
### Paper Reference
|
||
|
|
|
||
|
|
- **Title:** (Colgate-Palmolive + PyMC Labs, 2025)
|
||
|
|
- **arXiv:** https://arxiv.org/abs/2510.08338
|