diff --git a/raw/xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md b/raw/xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md new file mode 100644 index 0000000..e484288 --- /dev/null +++ b/raw/xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md @@ -0,0 +1,64 @@ +--- +type: xpost +source_url: https://x.com/HowToAI_/status/2065118982659883350 +retrieved: 2026-06-12 +author: "@HowToAI_" +is_thread: true +quote_count: 872 +retweet_count: 139 +paper_url: https://arxiv.org/abs/2510.08338 +tags: [llm, market-research, semantic-similarity-rating, colgate, synthetic-consumers] +--- + +# Colgate: LLMs Predict Purchase Intent at 90% Accuracy via Semantic Similarity Rating (SSR) + +**Author:** How To AI (@HowToAI_) +**Posted:** 2026-06-11 +**Source:** https://x.com/HowToAI_/status/2065118982659883350 + +## Summary + +Colgate-Palmolive (in collaboration with PyMC Labs, 2025) published a paper demonstrating that LLMs can predict real purchase intent with ~90% correlation attainment using a method called **Semantic Similarity Rating (SSR)**. + +### The Problem + +Asking an LLM directly "Rate this product from 1 to 5" produces safe, middle-of-the-road garbage — mediocre numerical ratings that don't reflect real consumer behavior. + +### The SSR Method + +Instead of asking for a number, researchers: +1. Gave the LLM a **demographic profile** (age, income bracket, etc.) +2. Showed it a **product concept** +3. Asked it to **role-play as a consumer** and write down raw, unfiltered thoughts +4. Used a **semantic model** to translate those written thoughts into a numerical score + +### Results + +- Tested against **57 real corporate surveys** and **9,300 actual human responses** +- Synthetic AI consumers matched real human buying behavior with **~90% reliability** +- Perfectly mirrored how different age brackets and income levels react to price changes +- Provided detailed, qualitative feedback that was deeper and more critical than what actual humans wrote + +### Key Clarifications (from thread) + +- The metric is **correlation attainment**: the AI panel hits ~90% of human test-retest reliability (i.e., ~90% of the way to how consistent real people are with their own answers on a retest) +- It measures **survey purchase intent**, not actual purchase behavior +- Works best in **categories the model already knows well** + +### Implications + +- Destroys the economics of traditional market research +- Can simulate 1,000 hyper-targeted customer interviews overnight +- Can A/B test pricing across every demographic instantly +- No need to wait a month to see if a product will sell + +### Critical Perspectives (from comments) + +- @miromusing: Attempted replication struggled to get the same results — approach is useful but not yet consistent/reliable enough to replace real research +- @cenkercakin: The LLM is trained on data from the very resource it replaces — what happens when that data source dries up? +- @chris_byrne: Prof. Juster invented the Buyer Intention Scale (0-10) in 1966; Bain later rebadged it as NPS + +### Paper Reference + +- **Title:** (Colgate-Palmolive + PyMC Labs, 2025) +- **arXiv:** https://arxiv.org/abs/2510.08338 diff --git a/wiki/concepts/semantic-similarity-rating-ssr.md b/wiki/concepts/semantic-similarity-rating-ssr.md new file mode 100644 index 0000000..dddd6dd --- /dev/null +++ b/wiki/concepts/semantic-similarity-rating-ssr.md @@ -0,0 +1,63 @@ +--- +created: 2026-06-12 +updated: 2026-06-12 +sources: [xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md] +tags: [llm, market-research, synthetic-consumers, semantic-similarity-rating] +--- + +# Semantic Similarity Rating (SSR) — Synthetic Consumer Research + +## Overview + +Semantic Similarity Rating (SSR) is a method developed by Colgate-Palmolive and PyMC Labs (2025) that uses LLMs to predict consumer purchase intent with ~90% correlation attainment. Instead of asking an LLM for a direct numerical rating (which produces mediocre results), SSR uses role-play and semantic analysis to extract realistic consumer preferences. + +## The Problem with Direct LLM Ratings + +Asking an LLM "Rate this product from 1 to 5" produces safe, middle-of-the-road responses that don't reflect real consumer behavior. LLMs tend to hedge and produce average scores when asked for direct numerical ratings. + +## The SSR Method + +1. **Demographic profiling** — The LLM is given a specific demographic profile (age, income bracket, lifestyle) +2. **Product concept exposure** — The LLM is shown a product concept or description +3. **Role-played free-text response** — The LLM is asked to role-play as that consumer and write down raw, unfiltered thoughts about the product +4. **Semantic translation** — A semantic model converts the free-text thoughts into a numerical score + +## Key Results + +- Tested against **57 real corporate surveys** and **9,300 actual human responses** +- Synthetic AI consumers matched real human buying behavior with **~90% correlation attainment** +- Accurately mirrored how different age brackets and income levels react to price changes +- Provided qualitative feedback that was deeper and more critical than human-written responses + +## Important Caveats + +- The metric is **correlation attainment** (not raw accuracy): the AI panel hits ~90% of human test-retest reliability +- Measures **survey purchase intent**, not actual purchase behavior +- Works best in **categories the model already knows well** during training +- Independent replication attempts have struggled to match the paper's results (per @miromusing) + +## Implications for Market Research + +- **Speed:** Can simulate 1,000 hyper-targeted customer interviews overnight +- **Cost:** Eliminates the need for expensive survey panels +- **Granularity:** Can A/B test pricing across every demographic instantly +- **Depth:** Provides richer qualitative feedback than typical survey responses + +## Risks & Limitations + +- **Data feedback loop:** LLMs are trained on data from the very market research they would replace — if that data source dries up, future models may lose accuracy +- **Replicability:** Early independent attempts suggest the method is not yet consistent enough to fully replace traditional research +- **Category dependency:** Performance degrades for niche or novel product categories +- **Survey vs. reality:** Measures stated purchase intent, not actual purchase behavior (a well-known gap in traditional research too) + +## Related Concepts + +- [LLM Knowledge Base](concepts/llm-knowledge-base.md) — foundational pattern for LLM-curated knowledge +- Synthetic data generation +- AI-powered market research +- Role-play prompting techniques + +## Sources + +- [X Post by @HowToAI_](https://x.com/HowToAI_/status/2065118982659883350) (2026-06-11) +- [arXiv Paper](https://arxiv.org/abs/2510.08338) — Colgate-Palmolive + PyMC Labs, 2025 diff --git a/wiki/index.md b/wiki/index.md index cd89a19..2000a0f 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -33,6 +33,7 @@ | [Subconscious Agent v2.1](concepts/subconscious-agent.md) | Hard Synthesis, Execution Gap, bekannter Bug, Outcomes | raw/other/2026-06-07_subconscious-runner-script.md | | [Pro-Leben-Direktive](concepts/pro-leben-directive.md) | Philosophischer Rahmen, Kernwerte, Anti-Positionen, Anwendung | context-tree | | [Qualitäts-Standard](concepts/quality-standard.md) | "Heilige Scheiße, das ist fertig" | context-tree | +| [Semantic Similarity Rating (SSR)](concepts/semantic-similarity-rating-ssr.md) | LLM-basierte Kaufintentions-Vorhersage mit 90% Korrelation | xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md | ## Decisions @@ -48,3 +49,4 @@ | `raw/blog/2026-06-04_anthropic-fordert-ki-pause.md` | blog | Anthropic fordert weltweite Aussetzung der KI-Entwicklung | | `raw/blog/2026-06-05_ai-news-roundup-may-2026.md` | blog | Latest AI & Technology News Roundup – May 2026 | | `raw/other/2026-06-07_subconscious-runner-script.md` | other | Subconscious Agent Evidence Runner Script v2.1 | +| `raw/xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md` | xpost | Colgate: LLMs Predict Purchase Intent at 90% via SSR | diff --git a/wiki/log.md b/wiki/log.md index 335f412..9b2d52c 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -60,3 +60,11 @@ - Fokus auf bleibende Erkenntnisse; temporäre/archivierte Details (einzelne Subconscious-Runs, Tagebuch-Einträge) nicht migriert - Duplikate (identische Facts in architecture + ecosystem) dedupliziert - Alte Tool-Integrationen (ByteRover, GBrain, SearXNG) historisch dokumentiert, nicht als aktive Empfehlungen + +## [2026-06-12] Ingest | Colgate SSR — LLM-basierte Kaufintentions-Vorhersage +**Type:** ingest | **Scope:** raw/xpost, wiki/concepts +**Actions:** +- raw: `raw/xpost/2026-06-11_colgate-llm-purchase-intent-ssr.md` (created — X-Post von @HowToAI_ über Colgate/PyMC Labs SSR-Studie) +- wiki: `concepts/semantic-similarity-rating-ssr.md` (created — Konzeptseite zu Semantic Similarity Rating) +- index: updated (neuer Concepts-Eintrag + Raw Sources-Eintrag) +- log: updated