Gorgias Releases 8,000 Open Conversations: AI vs Human Support Stats
The old playbook for B2B SaaS customer acquisition is to put "3x performance improvement" in your pitch deck. Gorgias (ARR ~$100M) did the opposite: it open-sourced its test code and ran its product head-to-head against 18 competitors using 8,356 real customer conversations. This kind of "publicly exposing your weaknesses" move works better than any marketing spin.
Data breakdown: lost on response speed, won on overall quality
- Pre-sales stumbles: Envive scored 72, Gorgias 65. The reason is straightforward: Gorgias averaged 18.4 seconds per response, with 28% of answers taking over 20 seconds; Envive clocked in at just 7.9 seconds. In shopping scenarios, slowness means lost customers.
- Support dominance with blind spots: Gorgias took the top overall spot, but its automation resolution rate—a core metric—was surpassed by Yuma. Gorgias admitted this on its own page instead of hiding it.
- Honest handling of weighting: The report explicitly notes that if pre-sales scores were calculated using support-scene weights (where speed counts for only 10%), Gorgias would have scored 74.3 and taken first place. Instead, it chose the stricter standard (speed at 25%) because it knows buyers care more about speed.
Methodology: how to replicate this trust
- Full-stack open source: Publish the testing framework (Harness) on GitHub so buyers can reproduce the results and verify there's no score manipulation.
- Real-time data: Cover 212+ independent sites and refresh data weekly to avoid the stale feeling of "static benchmarks."
- Own your disadvantages: Put the metrics you lost in a prominent position. As SaaStr's founder put it: "You set the scoring criteria, so there's bias—state the bias, then make everything verifiable."
For teams doing competitive analysis on AI Agents, this approach is worth copying: buyers now use AI Agents to filter vendors themselves, and transparent comparison data is the ticket to making it onto AI procurement shortlists. Showing data beats shouting slogans every time.
Source · SaaStrAI: Read the original article →