How often is Talqo wrong,
and how would you know?
It is a fair question and most answers to it are marketing. Here is ours, with the method, the number, and the calls it got wrong — including the one it is still getting wrong.
Last run 11 August 2026 · 46 of 47 checks correct · 12 of 13 scored calls with nothing wrong or missing.
Re-run from the repo with node benchmark/run.js && node benchmark/score.js. When the number moves, this page moves.
The method, so you can attack it
- The calls are synthetic and we say so. Not one line is copied or reconstructed from a customer conversation. A benchmark built on real calls could not be published — and a benchmark you cannot read is one you have to take on trust, which is the thing we are trying to avoid.
- They are engineered to be hard. Interruptions mid-sentence, hedged answers that look like commitments, Danish, Swedish and Norwegian dropped into English, numbers stated then revised, threads that start and never finish, and several "we'll see" moments that must not be scored as agreement. A benchmark of tidy calls measures nothing, because no real call is tidy.
- The answer sheets were written before the run and were not edited afterwards to match what came back.
- It runs the product, not a model. Every transcript goes through the same endpoint your calls would: the same prompt, the same parser, the same conservative downgrades. Prompting a model directly would measure something we do not ship.
- Three categories, and they are not equal. A missed fact costs a rep two minutes. A wrong fact travels — into a roll-up, into a board pack, out of the building. We count them separately because they are not the same failure.
The one it got wrong
A competitor's price landed in a Talqo field
On a call where the buyer opened with a rival's quote — "two hundred and ninety a seat, per month" — and asked us to match it, that 290 ended up in the structured price field on the deal record. We never quoted it. The rep on the call explicitly declined to.
How you would have caught it: the same record left the deal value empty, marked the price
pushback — not accepted, and its own written summary says
"Sigrid opened with a competitive price from a rival vendor at $290/seat/month with free integration, and asked Talqo to match it." Nothing downstream treated it as agreed money.
But a number we did not quote should not be in that field at all, and it is on the list to fix.
What it declined to say
Across the corpus it refused, among others: a buyer's off-hand guess at their own current costs ("that's a feeling, not a number"); a superseded price that was revised down later in the same call; an unnarrowed 350–450k range; a procurement threshold that looked like a deal value; a rep's own go-live promise; and every one of the "we'll see" endings. It did not once over-claim an acceptance.
A whole messy call, and everything Talqo said about it
The transcript below is one of the thirteen — the hardest kind: a procurement conversation that circles, a budget thread interrupted by someone walking into the room and never picked up again, and an ending where both people say out loud that nothing has been agreed. On the right is Talqo's output, unedited, including the fields it left empty.
- Deal value
- Not stated — no price was quoted on this call, so none was recorded.
- Price acceptance
- None — nothing to accept.
- Expected close date
- Not stated — the budget round "is normally October" and may have moved. That is not a date.
- Stage
- Needs Analysis
- Summary
- Rep spoke with Kjell Værnes about ~40 field service engineers who informally quote customers but whose on-site intelligence never reaches the business — a clear revenue leakage problem. No price was given at Kjell's request because procurement requires a formal requirement document before any number can be introduced. Rep agreed to send a template of requirement-document categories instead, and Kjell committed to checking the budget-round timeline with a colleague named Solveig.
- Next steps
- Rep sends requirement-document category framework to Kjell. Kjell asks Solveig about whether the budget round has moved from its usual October slot and feeds that back to the rep.
- Qualification
-
budget Unknown Budget round is likely October but may have moved; no budget amount or confirmed allocation discussed. authority Medium Kjell controls the requirement document and the business case but procurement runs the formal process; Solveig also appears relevant to timing. need High Kjell confirmed in his own words that lost on-site customer intelligence is the exact problem, with a vivid example of a missed line-replacement opportunity. timeline Low Budget round is normally October but Kjell said it may have moved this year and needs to check with Solveig — no confirmed date.
Why it prefers to be cautious
A false "unproven" costs a rep two minutes: they look at the deal, see we did not score something, and either supply the fact or ignore it. A false "validated" costs a sales leader their credibility, because it travels — and the first anyone hears of it is when a defended number turns out not to have been said by anybody.
So every ambiguous case resolves downward, and that rule lives in the code rather than in a prompt: an acceptance we cannot quote back to you is not an acceptance; a price met with silence is not a price agreed; an answer we do not recognise resolves to the least claiming state, never the friendliest. Across 13 calls built to tempt it, it never over-claimed once. That is the property we test for, and it is the reason the number above has a zero in the "missed" column and a one in "wrong" rather than the other way round.
Want this run against your own call instead? That is the offer on the front page — send one, and you get the same output, refusals included.