Evaluation set builder.
A good exam asks each thing once, even when people phrase it many ways, and leaves no topic out.
- Test sets
- 2 min
- Live demo
The analogy
A teacher builds an exam from the questions students asked all term. Many are the same question in other words, so she keeps one of each. If she trims too hard, she also throws out a question from another topic that only looked similar.
Here the questions are thirty fictional customer messages about six banking topics. The slider decides how alike two messages must be before the second one is dropped as a repeat.
In the diagram below
- the students' questions
- 30 customer messages
- the same question in other words
- a repeat of the same topic
- a look-alike from another topic
- a message dropped by mistake
- the exam
- the test set, three questions per topic
Try it
Thirty messages, five for each of six topics. Each one is compared with the messages already kept, in order.
Your move / Predict, then test
Before you run it, place a bet: dropping messages that are 80% alike or more, is a message from another topic thrown out as a repeat?
Choose your prediction, run the experiment, then reveal the result. You can also explore without predicting.
Each square is one message, in order. Green stays, blue was a repeat of its own topic, red was thrown out because it looked like a message from another topic.
Place your bet and press Run it.
Cleaned or as it came
The same exam of 18 questions, three per topic. One side cleans the repeats with your setting; the other takes the messages as they came.
Where it fits
Worth it
When a team tests an assistant with questions taken from real conversations and wants each topic counted once. I picture a bank's support assistant before a release.
Not needed
When the test questions are written by hand, one per case, and nobody copies them from conversation logs.
What it proves
I watched both ways an exam goes wrong: repeats that inflate the score, and questions from another topic that quietly drop out. The slider shows the point where one problem turns into the other.
For engineers
- Token-set Jaccard similarity with greedy, order-keeping deduplication: a message stays only if it is below the threshold against every message already kept.
- Messages are grouped by their labelled topic and the first three of each go to the test set. The topic label stands in for an intent classifier.
- The status of every message at each slider stop is pinned in a fixture read by the TypeScript and Python suites. Counting repeats in the exam uses the committed 0.8 cut.
- Stack: Next.js 16, TypeScript, Python, Vitest, pytest.