What calibrated means#
Take every Truth answer the model gave near 0.9. If about nine in ten of those statements are actually true, the probabilities are calibrated. That is what makes a threshold meaningful: at 0.9 you accept about one wrong answer in ten among the ones that pass, and you can reason about that before you ship. Calibration is measured on our evaluation sets; it is an estimate, not a guarantee, so measure the error rate at your threshold on your own data before you rely on it.
Where confidence lives in each type#
| Type | Field to gate on | Range and meaning |
|---|---|---|
| Choice | confidence | 0 when all options are equally likely, 1 when one option has all the probability |
| Score | confidence, or a sum of probabilities | How tightly the levels sit around the most likely one; tail sums answer "at least level n" |
| Truth | truth | The probability that the statement is true; distance from 0.50 is how sure it is |
| Number | confidence, or a sum of probabilities | The probability of number; add its neighbours for the chance the true value is within one |
Choice confidence#
confidence = (K · p_max − 1) / (K − 1)
- K
- the number of options
- p_max
- the probability of the chosen option
Score confidence#
confidence = 1 − E[|i − m|] / D(K)
D(K) = floor(K · K / 4) / K
- K
- the number of levels
- m
- the most likely level, the lowest one on a tie
- E[|i − m|]
- the expected distance from m: sum(p_i · |i − m|)
Both are computed from the unrounded probabilities, then rounded to 2 decimals. A score's confidence is 1 when all the probability sits on one level, and probability next to the most likely level costs little. With two levels the two formulas agree. The Score page works one through.
Choosing a threshold#
- 1.Start at 0.7 for choice and number confidence and 0.8 for Truth gates.
- 2.Run a few hundred real inputs whose outcome you already know, such as resolved tickets or settled disputes.
- 3.For each threshold, measure two numbers: how many inputs pass, and how many of those are wrong.
- 4.Pick the threshold where the error rate is one your product can live with. Above it your code acts instantly; below it, it takes the safe action or re-checks with the full model.
- 5.Re-check after you change the questions or move to a new model version.
Reading the numbers#
- Two decimals. Every number in an answer is a JSON float rounded to 2 decimals, and whole values keep their decimal point:
1.0,0.0,3.0. Parse them as floats. - One exception:
number. A number answer'snumberis a value of your grid, written as the grid writes it:7on a whole-number grid,3.5on a grid of 0.5. Itsexpectedandconfidencefollow the float rule (7.0, not7), and itsprobabilitiesare listed to 3 decimals. In a typed client, readnumberas a float; it always equals a grid value. - Clean fields.
confidence,scoreandtruthare always plain two-decimal values, andchoiceis always one of your option names. probabilitiescan print long. After rounding, the largest entries of a choice's or a score's list are nudged by 0.01 toward a sum of 1, and a nudged entry can print as0.8200000000000001. A few of those lists sum to 0.99 or 1.01. A number answer's list is to 3 decimals and is not nudged: its values sum to at most 1, and less over a wide grid. Thresholds and comparisons are unaffected; round before you display.- Read by key. Look answers up by question key and probabilities by option name or level, never by position: no order is part of the contract.