The fine-tuning quality gate

The four tests#

• Held-out accuracy

  Passes when

  Both: your version's accuracy on the held-out records is at least 0.02 above the base's, and the interval of the difference lies wholly above 0

  Why

  It must be better on your decisions, by more than chance: random labels cannot pass

• Held-out calibration

  Passes when

  Its calibration error is at most 0.01 above the base's

  Why

  A 0.80 must still mean true about 8 times in 10

• Standard test sets

  Passes when

  Its mean score on our test sets for the same question types is no more than 0.01 below the base's; doing better never fails

  Why

  It must not forget what the base does well

• Refusals

  Passes when

  Identical to the base's

  Why

  The safety check sits outside the model, so this always holds; it is checked anyway

The held-out records are the ones the run set aside and never trained on, so the comparison is fair to both. Calibration error is how far a model's stated probabilities are from how often they come true; lower is better.

The scorecard#

• Held out

  Numbers

  The number of records; accuracy of your version and the base, the difference and its interval; the calibration error of each

• By question

  Numbers

  Each question id: its type, its records, and the accuracy of your version and the base

• Standard test sets

  Numbers

  Each set by name and question type, your version against the base

• Gate

  Numbers

  Each test, passed or not, with its value and its threshold; and the reason when the version did not pass

The scorecard holds aggregate numbers only, never a record. Read it by question: a version can gain a lot on one question and nothing on another, which tells you where your labels carry a policy the base did not know.

When the gate says no_gain#

no_gain means the version was trained and scored, and the base is still the better choice on your data, for example "Your held-out accuracy 0.81 against the base's 0.80: not enough to ship." The version keeps its scorecard but no trained weights, so it can never be deployed; the run costs $50, and the compute part is not charged.

• Accuracy close to the base's on every question

  Likely cause

  The base already decides the way you do

  Try

  Keep the base and save the cost; sharpen instructions where it errs

• Accuracy up on some questions, down on others

  Likely cause

  Some questions have inconsistent labels

  Try

  Check the conflicts count and relabel those questions

• A gain too small to pass, on a small held-out split

  Likely cause

  One held-out record is 0.02 of accuracy at 50 records

  Try

  Add records: a larger held-out split measures a real gain

• Calibration worse than the base's

  Likely cause

  Truth answers written as guesses, or heavy weights on few records

  Try

  Use 0 and 1 where your label is certain; even out the weights

• Standard test sets lower

  Likely cause

  The dataset is narrow and repetitive

  Try

  Add variety: more states, more options answered

• Use your fine-tuned modelDeploy it and call it by name.Read
• Prepare your datasetThe record format, every rule, and how many records you need.Read