Score

A score question places the input on an ordered scale you define. It returns the expected level, the probability of every level and a confidence.

Score“How urgent is this?”

0.01
0.09
0.48
0.42

0 routine
1 today
2 urgent
3 critical

score 2.31 of 3confidence 0.47

Request#

JSON

{
  "urgency": {
    "type": "score",
    "instructions": "How urgent is this?",
    "criteria": ["routine", "today", "urgent", "critical"]
  }
}

criteria is an array of levels from lowest to highest, up to 10 of them (two or more make a scale). Level indexes start at 0 in the order you send them.

Response#

JSON

{
  "urgency": {
    "type": "score",
    "score": 2.31,
    "confidence": 0.47,
    "legend": {
      "0": "routine",
      "1": "today",
      "2": "urgent",
      "3": "critical"
    },
    "probabilities": { "0": 0.01, "1": 0.09, "2": 0.48, "3": 0.42 }
  }
}

scorenumber: The expected level index: the sum of each index times its probability.
probabilitiesobject: Level index to probability, in level order.
legendobject: Level index to your level text, exactly as sent.
confidencenumber: How tightly the probability sits around the most likely level, from 0 to 1.

Formulas#

score = sum(i · p_i)

confidence = 1 − E[|i − m|] / D(K)

D(K) = floor(K · K / 4) / K

i: the level index, from 0
p_i: the probability of level i
K: the number of levels
m: the most likely level, the lowest one on a tie
E[|i − m|]: the expected distance from m: sum(p_i · |i − m|)
D(K): the largest that distance can be on K levels

Confidence is 1 when all the probability sits on one level. Probability on a level next to the most likely one costs little, and probability far from it costs a lot, so a scale split between two neighbours still reads as fairly sure. With two levels it equals the choice formula.

Worked example

score = 0 · 0.01 + 1 · 0.09

+ 2 · 0.48 + 3 · 0.42 = 2.31

m = 2 (urgent)

E = 0.01 · 2 + 0.09 · 1

+ 0.48 · 0 + 0.42 · 1 = 0.53

D(4) = floor(4 · 4 / 4) / 4 = 1

confidence = 1 − 0.53 / 1 = 0.47

These are the probabilities from the response above. A middling confidence with a high score is common and useful: the model is sure the ticket is at least urgent, and split between urgent and critical.

Using a score#

As a number: sort or prioritise by score directly. It is continuous, so 2.31 ranks above 2.12.
As a level: round(score) gives the nearest level; read its name from legend.
As a tail probability: add the probabilities of the levels you care about. Here the chance of urgent or worse is 0.48 + 0.42 = 0.90, which is often the gate you actually want.

Request#

JSON

{
  "urgency": {
    "type": "score",
    "instructions": "How urgent is this?",
    "criteria": ["routine", "today", "urgent", "critical"]
  }
}

criteria is an array of levels from lowest to highest, up to 10 of them (two or more make a scale). Level indexes start at 0 in the order you send them.

Response#

JSON

{
  "urgency": {
    "type": "score",
    "score": 2.31,
    "confidence": 0.47,
    "legend": {
      "0": "routine",
      "1": "today",
      "2": "urgent",
      "3": "critical"
    },
    "probabilities": { "0": 0.01, "1": 0.09, "2": 0.48, "3": 0.42 }
  }
}

scorenumber

The expected level index: the sum of each index times its probability.

probabilitiesobject

Level index to probability, in level order.

legendobject

Level index to your level text, exactly as sent.

confidencenumber

How tightly the probability sits around the most likely level, from 0 to 1.

Formulas#

score = sum(i · p_i)

confidence = 1 − E[|i − m|] / D(K)

D(K) = floor(K · K / 4) / K

i: the level index, from 0
p_i: the probability of level i
K: the number of levels
m: the most likely level, the lowest one on a tie
E[|i − m|]: the expected distance from m: sum(p_i · |i − m|)
D(K): the largest that distance can be on K levels

Worked example

score = 0 · 0.01 + 1 · 0.09

+ 2 · 0.48 + 3 · 0.42 = 2.31

m = 2 (urgent)

E = 0.01 · 2 + 0.09 · 1

+ 0.48 · 0 + 0.42 · 1 = 0.53

D(4) = floor(4 · 4 / 4) / 4 = 1

confidence = 1 − 0.53 / 1 = 0.47

Using a score#

As a number: sort or prioritise by score directly. It is continuous, so 2.31 ranks above 2.12.

As a level: round(score) gives the nearest level; read its name from legend.

As a tail probability: add the probabilities of the levels you care about. Here the chance of urgent or worse is 0.48 + 0.42 = 0.90, which is often the gate you actually want.