Scorers

Scorers define how AI outputs are evaluated. Scout provides built-in scorers for both human review (collecting feedback) and automated evaluation (programmatic checks).

Human Review Scorers

Human review scorers are used with the Feedback API to collect ratings from users.

thumbs

The thumbs scorer collects simple thumbs up/down feedback.

Example:

1
{
2
  "target_type": "workflow_run",
3
  "target_id": "wfr_abc123",
4
  "scorer_id": "thumbs",
5
  "score": 1,
6
  "metadata": {
7
    "user_id": "user_456",
8
    "workflow_id": "wf_xyz",
9
    "session_id": "sess_789"
10
  }
11
}

Native Scorers

Native scorers are used for automated evaluation runs. They programmatically check workflow outputs against expected values, enabling you to test your AI workflows at scale.

exact_match

Checks if the output exactly matches the expected value.

Config Options:

Example:

1
{
2
  "scorer_id": "exact_match",
3
  "config": {
4
    "case_sensitive": false,
5
    "strip_whitespace": true
6
  }
7
}

contains

Checks if the output contains the expected value as a substring.

Config Options:

Example:

1
{
2
  "scorer_id": "contains",
3
  "config": {
4
    "case_sensitive": false
5
  }
6
}

regex_match

Checks if the output matches a regular expression pattern.

Config Options:

Example:

1
{
2
  "scorer_id": "regex_match",
3
  "config": {
4
    "pattern": "^[A-Z][a-z]+$",
5
    "flags": ""
6
  }
7
}

field_exists

Checks if a specific field exists in a JSON output.

Config Options:

Example:

1
{
2
  "scorer_id": "field_exists",
3
  "config": {
4
    "field_path": "summary.title"
5
  }
6
}

type_check

Validates that the output is of the expected type.

Config Options:

Example:

1
{
2
  "scorer_id": "type_check",
3
  "config": {
4
    "expected_type": "object"
5
  }
6
}

Choosing a Scorer