Collecting Feedback

This guide walks you through setting up feedback collection. You’ll learn how to use the thumbs scorer for simple ratings and how to handle both external and internal feedback.

Getting Started

To collect feedback, you’ll need:

1. A target ID - the specific output you want feedback on (e.g., a workflow run ID)
2. A scorer ID - defines what type of feedback to collect (e.g., thumbs)
3. User identification - who is providing the feedback

Using the Thumbs Scorer

The thumbs scorer is Scout’s built-in scorer for simple thumbs up/down ratings. Use score: 1 for positive feedback and score: 0 for negative feedback.

Create Thumbs Up Feedback

$
curl -X POST https://api.scoutos.com/v2/feedback \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "target_type": "workflow_run",
>
    "target_id": "wfr_abc123",
>
    "scorer_id": "thumbs",
>
    "score": 1,
>
    "user": {
>
      "id": "user_456"
>
    }
>
  }'

Response:

1
{
2
  "id": "fb_abc123xyz",
3
  "target_type": "workflow_run",
4
  "target_id": "wfr_abc123",
5
  "scorer_id": "thumbs",
6
  "score": 1,
7
  "user": {
8
    "id": "user_456"
9
  },
10
  "status": "new",
11
  "created_at": "2024-12-29T10:30:00Z"
12
}

Create Thumbs Down Feedback with Comment

$
curl -X POST https://api.scoutos.com/v2/feedback \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "target_type": "workflow_run",
>
    "target_id": "wfr_abc123",
>
    "scorer_id": "thumbs",
>
    "score": 0,
>
    "comment": "The response was not relevant to my question",
>
    "user": {
>
      "id": "user_456"
>
    }
>
  }'

Response:

1
{
2
  "id": "fb_def456xyz",
3
  "target_type": "workflow_run",
4
  "target_id": "wfr_abc123",
5
  "scorer_id": "thumbs",
6
  "score": 0,
7
  "comment": "The response was not relevant to my question",
8
  "user": {
9
    "id": "user_456"
10
  },
11
  "status": "new",
12
  "created_at": "2024-12-29T10:31:00Z"
13
}

Collecting External User Feedback

When collecting feedback from your end-users (customers, app users, etc.), include a user object with their identifier.

Example with User Metadata

1
{
2
  "target_type": "workflow_run",
3
  "target_id": "wfr_abc123",
4
  "scorer_id": "thumbs",
5
  "score": 1,
6
  "user": {
7
    "id": "user_456",
8
    "metadata": {
9
      "plan": "enterprise",
10
      "region": "us-west"
11
    }
12
  }
13
}

Internal Team Reviews

For internal review workflows, Scout users can provide feedback and manage it through status tracking.

Creating Internal Feedback

When authenticated via JWT (e.g., from the Scout dashboard), feedback is automatically attributed to the authenticated user. No user object needed:

1
{
2
  "target_type": "workflow_run",
3
  "target_id": "wfr_abc123",
4
  "scorer_id": "thumbs",
5
  "score": 0,
6
  "comment": "Hallucination detected - need to review training data"
7
}

Managing Feedback Status

The status workflow helps you triage negative feedback that needs attention. Positive feedback is just signal - it can stay as new or be dismissed.

Update feedback status to track review progress:

$
curl -X PATCH https://api.scoutos.com/v2/feedback/fb_xyz789 \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "status": "reviewing",
>
    "assigned_to": "team_member_id",
>
    "internal_notes": "Investigating prompt template"
>
  }'

Status Workflow

Typical flow: new → reviewing → resolved / dismissed

Resolving Feedback

When resolving feedback, the resolved_at and resolved_by fields are automatically populated:

1
{
2
  "status": "resolved",
3
  "internal_notes": "Fixed by updating the prompt template in v2.3"
4
}

Filtering and Querying Feedback

List feedback with filters to analyze patterns:

$
curl "https://api.scoutos.com/v2/feedback?target_type=workflow_run&status=new&limit=50" \
>
  -H "Authorization: Bearer YOUR_API_KEY"

Common Filters

Best Practices

1. Use consistent user IDs - Stable identifiers help you track feedback per user over time.

2. Encourage comments on negative feedback - When users give a thumbs down, prompting for a comment provides actionable insights.

3. Set up a review cadence - Regularly review new feedback to identify patterns and prioritize improvements.

4. Track resolution - Use the status workflow to ensure feedback leads to action, not just collection.

5. Segment by metadata - Use user.metadata to analyze feedback by user segments (plan type, region, etc.).