Collecting Feedback

This guide walks you through setting up feedback collection. You’ll learn how to use the thumbs scorer for simple ratings and how to handle both external and internal feedback.

Getting Started

To collect feedback, you’ll need:

1. A target ID - the specific output you want feedback on (e.g., a workflow run ID)
2. A scorer ID - defines what type of feedback to collect (e.g., thumbs)
3. Metadata - context about the feedback including user identification

Using the Thumbs Scorer

The thumbs scorer is Scout’s built-in scorer for simple thumbs up/down ratings. Use score: 1 for positive feedback and score: 0 for negative feedback.

Create Thumbs Up Feedback

$
curl -X POST https://api.scoutos.com/feedback \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "target_type": "workflow_run",
>
    "target_id": "wfr_abc123",
>
    "scorer_id": "thumbs",
>
    "score": 1,
>
    "metadata": {
>
      "user_id": "user_456",
>
      "workflow_id": "wf_xyz",
>
      "session_id": "sess_789"
>
    }
>
  }'

Response:

1
{
2
  "ok": true,
3
  "feedback": {
4
    "id": "fb_abc123xyz",
5
    "target_type": "workflow_run",
6
    "target_id": "wfr_abc123",
7
    "scorer_id": "thumbs",
8
    "score": 1,
9
    "metadata": {
10
      "user_id": "user_456",
11
      "workflow_id": "wf_xyz",
12
      "session_id": "sess_789"
13
    },
14
    "status": "new",
15
    "created_at": "2024-12-29T10:30:00Z"
16
  }
17
}

Create Thumbs Down Feedback with Comment

$
curl -X POST https://api.scoutos.com/feedback \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "target_type": "workflow_run",
>
    "target_id": "wfr_abc123",
>
    "scorer_id": "thumbs",
>
    "score": 0,
>
    "comment": "The response was not relevant to my question",
>
    "metadata": {
>
      "user_id": "user_456",
>
      "workflow_id": "wf_xyz",
>
      "session_id": "sess_789"
>
    }
>
  }'

Response:

1
{
2
  "ok": true,
3
  "feedback": {
4
    "id": "fb_def456xyz",
5
    "target_type": "workflow_run",
6
    "target_id": "wfr_abc123",
7
    "scorer_id": "thumbs",
8
    "score": 0,
9
    "comment": "The response was not relevant to my question",
10
    "metadata": {
11
      "user_id": "user_456",
12
      "workflow_id": "wf_xyz",
13
      "session_id": "sess_789"
14
    },
15
    "status": "new",
16
    "created_at": "2024-12-29T10:31:00Z"
17
  }
18
}

Required Metadata by Target Type

Different target types require specific fields in metadata:

For workflow_run targets, you must include workflow_id and session_id in metadata. Other fields like user_id are optional but recommended.

Collecting External User Feedback

When collecting feedback from your end-users (customers, app users, etc.), include their identifier in the metadata.user_id field.

Example with Additional Context

You can include any additional context in metadata for segmentation:

1
{
2
  "target_type": "workflow_run",
3
  "target_id": "wfr_abc123",
4
  "scorer_id": "thumbs",
5
  "score": 1,
6
  "metadata": {
7
    "user_id": "user_456",
8
    "workflow_id": "wf_xyz",
9
    "session_id": "sess_789",
10
    "plan": "enterprise",
11
    "region": "us-west"
12
  }
13
}

Internal Team Reviews

For internal review workflows, Scout users can provide feedback and manage it through status tracking.

Creating Internal Feedback

When authenticated via JWT (e.g., from the Scout dashboard), feedback is automatically attributed to the authenticated user. You still need to provide the required metadata:

1
{
2
  "target_type": "workflow_run",
3
  "target_id": "wfr_abc123",
4
  "scorer_id": "thumbs",
5
  "score": 0,
6
  "comment": "Hallucination detected - need to review training data",
7
  "metadata": {
8
    "workflow_id": "wf_xyz",
9
    "session_id": "sess_789"
10
  }
11
}

Managing Feedback Status

The status workflow helps you triage negative feedback that needs attention. Positive feedback is just signal - it can stay as new or be dismissed.

Update feedback status to track review progress:

$
curl -X PATCH https://api.scoutos.com/feedback/fb_xyz789 \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "status": "reviewing",
>
    "assigned_to": "team_member_id",
>
    "internal_notes": "Investigating prompt template"
>
  }'

Status Workflow

Typical flow: new → reviewing → resolved / dismissed

Resolving Feedback

When resolving feedback, the resolved_at and resolved_by fields are automatically populated:

1
{
2
  "status": "resolved",
3
  "internal_notes": "Fixed by updating the prompt template in v2.3"
4
}

Filtering and Querying Feedback

List feedback with filters to analyze patterns:

$
curl "https://api.scoutos.com/feedback?target_type=workflow_run&status=new&limit=50" \
>
  -H "Authorization: Bearer YOUR_API_KEY"

Common Filters

Best Practices

1. Use consistent user IDs - Stable identifiers in metadata.user_id help you track feedback per user over time.

2. Encourage comments on negative feedback - When users give a thumbs down, prompting for a comment provides actionable insights.

3. Set up a review cadence - Regularly review new feedback to identify patterns and prioritize improvements.

4. Track resolution - Use the status workflow to ensure feedback leads to action, not just collection.

5. Segment by metadata - Use additional metadata fields to analyze feedback by user segments (plan type, region, etc.).