Creating Collections & Tables

Collections are where Scout stores your data for search and retrieval. Each collection holds one or more tables, and each table has a schema you define. Think of a collection like a database, and a table like a spreadsheet inside it.

This page covers how to create a collection, configure its tables and get data into them.

Use with Agents

1) Enable Tools

In the agent’s Tools tab, enable Collections tools for document creation and querying.

2) Add Instruction Snippet

When creating or updating table records:
 
1. Validate target collection and table before writing.
2. Confirm required fields are present: `id`, `content` and key metadata.
3. Use consistent values for taxonomy fields like `category` and `status`.
4. If data is incomplete, ask for missing fields before saving.
5. After writing, return the record IDs that were created or updated.

3) Prompt Examples

• “Create a release_notes record for version 2.8.0 with category platform.”
• “Upsert these three customer call summaries into customer_feedback.”
• “Check whether this document already exists by URL before creating a new row.”

4) Expected Behavior

• The agent writes to the correct schema
• The agent avoids duplicate records when identifiers already exist
• The agent reports what changed with record IDs
• The agent asks for missing required fields before saving

Creating a Collection

From the Dashboard

1. Navigate to Collections in the Scout dashboard
2. Click + New at the top
3. Fill in the details:
   • Name (required): A descriptive name for your collection
   • Description (optional): What this collection contains
   • Icon (optional): Visual identifier
4. Click Create

Scout provisions the underlying vector database for your collection. This typically takes about 30 seconds, and the UI shows the current status while it’s setting up.

Via the API

curl -X POST https://api.scoutos.com/v2/collections \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Knowledge Base",
    "description": "Product documentation and support articles"
  }'

Using Python:

from scoutos import Scout
 
client = Scout(api_key="YOUR_API_KEY")
 
collection = client.collections.create(
    name="Knowledge Base",
    description="Product documentation and support articles"
)
 
print(collection)  # {'id': 'col_...', 'name': 'Knowledge Base', ...}

Using TypeScript:

import { ScoutClient } from "scoutos";
 
const client = new ScoutClient({ apiKey: "YOUR_API_KEY" });
 
const collection = await client.collections.create({
  name: "Knowledge Base",
  description: "Product documentation and support articles"
});

Configuring Tables

Each Collection contains Tables. When you create a Collection, Scout automatically creates an “Untitled” table for you.

Adding Columns

Tables use columns to define their schema. Each column has a type and stores a specific kind of data.

To add a column:

1. Open your table
2. Click the + button in the table header row
3. Enter a column name and select a type

Column Types

Example: Documentation Table

For a documentation knowledge base, create these columns:

The content Column

The content column is special when vector indexing is enabled:

• Automatic Chunking — Long content is split into chunks of about 2,500 characters
• Vector Embeddings — Each chunk gets its own embedding for semantic search
• Search Results — Results link back to the parent document

Best Practice: Store your main searchable text in a content column for optimal semantic search.

Populating Tables

Pick the method that fits your workflow:

Method 1: Manual Entry

For small datasets, add documents directly:

1. Open your table
2. Click + Add Document or click in an empty row
3. Fill in the fields
4. Save

Method 2: Source Syncs

Connect external data sources that automatically sync to your table:

• Web Scraping — Crawl websites and sitemaps
• Notion — Sync Notion databases and pages
• Google Sheets — Sync spreadsheet rows

Learn more about Sources →

For Notion-specific setup, see Notion.

For spreadsheet sync setup, see Google Sheets.

Method 3: API / SDK

Programmatically add documents from your applications:

Python:

from scoutos import Scout
 
client = Scout(api_key="YOUR_API_KEY")
 
# Add a single document
client.documents.create(
    collection_id="col_abc123",
    table_id="tab_xyz789",
    documents=[{
        "id": "doc_001",
        "title": "Getting Started Guide",
        "content": "This comprehensive guide walks you through...",
        "url": "https://docs.example.com/getting-started",
        "category": "tutorial",
        "updated_at": 1704067200
    }]
)
 
# Add multiple documents at once
client.documents.create(
    collection_id="col_abc123",
    table_id="tab_xyz789",
    documents=[
        {"id": "doc_001", "title": "Doc 1", "content": "...", "category": "tutorial"},
        {"id": "doc_002", "title": "Doc 2", "content": "...", "category": "reference"},
        {"id": "doc_003", "title": "Doc 3", "content": "...", "category": "tutorial"}
    ]
)

TypeScript:

import { ScoutClient } from "scoutos";
 
const client = new ScoutClient({ apiKey: "YOUR_API_KEY" });
 
// Add a single document
await client.documents.create({
  collectionId: "col_abc123",
  tableId: "tab_xyz789",
  documents: [{
    id: "doc_001",
    title: "Getting Started Guide",
    content: "This comprehensive guide walks you through...",
    url: "https://docs.example.com/getting-started",
    category: "tutorial",
    updatedAt: 1704067200
  }]
});
 
// Add multiple documents at once
await client.documents.create({
  collectionId: "col_abc123",
  tableId: "tab_xyz789",
  documents: [
    { id: "doc_001", title: "Doc 1", content: "...", category: "tutorial" },
    { id: "doc_002", title: "Doc 2", content: "...", category: "reference" },
    { id: "doc_003", title: "Doc 3", content: "...", category: "tutorial" }
  ]
});

Method 4: Workflow Blocks

Save data from workflows using the Save Document to Table block:

1. Add the block to your workflow
2. Select your Collection and Table
3. Map workflow outputs to table columns
4. Run the workflow

Learn more about querying data →

Document Structure

Each document has:

{
  "id": "unique_document_id",    // Required: Unique identifier
  "content": "Main searchable text...",  // Indexed for semantic search
  "title": "Document Title",     // Custom metadata field
  "category": "tutorial",        // Custom metadata field
  "url": "https://...",          // Custom metadata field
  "created_at": 1704067200       // Custom metadata field
}

Content Field

The content field (or any field designated as the vector index field):

• Gets automatically chunked into smaller pieces
• Each chunk receives its own vector embedding
• Enables semantic search on the document

Metadata Fields

All other columns store structured data:

• Used for filtering search results
• Displayed in the UI
• Can be modified anytime

Managing Tables

View Table Schema

curl https://api.scoutos.com/v2/collections/{collection_id}/tables/{table_id}/schema \
  -H "Authorization: Bearer YOUR_API_KEY"

Update Table

curl -X PATCH https://api.scoutos.com/v2/collections/{collection_id}/tables/{table_id} \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Updated Table Name"
  }'

Delete Table

curl -X DELETE https://api.scoutos.com/v2/collections/{collection_id}/tables/{table_id} \
  -H "Authorization: Bearer YOUR_API_KEY"

Best Practices

Naming Conventions

• Use descriptive, consistent names for collections and tables
• Use snake_case for column names (created_at, user_id)
• Keep column names concise but clear

Schema Design

• Store searchable text in a content column
• Use metadata columns for filtering (categories, dates, tags)
• Include an id field for easy document updates
• Add timestamps (created_at, updated_at) for tracking

Content Optimization

For better semantic search:

• Write comprehensive, descriptive content
• Include context and details
• Avoid heavy abbreviations
• Consider including synonyms
• Structure content with clear headings

Next Steps

• Querying Data — Learn how to search your collections
• Sources — Automate data syncs from external sources
• Workflows — Use collections in your AI workflows

Built with ❤️ by Scout OS