Vector Databases vs. Standard Databases

Understanding the differences between vector databases and standard databases helps you choose the right data storage approach for your use case. Scout uses Collections, which provide vector database capabilities along with structured data storage.

Vector Databases (Collections)

Vector databases are designed to store and search through high-dimensional vector embeddings that represent the semantic meaning of text. In Scout, Collections automatically provision and manage vector databases for you.

How Vector Databases Work

1. Embedding Generation: Text content is converted into numerical vectors (embeddings) that capture semantic meaning
2. Similarity Search: Queries are converted to vectors and matched against stored vectors using distance metrics
3. Semantic Understanding: Can find relevant content even when exact keywords don’t match

When to Use Vector Databases

Use vector databases (Collections) when you need:

• Semantic Search: Finding documents based on meaning, not just exact keyword matches
  • Example: Searching for “customer complaints about refunds” to find relevant support tickets
• RAG (Retrieval-Augmented Generation): Retrieving context for LLM queries
  • Example: Building a chatbot that answers questions from your documentation
• Similarity Matching: Finding similar content or documents
  • Example: Finding articles similar to a given piece of content
• Natural Language Queries: Querying data using natural language instead of structured queries
  • Example: “Find all documents about pricing changes in Q4”
• Unstructured or Semi-Structured Text: Working with documents, articles, support tickets, or any text-heavy content
  • Example: Storing and searching through knowledge base articles, blog posts, or documentation

Key Features in Scout Collections

• Automatic Embedding: Text is automatically chunked and embedded
• Hybrid Search: Combines vector search with keyword-based (BM25) search
• Metadata Filtering: Use structured metadata alongside semantic search
• No Schema Required: Flexible document structure with dynamic metadata

Standard Databases

Standard relational databases (like PostgreSQL, MySQL) store data in structured tables with predefined schemas and use SQL for queries.

How Standard Databases Work

1. Structured Schema: Data is organized into tables with fixed columns and data types
2. SQL Queries: Use SQL to perform exact matches, joins, aggregations, and complex queries
3. ACID Transactions: Guarantees data consistency and reliability

When to Use Standard Databases

Use standard databases when you need:

• Exact Matching: Finding records by specific values or IDs
  • Example: Looking up a user by email address or customer ID
• Complex Queries: Joins, aggregations, and complex relational queries
  • Example: Calculating sales totals grouped by region and product category
• Structured Data: Working with well-defined, structured data
  • Example: User accounts, orders, inventory, financial records
• Transactional Operations: Inserting, updating, or deleting records with ACID guarantees
  • Example: Processing payments, updating account balances
• Analytics and Reporting: Running analytical queries on structured data
  • Example: Generating monthly revenue reports, user activity analytics
• Integration with Existing Systems: Connecting to existing databases or data warehouses

Key Features of Standard Databases

• SQL Queries: Full SQL support for complex queries
• Structured Schema: Fixed schema with defined relationships
• Transaction Support: ACID guarantees for data consistency
• Exact Matching: Fast lookups by primary keys or indexes

Comparison Table

How Collections Combine Both Approaches

Scout Collections are designed to handle both structured and unstructured data:

Structured Data Storage

Collections store data in tables with defined columns and data types:

• Structured Metadata: Store organized data like user IDs, dates, categories, tags
• Filtering and Sorting: Use metadata columns to filter and sort your documents
• Exact Matching: Find documents by specific field values
• Data Types: Support for strings, numbers, booleans, dates, and JSON

Vector Search Capabilities

When vector indexing is enabled, Collections also provide:

• Semantic Search: Find documents by meaning, not just keywords
• Hybrid Search: Combines semantic search with keyword (BM25) search
• Natural Language Queries: Search using conversational language
• Similarity Matching: Find conceptually similar content

Best of Both Worlds

Collections let you:

• Store structured metadata for filtering and organization (like standard databases)
• Enable semantic search on text content for intelligent retrieval (like vector databases)
• Combine both approaches in a single query (hybrid search with metadata filtering)

Example Workflow

1. Query Collection → Find relevant documents using semantic search
2. Filter by metadata → Narrow results by category, date, or other fields
3. LLM Block → Generate response using the filtered, relevant context

Decision Guide

Choose Vector Databases (Collections) if:

• ✅ You’re building a RAG application
• ✅ You need to search through documents or text content
• ✅ You want natural language queries
• ✅ Your data is unstructured or semi-structured
• ✅ You need semantic similarity matching
• ✅ You want to combine structured metadata with semantic search

Choose Standard Databases if:

• ✅ You have structured, relational data that requires complex joins
• ✅ You need exact matches and complex SQL queries
• ✅ You require ACID transactions for data consistency
• ✅ You’re integrating with existing database systems
• ✅ You need analytics and reporting on structured data
• ✅ You’re building transactional systems (payments, orders, etc.)

Note: Scout Collections can handle many structured data use cases through metadata columns, while also providing vector search capabilities. For purely transactional systems or complex relational queries, you may need to integrate with external standard databases.

Getting Started

• Learn how to create and use Collections for vector database operations
• Learn how to save documents to tables with structured data and vector search
• Learn how to query collections with semantic and hybrid search