Query Collection Table Block

Query and retrieve data from Scout collection tables

The Query Collection Table block enables users to perform advanced searches on a specified collection table within Scout. This block supports multiple search configurations, including vector and hybrid searches, and allows for dynamic query and filter construction using Jinja templating. It is ideal for retrieving data with fine-tuned control over ranking, relevance, and filtering criteria.

Configuration (Required)

Collection
stringRequired

Select from a dropdown of which collection to query.

Table
stringRequired

Select from a dropdown the table within the selected collection to query. This input supports Jinja templating for dynamic table selection.

Search Term
stringRequired

The term or phrase to search for in the selected table. This input supports Jinja templating for dynamic query construction. For example, use {{inputs.user_message}} to dynamically query based on user input.

Advanced Configuration and Usage

Vector Minimum Distance

Vector Minimum Distance
floatDefaults to 1

Threshold for filtering search results based on vector distance, which determines how closely the result matches the search term semantically. Lower values mean stricter matches, and the range is from 0 to 2. Default is 1.

  • Example Input: 0.8
  • Effect: Returns results that are highly semantically similar to the search term, excluding loosely related matches.
  • Use Case: Use a lower value (e.g., 0.5) when searching for highly specific matches, such as product names or technical terms.

Hybrid Search
booleanDefaults to false

Toggles between a hybrid search that combines vector search (semantic relevance) and BM25 (textual relevance). Default is false, which only performs vector-based search.

  • Example Input: true
  • Effect: Combines vector-based relevance and BM25 scoring for a balanced result.
  • Use Case: Enable this when you need results that account for both semantic meaning and exact keyword matches.

BM25 Rank By

bm25_rank_by
string

A list of attributes and corresponding queries for BM25 ranking. This allows you to control how textual relevance is calculated.

  • Example Input: [{"attribute": "content", "query": "hello"}]
  • Effect: Results are ranked higher if they contain the word “hello” in the “content” field.
  • Use Case: Use this to prioritize documents containing specific terms in specific fields.

Alpha

alpha
floatDefaults to 0.5

The weighting factor for hybrid search. A value between 0 and 1, where 0 relies fully on BM25 (text-based ranking) and 1 relies fully on vector search (semantic ranking). Default is 0.5.

  • Example Input: 0.7
  • Effect: Gives more weight to semantic matching while still considering textual relevance.
  • Use Case: Use higher alpha values (e.g., 0.8) when searching for documents with nuanced, semantic meanings.

Filters

filters
string

A list of filters to narrow down search results. Filters are specified as JSON arrays, e.g., ["attribute", "operator", "value"].

Filters are used to narrow down the results by applying specific conditions to attributes in your data. Below is a detailed list of examples, along with explanations for each use case:

Basic Filter Examples

  1. Equality Filter
    Input: ["status", "Eq", "active"]
    Explanation: Filters results to include only rows where the status attribute is equal to active.
    Use Case: Retrieve all active users, orders, or documents.

  2. Inequality Filter
    Input: ["rating", "Ne", "5"]
    Explanation: Excludes results where the rating attribute equals 5.
    Use Case: Exclude perfect ratings when analyzing feedback.

  3. Greater Than Filter
    Input: ["age", "Gt", "30"]
    Explanation: Filters results to include only rows where the age attribute is greater than 30.
    Use Case: Find users older than 30 for demographic analysis.

  4. Less Than Filter
    Input: ["price", "Lt", "50"]
    Explanation: Filters results to include only rows where the price attribute is less than 50.
    Use Case: Retrieve all products priced below $50 for a sale or discount offer.

Range Filter Examples

  1. Between Filter
    Input: ["date", "Between", [1672531200, 1703980800]] (Timestamps for “2023-01-01” and “2023-12-31”) Explanation: Filters results to include only rows where the date attribute falls within the specified range.
    Use Case: Retrieve all transactions or events within the year 2023.

  2. Not Between Filter
    Input: ["score", "NotBetween", ["60", "80"]]
    Explanation: Excludes results where the score attribute is between 60 and 80.
    Use Case: Exclude average scores to focus on high or low outliers.

Substring and Pattern Matching Filters

  1. Contains Filter
    Input: ["description", "Contains", "urgent"]
    Explanation: Filters results to include only rows where the description attribute contains the word “urgent”.
    Use Case: Find all tasks or tickets marked as urgent.

  2. Not Contains Filter
    Input: ["content", "NotContains", "draft"]
    Explanation: Excludes results where the content attribute contains the word “draft”.
    Use Case: Retrieve only published or final content.

  3. Starts With Filter
    Input: ["name", "StartsWith", "John"]
    Explanation: Filters results to include only rows where the name attribute starts with “John”.
    Use Case: Find all users or contacts with names beginning with “John”.

  4. Ends With Filter
    Input: ["email", "EndsWith", "example.com"]
    Explanation: Filters results to include only rows where the email attribute ends with “example.com”.
    Use Case: Retrieve all users with email addresses from a specific domain.

Null and Empty Filters

  1. Is Null Filter
    Input: ["last_login", "IsNull", null]
    Explanation: Filters results to include only rows where the last_login attribute is null (i.e., no value).
    Use Case: Find all users who have never logged in.

  2. Is Not Null Filter
    Input: ["last_login", "IsNotNull", null]
    Explanation: Filters results to include only rows where the last_login attribute is not null.
    Use Case: Retrieve all users who have logged in at least once.

Advanced Examples

  1. In Filter
    Input: ["priority", "In", ["high", "critical"]]
    Explanation: Filters results to include rows where the priority attribute is either high or critical.
    Use Case: Retrieve high-priority and critical tasks for immediate action.

  2. Not In Filter
    Input: ["category", "NotIn", ["archived", "deleted"]]
    Explanation: Filters results to exclude rows where the category attribute is archived or deleted.
    Use Case: Retrieve only active and visible items.

  3. Custom Attribute Filter
    Input: ["custom_field", "Eq", "specific_value"]
    Explanation: Filters results to include rows where the custom_field attribute matches specific_value.
    Use Case: Retrieve data for custom-defined attributes in your collection.


Best Practices for Filters

  • Combine Filters: Use logical combinations to create complex filtering rules.
  • Dynamic Filters: Use Jinja templating to construct filters dynamically, e.g., ["status", "Eq", "{{inputs.user_message}}"].
  • Test Incrementally: Start with simple filters to validate results before adding more complexity.
  • Optimize for Performance: Use specific filters (e.g., equality or range filters) to reduce the result set and improve search speed.

Outputs

The block outputs a list of results from the queried collection table, based on the specified search criteria.

Example Output
1[
2 {
3 "details": {
4 "vector_distance": 0.15098750591278076,
5 "hybrid_score": null
6 },
7 "record": {
8 "type": "table_query_record",
9 "id": "document/doc_cb413d5611e487506a4d74f5bd4d2dcc/chunk/0:1",
10 "parent": {
11 "id": "doc_cb413d5611e487506a4d74f5bd4d2dcc",
12 "type": "document"
13 },
14 "attributes": {
15 "url": "https://docs.scoutos.com/docs/workflows",
16 "page_title": "Workflow overview — Scout | Documentation",
17 "content": "Workflow overview\n\nUnderstanding the power and flexibility of Scout workflows\n\nJust getting started? Check out our Build a RAG app in under 5 minutes quick start guide!\n\nWorkflows are a powerful tool in the Scout platform, designed to automate processes, streamline tasks, and integrate seamlessly with various data sources and services. By utilizing workflows, users can create sophisticated sequences of actions that respond to specific inputs, making them ideal for a wide range of use cases like data processing, automated reporting, and integration with third\\-party APIs.\n\nWorkflows consist of configurable components known as blocks, which can be customized to suit different needs and chained together to create complex logic flows. Each workflow can be tailored with specific parameters, allowing for dynamic input handling and tailored output generation. You can execute a workflow from the console in the Scout UI, Scout’s embeddable Copilot, a Slackbot, API, SDK and more. This flexibility makes workflows suitable for both straightforward tasks and more intricate operations that require nuanced logic.\n\nIn addition to their versatility, workflows provide a comprehensive environment for monitoring and managing operations, complete with a console for real\\-time execution, a revision history audit log, and detailed workflow execution logs for troubleshooting. Environment variables can be set to manage configurations across different environments, ensuring robust and adaptable solutions.\n\nWas this page helpful?Creating WorkflowsUp Next\n\nBuilt with"
18 }
19 }
20 },
21 {
22 "details": {
23 "vector_distance": 0.21071159839630127,
24 "hybrid_score": null
25 },
26 "record": {
27 "type": "table_query_record",
28 "id": "document/doc_f9595153bd40bf10c1742634437495ad/chunk/0:1",
29 "parent": {
30 "id": "doc_f9595153bd40bf10c1742634437495ad",
31 "type": "document"
32 },
33 "attributes": {
34 "url": "https://docs.scoutos.com/docs/workflows/creating-workflows",
35 "page_title": "Building Effective Workflows in Scout — Scout | Documentation",
36 "content": "On this page* Defining your objective\n* Data \\& integrations\n* Tactical\nWorkflowsBuilding Effective Workflows in Scout\n\nGuidance on creating optimized workflows in Scout applications\n\nJust getting started? Check out our Build a RAG app in under 5 minutes quick start guide!\n\n1\n\n### Defining your objective\n\nCreating workflows starts with a simple first step; clearly defining the objective of your workflow, understanding the inputs it will handle, and the outputs it is expected to produce. This helps in determining the necessary logic and components required. Consider mapping out the sequence of operations, which will guide the logical flow of your workflow, making it easier to identify the specific blocks needed.\n\n2\n\n### Data \\& integrations\n\nNext, ensure you have all required data sources and third\\-party service integrations accessible, as these will be crucial for your workflow’s functionality. At the very least, we’d suggest setting up a collection so that you can augment LLM queries with your specific data. If you’re going to production, and you want to make requests to external services, consider setting up environment variables.\n\n3\n\n### Tactical\n\nYou can create a workflow in the Scout UI by clicking “Workflows” in the Scout dashboard nav, then clicking ”\\+ New”. You can create workflows from scratch, or use one of our one click deploy Workflow Templates.\n\nWorkflows can be given names, icons (an avatar image), and a description.\n\nRequest enterprise access for SDK and API workflow creation.\n\nWas this page helpful?Workflow TemplatesUp Next\n\nBuilt with"
37 }
38 }
39 },
40 {
41 "details": {
42 "vector_distance": 0.2473444938659668,
43 "hybrid_score": null
44 },
45 "record": {
46 "type": "table_query_record",
47 "id": "document/doc_112b5b16b002c064e9a8062189fa71d5/chunk/0:1",
48 "parent": {
49 "id": "doc_112b5b16b002c064e9a8062189fa71d5",
50 "type": "document"
51 },
52 "attributes": {
53 "url": "https://docs.scoutos.com/docs/workflows/workflow-templates",
54 "page_title": "Workflow Templates in Scout — Scout | Documentation",
55 "content": "On this page* Getting Started\n* Out\\-of\\-the\\-Box Functionality\n* Configuration Requirements\n* Running Your Workflow\n* Publishing Your Own Workflow Template\nWorkflowsWorkflow Templates in Scout\n\nEffortlessly Automate and Optimize Your Processes with Workflow Templates\n\nScout’s workflow templates are reusable patterns designed to help kick start your building process. By utilizing these templates, you can quickly create use case specific workflows that are sophisticated and proven.\n\n## Getting Started\n\nIn the Workflows section of Scout, click the Browse Templates button.\n\nYou’ll be presented with the available workflow templates. When you click into a template, you will see a visual of the workflow architecture and a description of the workflow.\n\n## Out\\-of\\-the\\-Box Functionality\n\nCertain workflow templates, such as those involving only Search Engine Results Page (SERP block) and Large Language Model (LLM block) blocks, function with minimal configuration.\n\nThese two blocks interface with Google search and an LLM respectively, and don’t require that you set anything up before using them.\n\n## Configuration Requirements\n\nSome workflow templates require additional configuration, particularly those involving Collection blocks. Collection blocks are used to query and save data from within workflows. In order to build the most effective AI workflow, particularly for RAG customer service and technical domain use cases, you must create your own collection and upload the most relevant information to it.\n\n## Running Your Workflow\n\nYou can initiate and monitor workflow runs directly from the scout dashboard, which allows for immediate feedback and is ideal for testing and debugging workflows. For detailed instructions on running workflows from the console, refer to the Console section.\n\n## Publishing Your Own Workflow Template\n\nThe ability to self serve publish your own workflow teamplate is coming soon. To publish your own workflow template today, send us a message with the link to the workflow and some text context about the workflow’s objectives, descriptions of the inputs and outputs, and any data sources that are necessary to integrate.\n\nWas this page helpful?Workflow Logic \\& StateUp Next\n\nBuilt with"
56 }
57 }
58 },
59 {
60 "details": {
61 "vector_distance": 0.25603729486465454,
62 "hybrid_score": null
63 },
64 "record": {
65 "type": "table_query_record",
66 "id": "document/doc_6def7fa84a637fe5a0cd71bee352839a/chunk/0:1",
67 "parent": {
68 "id": "doc_6def7fa84a637fe5a0cd71bee352839a",
69 "type": "document"
70 },
71 "attributes": {
72 "url": "https://docs.scoutos.com/docs/workflows/console",
73 "page_title": "Workflow Console in Scout — Scout | Documentation",
74 "content": "WorkflowsWorkflow Console in Scout\n\nReal\\-time execution and monitoring of Scout workflows\n\nThe workflow console provides a dynamic and interactive environment for executing and monitoring your workflow processes in real\\-time. The console offers a user\\-friendly interface where you can input data, initiate workflow runs, and observe the execution flow step\\-by\\-step, making it an invaluable tool for testing, debugging, and refining your workflows.\n\nOn the upper left\\-hand side of the workflow canvas, click the play icon to open the console.\n\nThe console displays a form\\-like interface where you can enter the inputs defined in your workflow’s input block. This allows you to test different scenarios by providing various data inputs, ensuring that your workflow is robust and versatile.\n\nClick “Run” below the input to execute the workflow. As the workflow runs, each block is executed in sequence, and the console provides detailed logs and status updates for every step. This real\\-time feedback helps you identify and troubleshoot any issues within your workflow, enhancing the accuracy and efficiency of your processes.\n\nThe console not only shows the outputs generated by each block but also provides execution statistics and logs. This includes information about execution time, data transformations, and any errors encountered, enabling you to optimize the workflow for better performance.\n\nWas this page helpful?HistoryUp Next\n\nBuilt with"
75 }
76 }
77 },
78 {
79 "details": {
80 "vector_distance": 0.2602657675743103,
81 "hybrid_score": null
82 },
83 "record": {
84 "type": "table_query_record",
85 "id": "document/doc_70f35005df6ff97cee498b9ff4e00419/chunk/0:1",
86 "parent": {
87 "id": "doc_70f35005df6ff97cee498b9ff4e00419",
88 "type": "document"
89 },
90 "attributes": {
91 "url": "https://docs.scoutos.com/docs/workflows/history",
92 "page_title": "Workflow History — Scout | Documentation",
93 "content": "On this page* Accessing the workflow history\n* Version restoration\n* Draft management\n* Audit logs\nWorkflowsWorkflow History\n\nTracking and managing workflow changes in Scout\n\nScout provides a comprehensive audit trail, allowing you to track the evolution of your workflow over time. It logs edits, detailing who made changes and when, providing transparency and agility in your workflow development process.\n\n### Accessing the workflow history\n\nTo access the history of a workflow, navigate to the upper left\\-hand side of the workflow canvas and click on the time icon. This action opens the history panel, which displays a chronological list of all modifications made to the workflow.\n\n#### Version restoration\n\nYou can restore previous versions of your workflow. Click the three dots, then click restore to reinstate it as the active workflow. This capability is particularly useful for reverting changes that may have introduced errors or for exploring different iterations of a workflow design.\n\n#### Draft management\n\nNew drafts are autosaved, allowing you to experiment with different configurations without affecting the live version of the workflow.\n\n#### Audit logs\n\nThe history section functions as an audit log, providing a detailed account of each change, including additions, deletions, and modifications to workflow components. Each entry is timestamped and tagged with the user who performed the action, offering a clear view of the workflow’s development timeline.\n\nWas this page helpful?LogsUp Next\n\nBuilt with"
94 }
95 }
96 }
97]

Example Search Configurations

  1. Search for Product Reviews:

    • Search Term: “great quality”
    • Vector Minimum Distance: 0.7
    • Hybrid Search: true
    • BM25 Rank By: [{"attribute": "review", "query": "great quality"}]
    • Filters: ["rating", "Gt", "4"]
    • Limit: 5

    Outcome: Returns up to 5 reviews that highly match “great quality” in their semantic meaning and have a rating greater than 4.

  2. Retrieve Active Users:

    • Search Term: “John Doe”
    • Filters: ["status", "Eq", "active"]
    • Limit: 10

    Outcome: Lists up to 10 active users named “John Doe” or with a similar semantic match.

  3. Find Overdue Tasks:

    • Search Term: “overdue”
    • Vector Minimum Distance: 0.8
    • Filters: ["due_date", "Lt", 1704067200] (Timestamp for “2025-01-01”)
    • Limit: 20

    Outcome: Returns tasks that are semantically related to “overdue” and have a due date before January 1, 2025.

  4. Search Customer Feedback for Complaints:

    • Search Term: “bad service”
    • Hybrid Search: true
    • BM25 Rank By: [{"attribute": "feedback", "query": "bad service"}]
    • Filters: ["category", "Eq", "complaints"]
    • Limit: 15

    Outcome: Retrieves up to 15 feedback entries categorized as complaints and semantically matching “bad service.”

  5. Identify High-Priority Support Tickets:

    • Search Term: “urgent issue”
    • Vector Minimum Distance: 0.6
    • Filters: ["priority", "Eq", "high"]
    • Limit: 10

    Outcome: Returns support tickets with high semantic relevance to “urgent issue” and marked as high priority.

  6. Search for Products in a Price Range:

    • Search Term: “wireless headphones”
    • Filters: [["price", "Gt", "50"], ["price", "Lt", "150"]]
    • Limit: 10

    Outcome: Finds up to 10 wireless headphones priced between 50and50 and 150.

  7. Filter Blog Posts by Tags:

    • Search Term: “AI trends”
    • Hybrid Search: true
    • BM25 Rank By: [{"attribute": "tags", "query": "AI trends"}]
    • Filters: ["tags", "Contains", "AI"]
    • Limit: 5

    Outcome: Returns up to 5 blog posts tagged with “AI” and semantically related to “AI trends.”

  8. Find Users by Registration Date:

    • Search Term: “new users”
    • Filters: [["registration_date", "Gt", 1704067200], ["registration_date", "Lt", 1735689600]] (Timestamps for “2025-01-01” and “2025-12-31”)
    • Limit: 50

    Outcome: Retrieves up to 50 users who registered between January 1, 2025, and December 31, 2025.

  9. Search for Missing Information:

    • Search Term: “missing data”
    • Vector Minimum Distance: 0.9
    • Filters: ["email", "IsNull", null]
    • Limit: 25

    Outcome: Finds entries semantically related to “missing data” where the email field is null.

  10. Search for Top Performing Sales Reps:

    • Search Term: “top performer”
    • Filters: ["sales", "Gt", "100000"]
    • BM25 Rank By: [{"attribute": "performance_review", "query": "top performer"}]
    • Hybrid Search: true
    • Alpha: 0.8
    • Limit: 5

    Outcome: Retrieves the top 5 sales reps with sales exceeding $100,000 and high semantic relevance to “top performer” in their performance review.


Best Practices

  1. Use Jinja Templating: Dynamically construct queries, filters, and ranking criteria based on user input or workflow variables for flexible and adaptive searches.

  2. Fine-Tune Parameters: Experiment with Vector Minimum Distance, Alpha, and BM25 Rank By to achieve the desired balance between precision and recall.

  3. Apply Filters Thoughtfully: Filters can greatly enhance search efficiency by narrowing the scope of results to the most relevant subset.

  4. Leverage Hybrid Search: Use Hybrid Search for complex queries requiring a balance of semantic understanding and keyword-based ranking.

  5. Limit Results: Set a reasonable Limit to prevent overwhelming outputs and ensure the relevance of returned data.

Built with