Web Page Scrape Block — Scout

The Web Page Scrape block enables users to extract HTML content from web pages by making HTTP requests. It supports the exclusion of specific elements using CSS selectors, providing flexibility in what content is retrieved. This block is useful for integrating web scraping capabilities into Scout workflows, enabling automated data extraction from web pages.

Configuration (Required)

URL to which the request will be made

stringRequired

The URL specifies the web page to be scraped. Ensure the URL is valid and accessible to successfully retrieve the content. This input supports Jinja templating for dynamic content.

Exclude Selectors

string

A comma-separated list of classes, ids, or tags to exclude from the scraped content. Use this to remove unwanted elements from the HTML output. This input supports Jinja templating for dynamic content.

See Workflow Logic & State > State Management for details on using dynamic variables in this block.

Outputs

The block returns the HTML content of the web page with specified elements excluded. This output can be used in subsequent workflow steps to facilitate further data processing or integration.

Usage Context

Use this block to scrape web page content while excluding unwanted elements. This is particularly useful for extracting clean data from web pages for processing within Scout workflows.

Best Practices

Ensure the URL is valid and accessible: Verify that the URL is correct and reachable to avoid errors during the scraping process.
Use the exclude selectors to remove unnecessary elements from the HTML content: Specify selectors to exclude unnecessary elements, ensuring cleaner and more relevant data extraction.
Handle exceptions such as network errors and timeouts to ensure robust scraping: Implement error handling to manage network issues and timeouts effectively, ensuring robust and reliable scraping operations.