Scraping a Website

The full URL of the page, domain or sitemap you want to scrape.

Select the scraper technology. Available options are:

• HTTP - Faster, but may not mix well with heavily client side rendered sites.
• Playwright - Heavier and slower, but renders dynamic content, which works well with heavily client side rendered sites.

Defines how text is extracted from the page. Options include:

• Custom - Your own custom scraping logic.
• Readability - Scout’s smart scraping logic to include relevant components.
• Trafilatura - A Python package and command-line tool used to gather text on the web.

CSS selectors for the HTML elements you want to include in the scraping. For example, targeting specific tags like body, h1, p.

CSS selectors for the HTML elements you want to exclude from the scraping process. For example, elements like .navbar, #header, footer.

Select this option if you want to automatically remove common elements such as headers and footers that often appear on multiple pages.

Exclude any pages with a lastmod date older than the specified date. This helps to avoid scraping outdated content.

Sets the maximum depth for the crawl. This limits how deep the scraper will follow links from the original page.

Element selectors that you want stripped from the Document.

Defines the maximum number of pages to scrape. Use this to prevent overly large scrapes.

A list of allowed paths or patterns that the scraper will include in the crawl.

Specify which domains are allowed in the crawl. Links to other domains will be excluded unless specified here.

A list of paths or patterns that the scraper will exclude from the crawl.