Web scraping & extraction

Vector ships three crawlers and 24 single-page extraction actors.

Crawlers

• beautifulsoup-scraper — crawl from start URLs, follow links to a depth, extract title/headings/links.
• website-content-crawler — crawl a site and return clean readable text for AI/RAG pipelines.
• link-extractor — one URL in, all links/images/scripts out.

curl -X POST ".../v1/actors/website-content-crawler/run-sync-get-dataset-items" \
  -H "X-API-Key: vk_live_..." -H "Content-Type: application/json" \
  -d '{"start_urls":["https://example.com"],"max_pages":10}'

Extraction recipes

Point any of these at a single URL: page-metadata, opengraph, twitter-card, json-ld, headings, page-outline, page-links, page-images, page-tables, page-forms, feed-discovery, page-text, tech-hints, favicon, social-links, and more.

Good citizen by default

Every page fetch respects robots.txt, throttles per host, and caps page counts. That's what keeps scraping sustainable and legitimate.