Spiders, Scraping, and Web Acquisition

CrawlKit treats web acquisition as a first-class data engineering workbench. Spiders define deterministic extraction, crawls manage frontier and runtime behavior, scraping jobs combine acquisition and extraction, and every output lands in datasets with schema, quality, lineage, policy, and audit.

Design principle: AI once, deterministic forever. AI can build, suggest fields, and repair selectors, but production runs use explicit spider definitions, scoped crawl rules, validated selectors, repeatable processors, and auditable runs.

Spider Studio model

Concept	What it controls
Spider kind	Page scrape, paginated list, product catalog, search results, feed/API/sitemap, social/media, file download, dataset recrawl, or custom.
Scope	Start URLs, allowed domains, allow/deny patterns, depth, max pages, and target URL patterns.
Handlers	URL-pattern-specific extraction logic, iterate selectors, detail links, and field mappings.
Fields	Name, selector, type, required/default behavior, attribute, processors, validation, and description.
Selectors	CSS, XPath, JSONPath, regex, meta/header/structured data, LLM-assisted suggestions, fallback, and scoped selectors.
Navigation	Follow links, pagination, infinite scroll, form submit, URL templates, API pagination, and sitemap traversal.
Processors	Strip, case, replace/regex, cast, parse date/URL/JSON, normalize, extract emails/URLs/numbers, split, slugify, titlecase.
Health	Field completeness, selector degradation, required-field failures, render requirements, challenge detection, and repair opportunities.

Build a spider with AI

curl -X POST https://api.crawlkit.app/api/v1/spiders/build \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "desired_fields": ["title", "price", "availability", "image_url", "sku"],
    "description": "Extract product catalog detail pages into a trusted dataset.",
    "instructions": [
      "Prefer stable semantic selectors over visual utility classes.",
      "Use fallback selectors for price and availability.",
      "Return null rather than hallucinating unavailable fields."
    ],
    "target_schema": {
      "title": "string",
      "price": "string",
      "availability": "string",
      "image_url": "url",
      "sku": "string"
    },
    "sample_urls": [
      "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "https://books.toscrape.com/catalogue/soumission_998/index.html"
    ],
    "test_after_build": true,
    "max_repair_attempts": 2,
    "min_success_rate": 0.9
  }'

Suggest fields before building

curl -X POST https://api.crawlkit.app/api/v1/spiders/suggest-fields \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"}'

Create, test, and monitor a deterministic spider

# Create explicit spider definition
curl -X POST https://api.crawlkit.app/api/v1/spiders \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "product_catalog_spider",
    "kind": "product_catalog",
    "scope": {
      "start_urls": ["https://books.toscrape.com/catalogue/category/books/travel_2/index.html"],
      "domains": ["books.toscrape.com"],
      "allowed_patterns": ["/products/*"],
      "denied_patterns": ["/cart", "/checkout"],
      "max_depth": 2
    },
    "definition": {
      "handlers": [{
        "url_pattern": "/products/*",
        "fields": [
          {"name":"title", "selector":"h1", "type":"text", "required":true},
          {"name":"price", "selector":"[data-price], .price", "type":"text"},
          {"name":"image_url", "selector":"img.product-image", "attribute":"src"}
        ]
      }],
      "navigation": {"follow_links":"a[href*='/products/']", "max_depth":2}
    }
  }'

# Test extraction on one URL
curl -X POST https://api.crawlkit.app/api/v1/spiders/SPIDER_ID/test \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"}'

# Inspect spider health and field completeness
curl https://api.crawlkit.app/api/v1/spiders/SPIDER_ID/health \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

Health degradation detection

Spider health is not just uptime. CrawlKit tracks field completeness, selector misses, render requirements, challenge rates, and repair opportunities so extraction can degrade visibly instead of silently poisoning a dataset.

Field selector miss: price selector matched 0 nodes across 5 consecutive pages
Required field failure: title missing on 3.2% of recent samples
JS required: static HTML lacks product JSON; headless render recovers fields
Challenge detected: 403/429/CAPTCHA on 10% of fetches
Repair candidate: add [data-price] and .price-current fallback selectors

Repair a degraded spider

curl -X POST https://api.crawlkit.app/api/v1/spiders/SPIDER_ID/repair \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "failures": [
      {"field_name":"price", "handler_name":"product_detail", "reason":"selector_matched_zero_nodes"},
      {"field_name":"availability", "handler_name":"product_detail", "reason":"required_field_missing"}
    ]
  }'

Run a crawl job

Crawl jobs apply budgets, strategies, pause/resume/cancel semantics, proxy/stealth controls, and frontier visibility. Create or select a site first; crawl jobs use the returned site_id.

# Create or register a site
curl -X POST https://api.crawlkit.app/api/v1/sites \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"books.toscrape.com", "framework":"unknown"}'

# Start a crawl
curl -X POST https://api.crawlkit.app/api/v1/crawl/jobs \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_id": "00000000-0000-0000-0000-000000000000",
    "budget": {"max_pages": 500, "max_concurrency": 8, "js_rendering": true},
    "strategy_config": {"base_strategy":"breadth_first", "max_depth": 3},
    "stealth_config": {"rotate_user_agent": true, "delay_ms": [250, 1500]}
  }'

# Inspect frontier stats
curl https://api.crawlkit.app/api/v1/crawl/frontier/JOB_ID \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

# Pause, resume, or cancel
curl -X POST https://api.crawlkit.app/api/v1/crawl/jobs/JOB_ID/pause \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

Spider-to-dataset-to-query pattern

Representative URL → AI field suggestion → spider definition → test extraction
  → crawl job / scrape job → output dataset → quality gate → lineage
  → DataFusion/DuckDB query → export / report / downstream application

Operational states to watch

State	Meaning	Safe action
Selector miss	A field selector matched zero nodes.	Run suggest-fields or repair with validation failures.
JS required	Static HTML lacks required data.	Enable rendering or use API/structured-data selectors.
Challenge detected	Target returned anti-bot, CAPTCHA, or rate-limit page.	Review robots/policy, adjust politeness, proxy/stealth, or stop.
Frontier backlog	Queued URLs exceed worker throughput.	Adjust budget, strategy, concurrency, or prioritization.
Quality gate failed	Extracted rows failed schema or business rules.	Quarantine rows, repair spider, rerun affected pages.

Related API tags

Spiders — AI build, suggest, repair, CRUD, test, health, factory.
Crawl — jobs, strategies, frontier, pause/resume/cancel, proxies.
Inspection — URL/HTML inspection and selector discovery.
Datasets — output storage for extracted rows.
Quality — gates, filters, dedup, decontamination.