Developer Guides

Start with CrawlKit’s real platform surface: Arrow-native query, ETL, durable workflows, spider management, scraping, datasets, quality, lineage, and governance. SEO, GSC, content, leads, and reports remain available as applied lenses on the same backbone.

⚡

Getting Started

Get an API key, call the API, then create a dataset, build a spider, run enrichment, and understand the platform loop.

🧪

Arrow + DataFusion

ETL, Parquet, DataFusion, DuckDB hybrid query, schemas, artifacts, exports, quality, and lineage.

🚀

Durable Workflows

Workflow templates, operation discovery, pipeline DAGs, schedules, run monitoring, worker progress, and lineage emission.

🕷

Spiders & Scraping

AI spider builder, deterministic definitions, selectors, repair, testing, crawl jobs, frontier, stealth/proxy controls, and datasets.

🔒

Authentication

API keys, dashboard session tokens, tenant selection, scopes, validation, usage, and safe token management.

🤖

Integrations

Use REST/OpenAPI directly or connect CrawlKit to agent tools, MCP-compatible clients, and the CrawlKit Pi extension.

Google Search Console

🛡

Authorization Pattern

Build reusable provider integrations with tenant binding, one-time state, public callbacks, and server-only credentials.

⏱

Rate Limits

Tier-based limits, retry strategy, response headers, and safe backoff for crawls, workflows, and data APIs.

🚧

Error Reference

Error response format, HTTP status codes, correlation IDs, and operational handling patterns.

Platform model

CrawlKit is a data engineering backbone plus capability workbenches plus downstream applications. Every source, spider, crawl, ETL job, enrichment, query, export, OpenRate result, SEO issue, social mention, or synthetic run should map to source, run, dataset, schema, quality, lineage, policy, and audit evidence.

Acquire → Extract → Refine → Serve → Prove

Live web / archives / files / APIs / social / domain sources
  → spiders / crawls / ETL / parsers / synthetic ops
  → typed datasets + artifacts + schemas
  → quality gates + enrichment + scoring + indexes
  → DataFusion/DuckDB queries + Flight + streams + exports
  → lineage + audit + governance + downstream applications

Complete API domains

The public OpenAPI reference currently exposes 383 operations across 54 domains. The highest-priority technical domains are:

Domain	Use it for
Query	DataFusion Arrow SQL, DuckDB hybrid views, schema/stats, exports.
ETL	Fetch, parse, validate, load, compress, ETL pipelines.
Workflow Templates	Reusable durable workflows, operation registry, runs, validation, cancellation.
Pipelines	DAG processing, schedules, step execution, progress/status, lineage emission.
Spiders + Crawl	AI spider build/repair, deterministic extraction, crawl jobs, frontier, proxies.
Datasets + Artifacts	Typed data products and generated file/object outputs.
Lineage + Quality + Schema Registry	Trust, validation, contracts, provenance, and policy evidence.
OpenRate	Healthcare price-transparency downstream application and ingestion pipelines.
Telemetry + Healing + Incidents	Operational evidence, self-healing, RCA, and incident workflows.

Machine-readable docs

/openapi.json — full generated OpenAPI contract.
/llms.txt — compact LLM navigation entrypoint.
/llms-full.txt — generated endpoint inventory grouped by domain.
/capabilities.html — human-readable full API surface map.