CrawlKit Docs

Nothing hidden. The reference now exposes 315 paths, 383 operations, 54 capability domains, and 532 schemas — including Query, ETL, Workflow Templates, Pipelines, Spiders, Crawl, Datasets, Artifacts, Lineage, Quality, Schema Registry, OpenRate, Telemetry, Governance, Healing, Synthetic Data, and SEO/GSC.

Core data-engineering loop

Seed URLs / APIs / files
  -> Crawl frontier + browser rendering + proxy strategy
  -> AI spider builder / selector discovery / repair / factory batches
  -> Typed datasets + artifacts + schema registry
  -> ETL parse/load/compress/fetch -> Parquet + Arrow RecordBatches
  -> DataFusion Arrow SQL + DuckDB hybrid query + exports
  -> Durable workflow templates + pipeline DAGs + schedules
  -> Quality gates + dedup + taxonomy + decontamination
  -> Lineage graph + governance + observability + downstream apps

🧪

Arrow, DataFusion, DuckDB, Parquet

Register Parquet paths, query Arrow-native DataFusion tables, run hybrid DuckDB SQL, inspect schemas/stats, and export results.

🚀

Durable workflows in full splendor

Reusable workflow templates, operation discovery, validation, durable runs, schedule control, worker progress, status updates, and lineage emission.

🕷

Spider management and scraping

AI spider build, selector suggestions, test previews, repair, health, import/export, spider factory, crawl jobs, frontiers, and proxy pools.

🗺

Complete API capability map

Every endpoint grouped by domain — all 383 operations from the generated OpenAPI contract, with representative routes.

Platform surfaces

Query

DataFusion Arrow SQL, DuckDB views, hybrid query, table registration, schema introspection, stats, and exports.

ETL

Parse JSON/CSV, write Parquet batches, load tables, compress/decompress files, fetch remote artifacts, and run ETL pipelines.

Workflow Templates

Create, validate, run, monitor, cancel, and discover operations for reusable durable workflows.

Pipelines

DAG pipelines, schedule start/stop/status, step execution, run progress/status, cancellation, and lineage registration.

Spiders

AI build, field suggestion, repair, factory batches, CRUD, tests, health, imports, and exports.

Crawl

Crawl jobs, strategies, frontier stats, pause/resume/cancel, proxy inventory, and stealth settings.

Datasets

Typed dataset CRUD, rows, imports, exports, and statistics for extraction and enrichment outputs.

Lineage

Assets, edges, upstream/downstream graph, consent records, and provenance across crawls, ETL, pipelines, and exports.

Quality

Heuristic filters, deduplication, profiles, scoring, taxonomy classification, benchmark decontamination, and quality gates.

OpenRate

Healthcare price transparency search, comparison, discrepancy detection, MRF/TiC ingestion, reference imports, and view refresh.

Telemetry

Events, sessions, traces, metrics, funnels, dashboards, destinations, projects, and OpenTelemetry ingestion.

Agent interface

Use REST/OpenAPI directly or call CrawlKit from agent tools, including the CrawlKit Pi extension for local agent workflows.

First useful calls

# Register Parquet as a DataFusion table
curl -X POST https://api.crawlkit.app/api/v1/query/register-table \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"table_name":"crawl_pages","parquet_path":"s3://crawlkit/tenant/pages/*.parquet"}'

# Query through the Arrow-native engine
curl -X POST https://api.crawlkit.app/api/v1/query/arrow \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"sql":"select status, count(*) from crawl_pages group by status","limit":100}'

# Build a spider from a representative URL
curl -X POST https://api.crawlkit.app/api/v1/spiders/build \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"target_url":"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html","desired_fields":["title","price","availability"],"test_after_build":true}'

Machines should start at /llms.txt and /openapi.json. Humans can start with Arrow + DataFusion, Workflows, or Spiders.

CrawlKit turns the web into queryable, governed, workflow-ready data.