Integrations

CrawlKit integrates through boring, durable primitives: REST, OpenAPI, JSON, Arrow IPC, Parquet exports, webhooks, workflow templates, and agent tools. Use it from applications, notebooks, data pipelines, AI agents, custom dashboards, and automation systems.

Integration map

REST and OpenAPI

Every platform capability is callable over HTTPS. Use API keys with Authorization: Bearer. Generate clients from /openapi.json, then treat CrawlKit as the control plane for acquisition, extraction, refinement, serving, and proof.

# Register Parquet as a DataFusion table
curl -X POST https://api.crawlkit.app/api/v1/query/register-table \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"table_name":"pages","parquet_path":"s3://crawlkit/pages/*.parquet"}'

# Build an AI-assisted spider, then run deterministic extraction later
curl -X POST https://api.crawlkit.app/api/v1/spiders/build \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"target_url":"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html","desired_fields":["title","price"],"test_after_build":true}'

# Validate a reusable durable workflow template
curl -X POST https://api.crawlkit.app/api/v1/workflow-templates/validate \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d @workflow-template.json

Data engineering stacks

For data platforms, integrate at the table and batch boundary. CrawlKit can parse raw inputs, write Parquet, register DataFusion views, run SQL, return Arrow IPC streams, export results, and attach lineage.

• Warehouses and lakes: export Parquet/CSV/JSON from query results or datasets.
• Python and notebooks: pull JSON for control-plane operations; use Arrow/Parquet for analytical payloads.
• Orchestrators: trigger workflow templates or ETL pipelines from Airflow, Dagster, Prefect, GitHub Actions, or custom jobs.
• Governance: query lineage, quality profiles, consent records, audit events, incidents, and run metadata alongside outputs.

Data format negotiation

Use JSON for control-plane calls and Arrow/Parquet for analytical payloads. The same data product can be served to dashboards, notebooks, lakehouse jobs, and agents without re-scraping.

Agent integrations

The Pi extension is the single source of truth for 263 tools across 28 categories. The crawlkit-pi-mcp stdio bridge imports that same registry and executor at runtime, giving Hermes Agent, Claude Code, Claude Desktop, OpenAI Codex, and OpenCode identical names, JSON Schemas, handlers, composites, workflow helpers, Arrow tools, and confirmation rules.

# Universal full-parity MCP bridge
export CRAWLKIT_API_KEY="ck_live_..."
npx -y --package @crawlkit/pi-extension crawlkit-pi-mcp

# Agent-oriented flow
crawlkit_status
crawlkit_tool_catalog
crawlkit_spider_pipeline_bootstrap(
  name="competitor-products",
  site_url="https://crawlkit.app",
  start_urls=["https://books.toscrape.com/catalogue/category/books/travel_2/index.html"],
  columns=[{"name":"title","column_type":"text"},{"name":"price","column_type":"number"}],
  run_pipeline=true,
  lineage=true
)

ChatGPT Actions and custom GPTs

Import openapi.json as an Actions schema. Start with read-only endpoints such as query views, report retrieval, quality profiles, lineage graphs, and run status. Add mutating endpoints only after you add approval rules and account scoping.

1. Create a scoped API key for the assistant.
2. Import https://docs.crawlkit.app/openapi.json.
3. Restrict instructions to approved operation tags.
4. Require confirmation before create, update, delete, crawl, publish, or run operations.

LangChain, CrewAI, and custom tools

For LLM frameworks, wrap small, task-specific actions rather than exposing the full API at once. Good tool boundaries are: inspect URL, build spider, test spider, create dataset, run workflow, query view, export dataset, get lineage, and check quality.

async function queryCrawlKit(sql) {
  const res = await fetch('https://api.crawlkit.app/api/v1/query/execute', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.CRAWLKIT_API_KEY}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ sql, limit: 100 })
  });
  if (!res.ok) throw new Error(await res.text());
  return await res.json();
}

Workflow and webhook patterns

Use CrawlKit workflows as the durable system of record for data acquisition and transformation. External systems should trigger templates, pass parameters, then monitor run IDs. Downstream systems can poll run status or consume events from your telemetry/alerting bridge.

POST /api/v1/workflow-templates/{template_id}/run
{
  "parameters": {
    "target_url": "https://crawlkit.app",
    "max_pages": 500,
    "quality_policy": "production"
  },
  "output_dataset_id": "ds_..."
}

Security patterns

• Use a separate API key per integration and rotate keys regularly.
• Give automation the smallest set of operations it needs.
• Require human confirmation for crawls, publishes, deletes, and external notifications.
• Persist run IDs, dataset IDs, and lineage asset IDs in your app instead of copying raw intermediate state.
• Use quality gates before serving or exporting derived datasets.

Provider authorization integrations

Browser-based provider connections use a shared tenant-safe authorization pattern: authenticated initiation, tenant-owned subject verification, hash-only one-time state, a public GET callback, exact redirect reuse, and server-only credential storage. Google Search Console is the reference implementation.

• Set up Google Search Console
• Use Google Search Console data
• Implement another authorization provider

Recommended starting points