Durable Workflows

Workflows are the orchestration layer that makes CrawlKit more than scraping and ETL. They compose crawling, spiders, ETL, extraction, enrichment, quality, DataFusion/Flight, exports, synthetic data, OpenRate, SEO/GSC, social, telemetry, and governance into repeatable, durable, lineage-emitting data products.

Workflow layers

Operation registry

The Workflow Composer exposes concrete operation cards, not vague automation. These operation families map to real handlers in the CrawlKit platform.

Data Acquisition
  crawl.crawl, crawl.crawl_distributed, crawl.frontier_seed,
  crawl.frontier_claim, crawl.delta_check, crawl.extract_content,
  crawl.extract_pipeline, crawl.framework_detect, crawl.warc_read, crawl.cc_search

ML / Quality / Synthetic
  quality.heuristic_filter, quality.fuzzy_dedup, quality.decontaminate,
  quality.chunk_text, quality.score_quality, ml.synthetic_generate,
  ml.synthetic_verify, ml.cqsm_score, ml.eeat_score, ml.kosm_score

Output / Serving
  output.export_data, output.training_format, output.dataset_card,
  output.hf_push, output.stream_register, output.stream_quality_filter,
  output.stream_format, output.stream_emit

Analytics / Flight
  analytics.sql_query, analytics.schema_describe,
  analytics.flight_serve, analytics.flight_fetch

Platform / Dataset / Telemetry
  platform.dataset_create, platform.dataset_insert_rows,
  platform.dataset_validate_rows, platform.dataset_query,
  platform.telemetry_process, platform.telemetry_sink

OpenRate
  openrate.cms_discover, openrate.change_detect, openrate.resilient_fetch,
  openrate.parse_hospitals, openrate.parse_tic, openrate.parse_rates,
  openrate.parse_payers, openrate.parse_reference, openrate.enrich_pipeline,
  openrate.upsert_batch, openrate.refresh_views, openrate.complete_run

Discover operations

curl https://api.crawlkit.app/api/v1/workflow-templates/operations \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

curl https://api.crawlkit.app/api/v1/workflow-templates/operations/analytics.sql_query \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

Validate and create a workflow template

# Validate an inline template before storing it
curl -X POST https://api.crawlkit.app/api/v1/workflow-templates/validate \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "template": {
      "name": "web_scrape_to_trusted_dataset",
      "description": "Crawl, extract, quality-gate, query, and export.",
      "steps": [
        {"name":"seed", "operation":"crawl.frontier_seed", "config":{"start_urls":["https://crawlkit.app"]}},
        {"name":"crawl", "operation":"crawl.crawl_distributed", "depends_on":["seed"], "config":{"max_pages":500}},
        {"name":"extract", "operation":"crawl.extract_pipeline", "depends_on":["crawl"], "config":{"schema":"product_schema"}},
        {"name":"quality", "operation":"quality.heuristic_filter", "depends_on":["extract"], "config":{"rules":[{"field":"title","rule":"not_empty"}]}},
        {"name":"export", "operation":"output.export_data", "depends_on":["quality"], "config":{"format":"parquet"}}
      ]
    }
  }'

# Store the reusable template
curl -X POST https://api.crawlkit.app/api/v1/workflow-templates \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"template": {"name":"web_scrape_to_trusted_dataset", "steps": []}}'

Run and monitor

# Run a template with parameters
curl -X POST https://api.crawlkit.app/api/v1/workflow-templates/TEMPLATE_ID/run \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "parameters": {"target_domain":"crawlkit.app"},
    "output_dataset_id": "00000000-0000-0000-0000-000000000000"
  }'

# List runs
curl "https://api.crawlkit.app/api/v1/workflow-templates/runs?status=running" \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

# Get a run
curl https://api.crawlkit.app/api/v1/workflow-templates/runs/RUN_ID \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

# Cancel a run
curl -X POST https://api.crawlkit.app/api/v1/workflow-templates/runs/RUN_ID/cancel \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

Pipeline DAG internals

Pipeline endpoints expose the operational details that durable workflows need: schedules, execute-step, progress updates, status updates, and lineage emission.

# Start a schedule
curl -X POST https://api.crawlkit.app/api/v1/pipelines/PIPELINE_ID/schedule/start \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"cron":"0 */6 * * *", "timezone":"UTC"}'

# Emit lineage after a run
curl -X PUT https://api.crawlkit.app/api/v1/pipelines/runs/RUN_ID/lineage \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "template_name":"web_scrape_to_trusted_dataset",
    "output_dataset_id":"00000000-0000-0000-0000-000000000000",
    "step_assets":[
      {"step_index":0, "step_name":"crawl", "operation":"crawl.crawl_distributed", "rows_output":500},
      {"step_index":1, "step_name":"export", "operation":"output.export_data", "rows_output":500}
    ]
  }'

Canonical workflow templates

Workflow proof model

Workflow template → Durable run → Step timeline → Artifacts
  → Output dataset/assets → Quality gates → Lineage edges → Audit/telemetry

Related API tags

• Workflow Templates
• Pipelines
• ETL
• Lineage
• OpenRate Admin