Arrow + DataFusion Data Engineering

CrawlKit’s data plane is built for modern analytical workloads: ETL jobs parse and normalize data into Parquet/Arrow-shaped batches, DataFusion executes Arrow-native SQL, DuckDB handles hybrid Parquet views, and every output can be tied to datasets, schemas, quality gates, lineage, artifacts, and exports.

Positioning: This is the Serve and Query side of the CrawlKit backbone. It is what turns crawls, spiders, scrapes, social monitors, OpenRate ingestion, and synthetic data into usable data products.
Why Arrow? DataFusion gives CPU-efficient columnar SQL over Arrow batches; DuckDB hybrid views make Parquet joins practical; Arrow IPC/Flight-style streams let high-throughput clients consume results without flattening everything into JSON.

Core primitives

PrimitiveAPI surfaceJob
Dataset/datasetsTyped rows, imports, exports, stats, and extraction/enrichment output.
Artifact/artifactsRegister, list, download, and sign generated files and intermediate outputs.
ETL/etl/*Fetch, parse JSON/CSV, load Parquet, compress/decompress, and run ETL pipelines.
Query/query/*DuckDB, DuckDB hybrid, DataFusion table registration, Arrow-native SQL, schema, stats, and export.
Schema Registry/schemas/*Versioned schemas and validation contracts for datasets, artifacts, and workflow outputs.
Quality/quality/*Filters, dedup, profiles, scoring, classification, decontamination, and quality gates.
Lineage/lineage/*Asset graph, edges, consent, and evidence that proves where data came from.

ETL: parse, load, compress, fetch

# Parse CSV or JSON into Parquet/Arrow batches
curl -X POST https://api.crawlkit.app/api/v1/etl/parse \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "format": "json",
    "input_path": "s3://raw/source/events.json",
    "output_prefix": "s3://processed/events/",
    "record_path": "records",
    "batch_size": 50000,
    "use_simd": true
  }'

# Load Parquet batches into a queryable table
curl -X POST https://api.crawlkit.app/api/v1/etl/load \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "parquet_prefix": "s3://processed/events/",
    "table_name": "events",
    "column_names": ["event_id", "timestamp", "url", "payload"]
  }'

DataFusion: register Parquet and run Arrow-native SQL

# Register a Parquet path as a DataFusion table
curl -X POST https://api.crawlkit.app/api/v1/query/register-table \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "table_name": "pages",
    "parquet_path": "s3://crawlkit/web/pages/*.parquet"
  }'

# Query through DataFusion and return JSON rows from Arrow RecordBatches
curl -X POST https://api.crawlkit.app/api/v1/query/arrow \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sql": "select status_code, count(*) as pages from pages group by status_code order by pages desc",
    "limit": 100
  }'

DuckDB hybrid: register views and export results

# Register a Parquet glob as a DuckDB view
curl -X POST https://api.crawlkit.app/api/v1/query/register-view \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "view_name": "openrate_rates",
    "parquet_path": "s3://openrate/rates/**/*.parquet"
  }'

# Run hybrid SQL
curl -X POST https://api.crawlkit.app/api/v1/query/hybrid \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sql": "select payer_name, median(rate) from openrate_rates group by payer_name",
    "limit": 1000
  }'

# Export query output
curl -X POST https://api.crawlkit.app/api/v1/query/export \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sql": "select * from openrate_rates where state = '\''NY'\''",
    "output_path": "s3://exports/openrate-ny-rates.parquet",
    "format": "parquet"
  }'

Quality and proof

Use quality gates before serving outputs to applications, reports, streams, training data, or customers.

curl -X POST https://api.crawlkit.app/api/v1/quality/gate \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "00000000-0000-0000-0000-000000000000",
    "policy": {
      "rules": [
        {"field": "url", "rule": "not_null", "severity": "blocking"},
        {"field": "content_hash", "rule": "unique", "severity": "blocking"}
      ]
    },
    "quarantine": true
  }'

Backbone strip for data engineering

Source artifact → ETL run → Parquet batches → Dataset / DataFusion table
  → Schema registry → Quality gate → Query view / Export / Flight endpoint
  → Lineage graph → Audit evidence

Related API tags

PreviousGetting Started NextWorkflows