Data Pipeline Tutorial

Turn any website into structured, enriched data. This tutorial walks through the full CrawlKit data platform: create a dataset, deploy a spider, run a crawl, enrich the results, and export — all connected through a single pipeline.

This is CrawlKit's core capability. The data enrichment platform powers everything from lead generation to competitive intelligence. Once you understand this workflow, you can build pipelines for any vertical.

Pipeline Architecture

A CrawlKit data pipeline connects these stages in a DAG (directed acyclic graph):

Seed URLs  →  Crawl  →  Extract (Spider)  →  Enrich  →  Filter  →  Export
   |             |            |                  |            |           |
   URLs       Fetch HTML   CSS/XPath         17+ sources   Quality    CSV/JSON/
   Lists      JS render    selectors         Company data   gates     Webhook/
   Sitemaps   Pagination   Data mapping      SEO metrics    Dedup     Database

Step 1: Create a Dataset with Schema

A dataset is a structured table that holds your extracted and enriched data. Define columns with types to enforce data quality from the start.

curl

curl -X POST https://api.crawlkit.app/api/v1/datasets \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "saas_companies",
    "description": "B2B SaaS companies extracted from product directories",
    "site_url": "https://crawlkit.app",
    "columns": [
      { "name": "company_name", "data_type": "text" },
      { "name": "domain", "data_type": "text" },
      { "name": "description", "data_type": "text" },
      { "name": "pricing_model", "data_type": "text" },
      { "name": "monthly_traffic", "data_type": "integer" },
      { "name": "tech_stack", "data_type": "jsonb" },
      { "name": "employee_count", "data_type": "integer" },
      { "name": "funding_total", "data_type": "text" },
      { "name": "contact_email", "data_type": "text" },
      { "name": "social_profiles", "data_type": "jsonb" }
    ]
  }'

Response

{
  "dataset_id": "ds_7k2m3n4p5q",
  "name": "saas_companies",
  "columns": [
    { "name": "company_name", "data_type": "text", "nullable": true },
    { "name": "domain", "data_type": "text", "nullable": true },
    { "name": "description", "data_type": "text", "nullable": true },
    { "name": "pricing_model", "data_type": "text", "nullable": true },
    { "name": "monthly_traffic", "data_type": "integer", "nullable": true },
    { "name": "tech_stack", "data_type": "jsonb", "nullable": true },
    { "name": "employee_count", "data_type": "integer", "nullable": true },
    { "name": "funding_total", "data_type": "text", "nullable": true },
    { "name": "contact_email", "data_type": "text", "nullable": true },
    { "name": "social_profiles", "data_type": "jsonb", "nullable": true }
  ],
  "row_count": 0,
  "created_at": "2026-03-13T10:00:00Z"
}

Supported column types: text, integer, float, boolean, timestamp, jsonb, url, email.

Step 2: Create a Spider to Extract Data

A spider defines how to extract structured data from web pages. You provide CSS or XPath selectors that map page elements to your dataset columns. CrawlKit handles pagination, navigation, and data normalization.

curl

curl -X POST https://api.crawlkit.app/api/v1/spiders \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "saas_directory_spider",
    "site_url": "https://crawlkit.app",
    "scope": {
      "start_urls": ["https://www.producthunt.com/topics/saas"],
      "allowed_domains": ["producthunt.com"],
      "url_patterns": ["/products/*"]
    },
    "definition": {
      "handlers": [
        {
          "url_pattern": "/products/*",
          "selectors": {
            "company_name": { "css": "h1.product-title", "attribute": "text" },
            "domain": { "css": "a.product-website", "attribute": "href" },
            "description": { "css": "div.product-description p", "attribute": "text" },
            "pricing_model": { "css": "span.pricing-badge", "attribute": "text" }
          }
        }
      ],
      "navigation": {
        "pagination": { "css": "a.next-page", "max_pages": 20 },
        "follow_links": { "css": "a.product-card-link" }
      }
    },
    "dataset_id": "ds_7k2m3n4p5q"
  }'

Response

{
  "spider_id": "sp_3a4b5c6d7e",
  "name": "saas_directory_spider",
  "status": "created",
  "scope": {
    "start_urls": ["https://www.producthunt.com/topics/saas"],
    "allowed_domains": ["producthunt.com"],
    "url_patterns": ["/products/*"]
  },
  "dataset_id": "ds_7k2m3n4p5q",
  "created_at": "2026-03-13T10:05:00Z"
}

Step 3: Test the Spider

Before running a full crawl, test your spider against a single page to verify that selectors are working and data extraction is correct.

curl

curl -X POST https://api.crawlkit.app/api/v1/spiders/sp_3a4b5c6d7e/test \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "test_url": "https://www.producthunt.com/products/notion",
    "js_rendering": true
  }'

Response

{
  "success": true,
  "test_url": "https://www.producthunt.com/products/notion",
  "extracted_data": {
    "company_name": "Notion",
    "domain": "https://www.notion.so",
    "description": "Connected workspace for wiki, docs & projects",
    "pricing_model": "Freemium"
  },
  "selectors_matched": 4,
  "selectors_failed": 0,
  "navigation_links_found": 12,
  "warnings": [],
  "render_time_ms": 2340
}

Tip: If selectors fail, use the POST /api/v1/inspection/selectors endpoint to test CSS/XPath selectors interactively against any URL.

Step 4: Start a Crawl Job

Launch a crawl job that applies your spider across all matching pages. The crawl engine handles rate limiting, retries, proxy rotation, and respects robots.txt.

curl

curl -X POST https://api.crawlkit.app/api/v1/crawl/jobs \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://www.producthunt.com",
    "spider_id": "sp_3a4b5c6d7e",
    "strategy_id": "breadth_first",
    "budget": {
      "max_pages": 500
    },
    "js_rendering": true,
    "respect_robots_txt": true
  }'

Response

{
  "job_id": "cj_8f9g0h1i2j",
  "status": "running",
  "spider_id": "sp_3a4b5c6d7e",
  "strategy": "breadth_first",
  "budget": { "max_pages": 500 },
  "progress": {
    "pages_crawled": 0,
    "pages_queued": 1,
    "pages_failed": 0,
    "rows_extracted": 0
  },
  "started_at": "2026-03-13T10:15:00Z"
}

Check crawl status

curl "https://api.crawlkit.app/api/v1/crawl/jobs/cj_8f9g0h1i2j" \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

{
  "job_id": "cj_8f9g0h1i2j",
  "status": "completed",
  "progress": {
    "pages_crawled": 487,
    "pages_queued": 0,
    "pages_failed": 13,
    "rows_extracted": 342
  },
  "duration_secs": 245.6,
  "completed_at": "2026-03-13T10:19:06Z"
}

Step 5: Enrich the Data

Now that you have raw extracted data, enrich it with additional signals from CrawlKit's 17+ enrichment sources. Enrichment adds company data, tech stacks, SEO metrics, social signals, and more to each row.

curl

curl -X POST https://api.crawlkit.app/api/v1/enrichment/enrich \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "dataset_id": "ds_7k2m3n4p5q",
    "source_name": "company_data",
    "match_column": "domain",
    "enrich_columns": ["employee_count", "funding_total", "tech_stack"]
  }'

Response

{
  "job_id": "ej_4k5l6m7n8o",
  "status": "processing",
  "dataset_id": "ds_7k2m3n4p5q",
  "source": "company_data",
  "rows_to_enrich": 342,
  "estimated_completion_secs": 120
}

Available Enrichment Sources

Source	Data Provided	Match Column
web_scrape	Re-crawl target URLs for additional data points	`url` or `domain`
company_data	Employee count, founding year, funding, revenue range, industry	`domain`
contact_data	Email addresses, phone numbers, job titles, LinkedIn profiles	`domain` or `company_name`
seo_metrics	Domain authority, monthly traffic, backlink count, referring domains	`domain`
social_signals	Social media profiles, follower counts, engagement rates	`domain` or `company_name`
tech_detection	Technology stack: frameworks, CMS, analytics, CDN, hosting	`domain`

Run multiple enrichment passes to layer data:

# Enrich with tech stack detection
curl -X POST https://api.crawlkit.app/api/v1/enrichment/enrich \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "dataset_id": "ds_7k2m3n4p5q",
    "source_name": "tech_detection",
    "match_column": "domain",
    "enrich_columns": ["tech_stack"]
  }'

# Enrich with SEO metrics
curl -X POST https://api.crawlkit.app/api/v1/enrichment/enrich \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "dataset_id": "ds_7k2m3n4p5q",
    "source_name": "seo_metrics",
    "match_column": "domain",
    "enrich_columns": ["monthly_traffic"]
  }'

Step 6: Build a Pipeline

A pipeline connects all the stages into a single, repeatable workflow. Define the DAG of stages — seed, crawl, extract, enrich, filter, export — and CrawlKit orchestrates the entire flow with durable execution (powered by Restate).

curl

curl -X POST https://api.crawlkit.app/api/v1/pipelines \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "saas_intelligence_pipeline",
    "site_url": "https://crawlkit.app",
    "description": "Weekly SaaS company intelligence from Product Hunt",
    "stages": [
      {
        "id": "seed",
        "type": "seed",
        "config": {
          "urls": ["https://www.producthunt.com/topics/saas"],
          "mode": "sitemap_and_list"
        }
      },
      {
        "id": "crawl",
        "type": "crawl",
        "depends_on": ["seed"],
        "config": {
          "spider_id": "sp_3a4b5c6d7e",
          "max_pages": 500,
          "js_rendering": true,
          "strategy": "breadth_first"
        }
      },
      {
        "id": "extract",
        "type": "extract",
        "depends_on": ["crawl"],
        "config": {
          "spider_id": "sp_3a4b5c6d7e",
          "dataset_id": "ds_7k2m3n4p5q"
        }
      },
      {
        "id": "enrich_company",
        "type": "enrich",
        "depends_on": ["extract"],
        "config": {
          "source": "company_data",
          "match_column": "domain",
          "enrich_columns": ["employee_count", "funding_total"]
        }
      },
      {
        "id": "enrich_tech",
        "type": "enrich",
        "depends_on": ["extract"],
        "config": {
          "source": "tech_detection",
          "match_column": "domain",
          "enrich_columns": ["tech_stack"]
        }
      },
      {
        "id": "enrich_seo",
        "type": "enrich",
        "depends_on": ["enrich_company", "enrich_tech"],
        "config": {
          "source": "seo_metrics",
          "match_column": "domain",
          "enrich_columns": ["monthly_traffic"]
        }
      },
      {
        "id": "export",
        "type": "export",
        "depends_on": ["enrich_seo"],
        "config": {
          "format": "json",
          "destination": "webhook",
          "webhook_url": "https://your-app.com/api/pipeline-results"
        }
      }
    ],
    "schedule": {
      "cron": "0 6 * * 1",
      "timezone": "America/New_York"
    }
  }'

Response

{
  "pipeline_id": "pl_1a2b3c4d5e",
  "name": "saas_intelligence_pipeline",
  "status": "created",
  "stages": 7,
  "schedule": {
    "cron": "0 6 * * 1",
    "timezone": "America/New_York",
    "next_run": "2026-03-17T06:00:00-04:00"
  },
  "created_at": "2026-03-13T10:30:00Z"
}

Step 7: Run the Pipeline

Trigger a pipeline run manually or wait for the scheduled cron. Each run creates a durable workflow that tracks progress across all stages.

curl

curl -X POST https://api.crawlkit.app/api/v1/pipelines/pl_1a2b3c4d5e/run \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

Response

{
  "run_id": "pr_6f7g8h9i0j",
  "pipeline_id": "pl_1a2b3c4d5e",
  "status": "running",
  "stages": {
    "seed": { "status": "completed", "output": { "urls_generated": 1 } },
    "crawl": { "status": "running", "progress": { "pages_crawled": 42, "pages_queued": 458 } },
    "extract": { "status": "pending" },
    "enrich_company": { "status": "pending" },
    "enrich_tech": { "status": "pending" },
    "enrich_seo": { "status": "pending" },
    "export": { "status": "pending" }
  },
  "started_at": "2026-03-13T10:35:00Z"
}

Check run status

curl "https://api.crawlkit.app/api/v1/pipelines/pl_1a2b3c4d5e/runs/pr_6f7g8h9i0j" \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

{
  "run_id": "pr_6f7g8h9i0j",
  "status": "completed",
  "stages": {
    "seed": { "status": "completed", "duration_secs": 0.2 },
    "crawl": { "status": "completed", "duration_secs": 245.6, "output": { "pages_crawled": 487 } },
    "extract": { "status": "completed", "duration_secs": 12.3, "output": { "rows_extracted": 342 } },
    "enrich_company": { "status": "completed", "duration_secs": 89.1, "output": { "rows_enriched": 298 } },
    "enrich_tech": { "status": "completed", "duration_secs": 67.4, "output": { "rows_enriched": 312 } },
    "enrich_seo": { "status": "completed", "duration_secs": 45.2, "output": { "rows_enriched": 330 } },
    "export": { "status": "completed", "duration_secs": 2.1, "output": { "rows_exported": 342 } }
  },
  "total_duration_secs": 461.9,
  "completed_at": "2026-03-13T10:42:42Z"
}

Step 8: Export Results

Export your enriched dataset in any format. Supports CSV, JSON, JSONL, and webhook delivery.

curl

curl -X POST https://api.crawlkit.app/api/v1/datasets/ds_7k2m3n4p5q/export \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "format": "json",
    "filters": {
      "monthly_traffic": { "gte": 10000 },
      "employee_count": { "gte": 10, "lte": 500 }
    },
    "columns": ["company_name", "domain", "monthly_traffic", "tech_stack", "employee_count", "funding_total"]
  }'

Response

{
  "export_id": "ex_2b3c4d5e6f",
  "format": "json",
  "row_count": 87,
  "filtered_from": 342,
  "data": [
    {
      "company_name": "Notion",
      "domain": "notion.so",
      "monthly_traffic": 28400000,
      "tech_stack": ["React", "Node.js", "PostgreSQL", "AWS", "Cloudflare"],
      "employee_count": 450,
      "funding_total": "$343.5M"
    },
    {
      "company_name": "Linear",
      "domain": "linear.app",
      "monthly_traffic": 1200000,
      "tech_stack": ["React", "TypeScript", "PostgreSQL", "Vercel"],
      "employee_count": 85,
      "funding_total": "$52M"
    },
    {
      "company_name": "Resend",
      "domain": "resend.com",
      "monthly_traffic": 340000,
      "tech_stack": ["Next.js", "TypeScript", "PostgreSQL", "AWS"],
      "employee_count": 28,
      "funding_total": "$5.2M"
    }
  ]
}

Industry Verticals

CrawlKit's data platform is designed to work across any industry. Here are 8 verticals with proven spider templates and enrichment strategies:

E-commerce

Extract product catalogs, pricing, reviews, inventory levels. Selectors for product title, price, SKU, availability, ratings. Enrich with competitor pricing and market positioning.

News & Media

Monitor news sources, extract articles with author, date, category, and content. Build media monitoring dashboards. Enrich with social sharing metrics and sentiment analysis.

Forums & Communities

Extract discussions, answers, user profiles, and engagement metrics from forums like Reddit, StackOverflow, and niche communities. Identify trending topics and expert contributors.

Job Boards

Collect job postings with title, company, salary, location, requirements, and tech stack. Track hiring trends, skill demand, and salary benchmarks across industries.

Real Estate

Extract property listings with price, location, features, images, and agent info. Monitor price trends, days on market, and neighborhood data. Enrich with demographic data.

SaaS & Tech

Build competitive intelligence from product directories, review sites, and company pages. Extract pricing, features, tech stacks, and growth signals. This tutorial's example vertical.

Social & Influencers

Monitor social profiles, extract follower counts, engagement rates, and content patterns. Identify influencers, track brand mentions, and analyze audience demographics.

Research & Academic

Collect publications, citations, author profiles, and institutional data. Build research databases, track publication trends, and identify collaboration networks.

Enrichment Sources Deep Dive

Each enrichment source provides a different data layer. Combine multiple sources to build a comprehensive profile for every record in your dataset.

Web Scrape

Re-visit extracted URLs to collect additional data points that the initial spider did not capture. Useful for multi-page entities (e.g., product details spread across tabs).

{ "source_name": "web_scrape", "match_column": "domain", "config": { "selectors": { "about_text": "div.about-section p" } } }

Company Data

Resolves domains to company profiles with employee count, founding year, total funding, revenue range, industry classification, and headquarters location.

{ "source_name": "company_data", "match_column": "domain", "enrich_columns": ["employee_count", "funding_total"] }

Contact Data

Finds contact information associated with a domain or company name: email addresses (pattern-based and verified), phone numbers, key personnel with job titles.

{ "source_name": "contact_data", "match_column": "domain", "enrich_columns": ["contact_email"] }

SEO Metrics

Provides domain authority, estimated monthly organic traffic, total backlinks, referring domains, top-ranking keywords, and content gap analysis.

{ "source_name": "seo_metrics", "match_column": "domain", "enrich_columns": ["monthly_traffic"] }

Social Signals

Discovers and enriches social media profiles: Twitter/X, LinkedIn, Facebook, Instagram, YouTube. Returns follower counts, posting frequency, and engagement rates.

{ "source_name": "social_signals", "match_column": "domain", "enrich_columns": ["social_profiles"] }

Tech Detection

Detects the technology stack of any website: frontend frameworks (React, Vue, Angular), CMS (WordPress, Shopify), analytics tools, CDN providers, hosting infrastructure, and third-party services.

{ "source_name": "tech_detection", "match_column": "domain", "enrich_columns": ["tech_stack"] }

What's Next

Create a content pipeline that uses your extracted data to generate targeted content
Explore the Datasets API, Spiders API, and Pipelines API reference
Check out Enrichment API for the full list of sources and configuration options
Use Crawl API for advanced crawl strategies, frontier management, and proxy configuration