Getting Started

Go from zero to the first useful CrawlKit loop: authenticate, inspect a URL, create a typed dataset, deploy a spider, and run enrichment. From there, continue into Arrow/DataFusion query, durable workflows, scraping, quality, and lineage.

Platform mental model: CrawlKit turns sources into governed data products: Source → Acquisition Job → Dataset/Asset → Quality → Query/Serve → Lineage/Audit.

1. Sign Up and Get Your API Key

  1. Go to crawlkit.app and create an account.
  2. Navigate to Settings → API Keys in the dashboard.
  3. Click Create API Key and give it a descriptive name (e.g., "Development").
  4. Copy the key — it starts with ck_live_ and is only shown once.
Free tier included. Every account gets 10 SEO checks per day and 1 full site audit per week at no cost. Upgrade to Pro for unlimited access.

2. Check a URL for SEO Issues

The /check-url endpoint crawls a single URL and returns a full SEO analysis with issues, severity scores, and framework-specific remediation steps.

# Check a URL for SEO issues
curl -X POST https://api.crawlkit.app/api/v1/check-url \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://crawlkit.app",
    "js_rendering": false
  }'

Response

{
  "url": "https://crawlkit.app",
  "score": 72,
  "issues": [
    {
      "category": "missing_meta_description",
      "severity": "high",
      "message": "Page is missing a meta description",
      "affected_urls": ["https://crawlkit.app"],
      "remediation": {
        "description": "Add a meta description between 120-160 characters",
        "effort": "trivial",
        "file_changes": [
          {
            "path": "app/layout.tsx",
            "find": "export const metadata = {",
            "replace": "export const metadata = {\n  description: 'Your page description here',"
          }
        ]
      }
    },
    {
      "category": "missing_open_graph",
      "severity": "medium",
      "message": "Open Graph tags are missing",
      "affected_urls": ["https://crawlkit.app"]
    }
  ],
  "metadata": {
    "title": "Example Domain",
    "word_count": 46,
    "internal_links": 1,
    "external_links": 1,
    "load_time_ms": 312
  }
}

3. Create a Dataset

Datasets are structured tables where you store crawled and enriched data. Define columns with types like text, integer, float, boolean, jsonb, or timestamp.

# Create a dataset for competitor data
curl -X POST https://api.crawlkit.app/api/v1/datasets \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "competitor_analysis",
    "description": "Competitor domains with traffic and tech stack data",
    "columns": [
      {"name": "domain", "column_type": "text", "required": true},
      {"name": "monthly_traffic", "column_type": "integer"},
      {"name": "domain_rating", "column_type": "float"},
      {"name": "tech_stack", "column_type": "json"},
      {"name": "last_crawled", "column_type": "timestamp"}
    ]
  }'

Response

{
  "id": "ds_a1b2c3d4e5f6",
  "name": "competitor_analysis",
  "description": "Competitor domains with traffic and tech stack data",
  "columns": [
    {"name": "domain", "column_type": "text", "required": true},
    {"name": "monthly_traffic", "column_type": "integer"},
    {"name": "domain_rating", "column_type": "float"},
    {"name": "tech_stack", "column_type": "json"},
    {"name": "last_crawled", "column_type": "timestamp"}
  ],
  "row_count": 0,
  "created_at": "2026-03-13T10:00:00Z"
}

4. Deploy a Spider

Spiders crawl websites and extract structured data using CSS or XPath selectors. Define what to extract and where to crawl.

# Create a spider to extract product data
curl -X POST https://api.crawlkit.app/api/v1/spiders \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "product_scraper",
    "kind": "product_catalog",
    "scope": {
      "start_urls": ["https://books.toscrape.com/catalogue/category/books/travel_2/index.html"],
      "domains": ["books.toscrape.com"],
      "allowed_patterns": ["/products/*"],
      "max_depth": 3
    },
    "definition": {
      "handlers": [
        {
          "url_pattern": "/products/*",
          "fields": [
            {"name": "title", "selector": "h1.product-title", "type": "text"},
            {"name": "price", "selector": ".price-current", "type": "text"},
            {"name": "description", "selector": ".product-desc", "type": "text"},
            {"name": "image_url", "selector": "img.product-image", "attribute": "src"}
          ]
        }
      ],
      "navigation": {
        "follow_links": ".pagination a",
        "max_depth": 3
      }
    }
  }'

Response

{
  "id": "sp_x7y8z9w0",
  "name": "product_scraper",
  "kind": "product_catalog",
  "status": "active",
  "scope": {
    "start_urls": ["https://books.toscrape.com/catalogue/category/books/travel_2/index.html"],
    "domains": ["books.toscrape.com"]
  },
  "created_at": "2026-03-13T10:05:00Z"
}

5. Run Enrichment

Enrich your dataset rows with data from 17+ sources — company data, tech stacks, SEO metrics, social profiles, and more.

# Enrich dataset rows with tech stack detection
curl -X POST https://api.crawlkit.app/api/v1/enrichment/enrich \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "books.toscrape.com",
    "context": {
      "dataset_id": "ds_a1b2c3d4e5f6",
      "source": "tech_stack_detector"
    }
  }'

Response

{
  "domain": "books.toscrape.com",
  "status": "completed",
  "enriched": true,
  "signals": {
    "technologies": ["Next.js", "Stripe"],
    "category": "commerce"
  },
  "created_at": "2026-03-13T10:10:00Z"
}

Next Steps

PreviousGuides NextAuthentication