Changelog

Release history for the CrawlKit API. All dates are in UTC. Breaking changes are flagged explicitly.

v1.0 API Reference & Developer Docs March 13, 2026

Initial public release of the CrawlKit API and developer documentation site.

383 API operations across 54 capability domains — Arrow/DataFusion query, ETL, workflow templates, spiders, crawl, datasets, lineage, quality, OpenRate, SEO/GSC, content, social monitoring, DNS, and more
OpenAPI 3.1 spec with Scalar-powered interactive API reference at /reference/
Agent tool surface covering the full 383-operation platform for Pi, Claude Code, Gemini CLI, ChatGPT, Codex, and MCP-compatible clients
Developer guides — authentication, rate limits, error handling, integrations, getting started
End-to-end tutorials — site audit, keyword tracking, data pipeline, content pipeline
SDK examples in TypeScript, JavaScript, Python, Rust, and curl for every endpoint
Interactive demo gallery with SSE-streamed terminal output (Phase 2b)

Developer documentation site launched at docs.crawlkit.app.

docs.crawlkit.app goes live with guides, tutorials, SDK examples, and interactive demos
Dark-mode-first design with CrawlKit brand system (Inter + JetBrains Mono, accent #F05023)
Layout system with sticky header, sidebar navigation, breadcrumbs, and prev/next pager
SEO optimized — sitemap.xml, robots.txt, structured data, canonical URLs, Open Graph tags
Responsive across desktop, tablet, and mobile viewports

Full data enrichment platform with datasets, spiders, enrichment sources, and DAG-based pipeline engine.

Dataset API — create typed datasets with custom columns, add/list/export rows, statistics
Spider system — visual spider builder with CSS/XPath selectors, 8 vertical templates (e-commerce, news, jobs, real estate, SaaS, directories, forums, recipes), health monitoring
17+ enrichment sources — company data, tech stacks, SEO metrics, social profiles, traffic estimates, contact info
Pipeline engine — DAG-based execution with seed, crawl, extract, enrich, filter, export stages. Create, run, pause, cancel, and monitor pipeline runs
Web crawling — intelligent crawl strategies, frontier management, budget controls, proxy rotation. Start, pause, resume, cancel crawl jobs
Crawl strategies — configurable base strategies with max depth, scope rules, and URL filtering

Reddit and LinkedIn brand mention tracking with sentiment analysis and signal scoring.

Reddit monitoring — create monitors with keyword/subreddit filters, poll for new mentions, sentiment analysis, signal scoring
LinkedIn monitoring — track company mentions, industry discussions, and competitor activity with engagement metrics
Mention management — list, filter, and inspect individual mentions with full context and metadata
CRUD for monitors — create, list, get, update, delete monitors for both platforms

Lead scoring, buying signal detection, campaign management, and pipeline deal tracking.

Lead scoring — score leads based on firmographic data, tech stack signals, engagement patterns, and custom rules
Campaign management — create outreach campaigns, assign leads, track conversion funnels
Pipeline deals — manage deal stages, forecast revenue, track win/loss rates
Spider-to-lead workflow — spider a directory, enrich leads, score, and nurture in one pipeline

AI-powered content generation, SEO optimization, editorial calendar, and IndexNow integration.

Content briefs — generate keyword-targeted content briefs with competitor analysis, SERP intent, and recommended structure
Structured data — generate JSON-LD schema templates (Article, Product, FAQ, HowTo, etc.) for any page
Internal link suggestions — graph-based link recommendations with orphan rescue, depth reduction, and topical clustering
SERP analysis — analyze top-ranking competitors for any keyword with content gap detection
IndexNow — submit URLs to Bing, Yandex, Naver, and Seznam for instant indexing after content changes