Content Pipeline

From SEO brief to published, indexed content. Generate data-driven content briefs, create AI drafts, optimize for search, manage your editorial calendar, publish, and push to search engines via IndexNow.

Pro tier required. Content workspace features (AI generation, rewriting, and calendar) require a Pro subscription. Content briefs and IndexNow are available on all tiers.

Step 1: Generate a Content Brief

A content brief is a structured SEO plan for a target keyword. CrawlKit analyzes SERP results, competing content, and your existing keyword data to produce a brief with recommended structure, word count, headings, and topics to cover.

curl

curl -X POST https://api.crawlkit.app/api/v1/brief \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "keyword": "web scraping best practices",
    "content_type": "blog_post",
    "target_audience": "developers"
  }'

JavaScript / TypeScript

const response = await fetch("https://api.crawlkit.app/api/v1/brief", {
  method: "POST",
  headers: {
    "Authorization": "Bearer ck_live_YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    site_url: "https://crawlkit.app",
    keyword: "web scraping best practices",
    content_type: "blog_post",
    target_audience: "developers",
  }),
});

const brief = await response.json();
console.log(`Target word count: ${brief.recommended_word_count}`);
console.log(`Sections: ${brief.outline.length}`);

Response

{
  "brief_id": "br_5a6b7c8d9e",
  "keyword": "web scraping best practices",
  "search_intent": "informational",
  "difficulty": 42,
  "monthly_volume": 2400,
  "recommended_word_count": 2200,
  "recommended_title": "Web Scraping Best Practices: A Developer's Guide to Ethical, Reliable Data Collection",
  "meta_description": "Learn web scraping best practices including rate limiting, robots.txt compliance, error handling, and data quality. Production-ready patterns with code examples.",
  "outline": [
    {
      "heading": "What Is Web Scraping?",
      "type": "h2",
      "notes": "Brief definition. Distinguish from APIs and data feeds. 150 words.",
      "topics": ["definition", "use cases", "legal considerations"]
    },
    {
      "heading": "Respect robots.txt and Rate Limits",
      "type": "h2",
      "notes": "Cover robots.txt parsing, crawl delay, and adaptive rate limiting. Code example.",
      "topics": ["robots.txt", "rate limiting", "crawl delay", "politeness"]
    },
    {
      "heading": "Handle Errors and Retries Gracefully",
      "type": "h2",
      "notes": "Exponential backoff, retry budgets, circuit breakers. Code example.",
      "topics": ["error handling", "retries", "exponential backoff"]
    },
    {
      "heading": "Use Headless Browsers Only When Needed",
      "type": "h2",
      "notes": "When JS rendering is required vs. static HTML. Performance implications.",
      "topics": ["headless browsers", "JS rendering", "Playwright", "performance"]
    },
    {
      "heading": "Structure and Validate Extracted Data",
      "type": "h2",
      "notes": "Schema definition, data types, validation. Link to CrawlKit datasets.",
      "topics": ["data quality", "schema", "validation", "datasets"]
    },
    {
      "heading": "Monitor and Maintain Your Scrapers",
      "type": "h2",
      "notes": "Breakage detection, alerting, selector health. Link to spider health API.",
      "topics": ["monitoring", "maintenance", "selector health"]
    }
  ],
  "serp_analysis": {
    "top_results_analyzed": 10,
    "avg_word_count": 1850,
    "common_topics": ["robots.txt", "rate limiting", "data quality", "legal", "proxies"],
    "content_gaps": ["No top result covers pipeline automation", "None mention enrichment workflows"]
  },
  "internal_links_suggested": [
    { "anchor": "CrawlKit spider system", "url": "/features/spiders" },
    { "anchor": "data pipeline tutorial", "url": "/tutorials/data-pipeline" }
  ]
}

Step 2: Create a Document

Create a document in the content workspace. This serves as your working draft, linked to the content brief for context.

curl

curl -X POST https://api.crawlkit.app/api/v1/content-workspace/documents \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "title": "Web Scraping Best Practices: A Developer Guide",
    "brief_id": "br_5a6b7c8d9e",
    "slug": "web-scraping-best-practices",
    "status": "draft",
    "target_keyword": "web scraping best practices",
    "content_type": "blog_post",
    "scheduled_date": "2026-03-20T09:00:00Z"
  }'

Response

{
  "document_id": "doc_1x2y3z4a5b",
  "title": "Web Scraping Best Practices: A Developer Guide",
  "status": "draft",
  "brief_id": "br_5a6b7c8d9e",
  "slug": "web-scraping-best-practices",
  "target_keyword": "web scraping best practices",
  "word_count": 0,
  "seo_score": null,
  "scheduled_date": "2026-03-20T09:00:00Z",
  "created_at": "2026-03-13T11:00:00Z"
}

Step 3: AI-Generate Content

Generate a full draft from the content brief. CrawlKit's AI writer uses the brief's outline, SERP analysis, and your site's existing content style to produce a first draft.

curl

curl -X POST https://api.crawlkit.app/api/v1/content-workspace/documents/doc_1x2y3z4a5b/ai/generate \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "use_brief": true,
    "tone": "technical",
    "include_code_examples": true,
    "target_word_count": 2200
  }'

Response

{
  "document_id": "doc_1x2y3z4a5b",
  "status": "draft",
  "word_count": 2187,
  "content_preview": "# Web Scraping Best Practices: A Developer's Guide\n\nWeb scraping is the automated extraction of data from websites...",
  "sections_generated": 6,
  "code_examples_included": 4,
  "generation_time_secs": 8.3,
  "seo_score": 72,
  "seo_suggestions": [
    "Add the target keyword to the first paragraph",
    "Include 2-3 more internal links",
    "Add a FAQ section with structured data markup"
  ]
}

Step 4: AI-Rewrite and Optimize

Refine your content with AI-powered rewriting. Target specific improvements: SEO optimization, readability, tone adjustments, or keyword density. You can also provide custom instructions.

curl

curl -X POST https://api.crawlkit.app/api/v1/content-workspace/documents/doc_1x2y3z4a5b/ai/rewrite \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "seo_optimize",
    "instructions": "Increase keyword density for primary keyword. Add FAQ section. Improve the introduction to be more compelling.",
    "preserve_code_examples": true,
    "target_seo_score": 90
  }'

Response

{
  "document_id": "doc_1x2y3z4a5b",
  "status": "draft",
  "word_count": 2340,
  "previous_seo_score": 72,
  "new_seo_score": 88,
  "changes_summary": {
    "sections_modified": 3,
    "sections_added": 1,
    "keyword_density": {
      "primary": { "before": 0.8, "after": 1.4, "target": 1.5 },
      "secondary": { "before": 0.3, "after": 0.7 }
    },
    "readability": {
      "flesch_score": 52,
      "grade_level": "college"
    }
  },
  "improvements": [
    "Added FAQ section with 5 questions and schema.org markup",
    "Rewrote introduction with a hook and the target keyword",
    "Added 3 internal links to related documentation pages",
    "Improved meta description to include target keyword"
  ]
}

Rewrite modes: seo_optimize (improve search ranking signals), readability (simplify language), expand (add depth and detail), condense (shorten while preserving key points), tone_shift (adjust voice — technical, casual, formal).

Step 5: Review Editorial Calendar

View your content pipeline across all documents. The calendar shows scheduled, draft, in-review, and published content with SEO scores and target keywords.

curl

curl "https://api.crawlkit.app/api/v1/content-workspace/calendar?site_url=https://crawlkit.app&month=2026-03" \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

Response

{
  "month": "2026-03",
  "documents": [
    {
      "document_id": "doc_1x2y3z4a5b",
      "title": "Web Scraping Best Practices: A Developer Guide",
      "status": "draft",
      "scheduled_date": "2026-03-20T09:00:00Z",
      "target_keyword": "web scraping best practices",
      "seo_score": 88,
      "word_count": 2340,
      "content_type": "blog_post"
    },
    {
      "document_id": "doc_7c8d9e0f1g",
      "title": "How to Build a Data Enrichment Pipeline",
      "status": "in_review",
      "scheduled_date": "2026-03-17T09:00:00Z",
      "target_keyword": "data enrichment pipeline",
      "seo_score": 91,
      "word_count": 3100,
      "content_type": "blog_post"
    },
    {
      "document_id": "doc_2h3i4j5k6l",
      "title": "CrawlKit vs Scrapy: Which Web Scraping Tool Is Right for You?",
      "status": "published",
      "published_date": "2026-03-10T09:00:00Z",
      "target_keyword": "crawlkit vs scrapy",
      "seo_score": 85,
      "word_count": 1800,
      "content_type": "comparison"
    }
  ],
  "summary": {
    "total": 8,
    "published": 3,
    "in_review": 2,
    "draft": 3,
    "avg_seo_score": 84
  }
}

Step 6: Publish

Move a document to published status. If you have a webhook configured, CrawlKit can push the content to your CMS automatically.

curl

curl -X POST https://api.crawlkit.app/api/v1/content-workspace/documents/doc_1x2y3z4a5b/publish \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "publish_url": "https://docs.crawlkit.app/tutorials/data-pipeline.html",
    "notify_search_engines": true
  }'

Response

{
  "document_id": "doc_1x2y3z4a5b",
  "status": "published",
  "published_date": "2026-03-13T14:00:00Z",
  "publish_url": "https://docs.crawlkit.app/tutorials/data-pipeline.html",
  "search_engine_notification": {
    "indexnow_submitted": true,
    "engines_notified": ["bing", "yandex", "naver", "seznam"]
  }
}

Step 7: Submit to IndexNow for Instant Indexing

IndexNow notifies search engines (Bing, Yandex, Naver, Seznam) about new or updated URLs so they can be crawled and indexed faster. Unlike Google's Indexing API (which requires OAuth), IndexNow uses simple key-file authentication.

7a. Generate an IndexNow key

curl -X POST https://api.crawlkit.app/api/v1/indexnow/key/generate \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app"
  }'

{
  "key": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6",
  "key_url": "https://crawlkit.app/a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6.txt",
  "instructions": "Host the key file at the key_url path. The file should contain only the key value."
}

Important: You must host the key file at the root of your website. Create a text file at /a1b2c3d4e5...txt containing only the key value. Search engines will verify this file before accepting submissions.

7b. Verify key accessibility

curl "https://api.crawlkit.app/api/v1/indexnow/key/verify?site_url=https://crawlkit.app&key=a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6" \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

{
  "key": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6",
  "verified": true,
  "key_url": "https://crawlkit.app/a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6.txt",
  "status": 200
}

7c. Submit URLs

curl -X POST https://api.crawlkit.app/api/v1/indexnow/submit \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "key": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6",
    "urls": [
      "https://docs.crawlkit.app/tutorials/data-pipeline.html",
      "https://docs.crawlkit.app/guides/data-engineering.html"
    ]
  }'

Response

{
  "submitted": 2,
  "results": [
    {
      "url": "https://docs.crawlkit.app/tutorials/data-pipeline.html",
      "status": "submitted",
      "engines": ["bing", "yandex", "naver", "seznam"]
    },
    {
      "url": "https://docs.crawlkit.app/guides/data-engineering.html",
      "status": "submitted",
      "engines": ["bing", "yandex", "naver", "seznam"]
    }
  ]
}

Full Workflow Summary

Here is the complete content pipeline flow:

Step	Endpoint	Purpose
1	POST `/api/v1/brief`	Generate SEO content brief from keyword + SERP analysis
2	POST `/api/v1/content-workspace/documents`	Create document draft in the workspace
3	POST `/api/v1/.../ai/generate`	AI-generate full content from brief
4	POST `/api/v1/.../ai/rewrite`	Optimize for SEO, readability, or tone
5	GET `/api/v1/content-workspace/calendar`	Review editorial calendar and pipeline
6	POST `/api/v1/.../publish`	Publish content and notify search engines
7	POST `/api/v1/indexnow/submit`	Submit new URLs for instant indexing

What's Next

Explore the Content Workspace API reference for all document operations
Use keyword tracking to find content opportunities
Set up site audits to verify your content is being indexed correctly
Build data pipelines to fuel your content with original data and research
Check the full API reference for 383 operations across all capability domains