Keyword Tracking

Connect Google Search Console, import keywords, track rankings, and get AI-powered recommendations to close gaps in your keyword strategy.

Prerequisites: A CrawlKit API key (Pro tier for AI recommendations) and a Google Search Console property for your site.

Step 1: Connect Google Search Console

To import keyword data, you first need to connect your GSC property. This uses the standard OAuth 2.0 flow — CrawlKit stores the refresh token and handles token lifecycle automatically.

curl

curl -X POST https://api.crawlkit.app/api/v1/gsc/auth/url \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "redirect_uri": "https://your-app.com/callback",
    "scopes": ["webmasters.readonly"]
  }'

Response

{
  "authorization_url": "https://accounts.google.com/o/oauth2/v2/auth?client_id=...&redirect_uri=https%3A%2F%2Fyour-app.com%2Fcallback&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fwebmasters.readonly&response_type=code&access_type=offline"
}

Direct the user to the authorization_url. After they authorize, Google redirects to your callback URL with a code parameter. Exchange it:

curl -X POST https://api.crawlkit.app/api/v1/gsc/auth/callback \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "code": "4/0AeanS0a_OAUTH_CODE",
    "redirect_uri": "https://your-app.com/callback"
  }'

{
  "success": true,
  "properties_found": 3,
  "properties": [
    "sc-domain:crawlkit.app",
    "https://crawlkit.app/",
    "https://crawlkit.app/"
  ]
}

Step 2: Import Keywords from GSC

Pull your existing keyword performance data directly from Google Search Console. CrawlKit imports queries with clicks, impressions, CTR, and average position from the specified time range.

curl

curl -X POST https://api.crawlkit.app/api/v1/keywords/import/gsc \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "days": 28
  }'

JavaScript / TypeScript

const response = await fetch("https://api.crawlkit.app/api/v1/keywords/import/gsc", {
  method: "POST",
  headers: {
    "Authorization": "Bearer ck_live_YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    site_url: "https://crawlkit.app",
    days: 28,
  }),
});

const result = await response.json();
console.log(`Imported ${result.imported_count} keywords`);

Response

{
  "imported_count": 156,
  "date_range": {
    "start_date": "2026-02-13",
    "end_date": "2026-03-13"
  },
  "top_queries": [
    {
      "query": "web scraping api",
      "clicks": 342,
      "impressions": 8921,
      "ctr": 0.038,
      "position": 4.2
    },
    {
      "query": "data enrichment platform",
      "clicks": 187,
      "impressions": 5430,
      "ctr": 0.034,
      "position": 6.8
    },
    {
      "query": "seo audit tool api",
      "clicks": 124,
      "impressions": 3210,
      "ctr": 0.039,
      "position": 3.1
    }
  ]
}

Step 3: Import Keywords from CSV

Already tracking keywords in a spreadsheet? Import them via CSV. CrawlKit parses the CSV and merges with any existing keyword data for your site.

curl

curl -X POST https://api.crawlkit.app/api/v1/keywords/import/csv \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "csv_content": "query,volume,difficulty,intent\nweb scraping api,2400,45,transactional\ndata pipeline tool,1800,52,informational\ncrawl website python,3200,38,informational\nenrichment api,890,31,transactional\nlead generation api,1500,48,transactional"
  }'

Response

{
  "imported_count": 5,
  "merged_count": 1,
  "skipped_count": 0,
  "keywords": [
    {
      "query": "web scraping api",
      "volume": 2400,
      "difficulty": 45,
      "intent": "transactional",
      "source": "csv",
      "merged_with_gsc": true
    },
    {
      "query": "data pipeline tool",
      "volume": 1800,
      "difficulty": 52,
      "intent": "informational",
      "source": "csv",
      "merged_with_gsc": false
    }
  ]
}

Step 4: View Keyword Rankings

Retrieve your full keyword list with current rankings, search volume, and trend data. Filter by intent, difficulty range, or position.

curl

curl "https://api.crawlkit.app/api/v1/keywords?site_url=https://crawlkit.app&sort=position&limit=20" \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

JavaScript / TypeScript

const params = new URLSearchParams({
  site_url: "https://crawlkit.app",
  sort: "position",
  limit: "20",
});

const response = await fetch(
  `https://api.crawlkit.app/api/v1/keywords?${params}`,
  {
    headers: { "Authorization": "Bearer ck_live_YOUR_API_KEY" },
  }
);

const data = await response.json();
for (const kw of data.keywords) {
  console.log(`${kw.query}: position ${kw.position} (${kw.clicks} clicks)`);
}

Response

{
  "total": 161,
  "keywords": [
    {
      "id": "kw_a1b2c3d4",
      "query": "seo audit tool api",
      "position": 3.1,
      "previous_position": 4.8,
      "change": 1.7,
      "clicks": 124,
      "impressions": 3210,
      "ctr": 0.039,
      "volume": 1200,
      "difficulty": 42,
      "intent": "transactional",
      "url": "https://docs.crawlkit.app/guides/getting-started.html",
      "trend": "improving"
    },
    {
      "id": "kw_e5f6g7h8",
      "query": "web scraping api",
      "position": 4.2,
      "previous_position": 5.1,
      "change": 0.9,
      "clicks": 342,
      "impressions": 8921,
      "ctr": 0.038,
      "volume": 2400,
      "difficulty": 45,
      "intent": "transactional",
      "url": "https://docs.crawlkit.app/guides/spiders.html",
      "trend": "improving"
    }
  ],
  "summary": {
    "avg_position": 12.4,
    "top_3": 8,
    "top_10": 34,
    "top_100": 142,
    "improving": 89,
    "declining": 52,
    "stable": 20
  }
}

Step 5: Get AI Recommendations

CrawlKit analyzes your keyword portfolio, identifies gaps, and recommends high-impact keywords to target. This requires a Pro tier subscription.

curl

curl -X POST https://api.crawlkit.app/api/v1/keywords/recommend \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "max_recommendations": 10,
    "focus": "quick_wins"
  }'

Response

{
  "recommendations": [
    {
      "query": "web scraping service",
      "volume": 1900,
      "difficulty": 35,
      "intent": "transactional",
      "opportunity_score": 0.89,
      "reason": "High volume, low difficulty. You rank #14 for 'web scraping api' — a dedicated page targeting this variant could reach top 5.",
      "suggested_url": "/features/web-scraping-service",
      "content_type": "landing_page"
    },
    {
      "query": "how to enrich company data",
      "volume": 2100,
      "difficulty": 28,
      "intent": "informational",
      "opportunity_score": 0.84,
      "reason": "You have no content targeting enrichment tutorials. This high-volume informational query feeds into your transactional funnel.",
      "suggested_url": "/blog/how-to-enrich-company-data",
      "content_type": "blog_post"
    },
    {
      "query": "data pipeline api",
      "volume": 1400,
      "difficulty": 41,
      "intent": "transactional",
      "opportunity_score": 0.78,
      "reason": "Your pipeline documentation ranks #22. Optimizing existing content and adding structured data could move to page 1.",
      "suggested_url": "/features/pipeline",
      "content_type": "feature_page"
    }
  ],
  "strategy_summary": {
    "quick_wins": 4,
    "content_gaps": 3,
    "optimization_targets": 3,
    "estimated_monthly_traffic_gain": 2400
  }
}

Step 6: Track Competitive Landscape

Monitor how your competitors rank for the same keywords. The landscape endpoint provides a competitive snapshot with overlap analysis.

curl

curl "https://api.crawlkit.app/api/v1/landscape?site_url=https://crawlkit.app" \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY"

Response

{
  "site": "https://crawlkit.app",
  "competitors": [
    {
      "domain": "competitor-a.com",
      "overlap_keywords": 48,
      "overlap_percentage": 0.30,
      "avg_position_gap": -2.3,
      "winning_keywords": 18,
      "losing_keywords": 30,
      "top_winning": [
        { "query": "seo audit api", "your_position": 3, "their_position": 8 }
      ],
      "top_losing": [
        { "query": "web crawler tool", "your_position": 15, "their_position": 2 }
      ]
    },
    {
      "domain": "competitor-b.com",
      "overlap_keywords": 32,
      "overlap_percentage": 0.20,
      "avg_position_gap": 1.1,
      "winning_keywords": 14,
      "losing_keywords": 18
    }
  ],
  "keyword_gaps": [
    {
      "query": "headless scraping api",
      "volume": 1600,
      "competitor_positions": {
        "competitor-a.com": 4,
        "competitor-b.com": 7
      },
      "your_position": null,
      "gap_type": "missing"
    }
  ],
  "visibility_score": {
    "yours": 0.34,
    "competitor_avg": 0.41,
    "trend": "improving"
  }
}

Add or update competitors

curl -X PUT https://api.crawlkit.app/api/v1/landscape/competitors \
  -H "Authorization: Bearer ck_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site_url": "https://crawlkit.app",
    "competitors": [
      { "domain": "competitor-a.com" },
      { "domain": "competitor-b.com" },
      { "domain": "competitor-c.com" }
    ]
  }'

{
  "competitors_count": 3,
  "message": "Competitors updated. Initial crawl and analysis will begin within 5 minutes."
}

What's Next

Build a data pipeline to scrape competitor sites for deeper intelligence
Create a content pipeline to target keyword gaps with optimized content
Explore the Keywords API reference for batch updates and advanced filters