Keyword Research with AI Agents and Google Search Console

Manual keyword research is slow and disconnected from real user behavior. I use AI agents connected to Google Search Console data to automate keyword clustering, intent analysis, and content gap detection for client websites.

This article shows the complete workflow: from raw GSC data to actionable keyword clusters that drive content decisions.

The problem with manual keyword research

Traditional keyword research relies on guesswork. You look at competitor sites, use keyword tools with volume estimates, and try to predict what users might search for. But if you already have a website, Google is telling you exactly what people search for — and you're not listening.

Google Search Console provides actual search query data, grouped by page, with impressions, clicks, and average position. This is the real signal. The problem is extracting actionable insights from thousands of rows of query data.

That's where AI agents come in.

The architecture

The system consists of three components:

1. Data source: Google Search Console Search Analytics API
2. AI agent: A coding agent with access to the GSC data and the ability to execute scripts
3. Cluster output: A JSON mapping of queries to pages, intents, and optimization opportunities

The agent runs a Node.js script that authenticates with Google Cloud, fetches 12 months of search query data, and processes it into keyword clusters organized by page and intent.

Step 1: Setting up GSC access

Before anything else, you need API access to Search Console data. This requires:

1. A Google Cloud project with the Search Console API enabled
2. Application Default Credentials (ADC) configured with gcloud auth application-default login
3. The Search Console property added to your account

# Configure ADC credentials
gcloud auth application-default login

# Set the quota project (required since 2026)
export GOOGLE_CLOUD_QUOTA_PROJECT=your-project-id

Step 2: The keyword cluster script

The core of this workflow is a Node.js script that fetches GSC data and processes it. Here's how it works:

// Authenticate with GSC API
async function mintAdcToken() {
  const result = await execFileAsync('gcloud', [
    'auth', 'application-default', 'print-access-token'
  ]);
  return result.stdout.trim();
}

// Fetch query data for a specific period
async function fetchGscRows({ startDate, endDate }) {
  const accessToken = await mintAdcToken();
  const response = await fetch(
    'https://searchconsole.googleapis.com/webmasters/v3/sites/' +
    encodeURIComponent('sc-domain:your-site.com') +
    '/searchAnalytics/query',
    {
      method: 'POST',
      headers: {
        authorization: `Bearer ${accessToken}`,
        'content-type': 'application/json',
        'x-goog-user-project': process.env.GOOGLE_CLOUD_QUOTA_PROJECT
      },
      body: JSON.stringify({
        startDate,
        endDate,
        dimensions: ['query', 'page'],
        dataState: 'final',
        rowLimit: 25000
      })
    }
  );
  return (await response.json()).rows;
}

The script fetches up to 25,000 rows of query data spanning the last 12 months. It uses pagination to handle larger datasets.

Step 3: Processing the data into clusters

Once the data is fetched, the processing happens in several stages:

Map routes to pages: Each query result is mapped to a canonical route, stripping fragments and query parameters.

Aggregate by page: For each page, all queries are collected along with their impressions, clicks, and positions.

Identify intent clusters: Queries are grouped by intent. "What is X" queries are informative. "Buy X" or "hire X" queries are commercial. The intent is inferred from both the query phrasing and the page it leads to.

Detect cannibalization: When multiple pages rank for similar queries, the script flags potential cannibalization where pages compete for the same search terms.

Here's a simplified version of the clustering logic:

function clusterQueries(page, queries) {
  return {
    route: page,
    pageMetrics: calculatePageMetrics(queries),
    primaryTerm: queries[0].query, // Highest click volume
    cluster: queries.map((q, i) => ({
      query: q.query,
      clicks: q.clicks,
      impressions: q.impressions,
      ctr: q.clicks / q.impressions,
      role: i === 0 ? 'primary' : (i <= 9 ? 'secondary' : 'long-tail'),
      sharedWith: detectCannibalization(q.query, allPages)
    }))
  };
}

Step 4: Using the clusters for content decisions

The output is a JSON file that maps each page to its keyword clusters. This becomes the foundation for all content and SEO decisions.

Optimization targets: Pages with high impressions but low clicks are prime optimization targets. The title, meta description, or content may need improvement.

Internal linking strategy: Use the cluster data to identify which pages should link to which. If a page ranks for "website development services" and another ranks for "hire a web developer," they should link to each other.

Content gap identification: With query and page as dimensions, every returned row already pairs a query with a page. A gap here means the site may lack content that fits the search intent, even though an existing page earns impressions for that query. Review the current page and search results before deciding whether to improve it or create a separate page. This export cannot reveal demand for queries where the site has no recorded visibility.

Commercial vs. informational intent: The intent labels help decide whether a page should be a blog post (informative) or a landing page (commercial).

Example: real data from our site

Here's a real example from our Search Console data (August 2026):

(These queries are in Dutch — they come from a Dutch-language client site we manage. The workflow itself works the same in any language.)

What should we investigate?

• The price blog earns impressions for "website laten maken." It may answer cost questions well, while a service page may better serve someone ready to hire a developer. The query alone does not establish which intent dominates.
• The row for "goedkope website laten maken" shows one blog. To investigate cannibalization, inspect the other query-page pairs; this table alone does not demonstrate competing pages.
• "astro website laten maken" appears with a relevant service page and an average position of 7.0. That is useful visibility data, not evidence of inquiries or sales.
• Before proposing a new service page for "website laten maken," check whether a suitable one already exists and review the search results. Improving an existing page and its internal links may be the better option.

The script surfaces candidates for review. It does not establish that a new page is necessary or that it will bring in more business.

Automating the workflow with AI agents

The real power comes from automating this workflow. With an AI coding agent, you can:

1. Schedule regular runs: Run the script weekly or monthly to track changes
2. Generate reports: Have the agent analyze the data and generate optimization recommendations
3. Implement changes: Have the agent actually update titles, meta descriptions, and internal links based on the cluster data
4. Track results: Compare search performance and qualified inquiries before and after changes, using conversion tracking or CRM data alongside GSC

The agent doesn't replace your SEO judgment — it amplifies it. You still decide the strategy, but the agent handles the data analysis and implementation.

Measure inquiries as well as search visibility

For a service business, a new page should help generate relevant inquiries. GSC reports search performance; it does not tell you whether a visitor submitted a useful project inquiry or became a customer. Use analytics conversion tracking for successful form submissions and, where available, CRM records to assess lead quality.

Before making changes, record organic traffic and qualified inquiries for the relevant existing pages. After launch, compare the combined results for those pages and the new service page, not just the new page in isolation. Otherwise, moving inquiries from the blog to the service page can look like growth when the total has not changed.

Use comparable, non-overlapping periods of at least 30 days and allow longer when inquiry counts are low. Consider seasonality and other site or marketing changes; a before-and-after comparison alone does not prove causation. A higher ranking without more relevant inquiries is not enough to call the change a business success. These are proposed measurement criteria, not conversion results demonstrated by the table above.

Practical tips

• Always compare fixed periods: When measuring changes, use complete, non-overlapping periods. Don't compare this week with last week if they have different numbers of days.
• Track baseline metrics: Before making any SEO changes, record the current impressions, clicks, and positions for the affected pages and queries.
• Give changes time: Reassess after at least 30 days, and wait longer when the data is sparse. Avoid repeated changes before you have enough evidence to judge the previous one.
• Use the API, not the UI: The Search Analytics API gives you more data and more control than the web interface. It's worth the extra setup.

Conclusion

Keyword research doesn't have to be manual guesswork. By connecting AI agents to your real Google Search Console data, you can build a data-driven SEO workflow that continuously optimizes based on actual user behavior.

The key insight: you already have the data you need. You just need the right tools to process it.

This workflow is based on our experience running keyword research for client websites. The scripts described here are used in production on straffesites.com. New to keyword research? Start with How to do keyword research in 6 clear steps on the Straffe Sites blog.