Keyword Grouper: SERP-Driven Search Intent Clustering at Scale
An automated SEO pipeline that clusters 10,000+ keywords by search intent using SERP URL overlap, surfacing content gaps and priority targets that took analysts weeks to find by hand.
The Problem
SEO teams sat on keyword lists thousands deep with no scalable way to answer the questions that matter: which keywords represent the same search intent, where competitors rank but the client does not, and which targets offer the best traffic opportunity for the effort. Manual analysis of a large keyword set took weeks per engagement.
Intent is the hard part. Two keywords with different words can serve identical intent, and semantic similarity alone misjudges it—what actually reveals intent is what Google chooses to rank. But using that signal means fetching and comparing full SERP results for every keyword in the set, which is exactly the volume of work manual research cannot do.
What We Built
We built an automated pipeline that fetches 100 SERP results per keyword via SERPAPI with 100-thread concurrent processing, then groups keywords by URL overlap—seven or more common results marking shared intent—so clusters reflect how Google actually interprets queries rather than how the words look. Priority scoring combines search volume quartiles, value opportunity, and current ranking percentage into a Fibonacci-weighted score that surfaces quick wins, while traffic estimation applies position-based CTR curves to translate rankings into expected visits.
The pipeline is engineered for long unattended runs: snapshot checkpoints every N keywords persist to Google Cloud Storage so failed jobs resume from the last good state, and finished analyses upload to Snowflake in 15K-row batches for BI dashboards and historical trend tracking. Content-gap detection flags groups where competitors rank and the client does not; cannibalization detection flags multiple client pages competing for one intent.
Intent clustering from SERP overlap
Keywords sharing 7+ of their top 100 Google results are grouped as one intent, using the search engine’s own judgment as the clustering signal instead of surface word similarity.
Content gap and cannibalization detection
The pipeline flags intent groups where competitors rank and the client is absent, and groups where multiple client pages compete against each other for the same intent.
Opportunity-ranked prioritization
Fibonacci-weighted scoring across volume, value, and current rankings—paired with CTR-curve traffic estimates—orders the output by expected payoff, not alphabetically.
Resumable high-concurrency processing
100-thread SERP fetching with snapshot checkpoints to cloud storage means a 10,000-keyword run survives failures and resumes instead of restarting.
How It Works in Practice
An SEO strategist starting a new engagement with a 10,000-keyword export and a deadline for the content roadmap.
- 1
Load and fetch
The keyword set enters the pipeline, and concurrent workers pull 100 SERP results per keyword, checkpointing progress as they go.
- 2
Cluster by intent
URL-overlap analysis collapses thousands of keywords into intent groups—each one a single piece of content to write, not dozens of scattered targets.
- 3
Score the opportunities
Each group gets a priority score and a traffic estimate from CTR curves, with content gaps and cannibalization conflicts flagged automatically.
- 4
Deliver the roadmap
Results land in Snowflake for dashboards, and the strategist walks into the kickoff with a prioritized, evidence-backed content plan instead of a spreadsheet of hunches.
Screenshots
Engineering Approach
Let Google define intent
SERP URL overlap is a direct observation of how the ranking system interprets a query—clustering on it produces groups a content strategist can act on without second-guessing.
Built for unattended scale
Checkpointed, resumable jobs with lock-synchronized shared state make multi-hour, 10,000-keyword runs routine rather than fragile.
Outputs that plug into decisions
Priority scores, traffic estimates, and Snowflake delivery mean the pipeline ends where strategy begins—no manual post-processing between analysis and roadmap.
Have a similar problem?
Tell us what you're working on and we'll tell you honestly whether we can help.
