How to Scrape Google Search Results in 2026 (After /goto and the September Crackdown)

Last updated September 27, 2026. Figures cite their primary sources inline. The Search Console data is our own and is explained below.

The short version: you can still scrape Google search results in 2026, but only with a real browser, residential or mobile IPs in the market you're tracking, one sticky IP per query, slow pacing, and selective resolution of google.com/goto links. The alternative is buying the data from a SERP API. Plain HTTP scrapers stopped working when Google began requiring JavaScript in January 2025. Since around September 13, 2026, Google has also blocked rank trackers far more effectively. Read the legal section before you start: Google's terms prohibit automated queries.

Key facts (as of September 27, 2026)

• Google has required JavaScript to show Search results since January 2025, a change it said helps protect Search "from bots and evolving forms of abuse and spam" (TechCrunch).
• Google disabled the num=100 parameter on September 12–14, 2025, so tracking a keyword to position 100 takes 10 requests instead of 1 (Search Engine Land).
• Google confirmed on August 26, 2026 that result links now route through google.com/goto redirects. Resolving every link in one five-page ranking check takes 500–1,000 requests, according to rank-tracking firm Nozzle (Search Engine Roundtable, PPC Land).
• From about September 13, 2026, Nozzle lost roughly 80% of the Google data it could collect (Search Engine Roundtable).
• On July 20, 2026, a US federal court dismissed Google's DMCA claims against SerpApi for search results that contain no copyrighted content (N.D. Cal., case 4:25-cv-10826).
• Google's Custom Search JSON API is closed to new customers and shuts down on January 1, 2027 (Google for Developers).
• On novaproxy.io, Search Console impressions fell 64% and clicks rose 36% between late August and late September 2026, as automated rank checks disappeared (our data, below).

What is SERP scraping?

SERP scraping is the automated collection of search engine results pages: result titles, URLs and positions, plus features such as AI Overviews, local packs and People Also Ask. Rank trackers are the biggest users. SEO teams, price-intelligence companies, ad-verification vendors and AI data pipelines use the same data. The volume adds up fast: a tool that checks 10,000 keywords a day down to position 50 loads 50,000 Google results pages a day, one for every ten results. Google calls this "machine-generated traffic" and says in its spam policies that it includes "scraping results for rank-checking purposes". Every change below targets that traffic.

What changed for Google scraping in 2025–2026?

Four changes in 20 months broke most Google scrapers. In January 2025, Google stopped serving results without JavaScript. In September 2025, it removed the parameter that returned 100 results per request. In August 2026, it hid destination URLs behind google.com/goto redirects. And from around September 13, 2026, it blocked automated searches far more effectively. Each change raised the number of requests, the compute or the IP quality needed to collect one ranking. A scraper built before 2025 now fails at the first step. One built before August 2026 returns redirect links instead of URLs.

January 2025: no JavaScript, no results

In mid-January 2025, Google Search stopped serving results to clients that don't execute JavaScript. A Google spokesperson told TechCrunch that "enabling JavaScript allows us to better protect our services and users from bots and evolving forms of abuse and spam." Every scraper built on plain HTTP requests (Python's requests library, curl and most "Google scraper" snippets from before 2025) broke that week. Since then, a working Google scraper has meant a headless browser, which is heavier, slower and uses more bandwidth per results page.

September 2025: num=100 disappears

Between September 12 and 14, 2025, the &num=100 parameter, which returned 100 results in a single request, stopped working. Tracking a keyword to position 100 went from one request to ten. Search Console impressions dropped for most sites as a side effect: an analysis of 319 properties by Tyler Gargula of LOCOMOTIVE Agency, reported by Search Engine Land, found that 87.7% lost impressions and 77.6% lost unique ranking terms. Rank trackers loading deep pages had been generating those impressions all along.

August 2026: what is a google.com/goto link?

A google.com/goto link is a Google-owned redirect that Search now places in front of each result, in place of a direct link to the website. The link looks like https://www.google.com/goto?url=CAES…. The token in the link is encoded and can't be decoded locally. Visiting it returns a 302 redirect with the real address in the Location header. Observers first spotted these links on June 23, 2026. After roughly four months of intermittent testing, Google confirmed the rollout to Search Engine Roundtable on August 26, 2026, saying: "We have a long history of deploying technical measures against evolving forms of abuse, and we regularly take steps to protect our services and users." Nozzle's CEO Derek Perkins measured near-100% coverage across several US residential IP providers. Because HEAD requests are refused, every link needs a full GET, and he counted 500–1,000 requests to resolve all links in a single five-page ranking check (PPC Land). Rate limiting, not bandwidth, is now the binding constraint.

SerpApi announced on September 5, 2026 that its API returns direct destination URLs again. It didn't say how.

September 2026: the tracker crackdown

From around September 13, 2026, Google became markedly better at blocking search scrapers and third-party tracking tools, as Search Engine Roundtable reported on September 18. Nozzle saw roughly an 80% drop in the data it could collect, DataForSEO reported similar problems, and Sistrix's status page said on September 16 that its "data collection is currently running at a reduced rate." Google hasn't commented on the blocking itself. The wider web is moving the same way: see Cloudflare's new default block on AI crawlers.

And the official route is closing too

Google's own Custom Search JSON API is closed to new customers, and existing customers have until January 1, 2027 to move off it. Google points them to Vertex AI Search, which it recommends "for searching up to 50 domains". That's a site-search product, not a way to get general web results.

Why did rank trackers stop working in September 2026?

Rank trackers stopped working reliably in mid-September 2026 because Google began blocking automated search traffic far more effectively from around September 13. Nozzle reported losing about 80% of its Google data collection, and Sistrix said it was collecting at a reduced rate. That came on top of the google.com/goto redirects confirmed on August 26, 2026, which forced every tracker to rebuild how it resolves result URLs. The signs are gaps in position history, stale data or sudden "losses" dated September 10–16, 2026. If you see them, the tool is the likely cause, not your rankings. The quickest way to tell the difference is Search Console. Clicks come from real searchers, so if clicks and top-10 impressions held steady while deep impressions collapsed, your rankings are fine. Our own property shows exactly that pattern.

Comparing daily averages for September 16–24 with August 26–September 9, 2026:

• Total impressions fell 64%, in two steps: September 10 and September 16.
• Clicks rose 36%. Real searchers didn't go anywhere.
• Impressions at positions 51–100 fell 81%, while top-10 impressions were flat (−1%).
• Desktop impressions fell 74% and mobile only 16%, which fits automated desktop checks disappearing.
• Positions of individual pages didn't move.

Why did Search Console impressions drop in September 2026?

Search Console counts an impression whenever a result is on a loaded page of results. In Google's own definition, an impression is counted "whenever an item appears in the current page of results, whether or not the item is scrolled into view". A result on page 3 gets no impression if the searcher only views page 1. So impressions at position 85 require someone to load page 9 of the results, which almost no person does. A tracker checking thousands of keywords does it constantly. When Google started turning those tools away, the impressions they generated vanished. Clicks, which only real searchers produce, were unaffected. It's the same mechanism behind the num=100 drop in 2025, at a smaller scale.

What to do if your tracker looks broken:

1. Check your vendor's status page before changing anything. Sistrix, for one, flagged reduced collection on September 16.
2. Cross-check against Search Console clicks for top-10 queries, not total impressions. Clicks are counted server-side from real users.
3. Expect vendors to disagree for a while. Checking tools against real traffic, Glenn Gabe found that Ahrefs picked up a site's recovery, Semrush showed it with a lag and Sistrix didn't show it (via Search Engine Roundtable).
4. Annotate your reports at September 10–16, 2026 so nobody reads the step down as an algorithm hit.
5. Watch GA4 as well. SUSO Digital notes that when /goto redirects drop the referrer, GA4 can file organic visits as (direct) / (none). Search Console isn't affected.

What are the options for getting Google results in 2026?

There are three ways to get Google results in 2026: buy them from a SERP API, collect them yourself with a real browser and residential or mobile proxies, or use Google's official APIs. The official APIs are closing to general web search. APIs are simplest, but they fail when Google changes something, as September showed. Your own browser pool costs more engineering but gives you full control over location, device and which SERP features you capture.

Many SEO-tool teams run both: an API for the bulk of keywords and their own browser pool for locations, devices or SERP features the API doesn't cover, or as failover. The rest of this guide covers the second option.

How to scrape Google search results: a step-by-step guide

To scrape Google search results in 2026, you need five things working together: a headless browser that runs JavaScript, residential or mobile proxies in the market you track, one sticky IP per keyword, human-like pacing, and a parser that reads each result's visible URL instead of resolving every google.com/goto link. Get any one of them wrong and Google answers with its "unusual traffic" page. The six steps below explain each decision and its trade-offs, whatever language or framework you build in.

Step 1: Use a real browser, not an HTTP client

Google has required JavaScript since January 2025, so plain HTTP clients such as Python's requests library or curl get an empty shell or a challenge page. Use a headless browser automation tool: Playwright, Puppeteer and Selenium all work. Tell the browser to skip images, fonts and media. Your parser never reads them, and on per-GB proxy plans they make up much of the bill.

Step 2: Pick proxies that match the searcher you're emulating

Google shows different results by country, city, language and device, and it blocks datacenter IP ranges quickly. Residential proxies in the exact market you track solve both problems: requests come from real-user IPs, and the results match what a local searcher sees. Mobile proxies suit the hardest targets. Country-level targeting is enough for national rankings. For local packs and city-level rankings, target the state and city as well. Both Premium and Budget residential support country, state and city targeting. Before scraping anything, confirm that the proxy exits where you expect with our free proxy checker.

Step 3: Keep one IP per keyword and pace like a person

Use a sticky session so that every results page for one keyword loads from the same IP. That's how a person paging through results looks, and it keeps positions consistent across pages. Start a fresh session for the next keyword. With NovaProxy, the session ID and its lifetime are options on your proxy credentials, and the dashboard's proxy generator builds them for you. Leave several seconds between pages and longer between keywords. Set the country and language in the search itself as well, so the query, the IP and the language all agree.

Step 4: Read the breadcrumb before resolving any /goto link

Each result still shows its site in the breadcrumb line under the title, even when the link itself is a google.com/goto redirect. For most rank tracking, "which domain ranks where" is all you need, so read the breadcrumb and skip resolution. Resolving every link multiplies your requests roughly tenfold. Resolve only the links for your own domains and a handful of tracked competitors, using a full GET request because Google refuses HEAD requests. That keeps the cost to a few extra requests per page.

Step 5: Handle consent screens and blocks without hammering

Searches from EU and UK IPs often hit a "Before you continue to Google" consent screen first. Have your automation dismiss it, or use IPs in the market you actually track. If Google redirects to its "unusual traffic" page or returns a 429, stop that session, pause, and continue on a fresh session at a slower rate. Retrying immediately from the same IP burns the IP and teaches you nothing. The error table below covers each case.

Step 6: Keep the parser honest

Google changes its markup often, and its generated class names rotate. Anchor your parser to the page's structure (result headings and the links around them) rather than to class names. Build in two alarms: fail loudly when a page returns zero organic results, and save a screenshot of that page so you can tell a layout change from a consent screen or a block. Never record a zero-result page as a ranking drop.

Common Google scraping errors and how to fix them

Most Google scraping failures come down to four causes: a block page, a rate limit, a consent screen or a layout change. Unresolved /goto links have been a fifth since August 2026. Each shows a distinct symptom, and each has a different fix. Retrying harder from the same IP fixes none of them.

How much does it cost to scrape Google search results?

The cost of scraping Google search results depends on three things: how many results pages you load, how much each page weighs, and whether your proxy bills per GB or per day. Every page is a full browser load since num=100 went away, so tracking 1,000 keywords daily to position 20 means 2,000 page loads a day, or about 60,000 a month. On per-GB plans, blocking images, fonts and media is the biggest lever. On time-based unlimited plans, bandwidth stops mattering and request pacing becomes the limit.

GB per month = keywords × pages per keyword × checks per month × KB per page ÷ 1,048,576

Page weight varies with the query and with the SERP features Google shows, so measure it on your own keyword set rather than relying on a single benchmark. Browser automation tools can report the bytes transferred for each request. Sum them per page load, with and without asset blocking, and plug the median into the formula. For scale: 1,000 keywords tracked daily to position 20 is about 60,000 page loads a month, so every 100 KB of page weight adds roughly 5.7 GB.

Matching the NovaProxy plan to the workload:

• Small, steady keyword sets: rotating residential proxies from $2.29/GB, down to $1.69/GB at volume, or Residential Budget at $0.39/GB. Both include country, state and city targeting.
• Large daily batches: unlimited residential is billed by time instead of traffic, from $5.00/day, with unlimited concurrent connections. Once your daily check runs to many gigabytes, bandwidth stops being a line item.
• Hardest targets: mobile proxies (from $3.49/GB) exit through carrier networks where many real users share IPs, which makes them the most expensive IPs for a site to block.

Since /goto, rate limits bind before bandwidth does, as Nozzle's numbers show. A large pool you can spread requests across, in the countries you track (we cover 195+ locations), matters more than shaving cents off the per-GB price. You can buy in about 60 seconds with no account, pay by card or crypto, or start with the free trial, and add NovaProxy as a second pool next to your current provider. For rank-tracking specifics, see proxies for rank tracking.

Is it legal to scrape Google search results?

Short answer: Google's terms prohibit it, but in July 2026 a US federal court held that scraping public Google results with no copyrighted content doesn't violate the DMCA's anti-circumvention rules. The case continues over copyrighted material in Knowledge Panels. In practice, scraping Google is mainly a contract and access risk, and blocking is Google's main remedy. It's a copyright risk only for copyrighted elements. Collect only public results, never log in, don't collect personal data, and get legal advice before running anything at commercial scale.

Google's terms prohibit it. Google's spam policies define machine-generated traffic as "sending automated queries to Google", and state that this "includes scraping results for rank-checking purposes or other types of automated access to Google Search conducted without express permission." The policies add that "such activities violate our spam policies and the Google Terms of Service."

The main copyright theory has so far failed in court. On December 19, 2025, Google sued SerpApi in the Northern District of California (case 4:25-cv-10826), arguing that scraping past its "SearchGuard" anti-bot system violates the DMCA's anti-circumvention provisions. On July 20, 2026, Chief Judge Yvonne Gonzalez Rogers dismissed those claims, without leave to amend for search results that contain no copyrighted content: "To the extent that Google Search results do not contain any copyrighted content, SearchGuard cannot be said to effectively control access to a work protected under the Copyright Act." She allowed Google to amend its claims about copyrighted material in Knowledge Panels. Google filed an amended complaint in August 2026, and the case is ongoing.

This section is not legal advice.

Published September 2026, last updated September 27, 2026. We'll update this guide as Google's changes and the SerpApi case develop.