Visibility dropped after a redesign or migration: four quiet causes

Short answer: If traffic fell after a redesign, migration, plugin change or a staging site going live, check four technical causes first: a noindex rule on pages you want found, a canonical tag pointing somewhere else, broken URLs or messy redirects, and a robots.txt rule blocking crawlers. Each can stop search engines or AI search crawlers using a page while the site still looks fine to visitors.

None of them is guaranteed to be your cause, and they are not equally serious. A sitewide noindex can remove a whole site from Google; a single extra redirect hop usually costs little. Check, confirm with evidence, then fix.

First, rule out the boring causes

Before changing anything, confirm the drop is real and technical:

If Search Console shows impressions falling on particular pages after a site change, the four checks below are the place to start.

The four checks at a glance

Problem Symptom Diagnose Good result Repair Verify
noindex Pages drop out of Google’s index Page source, response headers, URL Inspection No noindex on pages you want found Remove the meta tag or header; turn off the CMS setting URL Inspection live test; Page indexing report over the following weeks
Wrong canonical Google shows another URL instead of yours Page source, URL Inspection Declared canonical is the page’s own live URL, and Google agrees Fix the template or plugin setting URL Inspection: user-declared and Google-selected canonical match
Broken URLs and redirects Old URLs return 404; long or looping redirect paths curl with redirects followed; Page indexing report Old URLs reach the final page in one permanent redirect Add or correct 301 redirects; update internal links Re-run curl; check “Not found (404)” and redirect errors in Search Console
robots.txt block Pages “blocked by robots.txt”; AI search crawler shut out Read the file; robots.txt report in Search Console No rule blocks crawlers you want on pages you want found Remove or narrow the rule Re-fetch the file; recheck after the crawler has re-read it

Crawling, indexing and being cited are different stages. A crawler has to be allowed to fetch a page, a search engine then decides whether to index it, and an AI answer may or may not use it as a source. A problem at an earlier stage blocks the later ones; a pass at an earlier stage does not guarantee the later ones.

1. Has a noindex rule crept onto a page?

A noindex rule asks search engines not to index the page. It is often switched on for a staging site, or by a CMS setting such as “discourage search engines”, and then survives the launch.

It can live in two places (Google Search Central):

How to check:

# Response headers, using a normal GET request
curl -s -o /dev/null -D - https://yoursite.com/key-page/ | grep -i "x-robots-tag"

# The robots meta tag in the HTML your server sends
curl -s https://yoursite.com/key-page/ | grep -io '<meta[^>]*name="robots"[^>]*>'

Then run URL Inspection in Google Search Console. Its live test “fetches and examines the URL in real time”, and the indexed result shows “the most recently indexed version of a page”. Google also says “There is no definitive test that can guarantee whether your page will be included in the Google index”, so treat a clean result as removing a barrier, not a promise (Search Console Help).

curl shows what your server sends to that one request. It does not show tags added later by JavaScript, and it is not Googlebot. Use it to find problems quickly, and URL Inspection to confirm what Google sees.

Check your home page, your main service or product pages, and any page you would want an AI assistant to quote. For a short definition of the rule, and why it needs the page to stay crawlable, see our noindex glossary entry.

2. Is a canonical tag pointing somewhere else?

A canonical tag tells search engines which URL you consider the main version of a page. Google describes rel="canonical" as “A strong signal that the specified URL should become canonical” (Google Search Central). It is still a signal. Google says “indicating a canonical preference is a hint, not a rule” and that “Google may choose a different page as canonical than you do” (Google Search Central).

Familiar ways it goes wrong after a site change:

A self-referencing canonical, where a page names its own URL, is what Google recommends on the canonical page. It is good practice, not a requirement for every URL, and a missing canonical is far less harmful than a wrong one.

How to check:

curl -s https://yoursite.com/key-page/ | grep -io '<link[^>]*rel="canonical"[^>]*>'

In URL Inspection, compare the User-declared canonical with the Google-selected canonical. If they differ, Google has chosen another URL. Spot-check one page from each template: home, a category, a product or service page and a blog post. A template fault repeats on every page that uses it.

3. Are key URLs broken, or redirecting badly?

Pages get renamed in redesigns. If an old URL now returns a 404, links and bookmarks pointing at it lead nowhere, and Google says that for a previously indexed URL “the indexing pipeline removes the URL from the index” (Google Search Central).

Redirects fix that. What matters is that they are correct, permanent where the move is permanent, and direct:

How to check:

# Every hop, with its status code and destination
curl -sIL https://yoursite.com/old-page/ | grep -iE "^(HTTP|location)"

# Just the final URL and the number of redirects
curl -sL -o /dev/null -w "%{http_code} %{num_redirects} %{url_effective}\n" https://yoursite.com/old-page/

Some servers answer a HEAD request (-I) differently from a normal page load, so if a result looks odd, re-run it without -I. Test the four forms of your home page (http:// and https://, with and without www): each should reach the final URL in one hop. In Search Console, check the Page indexing report for “Not found (404)” and redirect errors, and make sure internal links point at final URLs rather than old ones that redirect.

4. Is robots.txt blocking crawlers you need?

Robots.txt tells crawlers which paths they may fetch. One group can block everything:

User-agent: *
Disallow: /

This is common on staging sites and often survives a launch. Narrower rules cause quieter damage, such as blocking /blog/ or the folder your CSS and JavaScript load from.

Three things to know:

How to check:

  1. Open yoursite.com/robots.txt and read every group, not just the first Disallow you see.
  2. For each crawler you care about, Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot, find the group that applies and check what it blocks.
  3. In Search Console, the robots.txt report shows the file Google fetched and when.
  4. Check your firewall or CDN too. A bot-protection rule can refuse a crawler that robots.txt allows.

Our AI bot permissions guide has a full table of AI user agents and example rules.

Do these problems affect AI search too?

Often, yes, but by different routes. Google says that to be eligible as a supporting link in AI Overviews or AI Mode, “a page must be indexed and eligible to be shown in Google Search with a snippet”, and that “There are no additional technical requirements” (Google Search Central). So a noindex or a robots.txt block that keeps a page out of Google’s index also keeps it out of those features.

Other assistants run their own crawlers and indexes, and each publishes its own rules. A robots.txt block on an AI search crawler can keep you out of that engine’s search answers even when Google is unaffected. Fixing access makes a page eligible. It does not mean any assistant will cite it.

After a migration: a monitoring checklist

For the first few weeks after a redesign or migration:

Can I check all four at once?

The free SearchScore audit checks the site you enter for a noindex rule in the robots meta tag or X-Robots-Tag header, its canonical tags, its robots.txt rules including AI crawler groups, HTTPS and www consistency, and lists what to fix in a Fix-List. It is a quick first pass, not a replacement for URL Inspection or a full crawl after a migration. For a longer list of technical items, see the GEO technical checklist.

Sources

  1. Google Search Central, Block search indexing with noindex: https://developers.google.com/search/docs/crawling-indexing/block-indexing
  2. Google Search Console Help, URL Inspection tool: https://support.google.com/webmasters/answer/9012289
  3. Google Search Central, How to specify a canonical URL with rel=“canonical” and other methods: https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls
  4. Google Search Central, What is URL canonicalization: https://developers.google.com/search/docs/crawling-indexing/canonicalization
  5. Google Search Central, HTTP status codes, network and DNS errors: https://developers.google.com/search/docs/crawling-indexing/http-network-errors
  6. Google Search Central, Redirects and Google Search: https://developers.google.com/search/docs/crawling-indexing/301-redirects
  7. Google Search Central, Introduction to robots.txt: https://developers.google.com/search/docs/crawling-indexing/robots/intro
  8. Google Search Central, AI features and your website: https://developers.google.com/search/docs/appearance/ai-features

All Google sources checked 12 November 2026.

Frequently asked questions

How do I check if a page has a noindex rule?

Look for a robots meta tag containing noindex in the page's HTML, and check the HTTP response headers for an X-Robots-Tag, for example with curl. In Google Search Console, URL Inspection shows whether Google could index the page, and the Page indexing report lists pages excluded by noindex.

Does a robots.txt Disallow stop a page being indexed?

Not reliably. Google says robots.txt is not a mechanism for keeping a web page out of Google, and a blocked URL can still appear in results without a description. To keep a page out of the index, use noindex and leave the page crawlable so the rule can be read.

Is rel=canonical a command Google must obey?

No. Google says indicating a canonical preference is a hint, not a rule, and that it may choose a different page as canonical. URL Inspection shows both the canonical you declared and the one Google selected.

How many redirects will Google follow?

Google says its crawlers follow up to 10 redirect hops by default. That is a limit, not a target. Pointing old URLs straight at their final destination avoids wasted requests and is easier to maintain.

Will fixing these bring my traffic back straight away?

Not necessarily. A fix removes a barrier, but pages have to be recrawled and re-evaluated, and a traffic drop can have other causes, such as seasonality, ranking changes or tracking problems. Fix what you find, verify it, then watch the affected pages for several weeks.

Part of Pillar Article - see all guides in this series →