Google routinely crawls your mobile and desktop product pages and images to check for quality issues. If we're unable to perform these crawls, we won't be able to show your items in listings.
Additionally, for landing pages, if we detect crawling errors while fetching either the mobile or desktop landing page of an item, Google will disapprove the item for both mobile and desktop devices until we are able to access the landing page successfully.
On this page
Why this is happening
The most common reasons for product crawl issues are:
- Page not found (404) error: The provided URL is incorrect or broken and returns a 404 error. Please verify that the URL is accurate and your website is currently live.
- Server's robots.txt disallows access: Your robots.txt file is blocking access to Google crawlers. In order for us to access your whole site, ensure that your robots.txt file allows both user-agents "Googlebot" (used for landing pages) and "Googlebot-image" (used for images) to crawl your full site. You can learn more about how to configure robots.txt here. You can test your current configuration with the URL Inspection tool.
- Invalid URL: Your URL contains invalid characters or does not have the format of a valid link.
Note: Once you've resolved the issue, it may take up to 48 hours for your product to reappear in listings. For certain issues, you can speed up the review by requesting a website check on the "Product details" page under the "Needs attention" tab for an impacted product.
Other common issues
Here are a number of other issues that may also prevent Google from crawling your page:
- Page requires authentication: The URL provided is protected by some sort of authentication protocol that prevents Google from accessing the content.
- HTTP 4xx and 5xx responses: The server hosting your website returned an HTTP error that prevented us from accessing the content.
- Hostname not resolvable: We were unable to resolve the hostname of your server to an IP address and so could not access the page.
- Malformed HTTP response: The response from your server was garbled.
- Firewalls, geo-restrictions, or private IP: Your website may be behind a firewall, have geo-restrictions, or use a private IP that blocks our crawlers. You must allow crawling from the US because Google often operates from there. Ask your webmaster or hosting provider to update your firewall and server settings to allow the Googlebot user agent.
- Network error: A network connection issue occurred during the crawl.
- Timeout reading page: The server didn’t respond in time, so the crawl for this product stopped.
- Server redirects too often: The server redirected the request too many times, which caused the crawl to fail.
- Redirect errors: The server returned an invalid, empty, or excessively long URL that prevented the crawl.
- Robots.txt unreachable: We couldn’t access your robots.txt file, which stopped the crawl. Learn more about how to fix product crawl issues with robots.txt files.
- Regional landing pages not accessible: If using the Regional availability and pricing add-on feature, ensure that all landing pages with region ID can be crawled. For example, URLs like the following should be accessible to Google:
- https://mystorewebsite.com/p/myproductlandingpage?region_id=123456
- https://mystorewebsite.com/p?offer=myproduct®ion_id=123456