Need help with this?
Purchase All in One SEO Pack Pro to get dedicated support from us.
We regularly get asked about crawl or index errors in Google Search Console. This report shows you when Google is having problems crawling a URL on your site that they previously indexed.
Crawl errors can be broken down into a number of different types.
This means Google experienced an issue while crawling your site because of your server. Google will usually tell you what error code they received from the server. These are usually transient errors and go away, but if you get a lot of these errors, then this could indicate a server problem. In this case, you should contact your hosting company and let them know about the error codes Google is reporting.
If you’re seeing this error, then Google encountered a URL in your sitemap which is tagged as “noindex”. This value tells Google and other search engines that you do not want the page to be indexed.
If this is your intention, then you can safely ignore the error. Navigate to the relevant page and make sure to check the “Exclude From Sitemap” option in the All in One SEO Pack meta box so that the URL is no longer included in your sitemap.
If you however do want this page to be indexed, then you need to remove the “noindex” meta tag in either the General Settings or meta box (depending on whether you’ve enabled noindexing on a global scale for the relevant post type or only for that specific post). In a future release, we’ll exclude noindexed pages from the sitemap automatically.
This means Google couldn’t crawl your website because their user-agent was blocked. Make sure you’ve configured your robots.txt file in our Robots.txt module in accordance with Google’s guidelines for WordPress sites and make sure no firewall is blocking Google or server authentication.
404/Not Found Errors
The bulk of the errors you may see, and we get asked about, are 404 or so called “Not Found” errors. This means that Google has crawled the URL in the past but now when they crawl it they get a 404/Not Found error page because the URL no longer exists. Check each error and determine why Google cannot crawl the URL. Some common reasons are:
- You removed that content and it’s not coming back. Use the “Remove URLs” tool in the older version of Google Search Console (the new version does not include this tool yet) to remove the URL from their index. You also might want to verify that it is no longer included in your sitemap; otherwise Google will continue throwing errors.
- You removed that content and replaced it with other content. Use a redirection plugin (we recommend Redirection) to create a 301 redirect from the old URL to the new URL.
- The URL or your permalink structure changed. Use a redirection plugin to create a 301 redirect from the old URL to the new URL.
- It’s an incorrect link from someone else’s site. You cannot control how others link to your site, so if someone links to a page that doesn’t exist or typed the URL in wrong then you should use a redirection plugin to create a 301 redirect from the bad URL to the correct URL.
- There’s an incorrect link on your own site. Use a plugin such as Broken Link Checker to monitor your site for broken links and images and fix them when they are reported.
Make sure you fix crawl errors as soon as they are reported as they may have a significant influence on organic traffic to your site. If you use All in One SEO Pack to generate your XML Sitemap and you’re using the default “Dynamically Generate Sitemap” setting, then you can be assured that Google is always getting an updated version of your sitemap with all the correct URLs from your site.