Need help with this?
Purchase All in One SEO Pack Pro to get dedicated support from us.
We regularly get asked about the Crawl Errors report in Google Webmaster Tools. This report shows you when Google is having problems crawling a URL on your site that they previously indexed.
Crawl errors can be broken down into three types of error:
Server error: This is Google reporting they had a problem crawling your site because of your server. Google will usually tell you what error code they received from the server. These are usually transient errors and go away, but if you get a lot of these errors it could indicate a server problem and you should contact your hosting company and let them know what error codes Google is reporting.
Access denied: This is Google reporting they had a problem crawling your site because they were blocked. Make sure you’ve configured your robots.txt file in accordance with Google’s guidelines for WordPress sites and make sure their is no firewall blocking Google or server authentication.
404 / Not found errors: The bulk of the errors you may see, and we get asked about, are 404 or Not found errors. This is Google reporting that they previously crawled this URL but now when they crawl it they get a 404 Page no found error because that URL no longer exists. Check each error and determine why Google cannot crawl that URL. Some common reasons are:
- You removed that content and it’s not coming back. Use the Remove URLs tool in Google Webmaster Tools to remove that URL from their index.
- You removed that content and replaced it with other content. Use a plugin such as Redirection to create a 301 redirect from the old URL to the new URL.
- The URL or your permalink structure changed. Use a plugin such as Redirection to create a 301 redirect from the old URL to the new URL.
- It’s an incorrect link from someone else’s site. You cannot control how others link to your site, so if someone links to a page that doesn’t exist or typed the URL in wrong then use a plugin such as Redirection to create a 301 redirect from the bad URL to the correct URL.
- It’s an incorrect link on your own site. Use a plugins such as Broken Link Checker to monitor your site for broken links and images and fix them when they are reported.
Make sure you fix crawl errors as soon as they are reported and use a plugin such as Broken Link Checker to monitor your site for broken links and images. If you use All in One SEO Pack to create your XML Sitemap and you’re using the default setting of Dynamically Generate Sitemap then you can be assured that Google is always getting an up to date sitemap with all the correct URLs from your site.