Perhaps one of the most painful to read error messages on any website, if it’s your own at least, is the ’500 Internal Server Error’ message. It’s a pretty generic and therefore frustrating HTTP Status Code meaning no more than “something’s gone wrong” – you might come across them fairly regularly if you’re trying to be especially clever rubbish with htaccess, for example.
This post isn’t just a look at why 500 Errors are bad, that’d be too obvious, but sometimes 500 Errors can be on your website whilst the site appears to be functioning normally. This is therefore a small case study at an example of how your users could be reporting no problems, but the search engine is left wondering why your site is screwed – and your search traffic therefore tumbles as a result.
When A 500 Error Is Bad For Both Users And Search Engines
While irritating for you and your site users, it’s especially irritating for search engines. They can’t crawl pages with 500 errors, at all. However these Server Error pages are usually pretty easy to come across. Normally your site will be plastered with a beautiful “Internal Server Error” message in wonderful black and white.
If your site has enough regular visitors, hopefully you’re likely to know about this fairly promptly and can get on the case via the “Please help, the internet is meaningless to me without your site being up” type of emails.
However, 500 Errors aren’t always as simple as that – sometimes your site, as far as you can tell, is functioning completely normally… except for the 500 Error hidden within the page, making search engines cry. You might not notice the damage till you next log in to Analytics, only to find that your search traffic has ground to a halt. One page might not be a big disaster, but it could be that the error is being generated in a template used across your entire site for example; then you’re really in trouble.
When 500 Errors Are Sneaky
Or in other words: Sometimes, 500 Errors can be generated whilst the page loads seemingly as normal.
This is where the previously mentioned case study comes to point. Someone with a fairly active site purely through natural search had come to ask for advice on why his site had simply stopped generating traffic from search engines. Some pages were still ranking normally, but others had just stopped receiving traffic in entirety. In fact, those pages had been completely de-indexed. A large percentage of his content had been scraped onto multiple sites and our first worry was therefore that Panda had been overly brutal to him, or there was some other major penalty, and we’d have to go through to pain of resolving what could be a hugely extensive issue.
This was till after a lot of head scratching where we stumbled across a cheeky little error message displayed in the footer of only certain pages. This was generating a 500 Error for every page using that particular template. However the pages were still live, readable and fully loaded; no luxury of the horrible “Internal Server Error” black and white page making things obvious. This had meant that while those certain pages were still useable, the internal server error was resulting in Googlebot hitting a brick wall. After a couple of weeks of that, Googlebot had got bored of the brick wall and just de-indexed the pages altogether.
As you can see from the Analytics graph above, traffic from search engines had dropped off progressively as search engines decided to give up on more and more pages, de-indexing them one by one.
So How Do You Check For This Stuff?
There are tons of tools which will scour your website from top to bottom to allow you to see any errors. Xenu, for example, is a great tool often used for 404-Hunting, but can equally be used just as well for 500-Hunting. Xenu will look for links throughout your entire site, and then report back on the status of the resulting pages. That is, unless a high order page of your site is containing a 500 Error, in which case the tool will become just as stuck as Googlebot and will be unable to crawl any further. So if your home page contains a 500 Error for example, then this is no good for finding pages deeper than your front page. However you’d at least know that your index or main category page contains a 500 Error and you therefore have a major problem. Screaming Frog will do the job just as well too, but ironically their site is down with a 500 Error as I write this.
Ouch.
Google Webmaster Tools too, if you have access to that, will display any 500 Errors on your site from the ‘Unreachable’ tab of the ‘Crawl Errors’ page.
Ok, So I’ve Found Some Errors – Now What?
I’m not going to turn this post into a ’500 Error Checklist’, so if you’re not a web developer or don’t know what’s wrong, get someone technical onto it as soon as possible. The 500 Error can be pretty vague, so it could be a whole bunch of things which are potentially wrong. If your site is displaying a server error for long enough, search engines (Bing too) will just de-index the pages, they’re ruthless like that – so sort it sooner rather than later; you don’t necessarily know how long that page has been 500′ing for.
Once you’re happy the problem is solved, run Xenu or ScreamingFrog (or similar) again and make sure all is good.
From there, you need to tell search engines that the page/s is/are back up and running – as far as they’re concerned, the page is broken until they happen to stumble across it at a later date. Depending on how often search engines crawl your site, that could be weeks. This is when I point you across to a great post on the SEOmoz blog a few weeks back – Accidental Noindexation Recovery Strategy and Results. While that post is written from another perspective (pages which have been noindexed via a robots noindex tag), the recovery strategy is entirely relevant – follow those same steps and your traffic ought to return to normal within a couple of weeks.
Like so:
