Many web pages include a site search these days, that makes it possible for users to search content exactly on the page they are and not on the whole web, but what should one do, if no results were found?
Normally search engines would also index these pages, because they can’t see, that there is no result and we set the status to 200 which means “Okay, page found”. To prevent that, some webmasters use a “404 – File not found” header, so that the search engines would not index the page, but there are some major issues with that:
- A 404 should only be used for a page, that is not found. In the case of a search page with no results, a 404 would simply be wrong, because we show a page with content, only one with less content, than the user wanted.
- Links on 404 pages are lost. If the search engine bot finds a 404, all the content will not be fetched and all the links on that page (e.g. our result page with no results) would not be followed or indexed, so you loose a whole lot of link juice and maybe prevent indexing links, that can only be found on that particular page.
Okay, now that we know a 404 is bad, what should we do instead? Well, that’s easy, if you remember the META-Tags, because there we have the META robots tag, that lets us set the parameters “nofollow” and “noindex” and the last one is exactly what we need!
So, if you want a page not being indexed by Google, but all the links on the page should be followed, you should set the following in the head section of your page:
<META NAME=”ROBOTS” CONTENT=”NOINDEX,FOLLOW”>
This would prevent search engines from indexing the page, but no link juice or links get lost!