blocked web page
| |

How To Check To See If Blocked Pages Are Indexed

ByBill Sebald

You put a robots.txt on your site expecting it to keep Google out of certain pages. But you worry – did you do it correctly? Is Google following it? Is the index as tight as it could be?

Here’s a question for you. If you have a page blocked by robots.txt, will Google put it in the index? If you answered no, you’re incorrect. Google will indeed index a page blocked by robots.txt if it’s being linked by one of your pages (that do not have a rel=”nofollow”), or if it’s linked from another website.  It doesn’t usually rank well because Google can’t see what’s on the page, but it does get PageRank passed through it. What a waste! Google will probably give you a snippet like this:

robots.txt snippet

2 Ways To Check For Indexed Pages You Thought Were Blocked

Don’t worry, I have a couple relatively painless ways to check your own indexation.

Using Google.com (More Manual)

Visit your robots.txt file and look at the blocked directories.  Let’s use Toys R Us for example:

  1. Check http://www.toysrus.com/robots.txt
  2. Take the first blocked directory (as of 3/22/2015, it’s /search/)
  3. Query Google.com using site:toysrus.com inurl:/search/ (this will attempt to find any URL that has /search/ on the toysrus.com site)
  4. Take note of any listings stating a description for this result is not available because of this site’s robots.txt 
  5. Repeat with all the other blocked directories
  6. Find all linking pages and determine your best course of action (eg, a nofollow attribute, a meta-noindex, the “remove URL” from Google Webmaster Tools)

In some cases, this trick will result in noisy results. If you tried this example above, you probably didn’t find blocked URLs until about page 5, where you see “In order to show you the most relevant results, we have omitted some entries very similar to the 47 already displayed. If you like, you can repeat the search with the omitted results included.” Since Toys R Us uses “search” as a parameter, Google tries to show it. This is not the “search” we’re looking for.

Summary

SERPitude is certainly a great tool for purposes as well, but solid for understanding what the SERPs are showing your searchers. Now that you identified your blocked pages, the real fun comes in tracking them down, deindexing them, and plugging the links with a rel=”noindex”.  Go to it, Sherlock!

Similar Posts

Using Google Search Console Tools To Clean Up Your (404) Act
| |

Using Google Search Console Tools To Clean Up Your (404) Act

ByBill Sebald

This article is an entire rewrite from its original 2012 version. When Google launched the new Search Console (thus killing the legacy 404 report), it rendered the old version of this post useless. The bad news – the new URL inspector doesn’t work the same as the legacy report. In the previous report, you could…

Step-By-Step Google Disavow Process
| |

Step-By-Step Google Disavow Process

ByBill Sebald

Read Full Article

For one reason or another, plenty of sites are in the doghouse. The dust has settled a bit. Google has gotten more specific about the penalties and warnings through their notifications, and much of the confusion is no longer… as confusing. We’re now in the aftermath – the grass is slowly growing again and the sky is starting to clear. A lot of companies that sold black hat link building work have vanished (and seem to have their phone off the hook). Some companies who sold black hat work are now even charging to remove the links they built for you (we know who you are!). But at the end of the day, if you were snared by Google for willingly – or maybe unknowingly – creating “unnatural links,” the only thing to do is get yourself out of the doghouse.

optimize
|

Optimize NOW For Entities and Relationships

ByBill Sebald

I remember a few years ago blowing the mind of a boss with a theory that Google would eventually rank (in part) based on their own internal understanding of your object. If Wikipedia could know so much about an object, why couldn’t Google? In the end, I was basically describing semantic search and entities, something that has already lived as a concept in the fringe of the mainstream.

How Entities (and Knowledge Cards) Can Help With Query Intent
| |

How Entities (and Knowledge Cards) Can Help With Query Intent

ByBill Sebald

Entity optimization as a big SEO play isn’t quite upon us yet. It’s a slow, growing Google addition. I know – it frustrates me too. So much potential, of which I believe will greatly improve search results in the future. Google isn’t nearly showing the fruits of everything it knows through entities, whether through cards or search results – at least not relative to the way they rank on keywords alone. But can knowledge cards help bring qualified traffic while considering searcher intent? SEOs always talk about searchers intent. Anyone who’s been doing SEO for a while knows that building for intent can be a challenge.

seo report
| |

SEO Reporting: How to Build Meaningful Analytics Reports

ByBill Sebald

The SEO report. It’s a calling card for some agencies. These reports can be ornate or no-frills (everyone has their own style). Smart companies use APIs to compile reports without spending manual hours. Some rely on automatic SEO reporting tools. For other companies, it’s a time-intensive and considerably low-value exercise.

How To Work Relationships and Concepts Into Your Copy
| |

How To Work Relationships and Concepts Into Your Copy

ByBill Sebald

The folks at Mountain View made the conscious decision that keywords alone couldn’t deliver them the results they wanted to see (ahem, “their users wanted to see”). Google tried some different modeling but ultimately came around to semantic search (that is, using semantic technology to refine the query results). Now I said much of the industry has picked up on it. Not all. I still see a lot of pretending Panda, Penguin and Hummingbird never happened. That’s unfortunate for innocent clients around the world. But for most of us probably reading this, we’re students of a new lexicon. With words like “triples” and “entities” and “semiotics” and “topic modeling.”