Googlebots - sitemaps

I am close to launching a web site and so I have directed Google (via
Webmaster Tools) to index parts of the site via a sitemap (a rather
interesting approach to testing I might add).

Anyway, there are places where I require a login and generally if it
fails to find a logged in user, it does...
      request.env["HTTP_REFERER"]
      redirect_to :back
      return false

Now a GoogleBot doesn't submit a reply to the
request.env["HTTP_REFERRER"] so it dutifully sends me an e-mail via my
exception handler.

How do people handle these things?

Should I be doing something like...

my_url = request.env["HTTP_REFERRER"] || ":action => 'index'"
redirect_to my_url
return false

I am sure that at least at this point, I don't want to not send error
mails on 404's just as a sanity check.

Craig

Maybe I got you wrong, but wouldn't a rel="nofollow" in your login-links
be a solution for this situation?

Maybe I got you wrong, but wouldn't a rel="nofollow" in your login-links
be a solution for this situation?

I am close to launching a web site and so I have directed Google (via
Webmaster Tools) to index parts of the site via a sitemap (a rather
interesting approach to testing I might add).

Um, well; there are other link-checkers available, btw, but...

Anyway, there are places where I require a login and generally if it
fails to find a logged in user, it does...
request.env["HTTP_REFERER"]
redirect_to :back
return false

That seems like a really bad idea to me: referer isn't a required
header, period. Some paranoid clients won't send them at all, and
if for instance your legitimate user types in the URL, or invokes it
from a bookmark, there won't be a referer.

Most apps redirect to a login page if login is required. :slight_smile:

FWIW,

Craig White wrote:

I am close to launching a web site and so I have directed Google (via
Webmaster Tools) to index parts of the site via a sitemap (a rather
interesting approach to testing I might add).

[...]

And perhaps not a good one. Do you really want your not-yet-launched
pages in Google's index before launch?

Best,

Craig White wrote:
> I am close to launching a web site and so I have directed Google (via
> Webmaster Tools) to index parts of the site via a sitemap (a rather
> interesting approach to testing I might add).
[...]

And perhaps not a good one. Do you really want your not-yet-launched
pages in Google's index before launch?

Of course! You want to give Google (and other bots) time to find and index your site so that it's primed for launch. The timing is the only tricky part. I had to redesign a "deep" page to act more like an entry page because so many visitors got there first by coming from Google search results.

-Rob

Rob Biedenharn http://agileconsultingllc.com
Rob@AgileConsultingLLC.com