Topic: Blocking bots

[anthony]

hi, after fixing some errors i had on my site using Google Analytics, I started to recieve thousands of hits in my "noref_in". 6576 hits (sofar)from "http://www.google.com/bot.html" and it's messing up my stats.

Question: Should I block these? and if I do, will my pages stop showing up in google search?

thanks,
anthony

[ip0li]

Imho its not good idea to stop it since its google bot. Just ignore it. It should stop within cpl days. U can also see trades productivity (overall) and for each system trade so you are good I believe.

[anthony]

will do and thanks...

[anthony]

Help---

Googlebot is STILL after a week, bagging the crap out of my site. this is the information from the last 24hours,

"noref_in"-----Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)      6093 (94.79 %)

Help, what should I do, it is really messing up my numbers.

thanks,
anthony

[bbhrucy79]

Do you have anything in your header that may attract big G? Like, meta tags of 'revisit in one day', robot attractors like 'index. follow' etc. or maybe incorrect settings in you .htaccess?

For example if you don't have expiry header in your .htaccess I believe you get more bots. 

[bbhrucy79]

So if you are leaking out bad traffic like this to your trades, will they consider you 'cheater'?

I do not worry about the numbers and stats (bots mess up my stats big time as well) but I worry about this crap forwarded to my trades - not sure, but do trade scripts look at this traffic as 'cheating/bad traffic' or do they generally recognize that these are bots and do not 'punish' you for sending them?

As for stats, just create group named 'human traffic' and put inside all your trades and SE and bookmark traffic, and in this way you can check what is going on in terms of actual human traffic using 'history'.

[anthony]

yes i have

<META NAME="robot" CONTENT="index,follow"> on my index page and a robots.txt file that has the following lines in it.

User-agent: *
Disallow: /photopost/
Disallow: /tp/
Disallow: /click.php?
Allow: /

If you remove these will google index your website? i don't want to lose the little traffic i get from google..

[bbhrucy79]

you should remove <META NAME="robot" CONTENT="index,follow">

you wont loose traffic

have good SEO if you want seo traffic, but tags like above, and especially meta tag 'revisit in one day' etc, can hurt your site. Google marks your site as spam if you have the 'revisit in one day' meta tag, I experienced it.

your robots txt looks fine

hth

[anthony]

thanks............