Topic: Fliter SE Bots by useragents - you really missed that

[alex]

You forgot about se bots simulating surfers, they all going to noref and sometimes it generates a lot of noice in stats.
Those bots counted as noref and who knows how the hell they counted else when clicking.
There are a little string below from one of tiny sites.

Mozilla/4.0 (compatible; MSIE 7.0b;
Windows NT 6.0)   1458 (26.66 %)
http://www.cuil.com/twiceler/robot.html)   1022 (18.69 %)
Mozilla/5.0 (compatible; DotBot/1.1;
http://www.dotnetdotcom.org/,
crawler@dotnetdotcom.org)   447 (8.17 %)
Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)   112 (2.05 %)

sometimes it 4-5k with Googlebot/2.1 only.

[ip0li]

why would we filter it? Why is it wrong if they are shown there?

[alex]

- It drops total stats - productivity and everything. I even getting fucking alerts from day by day because of that.
- cant see real ppl coming with noref.
- what else?

[ip0li]

ok I will speak with kildoozer when he is back from vacation. added to the 2do list

[JuicyAsianRider]

yes its really good idea if bots were shown separate but NOT included in the stats total

[spudman]

did anything ever come of this?

i get so many googlebot hits and it does mess up stats, prod etc. what the best thing to do with this?

thanks

[ip0li]

we will do it in future builds. at the moment we are finalizing our other project and after that is done we are back in 100% to release next updates!