ASP.Net Detecting Bots, Crawlers
Sunday, October 25th, 2009This article discusses a method to detect bot and crawler visits to your site.
Recently, I was developing a model to collect visitor information to the site. The stats showed a mind boggling 10,000 visits a day. Stunned! This was highly unlikely for us. I tracked this through Google Analytics which showed a different number i.e. 2000. Well, we all are aware of the difference in script based (Google analytics) and server log based tracking arguments. I was experiencing the same. Reading through and understanding the concept, I got very suspicious on spider and crawler visits. Well, so the journey started to find out the and categorize the bot/crawler visits.
(more…)