Displaying posts tagged: spidering

Http:BL

No Comments
Looking through the logs logs of my web server last night I noticed some odd behaviour, a complete spidering of tamonten.com in a couple of mins from an ip address. No biggy, it's just a bot, but the UA string was that of a normal browser and at no point did it request robots.txt Tut tut ...