User Agent Info

Introduction

Each time a program (called user agent) requests a page from a web server it sets an environment variable called HTTP_USER_AGENT. This variable contains the user agent name, and, in some cases, additional information such as the operating system the user agent runs under.

Web Resources provides only information on robots. Robots are a special kind of user agents which are, as opposed to web browsers, not directly human-controlled but rather autonomously acting programs which search the web for information. We categorized them into four categories.

Robot Categories

Search Engine Spiders

Search Engine Spiders are used by search engines to index web sites for their databases.

E-Mail Collectors

E-Mail Collectors are user agents used to harvest e-mail addresses from a web pages, usually to flood these e-mail addresses with unsolicited commercial e-mail later.

Code Validators

Code Validators are very versatile. They are deployed in services such as dead link checking, user-friendliness tests, HTML validation, banner exchange code integrity checks, and many more.

Miscellaneous User Agents

Miscellaneous User Agents which don't deserve a category of their own, as well as user agents of unknown purpose.

Alphabetical Index

Alphabetical Index of all database entries.