Crawler

Definition by Best Host Ratings

Also referred to as a spider. It is an automated program used by search engines to collect data for web sites. A crawler will download your web page and will analyze its content and links. Then it will usually visit each link and so on. The data collected is used for building the search engine index which is used for providing search results to the users.

Suggest a term for our Glossary

Previous: CPU

Next: D Channel