What is Crawler?

Crawler — Also known as spider, an automated software that retrieves webpages and follows the hyperlinks contained in them. Used to generate indexes used by search engines.