What is Spider?

Spider — An automated software that retrieves webpages and follows the hyperlinks contained in them. Used to generate indexes used by search engines.