Crawling a Web of Links
How a search engine finds pages before it can search them
About 40 minutesMediumThe Internet and Search
What your child can do at the end
Describe how a crawler chooses what to fetch next
The brief
Before a search engine can answer anything, something has to go and read the web. A crawler starts at one page, reads it, notes every link it finds, and walks to those pages next. Pages nobody links to are invisible to it. Pages that ask to be skipped are skipped. In this lab you drive a crawler across a web of 25 pages and then look at the index it builds.
Skills it develops
algorithmic thinkingdata basicsdecomposition
How it is assessed
3 graded challenges inside the lab, and 8 questions at the end that check the idea rather than the clicks. Every attempt is saved, so a retry after a mistake counts as the learning it is.