How A Search Engine Finds Your Answer Faster Than A Blink

A search engine does not read the web when you ask it a question. It read the web weeks ago, and what it kept was not the pages themselves but a set of claims about them: this page contains these words, this often, in these positions, and other pages link to it with these words.

That substitution — pages for statistics about pages — is the whole trick. It is also where most of the interesting engineering lives.

The Bet

The index is a bet that a small amount of information about a page is enough to decide whether the page is worth showing. The bet pays off most of the time. When it does not, you get the familiar experience of knowing the answer is out there and being unable to phrase the question the way the machine filed it.

An index is a wager that the words someone will use are close enough to the words someone already wrote.

The wager is not made once. It is remade continuously, because the web does not hold still and neither do the ways people describe things.

Why Latency Is The Visible Part

Users notice speed long before they notice ranking quality. A ranking that is slightly worse is hard to perceive; a result that takes two seconds to appear is immediately obvious. That asymmetry pushes enormous effort into the retrieval path — the part that runs between your keystroke and the first byte.

The work splits cleanly:

• Offline, where a document is parsed, tokenised, and written into structures that are expensive to build and cheap to read.
• Online, where a query is turned into lookups against those structures, and the results are merged and ranked.

Almost every design decision in a search system is about moving work from the second list to the first.

The Part That Surprises People

The structures built offline are not neutral summaries. Choosing to index a word is choosing to be able to find it. Choosing to drop stop words is choosing that a query made only of stop words will behave strangely — which is why the phrase you half-remember is often the hardest thing in the world to search for.

An index is a promise about what can be found, and every promise is also an admission of what cannot.