Search Console tells you a page is "Indexed" or "Not indexed," and it's easy to treat that as the whole story. Green means fine, grey means problem.
But an index isn't a list of approved URLs. It's a data structure built for one job: finding the right documents for a query fast. Once I started thinking about it that way, a lot of everyday SEO advice made more sense, and some of it stopped making sense at all.
Crawling, indexing and serving are three different jobs
Google describes Search in three stages: crawling, indexing and serving results. They're easy to blur together, but they fail in different ways.
Crawling is fetching the page. Indexing is analyzing what's on it and storing it in a way that can be searched. Serving is finding and ranking the stored pages when someone searches.
A page can be crawled and not indexed. It can be indexed and almost never served. This is the same reason I keep saying a sitemap is not the website: getting a URL discovered is only the first of several separate steps.
The book index is the right picture
Google's own explanation uses the index at the back of a book, and it's still the best way to explain it. Google says its index covers hundreds of billions of pages and is well over 100,000,000 gigabytes.
The important part of that analogy is the direction. A book index doesn't list pages and then the words on them. It lists words, and then the pages they appear on. Search engines do the same thing, and that reversal is the whole trick.
Forward index vs. inverted index
Imagine three tiny pages on a travel site:
- Page 1: "Private food tour in Rome"
- Page 2: "Rome walking tour for families"
- Page 3: "Private boat tour in Lisbon"
A forward index stores what's on each page: page 1 contains private, food, tour, Rome, and so on. That's how we naturally think about content, but it's slow to search. To answer "private tour Rome," you'd have to read every page, every time.
An inverted index flips it around:
| Term | Pages |
|---|---|
| private | 1, 3 |
| tour | 1, 2, 3 |
| rome | 1, 2 |
| lisbon | 3 |
Each row is called a posting list, the list of documents where a term appears. To answer "private tour Rome," the engine jumps straight to those three posting lists and keeps the pages that appear in all of them. Here, that's page 1. This step is called intersecting posting lists, and it's fast because the engine never touches pages that don't contain the words.
One of the clearest explanations of this I know is an old 2005 Google article for librarians, written by Matt Cutts. The tools have changed enormously since then. The basic idea of matching first and ranking second hasn't.
What happens to words before they're stored
Text doesn't go into the index exactly as written. It's broken into tokens and normalized first, so that variations can still match.
Stemming chops words down to a rough root: running becomes run. Lemmatization does something similar but uses the dictionary form, so ran can also map to run. Both help a page about "planning a trip" match a search for "trip planner."
Older SEO lessons often add a third step: removing stop words like the, of and in. That used to be common. But the standard information retrieval textbook notes that web search engines generally don't use stop lists anymore. Small words carry meaning in phrases: "flights to Paris" and "flights from Paris" are different searches, and so are band names, book titles and product names.
So I don't strip "small" words from titles or headings to make them look more keyword-dense. They're doing work.
Where the words appear matters too
A simple posting list only says which pages contain a word. Most real indexes also store where in the page it appears. That's what makes phrase matching and proximity possible: "civil war" as two neighbouring words means something different from a page that mentions civil in one paragraph and war in another. The 2005 article made exactly this point, along with the hint that words in a page's title suggest what the page is about.
This is also part of why keyword stuffing never worked as well as people thought. Scoring functions like BM25 reward a term's presence, then quickly stop rewarding repetition.
Passages: the page isn't always the unit
For a long time, I thought about relevance at the page level. In 2020, Google announced it could better understand individual passages within a page, and said the change would affect around 7 percent of search queries. Google later described this as passage ranking: pages are still indexed as pages, but a specific section can be what makes a page relevant to a specific question.
That matters for long guides. A great answer buried in paragraph 14, under a vague heading, is harder to match to the question it answers than the same answer in a section built around it. I tested this more directly in my passage retrieval experiment.
Then vectors joined the index
The inverted index matches words. Vector search matches meaning: text is turned into embeddings, and the engine finds content whose meaning sits close to the query, even when the words don't overlap.
Modern systems usually run both, which is what people mean by hybrid retrieval. Newer methods like MUVERA are about making that vector side fast enough to use at scale. And retrieval-augmented AI answers, the RAG systems behind a lot of AI search, still start with this same step: find the right pieces of content first, then generate.
The inverted index didn't go away. It got a partner.
What I actually check because of this
Understanding the index changed a few habits for me:
I ask "indexed for what?" An indexed page can only be matched to searches whose words or meaning are actually on it. If the key fact lives only in an image, a PDF the crawler skips, or a tab that loads later, it may not be in the index the way you assume.
I write each section around one question. Clear headings and an answer near the top of each section give passage-level matching something to work with.
I keep exact names exact. Product names, place names and brand names should appear the way people search for them, small words included.
I look for near-duplicates. During indexing, Google groups similar pages and chooses a canonical one. Two pages answering the same question don't double your chances. Often one of them simply isn't the one shown, which is why I review overlapping articles before writing a third.
One claim I'd drop
Old indexing lessons often include a line like "Google indexes less than 5% of the internet." I've never found a reliable, current source for that number, and it mixes up very different things: private pages, logged-in content, duplicates and pages no one should index. I'd leave it out of any training deck.
The short version
"Indexed" means a page has been processed into a structure built for matching words, positions and, increasingly, meaning. That's why clear wording, focused sections and exact names still matter, even in AI search. The index can only return what you actually put on the page.
Download the illustrated explainer
Read the research summary and download the PDF for the diagrams, source list and practical checks.
