During a conversation with an web design client about the number of competing web pages in their given business sector, they asked: “So how many web pages are actually out there in total?”
I had to be honest and say that I could not quote an exact figure and that, even if I could, it would be significantly out of date within a week! However, it prompted me to see what Google had to say about the subject.
I remembered that back in the early days of Google (I was involved in web design and marketing before Google even existed!), they used to quote a figure of how many pages were in their index at the bottom of the Google results page. For a variety of reasons, they no longer do this which is a pity really, because it would have been interesting to watch it grow over time.
The main reason, and this is perfectly understandable, is the problem of duplicate content. There are so web pages and indeed websites that carry duplicate content, that it is now impossible to identify how many ‘real’ web pages are actually out there.
Getting A Handle On Size Of The Google Index
Having said that, it is possible to get something of an idea of web page numbers and the scale of growth that has taken place over the last 10-15 years. Here are some key figures:
- In 2000 Google already claimed to have 1 billion URLs in its index.
- Recent estimates from third parties outside of Google suggest that that has increased more than tenfold.
- As for what the guys at Google say, they recently claimed that their index is now well over 100,000,000 gigabytes. Moreover, they claimed to have spent over one million computing hours to build the Google index.