What algorithm does big search engines like google use?

You can read on the original paper The Anatomy of a Large-Scale Hypertextual Web Search Engine which described the PageRank algorithm, which is the core of the Google search engine they created (after they didn't find anyone interested in buying it, ironically). Or, if you want something lighter, you can simply read about it on Wikipedia.
The actual algorithm Google uses nowadays is more complex, taking into account many more things, but PageRank would be the go-to point. As the way it works, they have the whole web crawled (well, big part of it), and indexed. They use Map-Reduce jobs for processing.