How Search Engines Work?

The particular software that does this is commonly referred to as the crawler, bot or SE spider. The program travels or crawls through-out the internet through links and collects web pages with their content and brings them to the SEs database. This part of the SE function identifies new web pages and changes to old web pages. It is important to have your website crawlable. If it is not your website is as good as non-existent.

In the SE database the crawled pages are indexed or arranged in a manner that is somewhat sorted. A lay example of what the engines do at this stage is the index at the back of a book - particularly academic books. The index has a word or phrase and a following string of pages where that phrase appears. An index for only one book is a couple of pages. Now think of a SE index as a collection of millions of indices from millions of books harmonized.
Harmonized because, there are thousands of books on similar topics. The index takes into consideration several variables. For example is a particular word in the webpage in bold? Is it in the title tag? Is it within the content or on the navigation? The index is the heart of the SEs. It is the index that will mainly determine which page ranks where for whatever query.

The exact workings and what variables are considered are kept a secret for two man reasons. One is to avoid unfair manipulation by black hat SEOs to rank undeserving websites. And two is to avoid giving competitors "useful ideas" of serving better quality search to users.

It might seem to you that when you enter a search query into a Yahoo Search that is when Yahoo's SE goes out into the web to retrieve results. But most of the job was already done by the crawler and the indexer. When you perform a search you activate the third function of the SE, query processing.

The query works together with the index to serve the most relevant results to that particular query or term from the SEs dataase. You will commonly come across the term SE Algorithm or Algo referring to this operation.

Designing a website so that the SE indexes finds it most relevant to your websites keywords is what entails SEO. Making sure particular visitors click your website's link on the results that are served to the user by the SEs, rather than your competitors, entails SEM.