Today we will discuss what is a search engine and how to crawl your website in depth. A web search engine is software designed to direct and perform Internet searches by a user on the World Wide Web, following a set of rules (known as algorithms) which the purpose is to present the search results page (called the search engine results page or SERP), where the algorithm returns what it “thinks” will help answer the user’s query (which is the search result). Also known as a question).

When we hear “search engine,” most of us think of Google. After all, it gets over 63,000 searches per second, and many people think that the “Internet” is what you Google it.

 However, the results we see show only what Google managed to find and decided to include in the search results. But most of the content on the Internet is not on Google.

Why?

Because Google does not approve of everything, for example, it blocks websites that contain prohibited content or follow illegal practices.

Google doesn’t like all content (for example, a web page that doesn’t value users according to its own criteria). The website does not want to appear in the search results, so it informs the crawler not to index or include it in the Google search.

This means that many websites cannot be easily found or found by search engines and in order to access them you need to know their exact address. Entertaining fact: It is estimated that between 96 and 99% of the entire Internet belongs to the Deep Web.

To understand this topic, we will explain what a search engine is, how it works, which are the most popular, and which are the major companies …

Sounds very technical and boring, right?

Basically, a web search engine is a program that ‘surfaces’ the results of many websites, which it deems most appropriate to meet the need or reason that someone typed their search query. What happened?

For example, when you search for “How to Make a Grilled Cheese Sandwich”, the search engine goes through the entire index of Google (which is similar to the digital version of the library) and gives you many recipes, videos, Displays pictures, etc.

The fact is that the engine doesn’t know exactly what you want to know when you search for “How to Make a Grilled Cheese Sandwich”, but it does show you websites that help it solve your “problem”. Consider the best resources to do. Or question?

How do Internet search engines work?

All search engines compete with each other: Yahoo, Google, and Bing are independent and fight for the attention of browsers, so they did not come together to unanimously decide how the search engine should work.

Therefore, every search engine has a “secret formula” (extremely complex mathematical calculations called algorithms) that every company has created to provide users with the best search results.

An example of how a search engine works

Let’s give an example of how a search engine works: it crosses the World Wide Web after hyperlinks (also called “links”). A link is something you can click to go to another website/webpage.

This example of SERP Co shows that search engine web indexers (also called web crawlers, crawlers, or spiders) when they follow the path that the link takes them to, have everything inside every web page. Read HTML code, CSS, Javascript, etc.

So when you see this: As web spiders read the page, they begin to form an “understanding” of the website as a whole and on each page.

In this example, when the web spiders go through the SERP Co site and read it, they assume that it is a digital marketing agency because the title tag and number 1 header on the page indicate it.

Once the search engine understands the website, it archives it in a large database called index or index, which acts as a digital library where all the internet websites are kept (note: as we mentioned in the beginning) Mentioned, it only archives those you track).

How does the traditional search algorithm work?

Suppose you search for “inbound marketing agency” and the search engine goes through its index, searching all the websites that it has crawled that are related to your search.

It shows you all the websites it has found but is organized in a special way: each search results page (SERP) can only accommodate 10 organic results. And the search engine should choose who to show first, second, third, etc.

To do this, it relies on a number of ranking factors to help determine which is the “best” or “most relevant” result. This is where the algorithm comes in: the set of rules that determine how a website is perceived and preferred by search engines.

Unconventional search engine.

In general, when the term “search engine” is used, most people think of text-based searches, such as google.com (a crawler-based search engine).

However, due to continuous technological evolution, we have seen the rise of many other types of search engines and therefore have also seen different search engine algorithms.

Basically, anything that “finds something” is a search engine, but it doesn’t have to be the same algorithm that text-based search engines use.

Let’s look at some examples:

• Search engines that use web crawlers (crawler-based search engines)
• Directories
• Meta search engine
• Hybrid search engine
• Maps
• Multimedia
• Voice search engine
• Shopping seekers

Web crawler-based search engine

As we mentioned earlier, these types of search engines (including Google) use software and algorithms to search and rank websites. These “tracks” (like the digital version of clicking on a link) form a huge “network” of links and rankings that take links from one page to another and from one website to another. ۔ This is why crawlers are known as “crawlers”, “spiders” or “bots”.

Crawlers travel through websites, download them for inclusion in their index, and analyze the information they contain to rank them correctly (and then search for such information). Show people).

Examples of web crawler-based search engines:

• Google
• Yahoo
• Bang
• Dogpile
• High view
• Overture
• Hot boot
• Jade
• Conodle
• Cosmics

Directories

A search engine like a directory forces people to decide which categories websites should belong to. In these types of search engines, people manually add and sort websites, check the quality of sites, and rank them based on a default set of values ​​or rules

Meta search engine

We can say that meta-search engines are somewhat similar to the story of the Film Inception because their job is to analyze the search engine results. That is, they take results from other search engines, combine them, and then display them on the results page.

Here are some examples of meta-search engines:

• Answers.com
• deeperweb
• Dogpile
• Excited
• Harvester 42
• metacrawler
• Mobissimo
• Turbo 10
• Web crawler

Hybrid search engine

Hybrid search engine is a combination of a directory-type search engine and a web crawler.

what is a search engine: Map finders

These types of search engines help users find business locations on the map. It typically combines user-submitted data with algorithmic preferences, car cameras, street views, satellite data, user reviews, and more. 

Examples of map browsers:

• Google Maps
• Apple Maps
• Bing Maps
• MapQuest
• City Mapper
• here.com

Media browsers

Image and multimedia file search engines allow you to search results on a variety of data. As computers continue to learn what an image is, layers of data can be incorporated into media content to help them understand, classify, and organize.

This additional data, called metadata, may contain text descriptions, links to other related content, or even additional blocks of code that tell machines what an image, video, GIF, etc. really is. ۔ If you are interested in this topic, find out more at schema.org.

And here are some examples of multimedia search engines:

• Bing videos
• Blinkx
• Find voices.
• Google Images
• Google Video
• Image Search
• Podscope
• Science stage

Voice search engine

Voice Search allows us to “talk” to devices and get feedback in return. You may already be familiar with these search engines:

• Amazon-Alexa
• Google Home
• Apple’s head