Personal Search Engines - The Internet Free-For-All

January 06, 2006

In a previous article I discussed the monopoly that Google has become in terms of Internet search. Basically just about anyone performing a search is opting to use Google to find relevant websites. As a friend has reminded me, monopolies "cause the market to be inefficient by limiting competition therefore limiting innovation". So, we are left with Google's algorithms to determine which sites are relevant, and which are not. With billions and billions of websites vying for the top spot on Google, chances are they will not break into the first page.

I commented that a solution to this "problem" is a Custom Search Engine, and I'd like to discuss exactly what this is.

Custom Search Engines

Websites and web pages are made up of some kind of code (HTML, XML, XHTML, etc.). Code is basically just characters and words with some sort of structure that dictates to a browser how the code should be displayed. Because of this, the fact that all web pages are made up of defineable characters, words, and code, it becomes possible to parse this code data and manipulate it. When Google runs its algorithms, it looks for snippets of code, groups of words, special tags, etc. to determine how a page or website should rank.

In addition, there are other factors that determine how a web page is ranked. Things such as how many incoming links point to your site, the "quality" of those links, traffic to a web page, et. al. are used to determine a page's rank.

What we are left with are tons of variables that are fed into an algorithm, an equation of sorts, and one single number is output which becomes a page's rank.

So what do you do if you don't agree with Google's (or any other search engine's for that matter) algorithm to determine how to rank web pages? Nothing! You can't do anything. You are stuck and have to use Google's algorithm and results to find what you're looking for.

But what if you could create your own algorithm? What if YOU could determine how pages are ranked...which tags are most important, whether incoming links should matter, or if hidden text should be considered a penalty? The answer is that YOU could determine how web pages are ranked.

What would a Custom Search Engine look like?

I can envision a user-interface which, once you log in, presents you with a configuration screen(s). Listed will be every HTML tag, and many different variables (incoming link count, etc.) that can be used to determine how to rank websites. Perhaps radio buttons with numbers 1 through 10 which determine how relevant that particular attribute or variable is to search results. If you think keywords in bold are very important, give the bold tag a 10. If you think invisible text should be considered a penalty, check off the box marked "penalty". Once the configuration is complete (different configurations could be saved of course allowing multiple configurations for trial-and-error), the user is now ready to test it out and see what kind of results are obtained. Easier said than done, but I believe this is the answer to all our search problems.

Implications

I think the biggest hurdle to overcome is that fact that a freely available index does not exist. Google and Yahoo! and MSN and the other search engines have spent years building up their website indexes to include an almost unfathomable number of web pages. Obviously Google is not about to hand over its index so we must look elsewhere. But, if this happens, if a freely available index of web pages is put together, we would be much closer to Custom Search Engines becoming a reality.

Then we get into the issues web developers and search engine optimizers would face. How do you optimize a site for millions of different configurations? I don't think you can, and I think it's a good thing. There would be no standards for how to put together a web page to rank higher on the search engines and an "Internet free-for-all" is created. Go ahead, repeat your keywords a couple hundred times at the bottom of your page with the same text color as the background color of the page...Put every keyword in an "H1" tag scattered throughout the text on your site...Create link farms with millions of bogus links pointing to your site...go for it!

What would happen is that, over time, developers could introduce plug-ins that would let users rank other types of data to determine results. End-users would create their ranking systems, save the configuration, and then pass it along to other end-users, and popular configurations would start to surface.

I believe that Custom Search Engines are the only way to break away from the search monopoly that Google has created. In this way, ANYONE can determine what kind of search results they want.

Technorati Tags: search engines  google  yahoo  msn search  internet search  custom search engines  personalized search