[font=Arial]As the second most popular search tool, Yahoo moves a tremendous amount of traffic and is a very credible alternative to Google. Yahoo receives over 2.76 billion page views per day from hundreds of millions of unique users. It boasts over 157 million registered users enjoying mail, shopping and discussion groups and increasingly personalized search and news services. For the past two years, Yahoo, Google and MSN have been embroiled in a hard-fought battle for the loyalty of search engine users forcing all three firms into the hyper-evolution we are witnessing today.

Over the next three weeks I will examine how the Big-3 spiders work, what they look for and how to best prepare your sites for multiple visits from the bots that rank them. First, I will start with Yahoos bot, SLURP.

Getting Found By Slurp
The first thing to know about Slurp is that, like its better known cousin, Google-bot, Slurp discovers sites by following links from one site to another, reading and recording nearly everything it finds in its path. The majority of websites referenced by Yahoo were originally included in its database because they were accessed by Slurp following links from another site.

Yahoo suggests adding an inbound link to all pages in your site to guarantee those pages will be discovered by Slurp. They also recommend an internal sitemap linked to the Index (or home) page of the site. To encourage Slurp to spend more time deep-crawling your content, Yahoo recommends the addition of good authoritative links pointing into your site, from highly reputable sources such as news sites, established business partners and other sites relevant to your business or service.

Manual submission of the site is only recommended if, for some reason or another, Slurp does not find the site on its own. This is increasingly rare however as server-logs show Slurp is one of the most active spiders out there. In other words, if a site Slurp has already indexed links to your site, Slurp will almost certainly be visiting very soon. Webmasters should never have to pay submission fees to get into the Yahoo index ... 99% of the Yahoo index is crawled by Slurp for free.

It is still important to make sure your site is ready to receive a visit from Slurp. To ensure Slurp is able to travel across your entire site, provide standard HREF text links as opposed to forms, Flash or java script navigation tools. Webmasters are encouraged to avoid tracking and communication methods that rely on using cookies across every page of the site. If you have a database driven site or a site that creates unique sessions for each user, avoid embedding session IDs in URLs. Lastly, use 404 pages to redirect users (and spiders) to the root (index) page if a page or site URL becomes invalid. Yahoo also asks Webmasters of sites with shopping carts to use robot.txt exclusions in the source of the shopping carts.

Where Your Site Has Been Included - Results May Vary
Yahoo has seen enormous change over the past seven years. What started as a paid-inclusion, human edited search directory, has grown into the second largest database of indexed content.

Yahoo is on the cutting edge of integrating several forms of media into their search offerings and will likely soon produce its own entertainment content like an online HBO. Yahoo is flirting with the concept of becoming an infotainment portal again, but the core of its offerings remains firmly rooted in search.

Yahoo search results come in multiple formats including -- Yahoo-Local, Yahoo-products, MY-Yahoo (personalized results), specific nation-based Yahoo, and the standard Yahoo.com. One of the Yahoo goals appears to be presenting individual search-users with results that best match their personal needs. For instance, Yahoo would like to present constantly updated geographic-specific references when a user searches for daily-use items such as groceries, repair-workers, real estate and other services one would normally use a telephone directory to find. Similarly, Yahoo wants to present the entire global database of references when a user searches for international news, trans-national products or vacation plans. Being certain your website gets served up for all levels of search, local, regional and global, will be important if you wish to serve a market larger than your general region or community.

Getting Rankings
The Yahoo search engine ranks sites based on a formula that is very similar to the algorithms used by rivals Google and MSN. Yahoo values many of the same elements other search engines do including keyword enriched domain names; titles; meta tags; and content. Yahoo also values keywords found in the anchor text of internal links, though the effect at Yahoo is not as powerful as it is on Google.

According to Yahoo, well optimized pages and sites will continue to get good results across all versions of their search engine. By opening your site to Yahoo Slurp and performing well-planned optimization services across every page, a good SEO can nearly always achieve Top placements on Yahoo. The trick is in offering Slurp the information it needs to read; record and rank your site. If that information is included on each page, a set of text-based links is woven through the site to provide easy passage for Slurp, and Yahoo is told what your business is; where your business is located; who your business serves; and your site should achieve strong rankings.

Due to the advent of personal; local; regional and global search results, it is highly recommended to add full contact and address information on every page of a site. This information should be as precise as possible and should include street address; unit or suite number; zip or postal code; state or province; county and country; full telephone information (including area codes); and if possible, the approximate longitude and latitude of your business location. (look up your longitude and latitude - shown here).

When writing for Slurp here are a few basic fundamentals:

1.  Have a descriptive URL.
2.  Use keyword enriched titles on each page of the site.
3.  Place keyword enriched description and keywords meta tags on each page of the site.
4.  Use robot.txt files to keep Slurp out of your shopping cart or log in pages.
5.  Place keyword enriched text in the first paragraphs of your site-copy.
6.  Use HREF links to direct Slurp through each page of the site.
7.  Add a sitemap page and be certain there is a link from the index page to the sitemap.
8.  Be certain that geographic specific information is mentioned on each page of the site.
9.  Always have a contact page that also lists geographic specific information.
10.  Write a press release and send it to as many blogs, news-wires and press release sites as possible.
11.  Acquire strong, relevant incoming links from sites with topics similar to yours.
12.  Update your site frequently.
13.  Enjoy and value your placements.
[/font]

wow great article.  I will read it again later, but for now...

how would I go about doing this:

4.  Use robot.txt files to keep Slurp out of your shopping cart or log in pages.

I need to forbid robots from looking at any page with an
&act=REG , or
&act=MAIL, or
&act=PRINT in it. 

???

Perhaps these 3 sites will give you a far more in-depth understanding about the robots.txt file(s) and everything associated with it:

Site 1 | Site 2 | Site 3

Great article!  I may just have to publish it in the newsletter if that is ok with you...let me know.

Thanks, Admin -- I'm real easy to get along with; heck, you can copy anything I put here and publish it whenever you want.  I'll follow up over the next 2 weeks with more in-depth articles so the posters can get a better grasp of search engines, and how they are rapidly changing.

I'm glad I stumbled onto this board -- there's a wealth of information coming from many people here!

@travelagentr, unfortunately, those articles you linked don't help me, I already know how to forbid a directory or a specific file extension, but I need to know how to forbid certain query strings which apparently is not possible because I can't find any documentation anywhere which explains how to apply a regular expression in robots.txt.  Thanks though.

Also It might interest you to know, that SLURP is actually an Inktomi product that Yahoo licenses from InktomiSearch.  Yahoo's SLURP is identified by the user agent "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp ) " while Inktomi's SLURP has a user agent of "Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)" or "Slurp.so/1.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)".  SLURP is also used by HotBot and a few others.

Additionally, Yahoo operates a few robots besides SLURP that you may encounter.  YahooFeedSeeker crawls looking only for RSS/XML feeds and can be indentified by the user agent "YahooFeedSeeker".  And YahooMMCrawler can be indentified by the user agent "Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)" or "Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)".

Dang, Mutilated -- thanks for the info on Slurp and Inktomi; I didn't know about that yet!

As for the "query strings" -- I'm just as much in the dark about that as you!

However, I did find a couple of places you can check on that subject:

DevShed Forums | Query strings

I was under the impression that Yahoo actually bought Inktomi.  They may have kept them as a seperate business entity, but Inktomi is part of the Yahoo family.

From inktomi.com....

Copyright © 1996-2004 Inktomi, a Yahoo! company. All Rights Reserved. 

appears you are correct

Travelagent

What a fantastic article. Sure learned a lot from this single article alone

Thanks for sharing

brandon

wow great article.  I will read it again later, but for now...

I'll follow some sugestions

You might find this useful in your endeavor:

The Robots Exclusion Protocol

and in the fight against competition slurp or Yahoo is undergoing algo changes as of recent there is rumour of the yahoo dance, no references yet but check it out there is a real fight going on msn yahoo and ggle although rated no problems i reckon msn have hotted up the battle and in response things have recently changed last few days, last week. see other forum threads

Great article on a particularly intriguing subject. Yahoo is a mistery to me. Everything on the Net is but certain things (including behaviour of the google-bot) I've learned to predict (to a certain extend)... Nothing of the sort with Slurp. It has indexed my pages long ago but refuses to give them any good ranking. I have google #1 pages which aren't shown in yahoo SERPs anywhere visible.

I read that Yahoo's ranking algo put's more weight on onpage factors while inbond links have lesser effect on SERP than in Google. Which is similar to MSN where we have very strong possisions. Strange.

I look forward to read more posts on this.