Site Reference Forums

CeeCee · 2008-08-07 09:15:28

This is the message I received when scrubtheweb.com analyzed my new site. I'll explain my problem with geocities after scrubtheweb's report.

----------scrubtheweb report----- geocities.com/for_the_usa
Note! This site does NOT provide the robots.txt file or the robots.txt file is not constructed properly. It is
recommended all sites provide this file.
Redirection like this can cause problems with all major search engines and is the reason for our warning. The report below is based on the information found at the final URL location shown above and is the URL we recommend is submitted to search engines until you solve this problem.
It is recommended you verify the URL before submitting to search engines. Not many search engines will follow redirects. Note! The redirected URL may not have been checked against the robots.txt file to see if it is accessible by robots. You should verify the above redirected URL is allowing search engine access before submitting.
---------

I can't access the above redirected URL. It is no longer in file manager. When I created the page, it had to be named ....com/for_the_usa/index.html

As far as I know, I have no control over the fact that ...com/for_the_usa is a redirect page. It still can be accessed if you type in the URL. It also shows up in my stats.

index.html works also and is in the stats.

I don't have a robots.txt tag. My concern is that my robots.txt tag would have to be at ...com/for_the_usa/index.html
not at ....com/for_the_usa so I hope I would not accidentally tell the robots not to index ....com/for_the_usa/index.html.

Also since I don't have access to the page without the word, index.html, how can I tell the search engines it is not a redirect page. Geocities seems to insist that it is a redirect page.

Geocities did not respond to my concerns. I hope you can help.

I wonder if this is just a geocities "thing" and can be ignored! I'm sure many site owners have this problem with geocities.

Last edited by CeeCee (2008-08-07 09:42:15)

howrank · 2008-08-07 09:49:30

Robots.txt is a file named.... robots.txt that gives direction to bots. Lack of one will just allow all bots to crawl. not a big deal.

http://en.wikipedia.org/wiki/Robots.txt

If you do not want to have a page indexed try using
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

in the M <head> of the page.

Last edited by howrank (2008-08-07 09:50:45)

CeeCee · 2008-08-07 15:33:59

So scrubtheweb is making much ado over nothing? They made me feel like I was doing something terrible.

I just started the site a couple of weeks ago and only have 6 pages so far. I want all of them spidered. I don't anticipate having any pages which I don't want to be public. It is largely a place to put my amazon.com bookstore but I include a few links and free information which I think my visitors will like.

Oh, I have another question.
I already submitted
...com/for_the_usa Do I need to go back and submit the index page?
....com/for_the_usa/index.html

Scrubby really has me confused!

Thanks for the info including the link which I visited.
CeeCee

Last edited by CeeCee (2008-08-07 15:39:17)

Northie · 2008-08-07 16:41:25

the robots.txt file is a different issue to the redirect

Doing a quick header scan for [http://geocities.com/for_the_usa] gives

Code:

 [0] => HTTP/1.1 301 Moved Permanently

 [1] => Date: Thu, 07 Aug 2008 21:34:03 GMT

 [2] => P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV"

 [3] => Location: http://geocities.com/for_the_usa/

 [4] => Cache-Control: private

 [5] => Connection: close

 [6] => Content-Type: text/html

 [7] => HTTP/1.1 200 OK

 [8] => Date: Thu, 07 Aug 2008 21:34:04 GMT

 [9] => P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV"

[10] => X-Host: w73.geo.scd.yahoo.com

[11] => X-INKT-URI: http://us.geocities.com/for_the_usa/index.html

[12] => X-INKT-SITE: http://us.geocities.com/for_the_usa

[13] => Last-Modified: Wed, 06 Aug 2008 15:00:38 GMT

[14] => Accept-Ranges: bytes

[15] => Content-Length: 8444

[16] => Cache-Control: private

[17] => Connection: close

[18] => Content-Type: text/html

Notice that you are immediately redirected to the site, but with a trailing slash ('/') - points [0] and [3]

The slash usually means the root level of a folder, but because your site isn't really a folder on geocities (all database driven) it means that 'for_the_usa' is part of the querystring, and to make it look like a folder the user agent is redirected to the url with a trailing '/'. This ensures that when you link to books and music, for example, you are in fact linking to

geocities.com/for_the_usa/books-music.html

and not

geocities.com/for_the_usabooks-music.html

LinkPopularity · 2008-08-07 23:59:15

to own your own domain name and a hosting is really cheap; the limitations from using some kind of free web service are too big ... if you look around a bit you can have website for $50 per year or, that can't be too much ...

CeeCee · 2008-08-08 09:26:32

Hi Northie,

Thanks for the information.

So I don't need to (and can't) do anything about the page with and without
/index.html, right?

Just curious, how can I do the header scan which you did?

CeeCee

Northie · 2008-08-12 16:28:42

CeeCee wrote:
Hi Northie,

Thanks for the information.

So I don't need to (and can't) do anything about the page with and without
/index.html, right?

Just curious, how can I do the header scan which you did?

CeeCee

http://www.php.net/manual/en/function.get-headers.php

if you need to install php, i suggest installing xampp from apache friends on your computer. This will allow you to simulate a web host on your own computer for development and testing purposes

howrank · 2008-08-15 23:11:33

You ever get this thing figured out?

Sponsored Results

Site Reference Forums

#1 2008-08-07 09:15:28

Geocities insists on redirect. What can I do?

#2 2008-08-07 09:49:30

Re: Geocities insists on redirect. What can I do?

#3 2008-08-07 15:33:59

Re: Geocities insists on redirect. What can I do?

#4 2008-08-07 16:41:25

Re: Geocities insists on redirect. What can I do?

Code:

#5 2008-08-07 23:59:15

Re: Geocities insists on redirect. What can I do?

#6 2008-08-08 09:26:32

Re: Geocities insists on redirect. What can I do?

#7 2008-08-12 16:28:42

Re: Geocities insists on redirect. What can I do?

CeeCee wrote:

#8 2008-08-15 23:11:33

Re: Geocities insists on redirect. What can I do?

Active Discussions

Review Requests and Introductions

Board footer