Sunday, 21 December 2014 17:18
beggarly

Google submission fails - robots.txt unreachable

An open forum for opinions and general questions

Google submission fails - robots.txt unreachable

Postby ytcadmin » Sun Mar 23, 2008 2:20 am

Hi,
I've installed your Xmap 1.0.4 component into my Joomla 1.0.13 site, and added a menu item pointing to the component. This displays the html version nicely. Thanks.

However the submission of the xml page to Google is producing errors.

I have found the xml page url via the preferences section, and submitted
index.php?option=com_xmap&sitemap=1&view=xml&no_html=1
to google.

The sitemap submission fails, producing the following error:

**start of error message***

Property - Status
Sitemap type - Web
Submitted - Mar 22, 2008
Last downloaded by Google - Mar 22, 2008
Total URLs in Sitemap - 0
Indexed URLs in Sitemap - -

Network unreachable: robots.txt unreachable
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

**end of error message***

The robots.txt file is in the root directory, and the google webmaster tool 'Analyze robots.txt' shows no problems with it at all.

In addition, I can click on the xml url from within the Xmap preferences screen and it loads fine within Internet Explorer. url:

http://www.yourtradingcoach.com/index.php?option=com_xmap&sitemap=1&view=xml&no_html=1

However after submitting index.php?option=com_xmap&sitemap=1&view=xml&no_html=1 to google, Google Webmaster Tools then displays the sitemap xml page as:

Currently viewing: http://yourtradingcoach.com/index.php?option=com_xmap&sitemap=1&view=xml&no_html=1

For some reason, Google Webmaster Tools has dropped the www. at the start of the url. Clicking this link doesn't work and gives the error message below:

**start of error message***

The XML page cannot be displayed
Cannot view XML input using XSL style sheet. Please correct the error and then click the Refresh button, or try again later.
________________________________________
Access is denied. Error processing resource 'http://www.yourtradingcoach.com/components/com_xmap/gss.xsl'.

**end of error message***

Any ideas?
Thanks,
ytcadmin
ytcadmin
Fresh Boarder
Fresh Boarder
 
Posts: 4
Joined: Sun Mar 23, 2008 2:42 am

Re:Google submission fails - robots.txt unreachabl

Postby pacmon » Tue Mar 25, 2008 3:45 pm

Login to your webmaster tools. Under the tools section on the left, goto 'Set Preferred Domain' and make sure you've selected to use www.yourdomain.com (as opposed to yourdomain.com) as your preferred. It's possible that this is where your problem is.
pacmon
Fresh Boarder
Fresh Boarder
 
Posts: 3
Joined: Fri Mar 21, 2008 11:51 pm

Re:Google submission fails - robots.txt unreachabl

Postby ytcadmin » Wed Mar 26, 2008 6:57 am

Hi pacmon,

Thanks for the reply. I've changed the 'preferred domain' as suggested, and resubmitted the sitemap. The sitemap was accepted this time, however shows zero url's. It appears though that it's still trying to access the non-www domain rather than the www one. Maybe it takes a while for this change to come into effect.

Interestingly the Analyze Robots tool also shows a blank file, because it's trying to access the non-www file rather than the www one.

I've added a 301 redirect for all non-www pages to www pages in my .htaccess page, and resubmitted the sitemap again. Hopefully this time even if it tries to get the non-www domain it'll be redirected to the correct one.

I'll provide an update tomorrow if this works.

Cheers,
ytcadmin
ytcadmin
Fresh Boarder
Fresh Boarder
 
Posts: 4
Joined: Sun Mar 23, 2008 2:42 am

Re:Google submission fails - robots.txt unreachabl

Postby ytcadmin » Thu Mar 27, 2008 5:16 am

Update:

The sitemap is accepted (Status OK), and last downloaded 3 hours ago, however it's still not working.

Although it's getting no errors ('Sitemap errors and warnings' states 'No errors or warnings found.' ), it appears to be not picking up any of the contents of the sitemap. The google webmaster tools Sitemap details page displays:

Total URLs in Sitemap 0
Indexed URLs in Sitemap -

So what's going wrong?

In looking at the 'details' screen, it's still listing the sitemap url without the www. That is,

http://yourtradingcoach.com/index.php?option=com_xmap&sitemap=1&view=xml&no_html=1

My prefered domain has been set to www.yourtradingcoach.com for about 36 hours now. For some reason it's still not picking up the www.

I don't imagine this is a problem because my .htaccess has a 301 redirect that works fine, to ensure any non-www name goes to the www equivalent. Is the sitemap system set up to not follow a 301 redirect? If so, how do I get google to look at the right url (this might not be the right place for that question)?

Any suggestions would be greatly appreciated.

Cheers,
ytcadmin
Last edited by ytcadmin on Thu Jan 01, 1970 12:00 am, edited 65535 times in total.
Reason: 1
ytcadmin
Fresh Boarder
Fresh Boarder
 
Posts: 4
Joined: Sun Mar 23, 2008 2:42 am

Re:Google submission fails - robots.txt unreachabl

Postby guilleva » Thu Mar 27, 2008 2:09 pm

I think google don't like the 301 redirections, the preferred domain is used in google's search result and not for the sitemap url. When you created the site on Google Webmasters Tools you wrote your domain name without "www." at te beginning, you'll to delete your site and re-create it with "www.".
User avatar
guilleva
Administrator
Administrator
 
Posts: 1527
Joined: Wed Sep 12, 2007 3:10 am
Location: San José, Costa Rica

Re:Google submission fails - robots.txt unreachabl

Postby ytcadmin » Fri Mar 28, 2008 3:38 am

Thanks guilleva,

I've deleted the non-www site from Google Webmaster Tools, and just got the www version there now. Still the same problems.

This appears to be a problem with my site & Google, rather than an Xmap problem, as the sitemap is being generated correctly - it's just not being picked up by google.

I'm attempting to resolve this through GoDaddy (like banging my head against a wall) and through Google Discussion Groups.

Once resolved, I'll post an answer here in case anyone else gets similar issues.

Thanks for all your help.

Cheers,
ytcadmin
ytcadmin
Fresh Boarder
Fresh Boarder
 
Posts: 4
Joined: Sun Mar 23, 2008 2:42 am


Return to General



Who is online

Users browsing this forum: Bing [Bot] and 1 guest