Tuesday, 30 September 2014 06:05
beggarly

Google blocked by robots.txt ?

An open forum for opinions and general questions

Google blocked by robots.txt ?

Postby adrien5555 » Tue Jul 24, 2012 4:03 pm

Hello all,

in GWT, when submitting Xmap xml sitemap, google indicates that Robots.txt blocks the crawling of the pages...

here is my robots.txt for the site www.anco.pro:

Code: Select all
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
SITEMAP: http://www.anco.pro/sitemap.xml


note : i just added this last line 'sitmap' following Gui's instructions on the forum, but it doesn't change things up to now.

Do you think this related to Xmap ?

joomla 2.5.4, Xmap 2, yoo template, ace sef
Last edited by adrien5555 on Wed Jul 25, 2012 6:26 pm, edited 1 time in total.
adrien5555
Fresh Boarder
Fresh Boarder
 
Posts: 6
Joined: Wed Jul 04, 2012 1:19 am

Re: Crawling blocked by robots.txt ?

Postby adrien5555 » Wed Jul 25, 2012 6:21 pm

hello,

here is a screenshot of GWT :
http://screencast.com/t/rjDOxm16

since I used the 'Sitemap index technique' described in the doc, Google displays no errors. But none of the URLs are indexed...

please guys, an idea on this ?
thx, adrien
adrien5555
Fresh Boarder
Fresh Boarder
 
Posts: 6
Joined: Wed Jul 04, 2012 1:19 am

Re: Google blocked by robots.txt ?

Postby adrien5555 » Thu Jul 26, 2012 1:03 pm

update :

/index.php?option=com_xmap&view=xml&id=1
is still blocked by robots.txt in Google Webmaster Tools

/sitemap.xml
is accepted, no errors, and URLs are finally indexed. So I removed the first one, leaving only sitemap.xml.

the file sitemap.xml was created thanks to the doc :
http://joomla.vargas.co.cr/en/documentation/34-xmap-2/how-to/108-xmap-sitemap-as-sitemapxml
read section 'Sitemap index'. Note that the first method described, using rewriting in htaccess didn't work for me.

hope it can be helpful ! :D

I dont know why, though, but only 58 out of 77 URLs are indexed...http://screencast.com/t/jLs1VyGueKd
adrien5555
Fresh Boarder
Fresh Boarder
 
Posts: 6
Joined: Wed Jul 04, 2012 1:19 am


Return to General



Who is online

Users browsing this forum: No registered users and 3 guests