Google blocked by robots.txt ?
3 posts
• Page 1 of 1
Google blocked by robots.txt ?
Hello all,
in GWT, when submitting Xmap xml sitemap, google indicates that Robots.txt blocks the crawling of the pages...
here is my robots.txt for the site www.anco.pro:
note : i just added this last line 'sitmap' following Gui's instructions on the forum, but it doesn't change things up to now.
Do you think this related to Xmap ?
joomla 2.5.4, Xmap 2, yoo template, ace sef
in GWT, when submitting Xmap xml sitemap, google indicates that Robots.txt blocks the crawling of the pages...
here is my robots.txt for the site www.anco.pro:
- Code: Select all
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
SITEMAP: http://www.anco.pro/sitemap.xml
note : i just added this last line 'sitmap' following Gui's instructions on the forum, but it doesn't change things up to now.
Do you think this related to Xmap ?
joomla 2.5.4, Xmap 2, yoo template, ace sef
Last edited by adrien5555 on Wed Jul 25, 2012 6:26 pm, edited 1 time in total.
- adrien5555
- Fresh Boarder

- Posts: 6
- Joined: Wed Jul 04, 2012 1:19 am
Re: Crawling blocked by robots.txt ?
hello,
here is a screenshot of GWT :
http://screencast.com/t/rjDOxm16
since I used the 'Sitemap index technique' described in the doc, Google displays no errors. But none of the URLs are indexed...
please guys, an idea on this ?
thx, adrien
here is a screenshot of GWT :
http://screencast.com/t/rjDOxm16
since I used the 'Sitemap index technique' described in the doc, Google displays no errors. But none of the URLs are indexed...
please guys, an idea on this ?
thx, adrien
- adrien5555
- Fresh Boarder

- Posts: 6
- Joined: Wed Jul 04, 2012 1:19 am
Re: Google blocked by robots.txt ?
update :
/index.php?option=com_xmap&view=xml&id=1
is still blocked by robots.txt in Google Webmaster Tools
/sitemap.xml
is accepted, no errors, and URLs are finally indexed. So I removed the first one, leaving only sitemap.xml.
the file sitemap.xml was created thanks to the doc :
http://joomla.vargas.co.cr/en/documentation/34-xmap-2/how-to/108-xmap-sitemap-as-sitemapxml
read section 'Sitemap index'. Note that the first method described, using rewriting in htaccess didn't work for me.
hope it can be helpful !
I dont know why, though, but only 58 out of 77 URLs are indexed...http://screencast.com/t/jLs1VyGueKd
/index.php?option=com_xmap&view=xml&id=1
is still blocked by robots.txt in Google Webmaster Tools
/sitemap.xml
is accepted, no errors, and URLs are finally indexed. So I removed the first one, leaving only sitemap.xml.
the file sitemap.xml was created thanks to the doc :
http://joomla.vargas.co.cr/en/documentation/34-xmap-2/how-to/108-xmap-sitemap-as-sitemapxml
read section 'Sitemap index'. Note that the first method described, using rewriting in htaccess didn't work for me.
hope it can be helpful !
I dont know why, though, but only 58 out of 77 URLs are indexed...http://screencast.com/t/jLs1VyGueKd
- adrien5555
- Fresh Boarder

- Posts: 6
- Joined: Wed Jul 04, 2012 1:19 am
3 posts
• Page 1 of 1
Who is online
Users browsing this forum: No registered users and 2 guests