This forum has moved to a new location and is in read-only mode. Please visit talk.octobercms.com to access the new location.

marcoelho977747
marcoelho977747

Hi, I just want to ask, what should I put in my robots.txt??

Crazymodder
Crazymodder

Hi you can use this plugin, which helps you generation robots.txt

http://octobercms.com/plugin/mohsin-txt

marcoelho977747
marcoelho977747

Oh, I see, thank you very much :DDDDDDDD

micwanyoike5462
micwanyoike5462

Place your robots.txt file at the root of your site and paste this in your .htaccess file after the line 'RewriteEngine On':

##
## Allow robots.txt
##
RewriteRule ^robots.txt - [L]
Alex.Nov
Alex.Nov

Create a new page, URL = robots.txt. And there to write what they wanted to write robors.txt

info14532
info14532

Creating a page with URL = robots.txt works but the response in the network requests of the browser is 404 instead of 200.

It is strange as the content of the file still shows. The only issue is the status code which means that google will not be able to see it.

I can create other txt files but any file with a URL of robots.txt always returns a 404

FFacebookPro
FFacebookPro

What I put in my development site is:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Additionally, depending on how you have the .htaccess file setup, you may need to give permission for Crawlers/Visitors to access it. In my case I had to add the following under White listed folders:

##
## White listed folders
##
ReWriteCond %{REQUEST_FILENAME} !/robots.txt

1-7 of 7

You cannot edit posts or make replies: the forum has moved to talk.octobercms.com.