Hi, I just want to ask, what should I put in my robots.txt??

Hi you can use this plugin, which helps you generation robots.txt

http://octobercms.com/plugin/mohsin-txt

Oh, I see, thank you very much :DDDDDDDD

Place your robots.txt file at the root of your site and paste this in your .htaccess file after the line 'RewriteEngine On':

##
## Allow robots.txt
##
RewriteRule ^robots.txt - [L]

Create a new page, URL = robots.txt. And there to write what they wanted to write robors.txt

Creating a page with URL = robots.txt works but the response in the network requests of the browser is 404 instead of 200.

It is strange as the content of the file still shows. The only issue is the status code which means that google will not be able to see it.

I can create other txt files but any file with a URL of robots.txt always returns a 404

What I put in my development site is:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Additionally, depending on how you have the .htaccess file setup, you may need to give permission for Crawlers/Visitors to access it. In my case I had to add the following under White listed folders:

##
## White listed folders
##
ReWriteCond %{REQUEST_FILENAME} !/robots.txt