Hi, what is your "best practice" with /robots.txt file?

I have created CMS page with URL /robots.txt and Response:: like this:

function onStart(){
    $content = "User-agent: *\nAllow:";
    return Response::make($content)->header('Content-Type', 'text/plain');
}

This allows me to easily change content of robots.txt from backend.

I know about Mohsin's TXT plugin, but I wanted template based solution.

Last updated 7 years ago

Wow, thanks, that's very easy. I've noticed you can make RSS, any xml exports like this as well.

https://octobercms.com/forum/post/best-practice-on-using-rainlabsitemap-with-multilanguage-cms-pages

Regarding to this: maybe it is possible to use 3 sitemap.xml's on the website, defined in robots.txt ?! Then each sitemap will have the language version of it, but the only small issue I see here is to change routes in sitemap plugin with hard coded language in it., and even this seems like a easely doable.

Excellent. Would be better that october let us create .txt file and automatically interpret it as plain text (actually it throw an error saying october can just accept .htm page), but your snipped do the job.

jan-vince said:

Hi, what is your "best practice" with /robots.txt file?

I have created CMS page with URL /robots.txt and Response:: like this:

function onStart(){ $content = "User-agent: *\nAllow:"; return Response::make($content)->header('Content-Type', 'text/plain'); }

This allows me to easily change content of robots.txt from backend.

I know about Mohsin's TXT plugin, but I wanted template based solution.

Instead of putting the content directly in php code, you can put it in the "Markup" tag with this code :

function onStart() {
  $content = $this->page->markup;
  return Response::make($content)->header('Content-Type', 'text/plain');
}

Last updated 7 years ago

Use this plugin http://octobercms.com/plugin/zen-robots

See also: https://octobercms.com/forum/post/robotstxt