Menü schliessen
Created: December 9th 2011
Last updated: May 1st 2020
Categories: Joomla,  Typo3
Author: Marcus Fleuti

Joomla and Typo3 robots.txt default configuration

Tags:  Joomla,  robots.txt,  Typo3
Donation Section: Background
Monero Badge: QR-Code
Monero Badge: Logo Icon Donate with Monero Badge: Logo Text
82uymVXLkvVbB4c4JpTd1tYm1yj1cKPKR2wqmw3XF8YXKTmY7JrTriP4pVwp2EJYBnCFdXhLq4zfFA6ic7VAWCFX5wfQbCC

Why use a robots.txt?

Actually, there are several reasons:

  1. Reduce bandwidth usage by prohibiting various search engines to index files and folders that don't need to be indexed
  2. Get better results in search results
  3. Control which content shall be displayed in search results (e.g. perhaps you don't want google to index your picture gallery or precious graphical illustrations)
  4. there may be more...

Robots.txt for Joomla

In our Joomla projects we found out that the following robots.txt configuration excludes all of the Joomla default folders that do not need to be indexed by default:

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /cgi-bin/
Disallow: /stats

Robots.txt for Typo3

We normally have a very narrow robots.txt setup in Typo3 which looks like this

User-agent: *
Disallow: /Proselect/
Disallow: /fileadmin/templates/
Disallow: /typo3/
Disallow: /t3lib/
Disallow: /typo3conf/
Disallow: /typo3temp/

Howto install the robots.txt

To install a robots text just copy/paste the above code lines into a textfile and name it - robots.txt
Copy that file to your web projects'  public html root folder and try to fetch the file by entering the URL to it in the browser:

http://yourdomain.com/robots.txt

If you can see the text in the browser you're all set!