+4 votes
in CMS Tips by (40.5k points)
If I check robots.txt of my website using a web browser, I can view it. But I do not see robots.txt in my website's home directory (public_html). It seems that WordPress is generating and using a virtual robots.txt file. How can I create my own robots.txt and use it?

1 Answer

+2 votes
by (349k points)
selected by
 
Best answer
I also found this behavior of WordPress. It generates a virtual robots.txt if you do not have a real robots.txt in the home directory of your website.

To stop WordPress from using the virtual robots.txt, you need to create a robots.txt file in the home directory (public_html). Thus, the web server will serve your robots.txt  instead of WordPress's virtual one.

Related questions


...