+3 votes
in Web & Google by (14.6k points)
What code should I use in the robots.txt to allow or disallow all pages of my website to all bots? I am using the Apache web server.

1 Answer

+1 vote
by (48.9k points)

If you want to allow every page to all bots, you can use the following code in the robots.txt file.

User-agent: *
Disallow:

If you want to disallow every page to all bots, you can use the following code in the robots.txt file.

User-agent: *
Disallow: /


...