robots.txt file basic setting | Optimizing All in One SEO Pack Pro | Support | Semper Plugins

Lost password?
Advanced Search
Forum Scope


Forum Options

Minimum search word length is 3 characters - maximum search word length is 84 characters
The forums are currently locked and only available for read only access
sp_Feed Topic RSS sp_TopicIcon
robots.txt file basic setting
July 11, 2018
12:03 pm
Gary Leggett
Forum Posts: 12
Member Since:
June 8, 2018
sp_UserOfflineSmall Offline


Is there a basic setting for the robots.txt file?  In google webmaster I get alot of errors concerning the robots.txt file with different sites concerning the “no index” error.

For example in one site’s robot.txt file there is this line among many:

Disallow: /wp-content   

seems like I would want google bots in the content section of a site?  So this site seems to need just a basic robots.txt file.


In another site this is all I have in the robots.txt file

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php



and this site has 51 errors concerning mostly concerning  /category

an example:…

google webmaster says this:  No: ‘noindex’ detected in ‘robots’ meta tag

Is this in the above robots.txt file displayed or is there a setting I’m missing someplace else?


July 11, 2018
12:20 pm
Steve Mortiboy
Forum Posts: 2204
Member Since:
January 5, 2012
sp_UserOfflineSmall Offline

** You currently do not have permission to see this post. To obtain access to these Premium Support Forums, please purchase a plugin and support subscription from Semper Plugins **

Forum Timezone: America/New_York
Most Users Ever Online: 964
Currently Online:
Currently Browsing this Page:
1 Guest(s)
Top Posters:
Member Stats:
Guest Posters: 36
Members: 23928
Moderators: 1
Admins: 7
Forum Stats:
Groups: 4
Forums: 28
Topics: 2143
Posts: 9332
Newest Members:
Manda Razaka
Moderators: semperplugins: 0
Administrators: Michael TorbertPeter BayliesSupport TeamSteve MortiboyMatthew RobinsonSteve KlasenArnaud Broes
Skip to toolbar