robots.txt file basic setting | Optimizing All in One SEO Pack Pro | Support | Semper Plugins


Lost password?
Advanced Search

— Forum Scope —

— Match —

— Forum Options —

Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed Topic RSS sp_TopicIcon
robots.txt file basic setting
July 11, 2018
12:03 pm
Gary Leggett
Forum Posts: 12
Member Since:
June 8, 2018
sp_UserOfflineSmall Offline


Is there a basic setting for the robots.txt file?  In google webmaster I get alot of errors concerning the robots.txt file with different sites concerning the “no index” error.

For example in one site’s robot.txt file there is this line among many:

Disallow: /wp-content   

seems like I would want google bots in the content section of a site?  So this site seems to need just a basic robots.txt file.


In another site this is all I have in the robots.txt file

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php



and this site has 51 errors concerning mostly concerning  /category

an example:…

google webmaster says this:  No: ‘noindex’ detected in ‘robots’ meta tag

Is this in the above robots.txt file displayed or is there a setting I’m missing someplace else?


July 11, 2018
12:20 pm
Steve Mortiboy
Forum Posts: 2051
Member Since:
January 5, 2012
sp_UserOfflineSmall Offline

** You currently do not have permission to see this post. To obtain access to these Premium Support Forums, please purchase a plugin and support subscription from Semper Plugins **

Forum Timezone: America/New_York

Most Users Ever Online: 964

Currently Online: niket
35 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

Member Stats:

Guest Posters: 35

Members: 21256

Moderators: 1

Admins: 8

Forum Stats:

Groups: 4

Forums: 28

Topics: 2002

Posts: 8665

Newest Members:

Robert Berry

Moderators: Dawa Liu

Administrators: Michael TorbertPeter BayliesSupport TeamSteve MortiboyMatthew RobinsonSteve KlasenArnaud BroesAshish Ravi

Skip to toolbar