robots.txt file basic setting | Optimizing All in One SEO Pack Pro | Support | Semper Plugins

Avatar
Lost password?
Advanced Search
Forum Scope


Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Feed Topic RSS sp_TopicIcon
robots.txt file basic setting
July 11, 2018
12:03 pm
Avatar
Gary Leggett
Member
AIOSEOP
Forum Posts: 12
Member Since:
June 8, 2018
sp_UserOfflineSmall Offline

Hi,

Is there a basic setting for the robots.txt file?  In google webmaster I get alot of errors concerning the robots.txt file with different sites concerning the “no index” error.

For example in one site’s robot.txt file there is this line among many:

Disallow: /wp-content   

seems like I would want google bots in the content section of a site?  So this site seems to need just a basic robots.txt file.

—-

In another site this is all I have in the robots.txt file

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: http://aidomes.com/sitemap.xml.gz

—————-

and this site has 51 errors concerning mostly concerning  /category

an example:

http://aidomes.com/category/s6…..us/page/2/

google webmaster says this:  No: ‘noindex’ detected in ‘robots’ meta tag

Is this in the above robots.txt file displayed or is there a setting I’m missing someplace else?

Thanks

July 11, 2018
12:20 pm
Avatar
Steve Mortiboy
Admin
Forum Posts: 2069
Member Since:
January 5, 2012
sp_UserOfflineSmall Offline

** You currently do not have permission to see this post. To obtain access to these Premium Support Forums, please purchase a plugin and support subscription from Semper Plugins **

Forum Timezone: America/New_York
Most Users Ever Online: 964
Currently Online:
30
Guest(s)
Currently Browsing this Page:
1 Guest(s)
Top Posters:
Member Stats:
Guest Posters: 35
Members: 21706
Moderators: 0
Admins: 8
Forum Stats:
Groups: 4
Forums: 28
Topics: 2025
Posts: 8763
Newest Members:
義榮 廖
Administrators: Michael TorbertPeter BayliesSupport TeamSteve MortiboyMatthew RobinsonSteve Klasen, Arnaud: 197Ashish Ravi
[i]
[i]
Skip to toolbar