June 8, 2018
Is there a basic setting for the robots.txt file? In google webmaster I get alot of errors concerning the robots.txt file with different sites concerning the “no index” error.
For example in one site’s robot.txt file there is this line among many:
seems like I would want google bots in the content section of a site? So this site seems to need just a basic robots.txt file.
In another site this is all I have in the robots.txt file
and this site has 51 errors concerning mostly concerning /category
google webmaster says this: No: ‘noindex’ detected in ‘robots’ meta tag
Is this in the above robots.txt file displayed or is there a setting I’m missing someplace else?