If you value SEO (search engine rankings), then your XF site should probably have solid robots.txt settings. I have spent a considerable amount of time learning about SEO and examining detailed crawl logs of my site. Based on my learnings I have updated my robots.txt file as shown below.
For those who do not know, a robots.txt file tells search engines what parts of the site they can and cannot explore. These are recommendations to the search engine crawlers which can be ignored, but...
Read more
Continue reading...
For those who do not know, a robots.txt file tells search engines what parts of the site they can and cannot explore. These are recommendations to the search engine crawlers which can be ignored, but...
Read more
Continue reading...