robots.txt за wordpress
За да се избегне дублиране на съдържанието и индексирането на ненужни страници прочетох, че е добре robots.txt да изглежда така:
HTML Code:
User-Agent: *
Disallow: /cgi-bin
Disallow: /awstats
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /category
Disallow: /author
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /*/trackback/$
Disallow: /feed/
Disallow: /feed/atom/
Disallow: /feed/rss/
Disallow: /*feed*
Disallow: /*/rss/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/feed/atom/$
Disallow: /rss/
Disallow: /wp-register.php
Disallow: /wp-login.php
Disallow: /comments
Disallow: /comments/feed/
Disallow: */comments
Disallow: /*/comments/feed/$
Disallow: /*?*
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Allow: /wp-content/uploads
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*
# allow Google adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
Понеже не съм 100% сигурен за някои неща, реших да го постна тук преди да съм сътворил простотията 
ФорумиMarch 13, 2010
Leave a Reply