Difference between revisions of "X-Cart:Setting up robots.txt"

From X-Cart 4 Classic
Jump to: navigation, search
m (Dynamic version (without HTML catalog))
 
(15 intermediate revisions by 2 users not shown)
Line 11: Line 11:
 
{{Note|The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.}}
 
{{Note|The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.}}
  
Configuring a store to make it properly visible to search engines takes placing a special text file named [http://en.wikipedia.org/wiki/Robots.txt robots.txt] into your website's root folder. (Please note that the file cannot reside in a sub-folder, such as <u>/store</u>). The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel.
+
Configuring a store to make it properly visible to search engines takes placing a special text file named [https://en.wikipedia.org/wiki/Robots.txt robots.txt] into your website's root folder. Please note that the file cannot reside in a sub-folder, such as <u>/store</u>; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:
 
+
<pre>
The content of robots.txt can vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:
+
Disallow: /store/admin/
 +
</pre>
 +
The content of robots.txt may vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:
  
 
* Dynamic store, generated on the fly from the database.
 
* Dynamic store, generated on the fly from the database.
* Static store, based on a static [http://www.x-cart.com/search.html?query=html+catalog HTML catalog].
+
* Static store, based on a static [https://www.x-cart.com/search.html?query=html+catalog HTML catalog].
  
Here is a working example for each case.
+
Further below you will find a working example for each of the above-said cases.
 +
 
 +
Note that if your store is available via HTTPS, it is strongly recommended to add "Host: https://<your_domain>" to your robots.txt file; for example:
 +
<pre>
 +
Host: https://example.com
 +
</pre>
 +
That is needed to tell Google that the main mirror for your site is the HTTPS version, even though HTTP is still available.
  
 
== Dynamic version (without HTML catalog) ==
 
== Dynamic version (without HTML catalog) ==
Line 33: Line 41:
 
<pre>
 
<pre>
 
User-agent: Googlebot
 
User-agent: Googlebot
Disallow: /*printable=Y*
 
 
Disallow: /*js=*
 
Disallow: /*js=*
 
Disallow: /*print_cat=*
 
Disallow: /*print_cat=*
Line 39: Line 46:
  
 
User-agent: *
 
User-agent: *
 +
Allow: *.js
 +
Allow: *.css
 +
Allow: *.jpg
 +
Allow: *.gif
 +
Allow: *.png
 
Disallow: /admin/
 
Disallow: /admin/
 
Disallow: /catalog/
 
Disallow: /catalog/
 
Disallow: /customer/
 
Disallow: /customer/
 
Disallow: /files/
 
Disallow: /files/
Disallow: /images/
 
 
Disallow: /include/
 
Disallow: /include/
 
Disallow: /mail/
 
Disallow: /mail/
Line 74: Line 85:
 
Disallow: /change_password.php
 
Disallow: /change_password.php
 
Disallow: /check_requirements.php
 
Disallow: /check_requirements.php
 +
Disallow: /choosing.php
 
Disallow: /cleanup.php
 
Disallow: /cleanup.php
 
Disallow: /cmpi_popup.php
 
Disallow: /cmpi_popup.php
 +
Disallow: /comparison.php
 +
Disallow: /comparison_list.php
 
Disallow: /config.php
 
Disallow: /config.php
 
Disallow: /cron.php
 
Disallow: /cron.php
Line 82: Line 96:
 
Disallow: /error_message.php
 
Disallow: /error_message.php
 
Disallow: /featured_products.php
 
Disallow: /featured_products.php
 +
Disallow: /admin/twofactor_token.php
 +
Disallow: /gak_admin.php
 +
Disallow: /provider/twofactor_token.php
 +
Disallow: /twofactor_token.php
 +
 +
Disallow: /giftcert.php
 
Disallow: /get_block.php
 
Disallow: /get_block.php
 
Disallow: /get_info.php
 
Disallow: /get_info.php
 
Disallow: /giftreg_manage.php
 
Disallow: /giftreg_manage.php
 +
Disallow: /giftregs.php
 
Disallow: /https.php
 
Disallow: /https.php
 
Disallow: /image.php
 
Disallow: /image.php
 
Disallow: /init.php
 
Disallow: /init.php
 
Disallow: /install.php
 
Disallow: /install.php
 +
Disallow: /login.php
 
Disallow: /magnifier_xml.php
 
Disallow: /magnifier_xml.php
 +
Disallow: /mailchimp_news.php
 
Disallow: /message.html
 
Disallow: /message.html
 
Disallow: /order.php
 
Disallow: /order.php
Line 144: Line 167:
 
<pre>
 
<pre>
 
User-agent: Googlebot
 
User-agent: Googlebot
Disallow: /*printable=Y*
 
 
Disallow: /*js=*
 
Disallow: /*js=*
 
Disallow: /*print_cat=*
 
Disallow: /*print_cat=*
Line 150: Line 172:
  
 
User-agent: *
 
User-agent: *
 +
Allow: *.js
 +
Allow: *.css
 +
Allow: *.jpg
 +
Allow: *.gif
 +
Allow: *.png
 
Disallow: /admin/
 
Disallow: /admin/
 
Disallow: /customer/
 
Disallow: /customer/
 
Disallow: /files/
 
Disallow: /files/
Disallow: /images/
 
 
Disallow: /include/
 
Disallow: /include/
 
Disallow: /mail/
 
Disallow: /mail/
Line 171: Line 197:
 
Disallow: /COPYRIGHT
 
Disallow: /COPYRIGHT
 
Disallow: /INSTALL
 
Disallow: /INSTALL
Disallow: /NEW-4.4.0
+
Disallow: /NEW-4.5.0
 
Disallow: /README
 
Disallow: /README
 
Disallow: /VERSION
 
Disallow: /VERSION
Line 184: Line 210:
 
Disallow: /change_password.php
 
Disallow: /change_password.php
 
Disallow: /check_requirements.php
 
Disallow: /check_requirements.php
Disallow: /choosing.php
 
 
Disallow: /cleanup.php
 
Disallow: /cleanup.php
 
Disallow: /cmpi_popup.php
 
Disallow: /cmpi_popup.php
Disallow: /comparison.php
 
Disallow: /comparison_list.php
 
 
Disallow: /config.php
 
Disallow: /config.php
 
Disallow: /cron.php
 
Disallow: /cron.php
Disallow: /default_icon.gif
 
Disallow: /default_image.gif
 
Disallow: /default_logo.gif
 
 
Disallow: /dispatcher.php
 
Disallow: /dispatcher.php
 
Disallow: /download.php
 
Disallow: /download.php
Line 200: Line 220:
 
Disallow: /get_block.php
 
Disallow: /get_block.php
 
Disallow: /get_info.php
 
Disallow: /get_info.php
Disallow: /giftcert.php
 
 
Disallow: /giftreg_manage.php
 
Disallow: /giftreg_manage.php
Disallow: /giftregs.php
 
Disallow: /help.php
 
Disallow: /home.php
 
 
Disallow: /https.php
 
Disallow: /https.php
 
Disallow: /image.php
 
Disallow: /image.php
Disallow: /index.php
 
 
Disallow: /init.php
 
Disallow: /init.php
 
Disallow: /install.php
 
Disallow: /install.php
Disallow: /login.php
 
 
Disallow: /magnifier_xml.php
 
Disallow: /magnifier_xml.php
Disallow: /manufacturers.php
 
 
Disallow: /message.html
 
Disallow: /message.html
Disallow: /minicart.php
 
Disallow: /news.php
 
Disallow: /offers.php
 
 
Disallow: /order.php
 
Disallow: /order.php
 
Disallow: /orders.php
 
Disallow: /orders.php
 
Disallow: /pages.php
 
Disallow: /pages.php
Disallow: /pconf.php
 
 
Disallow: /permission_denied.html
 
Disallow: /permission_denied.html
 
Disallow: /popup_address.php
 
Disallow: /popup_address.php
Line 226: Line 235:
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_estimate_shipping.php
 
Disallow: /popup_estimate_shipping.php
Disallow: /popup_fc_products.php
 
 
Disallow: /popup_image.php
 
Disallow: /popup_image.php
 
Disallow: /popup_info.php
 
Disallow: /popup_info.php
Line 234: Line 242:
 
Disallow: /preauth.php
 
Disallow: /preauth.php
 
Disallow: /prepare.php
 
Disallow: /prepare.php
Disallow: /probe.php
 
Disallow: /preview_image.gif
 
 
Disallow: /process_order.php
 
Disallow: /process_order.php
Disallow: /product.php
 
 
Disallow: /products.php
 
Disallow: /products.php
 
Disallow: /products_map.php
 
Disallow: /products_map.php
 
Disallow: /recommends.php
 
Disallow: /recommends.php
 
Disallow: /referer.php
 
Disallow: /referer.php
Disallow: /register.php
 
 
Disallow: /returns.php
 
Disallow: /returns.php
 
Disallow: /search.php
 
Disallow: /search.php
Line 248: Line 252:
 
Disallow: /shop_closed.html
 
Disallow: /shop_closed.html
 
Disallow: /shop_registration.php
 
Disallow: /shop_registration.php
Disallow: /sitemap.php
+
Disallow: /admin/twofactor_token.php
 +
Disallow: /gak_admin.php
 +
Disallow: /provider/twofactor_token.php
 +
Disallow: /twofactor_token.php
 +
 
 
Disallow: /slabel.php
 
Disallow: /slabel.php
 
Disallow: /smarty.php
 
Disallow: /smarty.php
 
Disallow: /survey.php
 
Disallow: /survey.php
 
Disallow: /top.inc.php
 
Disallow: /top.inc.php
Disallow: /vote.php
+
Disallow: /popup_cookie_settings.php
Disallow: /wysiwyg.php
+
Disallow: /flash_container.swf
 +
Disallow: /pplogin_return.php
 +
Disallow: /pplogin_popup.php
 +
Disallow: /mailchimp_news.php
 +
Disallow: /xmonitoring_api.php
 +
Disallow: /cloud_search_api.php
 +
Disallow: /klarna_popup_address.php
 +
Disallow: /xps_subscriptions.php
 +
Disallow: /saved_cards.php
 +
Disallow: /fcommerce.php
 +
Disallow: /shop_closed_evaluation.html
 +
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
 +
Disallow: /xps_cron.php
 +
Disallow: /amazon_checkout.php
 +
Disallow: /VERSION.CLOUD_SEARCH
 
</pre>
 
</pre>
  
 
'''See also:'''
 
'''See also:'''
 
* [[X-Cart:Customer_Reviews#Google_bot_can_vote.2Frate_products_when_the_.22Who_must_be_allowed_to_add_ratings.22_option_is_set_to_.22All_users.22|Google bot can vote/rate products when the "Who must be allowed to add ratings" option is set to "All users" ]]
 
* [[X-Cart:Customer_Reviews#Google_bot_can_vote.2Frate_products_when_the_.22Who_must_be_allowed_to_add_ratings.22_option_is_set_to_.22All_users.22|Google bot can vote/rate products when the "Who must be allowed to add ratings" option is set to "All users" ]]
 
== Blocking the secure part ==
 
 
Since the checkout is normally carried out over a secure protocol, the majority of e-commerce websites can be reached via both plain and secure HTTP. Some search robots can treat this as duplicate content, as all the pages available over HTTP can be accessed via the secure protocol as well. However, if your web server allows using mod_rewrite, all the secure pages can be easily blocked.
 
 
To do so, first of all, you need to create a file called .htaccess that contains the following code (or add this code to your existing .htaccess file):
 
 
=== .htaccess ===
 
 
<pre>
 
RewriteEngine on
 
RewriteCond %{HTTPS} on
 
RewriteRule ^robots\.txt$ robots-https.txt
 
</pre>
 
 
Next, create a file to be used instead of the '''robots.txt''' when the store is accessed via the secure protocol and name that file '''robots-https.txt''':
 
 
=== robots-https.txt ===
 
 
<pre>
 
User-agent: *
 
Disallow: /
 
</pre>
 
 
Finally, upload both these files along with the current version of '''robots.txt''' to the root of your website.
 
 
That's it.
 
 
If you ever need help setting up the robots.txt file for your online store, please, do not hesitate to contact our technical support for assistance. Our specialists will examine your store and make up a file that would be the most suitable from search engines' point of view.
 
  
 
== See also ==
 
== See also ==
  
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
+
* [https://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
* [http://www.google.com/support/webmasters/bin/topic.py?topic=8475 Using the robots.txt analysis tool]
+
* [https://www.google.com/support/webmasters/bin/topic.py?topic=8475 Using the robots.txt analysis tool]
* [http://www.searchtools.com/robots/robots-txt.html#checkers Robots.txt Checkers]
+
* [https://www.searchtools.com/robots/robots-txt.html#checkers Robots.txt Checkers]
  
 
[[Category:X-Cart user manual]]
 
[[Category:X-Cart user manual]]

Latest revision as of 18:03, 22 July 2020

About robots.txt

Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index.

Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.

Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.

Below we describe how to configure an X-Cart based online store for proper search engine indexing.

Note: The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.

Configuring a store to make it properly visible to search engines takes placing a special text file named robots.txt into your website's root folder. Please note that the file cannot reside in a sub-folder, such as /store; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:

Disallow: /store/admin/

The content of robots.txt may vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:

  • Dynamic store, generated on the fly from the database.
  • Static store, based on a static HTML catalog.

Further below you will find a working example for each of the above-said cases.

Note that if your store is available via HTTPS, it is strongly recommended to add "Host: https://<your_domain>" to your robots.txt file; for example:

 
Host: https://example.com

That is needed to tell Google that the main mirror for your site is the HTTPS version, even though HTTP is still available.

Dynamic version (without HTML catalog)

If you are not using HTML catalog, only the following pages should be included in the index:

  • home and category pages (home.php)
  • product pages (product.php)
  • static pages (pages.php)
  • manufacturers (manufacturers.php)

The rest of the pages available on the website are to be hidden from search engines. Keeping this in mind, here is the configuration we suggest for a store with dynamically generated content:

User-agent: Googlebot
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Allow: *.js
Allow: *.css
Allow: *.jpg
Allow: *.gif
Allow: *.png
Disallow: /admin/
Disallow: /catalog/
Disallow: /customer/
Disallow: /files/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /choosing.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /comparison.php 
Disallow: /comparison_list.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /admin/twofactor_token.php
Disallow: /gak_admin.php
Disallow: /provider/twofactor_token.php
Disallow: /twofactor_token.php

Disallow: /giftcert.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /giftregs.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /login.php
Disallow: /magnifier_xml.php
Disallow: /mailchimp_news.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

Static version (using HTML catalog)

For the static version, only the corresponding HTML part is to be included in the index, and all the dynamic (PHP-based) part can be hidden from search engines:

User-agent: Googlebot
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Allow: *.js
Allow: *.css
Allow: *.jpg
Allow: *.gif
Allow: *.png
Disallow: /admin/
Disallow: /customer/
Disallow: /files/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /magnifier_xml.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /admin/twofactor_token.php
Disallow: /gak_admin.php
Disallow: /provider/twofactor_token.php
Disallow: /twofactor_token.php

Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

See also:

See also