Difference between revisions of "X-Cart:Setting up robots.txt"

From X-Cart 4 Classic
Jump to: navigation, search
m (Reverted edits by Dohtur (talk) to last revision by Seyfin)
m (Undo revision 15064 by Dohtur (talk))
Line 1: Line 1:
 
== About robots.txt ==
 
== About robots.txt ==
  
Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information, relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, and therefore, it's quite reasonable to include only content-rich parts of your website in the search engines' index.
+
Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index.
  
Another important action, highly recommended for successful search engine indexing, is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.
+
Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.
  
 
Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.
 
Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.
Line 9: Line 9:
 
Below we describe how to configure an X-Cart based online store for proper search engine indexing.
 
Below we describe how to configure an X-Cart based online store for proper search engine indexing.
  
{{Note|The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may slightly vary.}}
+
{{Note|The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.}}
  
Configuring a store to make it properly visible to search engines takes placing a special text file named [http://en.wikipedia.org/wiki/Robots.txt robots.txt] into your website's root folder. (Please note that the file cannot reside in a sub-folder, such as <u>/store</u>). The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel.
+
Configuring a store to make it properly visible to search engines takes placing a special text file named [http://en.wikipedia.org/wiki/Robots.txt robots.txt] into your website's root folder. Please note that the file cannot reside in a sub-folder, such as <u>/store</u>; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:
 
+
<pre>
The content of robots.txt can widely vary to meet your specific needs. In our case, the main differences occur in two cases:
+
Disallow: /store/admin/
 +
</pre>
 +
The content of robots.txt can vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:
  
 
* Dynamic store, generated on the fly from the database.
 
* Dynamic store, generated on the fly from the database.
Line 61: Line 63:
 
Disallow: /COPYRIGHT
 
Disallow: /COPYRIGHT
 
Disallow: /INSTALL
 
Disallow: /INSTALL
Disallow: /NEW-4.4.0
+
Disallow: /NEW-4.5.0
 
Disallow: /README
 
Disallow: /README
 
Disallow: /VERSION
 
Disallow: /VERSION
Line 74: Line 76:
 
Disallow: /change_password.php
 
Disallow: /change_password.php
 
Disallow: /check_requirements.php
 
Disallow: /check_requirements.php
Disallow: /choosing.php
 
 
Disallow: /cleanup.php
 
Disallow: /cleanup.php
 
Disallow: /cmpi_popup.php
 
Disallow: /cmpi_popup.php
Disallow: /comparison.php
 
Disallow: /comparison_list.php
 
 
Disallow: /config.php
 
Disallow: /config.php
 
Disallow: /cron.php
 
Disallow: /cron.php
Disallow: /default_icon.gif
 
Disallow: /default_image.gif
 
Disallow: /default_logo.gif
 
 
Disallow: /dispatcher.php
 
Disallow: /dispatcher.php
 
Disallow: /download.php
 
Disallow: /download.php
Line 90: Line 86:
 
Disallow: /get_block.php
 
Disallow: /get_block.php
 
Disallow: /get_info.php
 
Disallow: /get_info.php
Disallow: /giftcert.php
 
 
Disallow: /giftreg_manage.php
 
Disallow: /giftreg_manage.php
Disallow: /giftregs.php
 
Disallow: /help.php
 
 
Disallow: /https.php
 
Disallow: /https.php
 
Disallow: /image.php
 
Disallow: /image.php
Disallow: /index.php
 
 
Disallow: /init.php
 
Disallow: /init.php
 
Disallow: /install.php
 
Disallow: /install.php
Disallow: /login.php
 
 
Disallow: /magnifier_xml.php
 
Disallow: /magnifier_xml.php
 
Disallow: /message.html
 
Disallow: /message.html
Disallow: /minicart.php
 
Disallow: /news.php
 
Disallow: /offers.php
 
 
Disallow: /order.php
 
Disallow: /order.php
 
Disallow: /orders.php
 
Disallow: /orders.php
 
Disallow: /pages.php
 
Disallow: /pages.php
Disallow: /pconf.php
 
 
Disallow: /permission_denied.html
 
Disallow: /permission_denied.html
 
Disallow: /popup_address.php
 
Disallow: /popup_address.php
Line 114: Line 101:
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_estimate_shipping.php
 
Disallow: /popup_estimate_shipping.php
Disallow: /popup_fc_products.php
 
 
Disallow: /popup_image.php
 
Disallow: /popup_image.php
 
Disallow: /popup_info.php
 
Disallow: /popup_info.php
Line 122: Line 108:
 
Disallow: /preauth.php
 
Disallow: /preauth.php
 
Disallow: /prepare.php
 
Disallow: /prepare.php
Disallow: /probe.php
 
Disallow: /preview_image.gif
 
 
Disallow: /process_order.php
 
Disallow: /process_order.php
 
Disallow: /products.php
 
Disallow: /products.php
Line 129: Line 113:
 
Disallow: /recommends.php
 
Disallow: /recommends.php
 
Disallow: /referer.php
 
Disallow: /referer.php
Disallow: /register.php
 
 
Disallow: /returns.php
 
Disallow: /returns.php
 
Disallow: /search.php
 
Disallow: /search.php
Line 135: Line 118:
 
Disallow: /shop_closed.html
 
Disallow: /shop_closed.html
 
Disallow: /shop_registration.php
 
Disallow: /shop_registration.php
Disallow: /sitemap.php
 
 
Disallow: /slabel.php
 
Disallow: /slabel.php
 
Disallow: /smarty.php
 
Disallow: /smarty.php
 
Disallow: /survey.php
 
Disallow: /survey.php
 
Disallow: /top.inc.php
 
Disallow: /top.inc.php
Disallow: /vote.php
+
Disallow: /popup_cookie_settings.php
Disallow: /wysiwyg.php
+
Disallow: /flash_container.swf
 +
Disallow: /pplogin_return.php
 +
Disallow: /pplogin_popup.php
 +
Disallow: /mailchimp_news.php
 +
Disallow: /xmonitoring_api.php
 +
Disallow: /cloud_search_api.php
 +
Disallow: /klarna_popup_address.php
 +
Disallow: /xps_subscriptions.php
 +
Disallow: /saved_cards.php
 +
Disallow: /fcommerce.php
 +
Disallow: /shop_closed_evaluation.html
 +
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
 +
Disallow: /xps_cron.php
 +
Disallow: /amazon_checkout.php
 +
Disallow: /VERSION.CLOUD_SEARCH
 
</pre>
 
</pre>
  
Line 177: Line 173:
 
Disallow: /COPYRIGHT
 
Disallow: /COPYRIGHT
 
Disallow: /INSTALL
 
Disallow: /INSTALL
Disallow: /NEW-4.4.0
+
Disallow: /NEW-4.5.0
 
Disallow: /README
 
Disallow: /README
 
Disallow: /VERSION
 
Disallow: /VERSION
Line 190: Line 186:
 
Disallow: /change_password.php
 
Disallow: /change_password.php
 
Disallow: /check_requirements.php
 
Disallow: /check_requirements.php
Disallow: /choosing.php
 
 
Disallow: /cleanup.php
 
Disallow: /cleanup.php
 
Disallow: /cmpi_popup.php
 
Disallow: /cmpi_popup.php
Disallow: /comparison.php
 
Disallow: /comparison_list.php
 
 
Disallow: /config.php
 
Disallow: /config.php
 
Disallow: /cron.php
 
Disallow: /cron.php
Disallow: /default_icon.gif
 
Disallow: /default_image.gif
 
Disallow: /default_logo.gif
 
 
Disallow: /dispatcher.php
 
Disallow: /dispatcher.php
 
Disallow: /download.php
 
Disallow: /download.php
Line 206: Line 196:
 
Disallow: /get_block.php
 
Disallow: /get_block.php
 
Disallow: /get_info.php
 
Disallow: /get_info.php
Disallow: /giftcert.php
 
 
Disallow: /giftreg_manage.php
 
Disallow: /giftreg_manage.php
Disallow: /giftregs.php
 
Disallow: /help.php
 
Disallow: /home.php
 
 
Disallow: /https.php
 
Disallow: /https.php
 
Disallow: /image.php
 
Disallow: /image.php
Disallow: /index.php
 
 
Disallow: /init.php
 
Disallow: /init.php
 
Disallow: /install.php
 
Disallow: /install.php
Disallow: /login.php
 
 
Disallow: /magnifier_xml.php
 
Disallow: /magnifier_xml.php
Disallow: /manufacturers.php
 
 
Disallow: /message.html
 
Disallow: /message.html
Disallow: /minicart.php
 
Disallow: /news.php
 
Disallow: /offers.php
 
 
Disallow: /order.php
 
Disallow: /order.php
 
Disallow: /orders.php
 
Disallow: /orders.php
 
Disallow: /pages.php
 
Disallow: /pages.php
Disallow: /pconf.php
 
 
Disallow: /permission_denied.html
 
Disallow: /permission_denied.html
 
Disallow: /popup_address.php
 
Disallow: /popup_address.php
Line 232: Line 211:
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_estimate_shipping.php
 
Disallow: /popup_estimate_shipping.php
Disallow: /popup_fc_products.php
 
 
Disallow: /popup_image.php
 
Disallow: /popup_image.php
 
Disallow: /popup_info.php
 
Disallow: /popup_info.php
Line 240: Line 218:
 
Disallow: /preauth.php
 
Disallow: /preauth.php
 
Disallow: /prepare.php
 
Disallow: /prepare.php
Disallow: /probe.php
 
Disallow: /preview_image.gif
 
 
Disallow: /process_order.php
 
Disallow: /process_order.php
Disallow: /product.php
 
 
Disallow: /products.php
 
Disallow: /products.php
 
Disallow: /products_map.php
 
Disallow: /products_map.php
 
Disallow: /recommends.php
 
Disallow: /recommends.php
 
Disallow: /referer.php
 
Disallow: /referer.php
Disallow: /register.php
 
 
Disallow: /returns.php
 
Disallow: /returns.php
 
Disallow: /search.php
 
Disallow: /search.php
Line 254: Line 228:
 
Disallow: /shop_closed.html
 
Disallow: /shop_closed.html
 
Disallow: /shop_registration.php
 
Disallow: /shop_registration.php
Disallow: /sitemap.php
 
 
Disallow: /slabel.php
 
Disallow: /slabel.php
 
Disallow: /smarty.php
 
Disallow: /smarty.php
 
Disallow: /survey.php
 
Disallow: /survey.php
 
Disallow: /top.inc.php
 
Disallow: /top.inc.php
Disallow: /vote.php
+
Disallow: /popup_cookie_settings.php
Disallow: /wysiwyg.php
+
Disallow: /flash_container.swf
 +
Disallow: /pplogin_return.php
 +
Disallow: /pplogin_popup.php
 +
Disallow: /mailchimp_news.php
 +
Disallow: /xmonitoring_api.php
 +
Disallow: /cloud_search_api.php
 +
Disallow: /klarna_popup_address.php
 +
Disallow: /xps_subscriptions.php
 +
Disallow: /saved_cards.php
 +
Disallow: /fcommerce.php
 +
Disallow: /shop_closed_evaluation.html
 +
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
 +
Disallow: /xps_cron.php
 +
Disallow: /amazon_checkout.php
 +
Disallow: /VERSION.CLOUD_SEARCH
 
</pre>
 
</pre>
  
Line 281: Line 268:
  
 
Next, create a file to be used instead of the '''robots.txt''' when the store is accessed via the secure protocol and name that file '''robots-https.txt''':
 
Next, create a file to be used instead of the '''robots.txt''' when the store is accessed via the secure protocol and name that file '''robots-https.txt''':
 
=== robots-https.txt ===
 
 
<pre>
 
User-agent: *
 
Disallow: /
 
</pre>
 
 
Finally, upload both these files along with the current version of '''robots.txt''' to the root of your website.
 
 
That's it.
 
 
If you ever need help setting up the robots.txt file for your online store, please, do not hesitate to contact our technical support for assistance. Our specialists will examine your store and make up a file that would be the most suitable from search engines' point of view.
 
  
 
== See also ==
 
== See also ==

Revision as of 19:00, 20 February 2017

About robots.txt

Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index.

Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.

Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.

Below we describe how to configure an X-Cart based online store for proper search engine indexing.

Note: The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.

Configuring a store to make it properly visible to search engines takes placing a special text file named robots.txt into your website's root folder. Please note that the file cannot reside in a sub-folder, such as /store; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:

Disallow: /store/admin/

The content of robots.txt can vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:

  • Dynamic store, generated on the fly from the database.
  • Static store, based on a static HTML catalog.

Here is a working example for each case.

Dynamic version (without HTML catalog)

If you are not using HTML catalog, only the following pages should be included in the index:

  • home and category pages (home.php)
  • product pages (product.php)
  • static pages (pages.php)
  • manufacturers (manufacturers.php)

The rest of the pages available on the website are to be hidden from search engines. Keeping this in mind, here is the configuration we suggest for a store with dynamically generated content:

User-agent: Googlebot
Disallow: /*printable=Y*
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Disallow: /admin/
Disallow: /catalog/
Disallow: /customer/
Disallow: /files/
Disallow: /images/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /magnifier_xml.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

Static version (using HTML catalog)

For the static version, only the corresponding HTML part is to be included in the index, and all the dynamic (PHP-based) part can be hidden from search engines:

User-agent: Googlebot
Disallow: /*printable=Y*
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Disallow: /admin/
Disallow: /customer/
Disallow: /files/
Disallow: /images/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /magnifier_xml.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

See also:

Blocking the secure part

Since the checkout is normally carried out over a secure protocol, the majority of e-commerce websites can be reached via both plain and secure HTTP. Some search robots can treat this as duplicate content, as all the pages available over HTTP can be accessed via the secure protocol as well. However, if your web server allows using mod_rewrite, all the secure pages can be easily blocked.

To do so, first of all, you need to create a file called .htaccess that contains the following code (or add this code to your existing .htaccess file):

.htaccess

RewriteEngine on
RewriteCond %{HTTPS} on
RewriteRule ^robots\.txt$ robots-https.txt

Next, create a file to be used instead of the robots.txt when the store is accessed via the secure protocol and name that file robots-https.txt:

See also