X-Cart:Setting up robots.txt

About robots.txt

Search engines allow webmasters to define what kind of web pages should be included into the database and become available within search results. Some pages may also be undesired for inclusion since they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, therefore, it's quite reasonable to include only content-rich part of your site into the index.

Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from database. For example, we may speak of printable versions and non-JavaScript versions as main sources of the same content with different presentations.

If we speak of online stores based on X-Cart, their main traffic sources are pages containing product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.

Below we describe how to setup X-Cart-based online store for proper search engine indexing.

Note: the description below is based on default X-Cart files and folders location. For customized stores, the configuration may slightly vary.

This is done by placing a special text file called robots.txt into the root folder of your domain (please note that it cannot be located within a subfolder, for example, /store). Such file can be created using any plain-text editor (like NotePad) and should be uploaded using an FTP client or online file manager of your hosting control panel.

If we speak of a content, for robots.txt it can easily be varied in accordance with your needs. In our case, main differences take place for two cases: dynamic store or the store based on HTML catalog. Working examples for each case are presented below:

Dynamic version (without HTML catalog)

If you're not using HTML catalog, only the following pages should be included into the index:

• home and category pages (home.php)
• product pages (product.php)
• static pages (pages.php)
• manufacturers (manufacturers.php)

All other pages are recommended for exclusion. Therefore, we suggest using this configuration:

User-agent: Googlebot
Disallow: /*printable=Y*
Disallow: /*js=*
Disallow: /*print_cat=*

User-agent: *
Disallow: /admin/
Disallow: /catalog/
Disallow: /customer/
Disallow: /files/
Disallow: /images/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.4.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /choosing.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /comparison.php
Disallow: /comparison_list.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /default_icon.gif
Disallow: /default_image.gif
Disallow: /default_logo.gif
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftcert.php
Disallow: /giftreg_manage.php
Disallow: /giftregs.php
Disallow: /help.php
Disallow: /https.php
Disallow: /image.php
Disallow: /index.php
Disallow: /init.php
Disallow: /install.php
Disallow: /login.php
Disallow: /magnifier_xml.php
Disallow: /message.html
Disallow: /minicart.php
Disallow: /news.php
Disallow: /offers.php
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /pconf.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_fc_products.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /probe.php
Disallow: /preview_image.gif
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /register.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /sitemap.php
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /wysiwyg.php

Static version (using HTML catalog)

For static version, it is suggested that only corresponding HTML part is included into index. Therefore, all of dynamic (PHP-based) part can be closed from search engines while all HTML pages will be kept for inclusion:

User-agent: Googlebot
Disallow: /*printable=Y*
Disallow: /*js=*
Disallow: /*print_cat=*

User-agent: *
Disallow: /admin/
Disallow: /customer/
Disallow: /files/
Disallow: /images/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.4.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /choosing.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /comparison.php
Disallow: /comparison_list.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /default_icon.gif
Disallow: /default_image.gif
Disallow: /default_logo.gif
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftcert.php
Disallow: /giftreg_manage.php
Disallow: /giftregs.php
Disallow: /help.php
Disallow: /home.php
Disallow: /https.php
Disallow: /image.php
Disallow: /index.php
Disallow: /init.php
Disallow: /install.php
Disallow: /login.php
Disallow: /magnifier_xml.php
Disallow: /manufacturers.php
Disallow: /message.html
Disallow: /minicart.php
Disallow: /news.php
Disallow: /offers.php
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /pconf.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_fc_products.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /probe.php
Disallow: /preview_image.gif
Disallow: /process_order.php
Disallow: /product.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /register.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /sitemap.php
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /wysiwyg.php

Blocking of the secure part

Most of e-commerce websites can be reached via both HTTP and HTTPs protocols since checkout is usually done securely. Such situation can be treated by search robots as a duplicate content issue as all HTTP pages can be accessed via HTTPs as well. However, if your web server allows using mod_rewrite, all of secure pages can easily be blocked.

First of all, you will need to create a file called .htaccess containing the following lines (or add them into the existing one):

.htaccess

RewriteEngine on
RewriteCond %{HTTPS} on
RewriteRule ^robots\.txt$ robots-https.txt 

Next, create a file which will be used instead of robots.txt via the secure protocol and name it robots-https.txt:

robots-https.txt

User-agent: *
Disallow: / 

Finally, you need to upload both files along with the actual version of robots.txt that you intend to use.

If you ever require our assistance with the robots.txt setup for your online store, please, do not hesitate to contact our technical support for assistance. Our specialists will examine your store and perform the configuration which will be the most suitable from the search engines point of view.

See also

• A Standard for Robot Exclusion
• Using the robots.txt analysis tool
• Robots.txt Checkers