X-Cart:Setting up robots.txt

About robots.txt

Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index.

Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.

Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.

Below we describe how to configure an X-Cart based online store for proper search engine indexing.

Note: The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.

Configuring a store to make it properly visible to search engines takes placing a special text file named robots.txt into your website's root folder. Please note that the file cannot reside in a sub-folder, such as /store; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:

Disallow: /store/admin/

The content of robots.txt can vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:

• Dynamic store, generated on the fly from the database.
• Static store, based on a static HTML catalog.

Here is a working example for each case.

Dynamic version (without HTML catalog)

If you are not using HTML catalog, only the following pages should be included in the index:

• home and category pages (home.php)
• product pages (product.php)
• static pages (pages.php)
• manufacturers (manufacturers.php)

The rest of the pages available on the website are to be hidden from search engines. Keeping this in mind, here is the configuration we suggest for a store with dynamically generated content:

User-agent: Googlebot
Disallow: /*printable=Y*
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Disallow: /admin/
Disallow: /catalog/
Disallow: /customer/
Disallow: /files/
Disallow: /images/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /magnifier_xml.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

Static version (using HTML catalog)

For the static version, only the corresponding HTML part is to be included in the index, and all the dynamic (PHP-based) part can be hidden from search engines:

User-agent: Googlebot
Disallow: /*printable=Y*
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Disallow: /admin/
Disallow: /customer/
Disallow: /files/
Disallow: /images/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /magnifier_xml.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

See also:

• Google bot can vote/rate products when the "Who must be allowed to add ratings" option is set to "All users"

Blocking the secure part

Since the checkout is normally carried out over a secure protocol, the majority of e-commerce websites can be reached via both plain and secure HTTP. Some search robots can treat this as duplicate content, as all the pages available over HTTP can be accessed via the secure protocol as well. However, if your web server allows using mod_rewrite, all the secure pages can be easily blocked.

To do so, first of all, you need to create a file called .htaccess that contains the following code (or add this code to your existing .htaccess file):

.htaccess

RewriteEngine on
RewriteCond %{HTTPS} on
RewriteRule ^robots\.txt$ robots-https.txt

Next, create a file to be used instead of the robots.txt when the store is accessed via the secure protocol and name that file robots-https.txt:

See also

• A Standard for Robot Exclusion
• Using the robots.txt analysis tool
• Robots.txt Checkers