X-Cart:Setting up robots.txt

About robots.txt

Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index.

Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.

Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.

Below we describe how to configure an X-Cart based online store for proper search engine indexing.

Note: The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.

Configuring a store to make it properly visible to search engines takes placing a special text file named robots.txt into your website's root folder. Please note that the file cannot reside in a sub-folder, such as /store; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:

Disallow: /store/admin/

The content of robots.txt may vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:

• Dynamic store, generated on the fly from the database.
• Static store, based on a static HTML catalog.

Further below you will find a working example for each of the above-said cases.

Note that if your store is available via HTTPS, it is strongly recommended to add "Host: https://<your_domain>" to your robots.txt file; for example:

 
Host: https://example.com

That is needed to tell Google that the main mirror for your site is the HTTPS version, even though HTTP is still available.

Dynamic version (without HTML catalog)

If you are not using HTML catalog, only the following pages should be included in the index:

• home and category pages (home.php)
• product pages (product.php)
• static pages (pages.php)
• manufacturers (manufacturers.php)

The rest of the pages available on the website are to be hidden from search engines. Keeping this in mind, here is the configuration we suggest for a store with dynamically generated content:

User-agent: Googlebot
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Allow: *.js
Allow: *.css
Allow: *.jpg
Allow: *.gif
Allow: *.png
Disallow: /admin/
Disallow: /catalog/
Disallow: /customer/
Disallow: /files/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /choosing.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /comparison.php 
Disallow: /comparison_list.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /admin/twofactor_token.php
Disallow: /gak_admin.php
Disallow: /provider/twofactor_token.php
Disallow: /twofactor_token.php

Disallow: /giftcert.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /giftregs.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /login.php
Disallow: /magnifier_xml.php
Disallow: /mailchimp_news.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

Static version (using HTML catalog)

For the static version, only the corresponding HTML part is to be included in the index, and all the dynamic (PHP-based) part can be hidden from search engines:

User-agent: Googlebot
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Allow: *.js
Allow: *.css
Allow: *.jpg
Allow: *.gif
Allow: *.png
Disallow: /admin/
Disallow: /customer/
Disallow: /files/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /magnifier_xml.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /admin/twofactor_token.php
Disallow: /gak_admin.php
Disallow: /provider/twofactor_token.php
Disallow: /twofactor_token.php

Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

See also:

• Google bot can vote/rate products when the "Who must be allowed to add ratings" option is set to "All users"

See also

• A Standard for Robot Exclusion
• Using the robots.txt analysis tool
• Robots.txt Checkers