X-Cart:Setting up robots.txt
Contents
About robots.txt
Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index.
Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.
Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.
Below we describe how to configure an X-Cart based online store for proper search engine indexing.
Configuring a store to make it properly visible to search engines takes placing a special text file named robots.txt into your website's root folder. Please note that the file cannot reside in a sub-folder, such as /store; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:
Disallow: /store/admin/
The content of robots.txt may vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:
- Dynamic store, generated on the fly from the database.
- Static store, based on a static HTML catalog.
Further below you will find a working example for each of the above-said cases.
Note that if your store is available via HTTPS, it is strongly recommended to add "Host: https://<your_domain>" to your robots.txt file; for example:
Host: https://example.com
That is needed to tell Google that the main mirror for your site is the HTTPS version, even though HTTP is still available.
Dynamic version (without HTML catalog)
If you are not using HTML catalog, only the following pages should be included in the index:
- home and category pages (home.php)
- product pages (product.php)
- static pages (pages.php)
- manufacturers (manufacturers.php)
The rest of the pages available on the website are to be hidden from search engines. Keeping this in mind, here is the configuration we suggest for a store with dynamically generated content:
User-agent: Googlebot Disallow: /*js=* Disallow: /*print_cat=* Disallow: /*mode=add_vote* User-agent: * Allow: *.js Allow: *.css Allow: *.jpg Allow: *.gif Allow: *.png Disallow: /admin/ Disallow: /catalog/ Disallow: /customer/ Disallow: /files/ Disallow: /include/ Disallow: /mail/ Disallow: /modules/ Disallow: /partner/ Disallow: /payment/ Disallow: /provider/ Disallow: /schemes/ Disallow: /shipping/ Disallow: /skin/ Disallow: /skin_backup/ Disallow: /sql/ Disallow: /upgrade/ Disallow: /var/ Disallow: /404.php Disallow: /CHANGELOG Disallow: /COPYRIGHT Disallow: /INSTALL Disallow: /NEW-4.5.0 Disallow: /README Disallow: /VERSION Disallow: /adaptive.php Disallow: /address_book.php Disallow: /adv_counter.php Disallow: /antibot_image.php Disallow: /auth.php Disallow: /banner.php Disallow: /bonuses.php Disallow: /cart.php Disallow: /change_password.php Disallow: /check_requirements.php Disallow: /choosing.php Disallow: /cleanup.php Disallow: /cmpi_popup.php Disallow: /comparison.php Disallow: /comparison_list.php Disallow: /config.php Disallow: /cron.php Disallow: /dispatcher.php Disallow: /download.php Disallow: /error_message.php Disallow: /featured_products.php Disallow: /giftcert.php Disallow: /get_block.php Disallow: /get_info.php Disallow: /giftreg_manage.php Disallow: /giftregs.php Disallow: /https.php Disallow: /image.php Disallow: /init.php Disallow: /install.php Disallow: /login.php Disallow: /magnifier_xml.php Disallow: /mailchimp_news.php Disallow: /message.html Disallow: /order.php Disallow: /orders.php Disallow: /pages.php Disallow: /permission_denied.html Disallow: /popup_address.php Disallow: /popup_ask.php Disallow: /popup_edit_label.php Disallow: /popup_estimate_shipping.php Disallow: /popup_image.php Disallow: /popup_info.php Disallow: /popup_magnifier.php Disallow: /popup_poptions.php Disallow: /postauth.php Disallow: /preauth.php Disallow: /prepare.php Disallow: /process_order.php Disallow: /products.php Disallow: /products_map.php Disallow: /recommends.php Disallow: /referer.php Disallow: /returns.php Disallow: /search.php Disallow: /send_to_friend.php Disallow: /shop_closed.html Disallow: /shop_registration.php Disallow: /slabel.php Disallow: /smarty.php Disallow: /survey.php Disallow: /top.inc.php Disallow: /popup_cookie_settings.php Disallow: /flash_container.swf Disallow: /pplogin_return.php Disallow: /pplogin_popup.php Disallow: /mailchimp_news.php Disallow: /xmonitoring_api.php Disallow: /cloud_search_api.php Disallow: /klarna_popup_address.php Disallow: /xps_subscriptions.php Disallow: /saved_cards.php Disallow: /fcommerce.php Disallow: /shop_closed_evaluation.html Disallow: /TERMSOFSERVICE.CLOUD_SEARCH Disallow: /xps_cron.php Disallow: /amazon_checkout.php Disallow: /VERSION.CLOUD_SEARCH
Static version (using HTML catalog)
For the static version, only the corresponding HTML part is to be included in the index, and all the dynamic (PHP-based) part can be hidden from search engines:
User-agent: Googlebot Disallow: /*js=* Disallow: /*print_cat=* Disallow: /*mode=add_vote* User-agent: * Allow: *.js Allow: *.css Allow: *.jpg Allow: *.gif Allow: *.png Disallow: /admin/ Disallow: /customer/ Disallow: /files/ Disallow: /include/ Disallow: /mail/ Disallow: /modules/ Disallow: /partner/ Disallow: /payment/ Disallow: /provider/ Disallow: /schemes/ Disallow: /shipping/ Disallow: /skin/ Disallow: /skin_backup/ Disallow: /sql/ Disallow: /upgrade/ Disallow: /var/ Disallow: /404.php Disallow: /CHANGELOG Disallow: /COPYRIGHT Disallow: /INSTALL Disallow: /NEW-4.5.0 Disallow: /README Disallow: /VERSION Disallow: /adaptive.php Disallow: /address_book.php Disallow: /adv_counter.php Disallow: /antibot_image.php Disallow: /auth.php Disallow: /banner.php Disallow: /bonuses.php Disallow: /cart.php Disallow: /change_password.php Disallow: /check_requirements.php Disallow: /cleanup.php Disallow: /cmpi_popup.php Disallow: /config.php Disallow: /cron.php Disallow: /dispatcher.php Disallow: /download.php Disallow: /error_message.php Disallow: /featured_products.php Disallow: /get_block.php Disallow: /get_info.php Disallow: /giftreg_manage.php Disallow: /https.php Disallow: /image.php Disallow: /init.php Disallow: /install.php Disallow: /magnifier_xml.php Disallow: /message.html Disallow: /order.php Disallow: /orders.php Disallow: /pages.php Disallow: /permission_denied.html Disallow: /popup_address.php Disallow: /popup_ask.php Disallow: /popup_edit_label.php Disallow: /popup_estimate_shipping.php Disallow: /popup_image.php Disallow: /popup_info.php Disallow: /popup_magnifier.php Disallow: /popup_poptions.php Disallow: /postauth.php Disallow: /preauth.php Disallow: /prepare.php Disallow: /process_order.php Disallow: /products.php Disallow: /products_map.php Disallow: /recommends.php Disallow: /referer.php Disallow: /returns.php Disallow: /search.php Disallow: /send_to_friend.php Disallow: /shop_closed.html Disallow: /shop_registration.php Disallow: /admin/twofactor_token.php Disallow: /gak_admin.php Disallow: /provider/twofactor_token.php Disallow: /twofactor_token.php Disallow: /slabel.php Disallow: /smarty.php Disallow: /survey.php Disallow: /top.inc.php Disallow: /popup_cookie_settings.php Disallow: /flash_container.swf Disallow: /pplogin_return.php Disallow: /pplogin_popup.php Disallow: /mailchimp_news.php Disallow: /xmonitoring_api.php Disallow: /cloud_search_api.php Disallow: /klarna_popup_address.php Disallow: /xps_subscriptions.php Disallow: /saved_cards.php Disallow: /fcommerce.php Disallow: /shop_closed_evaluation.html Disallow: /TERMSOFSERVICE.CLOUD_SEARCH Disallow: /xps_cron.php Disallow: /amazon_checkout.php Disallow: /VERSION.CLOUD_SEARCH
See also: