Difference between revisions of "X-Cart:Setting up robots.txt"
(6 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
== About robots.txt == | == About robots.txt == | ||
− | Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information | + | Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index. |
− | Another important action | + | Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats. |
Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections. | Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections. | ||
Line 9: | Line 9: | ||
Below we describe how to configure an X-Cart based online store for proper search engine indexing. | Below we describe how to configure an X-Cart based online store for proper search engine indexing. | ||
− | {{Note|The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may slightly | + | {{Note|The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.}} |
− | Configuring a store to make it properly visible to search engines takes placing a special text file named [http://en.wikipedia.org/wiki/Robots.txt robots.txt] into your website's root folder. | + | Configuring a store to make it properly visible to search engines takes placing a special text file named [http://en.wikipedia.org/wiki/Robots.txt robots.txt] into your website's root folder. Please note that the file cannot reside in a sub-folder, such as <u>/store</u>; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives: |
− | + | <pre> | |
− | The content of robots.txt | + | Disallow: /store/admin/ |
+ | </pre> | ||
+ | The content of robots.txt may vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases: | ||
* Dynamic store, generated on the fly from the database. | * Dynamic store, generated on the fly from the database. | ||
* Static store, based on a static [http://www.x-cart.com/search.html?query=html+catalog HTML catalog]. | * Static store, based on a static [http://www.x-cart.com/search.html?query=html+catalog HTML catalog]. | ||
− | + | Further below you will find a working example for each of the above-said cases. | |
+ | |||
+ | Note that if your store is available via HTTPS, it is strongly recommended to add "Host: https://<your_domain>" to your robots.txt file; for example: | ||
+ | <pre> | ||
+ | Host: https://example.com | ||
+ | </pre> | ||
+ | That is needed to tell Google that the main mirror for your site is the HTTPS version, even though HTTP is still available. | ||
== Dynamic version (without HTML catalog) == | == Dynamic version (without HTML catalog) == | ||
Line 39: | Line 47: | ||
User-agent: * | User-agent: * | ||
+ | Allow: *.js | ||
+ | Allow: *.css | ||
+ | Allow: *.jpg | ||
+ | Allow: *.gif | ||
+ | Allow: *.png | ||
Disallow: /admin/ | Disallow: /admin/ | ||
Disallow: /catalog/ | Disallow: /catalog/ | ||
Disallow: /customer/ | Disallow: /customer/ | ||
Disallow: /files/ | Disallow: /files/ | ||
− | |||
Disallow: /include/ | Disallow: /include/ | ||
Disallow: /mail/ | Disallow: /mail/ | ||
Line 61: | Line 73: | ||
Disallow: /COPYRIGHT | Disallow: /COPYRIGHT | ||
Disallow: /INSTALL | Disallow: /INSTALL | ||
− | Disallow: /NEW-4. | + | Disallow: /NEW-4.5.0 |
Disallow: /README | Disallow: /README | ||
Disallow: /VERSION | Disallow: /VERSION | ||
Line 77: | Line 89: | ||
Disallow: /cleanup.php | Disallow: /cleanup.php | ||
Disallow: /cmpi_popup.php | Disallow: /cmpi_popup.php | ||
− | Disallow: /comparison.php | + | Disallow: /comparison.php |
Disallow: /comparison_list.php | Disallow: /comparison_list.php | ||
Disallow: /config.php | Disallow: /config.php | ||
Disallow: /cron.php | Disallow: /cron.php | ||
− | |||
− | |||
− | |||
Disallow: /dispatcher.php | Disallow: /dispatcher.php | ||
Disallow: /download.php | Disallow: /download.php | ||
Disallow: /error_message.php | Disallow: /error_message.php | ||
Disallow: /featured_products.php | Disallow: /featured_products.php | ||
+ | Disallow: /giftcert.php | ||
Disallow: /get_block.php | Disallow: /get_block.php | ||
Disallow: /get_info.php | Disallow: /get_info.php | ||
− | |||
Disallow: /giftreg_manage.php | Disallow: /giftreg_manage.php | ||
Disallow: /giftregs.php | Disallow: /giftregs.php | ||
− | |||
Disallow: /https.php | Disallow: /https.php | ||
Disallow: /image.php | Disallow: /image.php | ||
− | |||
Disallow: /init.php | Disallow: /init.php | ||
Disallow: /install.php | Disallow: /install.php | ||
Disallow: /login.php | Disallow: /login.php | ||
Disallow: /magnifier_xml.php | Disallow: /magnifier_xml.php | ||
+ | Disallow: /mailchimp_news.php | ||
Disallow: /message.html | Disallow: /message.html | ||
− | |||
− | |||
− | |||
Disallow: /order.php | Disallow: /order.php | ||
Disallow: /orders.php | Disallow: /orders.php | ||
Disallow: /pages.php | Disallow: /pages.php | ||
− | |||
Disallow: /permission_denied.html | Disallow: /permission_denied.html | ||
Disallow: /popup_address.php | Disallow: /popup_address.php | ||
Line 114: | Line 118: | ||
Disallow: /popup_edit_label.php | Disallow: /popup_edit_label.php | ||
Disallow: /popup_estimate_shipping.php | Disallow: /popup_estimate_shipping.php | ||
− | |||
Disallow: /popup_image.php | Disallow: /popup_image.php | ||
Disallow: /popup_info.php | Disallow: /popup_info.php | ||
Line 122: | Line 125: | ||
Disallow: /preauth.php | Disallow: /preauth.php | ||
Disallow: /prepare.php | Disallow: /prepare.php | ||
− | |||
− | |||
Disallow: /process_order.php | Disallow: /process_order.php | ||
Disallow: /products.php | Disallow: /products.php | ||
Line 129: | Line 130: | ||
Disallow: /recommends.php | Disallow: /recommends.php | ||
Disallow: /referer.php | Disallow: /referer.php | ||
− | |||
Disallow: /returns.php | Disallow: /returns.php | ||
Disallow: /search.php | Disallow: /search.php | ||
Line 135: | Line 135: | ||
Disallow: /shop_closed.html | Disallow: /shop_closed.html | ||
Disallow: /shop_registration.php | Disallow: /shop_registration.php | ||
− | |||
Disallow: /slabel.php | Disallow: /slabel.php | ||
Disallow: /smarty.php | Disallow: /smarty.php | ||
Disallow: /survey.php | Disallow: /survey.php | ||
Disallow: /top.inc.php | Disallow: /top.inc.php | ||
− | Disallow: / | + | Disallow: /popup_cookie_settings.php |
− | Disallow: / | + | Disallow: /flash_container.swf |
+ | Disallow: /pplogin_return.php | ||
+ | Disallow: /pplogin_popup.php | ||
+ | Disallow: /mailchimp_news.php | ||
+ | Disallow: /xmonitoring_api.php | ||
+ | Disallow: /cloud_search_api.php | ||
+ | Disallow: /klarna_popup_address.php | ||
+ | Disallow: /xps_subscriptions.php | ||
+ | Disallow: /saved_cards.php | ||
+ | Disallow: /fcommerce.php | ||
+ | Disallow: /shop_closed_evaluation.html | ||
+ | Disallow: /TERMSOFSERVICE.CLOUD_SEARCH | ||
+ | Disallow: /xps_cron.php | ||
+ | Disallow: /amazon_checkout.php | ||
+ | Disallow: /VERSION.CLOUD_SEARCH | ||
</pre> | </pre> | ||
Line 156: | Line 169: | ||
User-agent: * | User-agent: * | ||
+ | Allow: *.js | ||
+ | Allow: *.css | ||
+ | Allow: *.jpg | ||
+ | Allow: *.gif | ||
+ | Allow: *.png | ||
Disallow: /admin/ | Disallow: /admin/ | ||
Disallow: /customer/ | Disallow: /customer/ | ||
Disallow: /files/ | Disallow: /files/ | ||
− | |||
Disallow: /include/ | Disallow: /include/ | ||
Disallow: /mail/ | Disallow: /mail/ | ||
Line 177: | Line 194: | ||
Disallow: /COPYRIGHT | Disallow: /COPYRIGHT | ||
Disallow: /INSTALL | Disallow: /INSTALL | ||
− | Disallow: /NEW-4. | + | Disallow: /NEW-4.5.0 |
Disallow: /README | Disallow: /README | ||
Disallow: /VERSION | Disallow: /VERSION | ||
Line 190: | Line 207: | ||
Disallow: /change_password.php | Disallow: /change_password.php | ||
Disallow: /check_requirements.php | Disallow: /check_requirements.php | ||
− | |||
Disallow: /cleanup.php | Disallow: /cleanup.php | ||
Disallow: /cmpi_popup.php | Disallow: /cmpi_popup.php | ||
− | |||
− | |||
Disallow: /config.php | Disallow: /config.php | ||
Disallow: /cron.php | Disallow: /cron.php | ||
− | |||
− | |||
− | |||
Disallow: /dispatcher.php | Disallow: /dispatcher.php | ||
Disallow: /download.php | Disallow: /download.php | ||
Line 206: | Line 217: | ||
Disallow: /get_block.php | Disallow: /get_block.php | ||
Disallow: /get_info.php | Disallow: /get_info.php | ||
− | |||
Disallow: /giftreg_manage.php | Disallow: /giftreg_manage.php | ||
− | |||
− | |||
− | |||
Disallow: /https.php | Disallow: /https.php | ||
Disallow: /image.php | Disallow: /image.php | ||
− | |||
Disallow: /init.php | Disallow: /init.php | ||
Disallow: /install.php | Disallow: /install.php | ||
− | |||
Disallow: /magnifier_xml.php | Disallow: /magnifier_xml.php | ||
− | |||
Disallow: /message.html | Disallow: /message.html | ||
− | |||
− | |||
− | |||
Disallow: /order.php | Disallow: /order.php | ||
Disallow: /orders.php | Disallow: /orders.php | ||
Disallow: /pages.php | Disallow: /pages.php | ||
− | |||
Disallow: /permission_denied.html | Disallow: /permission_denied.html | ||
Disallow: /popup_address.php | Disallow: /popup_address.php | ||
Line 232: | Line 232: | ||
Disallow: /popup_edit_label.php | Disallow: /popup_edit_label.php | ||
Disallow: /popup_estimate_shipping.php | Disallow: /popup_estimate_shipping.php | ||
− | |||
Disallow: /popup_image.php | Disallow: /popup_image.php | ||
Disallow: /popup_info.php | Disallow: /popup_info.php | ||
Line 240: | Line 239: | ||
Disallow: /preauth.php | Disallow: /preauth.php | ||
Disallow: /prepare.php | Disallow: /prepare.php | ||
− | |||
− | |||
Disallow: /process_order.php | Disallow: /process_order.php | ||
− | |||
Disallow: /products.php | Disallow: /products.php | ||
Disallow: /products_map.php | Disallow: /products_map.php | ||
Disallow: /recommends.php | Disallow: /recommends.php | ||
Disallow: /referer.php | Disallow: /referer.php | ||
− | |||
Disallow: /returns.php | Disallow: /returns.php | ||
Disallow: /search.php | Disallow: /search.php | ||
Line 254: | Line 249: | ||
Disallow: /shop_closed.html | Disallow: /shop_closed.html | ||
Disallow: /shop_registration.php | Disallow: /shop_registration.php | ||
− | |||
Disallow: /slabel.php | Disallow: /slabel.php | ||
Disallow: /smarty.php | Disallow: /smarty.php | ||
Disallow: /survey.php | Disallow: /survey.php | ||
Disallow: /top.inc.php | Disallow: /top.inc.php | ||
− | Disallow: / | + | Disallow: /popup_cookie_settings.php |
− | Disallow: / | + | Disallow: /flash_container.swf |
+ | Disallow: /pplogin_return.php | ||
+ | Disallow: /pplogin_popup.php | ||
+ | Disallow: /mailchimp_news.php | ||
+ | Disallow: /xmonitoring_api.php | ||
+ | Disallow: /cloud_search_api.php | ||
+ | Disallow: /klarna_popup_address.php | ||
+ | Disallow: /xps_subscriptions.php | ||
+ | Disallow: /saved_cards.php | ||
+ | Disallow: /fcommerce.php | ||
+ | Disallow: /shop_closed_evaluation.html | ||
+ | Disallow: /TERMSOFSERVICE.CLOUD_SEARCH | ||
+ | Disallow: /xps_cron.php | ||
+ | Disallow: /amazon_checkout.php | ||
+ | Disallow: /VERSION.CLOUD_SEARCH | ||
</pre> | </pre> | ||
'''See also:''' | '''See also:''' | ||
* [[X-Cart:Customer_Reviews#Google_bot_can_vote.2Frate_products_when_the_.22Who_must_be_allowed_to_add_ratings.22_option_is_set_to_.22All_users.22|Google bot can vote/rate products when the "Who must be allowed to add ratings" option is set to "All users" ]] | * [[X-Cart:Customer_Reviews#Google_bot_can_vote.2Frate_products_when_the_.22Who_must_be_allowed_to_add_ratings.22_option_is_set_to_.22All_users.22|Google bot can vote/rate products when the "Who must be allowed to add ratings" option is set to "All users" ]] | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
== See also == | == See also == |
Revision as of 18:33, 14 November 2017
Contents
About robots.txt
Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index.
Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.
Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.
Below we describe how to configure an X-Cart based online store for proper search engine indexing.
Configuring a store to make it properly visible to search engines takes placing a special text file named robots.txt into your website's root folder. Please note that the file cannot reside in a sub-folder, such as /store; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:
Disallow: /store/admin/
The content of robots.txt may vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:
- Dynamic store, generated on the fly from the database.
- Static store, based on a static HTML catalog.
Further below you will find a working example for each of the above-said cases.
Note that if your store is available via HTTPS, it is strongly recommended to add "Host: https://<your_domain>" to your robots.txt file; for example:
Host: https://example.com
That is needed to tell Google that the main mirror for your site is the HTTPS version, even though HTTP is still available.
Dynamic version (without HTML catalog)
If you are not using HTML catalog, only the following pages should be included in the index:
- home and category pages (home.php)
- product pages (product.php)
- static pages (pages.php)
- manufacturers (manufacturers.php)
The rest of the pages available on the website are to be hidden from search engines. Keeping this in mind, here is the configuration we suggest for a store with dynamically generated content:
User-agent: Googlebot Disallow: /*printable=Y* Disallow: /*js=* Disallow: /*print_cat=* Disallow: /*mode=add_vote* User-agent: * Allow: *.js Allow: *.css Allow: *.jpg Allow: *.gif Allow: *.png Disallow: /admin/ Disallow: /catalog/ Disallow: /customer/ Disallow: /files/ Disallow: /include/ Disallow: /mail/ Disallow: /modules/ Disallow: /partner/ Disallow: /payment/ Disallow: /provider/ Disallow: /schemes/ Disallow: /shipping/ Disallow: /skin/ Disallow: /skin_backup/ Disallow: /sql/ Disallow: /upgrade/ Disallow: /var/ Disallow: /404.php Disallow: /CHANGELOG Disallow: /COPYRIGHT Disallow: /INSTALL Disallow: /NEW-4.5.0 Disallow: /README Disallow: /VERSION Disallow: /adaptive.php Disallow: /address_book.php Disallow: /adv_counter.php Disallow: /antibot_image.php Disallow: /auth.php Disallow: /banner.php Disallow: /bonuses.php Disallow: /cart.php Disallow: /change_password.php Disallow: /check_requirements.php Disallow: /choosing.php Disallow: /cleanup.php Disallow: /cmpi_popup.php Disallow: /comparison.php Disallow: /comparison_list.php Disallow: /config.php Disallow: /cron.php Disallow: /dispatcher.php Disallow: /download.php Disallow: /error_message.php Disallow: /featured_products.php Disallow: /giftcert.php Disallow: /get_block.php Disallow: /get_info.php Disallow: /giftreg_manage.php Disallow: /giftregs.php Disallow: /https.php Disallow: /image.php Disallow: /init.php Disallow: /install.php Disallow: /login.php Disallow: /magnifier_xml.php Disallow: /mailchimp_news.php Disallow: /message.html Disallow: /order.php Disallow: /orders.php Disallow: /pages.php Disallow: /permission_denied.html Disallow: /popup_address.php Disallow: /popup_ask.php Disallow: /popup_edit_label.php Disallow: /popup_estimate_shipping.php Disallow: /popup_image.php Disallow: /popup_info.php Disallow: /popup_magnifier.php Disallow: /popup_poptions.php Disallow: /postauth.php Disallow: /preauth.php Disallow: /prepare.php Disallow: /process_order.php Disallow: /products.php Disallow: /products_map.php Disallow: /recommends.php Disallow: /referer.php Disallow: /returns.php Disallow: /search.php Disallow: /send_to_friend.php Disallow: /shop_closed.html Disallow: /shop_registration.php Disallow: /slabel.php Disallow: /smarty.php Disallow: /survey.php Disallow: /top.inc.php Disallow: /popup_cookie_settings.php Disallow: /flash_container.swf Disallow: /pplogin_return.php Disallow: /pplogin_popup.php Disallow: /mailchimp_news.php Disallow: /xmonitoring_api.php Disallow: /cloud_search_api.php Disallow: /klarna_popup_address.php Disallow: /xps_subscriptions.php Disallow: /saved_cards.php Disallow: /fcommerce.php Disallow: /shop_closed_evaluation.html Disallow: /TERMSOFSERVICE.CLOUD_SEARCH Disallow: /xps_cron.php Disallow: /amazon_checkout.php Disallow: /VERSION.CLOUD_SEARCH
Static version (using HTML catalog)
For the static version, only the corresponding HTML part is to be included in the index, and all the dynamic (PHP-based) part can be hidden from search engines:
User-agent: Googlebot Disallow: /*printable=Y* Disallow: /*js=* Disallow: /*print_cat=* Disallow: /*mode=add_vote* User-agent: * Allow: *.js Allow: *.css Allow: *.jpg Allow: *.gif Allow: *.png Disallow: /admin/ Disallow: /customer/ Disallow: /files/ Disallow: /include/ Disallow: /mail/ Disallow: /modules/ Disallow: /partner/ Disallow: /payment/ Disallow: /provider/ Disallow: /schemes/ Disallow: /shipping/ Disallow: /skin/ Disallow: /skin_backup/ Disallow: /sql/ Disallow: /upgrade/ Disallow: /var/ Disallow: /404.php Disallow: /CHANGELOG Disallow: /COPYRIGHT Disallow: /INSTALL Disallow: /NEW-4.5.0 Disallow: /README Disallow: /VERSION Disallow: /adaptive.php Disallow: /address_book.php Disallow: /adv_counter.php Disallow: /antibot_image.php Disallow: /auth.php Disallow: /banner.php Disallow: /bonuses.php Disallow: /cart.php Disallow: /change_password.php Disallow: /check_requirements.php Disallow: /cleanup.php Disallow: /cmpi_popup.php Disallow: /config.php Disallow: /cron.php Disallow: /dispatcher.php Disallow: /download.php Disallow: /error_message.php Disallow: /featured_products.php Disallow: /get_block.php Disallow: /get_info.php Disallow: /giftreg_manage.php Disallow: /https.php Disallow: /image.php Disallow: /init.php Disallow: /install.php Disallow: /magnifier_xml.php Disallow: /message.html Disallow: /order.php Disallow: /orders.php Disallow: /pages.php Disallow: /permission_denied.html Disallow: /popup_address.php Disallow: /popup_ask.php Disallow: /popup_edit_label.php Disallow: /popup_estimate_shipping.php Disallow: /popup_image.php Disallow: /popup_info.php Disallow: /popup_magnifier.php Disallow: /popup_poptions.php Disallow: /postauth.php Disallow: /preauth.php Disallow: /prepare.php Disallow: /process_order.php Disallow: /products.php Disallow: /products_map.php Disallow: /recommends.php Disallow: /referer.php Disallow: /returns.php Disallow: /search.php Disallow: /send_to_friend.php Disallow: /shop_closed.html Disallow: /shop_registration.php Disallow: /slabel.php Disallow: /smarty.php Disallow: /survey.php Disallow: /top.inc.php Disallow: /popup_cookie_settings.php Disallow: /flash_container.swf Disallow: /pplogin_return.php Disallow: /pplogin_popup.php Disallow: /mailchimp_news.php Disallow: /xmonitoring_api.php Disallow: /cloud_search_api.php Disallow: /klarna_popup_address.php Disallow: /xps_subscriptions.php Disallow: /saved_cards.php Disallow: /fcommerce.php Disallow: /shop_closed_evaluation.html Disallow: /TERMSOFSERVICE.CLOUD_SEARCH Disallow: /xps_cron.php Disallow: /amazon_checkout.php Disallow: /VERSION.CLOUD_SEARCH
See also: