Difference between revisions of "X-Cart:Setting up robots.txt"

From X-Cart 4 Classic
Jump to: navigation, search
(Static version (using HTML catalog))
 
(22 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
== About robots.txt ==
 
== About robots.txt ==
  
Search engines allow webmasters to define what kind of web pages should be included into the database and become available within search results. Some pages may also be undesired for inclusion since they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, therefore, it's quite reasonable to include only content-rich part of your site into the index.  
+
Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index.
  
Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from database. For example, we may speak of printable versions and non-JavaScript versions as main sources of the same content with different presentations.  
+
Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.
  
If we speak of online stores based on X-Cart, their main traffic sources are pages containing product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.  
+
Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.
  
Below we describe how to setup X-Cart-based online store for proper search engine indexing.  
+
Below we describe how to configure an X-Cart based online store for proper search engine indexing.
  
Note: the description below is based on default X-Cart files and folders location. For customized stores, the configuration may slightly vary.  
+
{{Note|The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.}}
  
This is done by placing a special text file called [http://en.wikipedia.org/wiki/Robots.txt robots.txt] into the root folder of your domain (please note that it cannot be located within a subfolder, for example, /store). Such file can be created using any plain-text editor (like NotePad) and should be uploaded using an FTP client or online file manager of your hosting control panel.  
+
Configuring a store to make it properly visible to search engines takes placing a special text file named [https://en.wikipedia.org/wiki/Robots.txt robots.txt] into your website's root folder. Please note that the file cannot reside in a sub-folder, such as <u>/store</u>; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:
 +
<pre>
 +
Disallow: /store/admin/
 +
</pre>
 +
The content of robots.txt may vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:
 +
 
 +
* Dynamic store, generated on the fly from the database.
 +
* Static store, based on a static [https://www.x-cart.com/search.html?query=html+catalog HTML catalog].
  
If we speak of a content, for robots.txt it can easily be varied in accordance with your needs. In our case, main differences take place for two cases: dynamic store or the store based on [http://www.x-cart.com/search.html?query=html+catalog HTML catalog]. Working examples for each case are presented below:
+
Further below you will find a working example for each of the above-said cases.
 +
 
 +
Note that if your store is available via HTTPS, it is strongly recommended to add "Host: https://<your_domain>" to your robots.txt file; for example:
 +
<pre>
 +
Host: https://example.com
 +
</pre>
 +
That is needed to tell Google that the main mirror for your site is the HTTPS version, even though HTTP is still available.
  
 
== Dynamic version (without HTML catalog) ==
 
== Dynamic version (without HTML catalog) ==
  
If you're not using HTML catalog, only the following pages should be included into the index:
+
If you are not using HTML catalog, only the following pages should be included in the index:
  
 
* home and category pages (home.php)
 
* home and category pages (home.php)
Line 24: Line 37:
 
* manufacturers (manufacturers.php)
 
* manufacturers (manufacturers.php)
  
All other pages are recommended for exclusion. Therefore, we suggest using this configuration:
+
The rest of the pages available on the website are to be hidden from search engines. Keeping this in mind, here is the configuration we suggest for a store with dynamically generated content:
  
 
<pre>
 
<pre>
 
User-agent: Googlebot
 
User-agent: Googlebot
Disallow: /*printable=Y*
 
 
Disallow: /*js=*
 
Disallow: /*js=*
 
Disallow: /*print_cat=*
 
Disallow: /*print_cat=*
 +
Disallow: /*mode=add_vote*
  
 
User-agent: *
 
User-agent: *
 +
Allow: *.js
 +
Allow: *.css
 +
Allow: *.jpg
 +
Allow: *.gif
 +
Allow: *.png
 
Disallow: /admin/
 
Disallow: /admin/
 
Disallow: /catalog/
 
Disallow: /catalog/
 
Disallow: /customer/
 
Disallow: /customer/
 
Disallow: /files/
 
Disallow: /files/
Disallow: /images/
 
 
Disallow: /include/
 
Disallow: /include/
 
Disallow: /mail/
 
Disallow: /mail/
Line 55: Line 72:
 
Disallow: /COPYRIGHT
 
Disallow: /COPYRIGHT
 
Disallow: /INSTALL
 
Disallow: /INSTALL
Disallow: /NEW-4.4.0
+
Disallow: /NEW-4.5.0
 
Disallow: /README
 
Disallow: /README
 
Disallow: /VERSION
 
Disallow: /VERSION
Line 71: Line 88:
 
Disallow: /cleanup.php
 
Disallow: /cleanup.php
 
Disallow: /cmpi_popup.php
 
Disallow: /cmpi_popup.php
Disallow: /comparison.php
+
Disallow: /comparison.php  
 
Disallow: /comparison_list.php
 
Disallow: /comparison_list.php
 
Disallow: /config.php
 
Disallow: /config.php
 
Disallow: /cron.php
 
Disallow: /cron.php
Disallow: /default_icon.gif
 
Disallow: /default_image.gif
 
Disallow: /default_logo.gif
 
 
Disallow: /dispatcher.php
 
Disallow: /dispatcher.php
 
Disallow: /download.php
 
Disallow: /download.php
 
Disallow: /error_message.php
 
Disallow: /error_message.php
 
Disallow: /featured_products.php
 
Disallow: /featured_products.php
 +
Disallow: /admin/twofactor_token.php
 +
Disallow: /gak_admin.php
 +
Disallow: /provider/twofactor_token.php
 +
Disallow: /twofactor_token.php
 +
 +
Disallow: /giftcert.php
 
Disallow: /get_block.php
 
Disallow: /get_block.php
 
Disallow: /get_info.php
 
Disallow: /get_info.php
Disallow: /giftcert.php
 
 
Disallow: /giftreg_manage.php
 
Disallow: /giftreg_manage.php
 
Disallow: /giftregs.php
 
Disallow: /giftregs.php
Disallow: /help.php
 
 
Disallow: /https.php
 
Disallow: /https.php
 
Disallow: /image.php
 
Disallow: /image.php
Disallow: /index.php
 
 
Disallow: /init.php
 
Disallow: /init.php
 
Disallow: /install.php
 
Disallow: /install.php
 
Disallow: /login.php
 
Disallow: /login.php
 
Disallow: /magnifier_xml.php
 
Disallow: /magnifier_xml.php
 +
Disallow: /mailchimp_news.php
 
Disallow: /message.html
 
Disallow: /message.html
Disallow: /minicart.php
 
Disallow: /news.php
 
Disallow: /offers.php
 
 
Disallow: /order.php
 
Disallow: /order.php
 
Disallow: /orders.php
 
Disallow: /orders.php
 
Disallow: /pages.php
 
Disallow: /pages.php
Disallow: /pconf.php
 
 
Disallow: /permission_denied.html
 
Disallow: /permission_denied.html
 
Disallow: /popup_address.php
 
Disallow: /popup_address.php
Line 108: Line 122:
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_estimate_shipping.php
 
Disallow: /popup_estimate_shipping.php
Disallow: /popup_fc_products.php
 
 
Disallow: /popup_image.php
 
Disallow: /popup_image.php
 
Disallow: /popup_info.php
 
Disallow: /popup_info.php
Line 116: Line 129:
 
Disallow: /preauth.php
 
Disallow: /preauth.php
 
Disallow: /prepare.php
 
Disallow: /prepare.php
Disallow: /probe.php
 
Disallow: /preview_image.gif
 
 
Disallow: /process_order.php
 
Disallow: /process_order.php
 
Disallow: /products.php
 
Disallow: /products.php
Line 123: Line 134:
 
Disallow: /recommends.php
 
Disallow: /recommends.php
 
Disallow: /referer.php
 
Disallow: /referer.php
Disallow: /register.php
 
 
Disallow: /returns.php
 
Disallow: /returns.php
 
Disallow: /search.php
 
Disallow: /search.php
Line 129: Line 139:
 
Disallow: /shop_closed.html
 
Disallow: /shop_closed.html
 
Disallow: /shop_registration.php
 
Disallow: /shop_registration.php
Disallow: /sitemap.php
 
 
Disallow: /slabel.php
 
Disallow: /slabel.php
 
Disallow: /smarty.php
 
Disallow: /smarty.php
 
Disallow: /survey.php
 
Disallow: /survey.php
 
Disallow: /top.inc.php
 
Disallow: /top.inc.php
Disallow: /wysiwyg.php
+
Disallow: /popup_cookie_settings.php
 +
Disallow: /flash_container.swf
 +
Disallow: /pplogin_return.php
 +
Disallow: /pplogin_popup.php
 +
Disallow: /mailchimp_news.php
 +
Disallow: /xmonitoring_api.php
 +
Disallow: /cloud_search_api.php
 +
Disallow: /klarna_popup_address.php
 +
Disallow: /xps_subscriptions.php
 +
Disallow: /saved_cards.php
 +
Disallow: /fcommerce.php
 +
Disallow: /shop_closed_evaluation.html
 +
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
 +
Disallow: /xps_cron.php
 +
Disallow: /amazon_checkout.php
 +
Disallow: /VERSION.CLOUD_SEARCH
 
</pre>
 
</pre>
  
 
== Static version (using HTML catalog) ==
 
== Static version (using HTML catalog) ==
For static version, it is suggested that only corresponding HTML part is included into index. Therefore, all of dynamic (PHP-based) part can be closed from search engines while all HTML pages will be kept for inclusion:
+
 
 +
For the static version, only the corresponding HTML part is to be included in the index, and all the dynamic (PHP-based) part can be hidden from search engines:
  
 
<pre>
 
<pre>
 
User-agent: Googlebot
 
User-agent: Googlebot
Disallow: /*printable=Y*
 
 
Disallow: /*js=*
 
Disallow: /*js=*
 
Disallow: /*print_cat=*
 
Disallow: /*print_cat=*
 +
Disallow: /*mode=add_vote*
  
 
User-agent: *
 
User-agent: *
 +
Allow: *.js
 +
Allow: *.css
 +
Allow: *.jpg
 +
Allow: *.gif
 +
Allow: *.png
 
Disallow: /admin/
 
Disallow: /admin/
 
Disallow: /customer/
 
Disallow: /customer/
 
Disallow: /files/
 
Disallow: /files/
Disallow: /images/
 
 
Disallow: /include/
 
Disallow: /include/
 
Disallow: /mail/
 
Disallow: /mail/
Line 168: Line 197:
 
Disallow: /COPYRIGHT
 
Disallow: /COPYRIGHT
 
Disallow: /INSTALL
 
Disallow: /INSTALL
Disallow: /NEW-4.4.0
+
Disallow: /NEW-4.5.0
 
Disallow: /README
 
Disallow: /README
 
Disallow: /VERSION
 
Disallow: /VERSION
Line 181: Line 210:
 
Disallow: /change_password.php
 
Disallow: /change_password.php
 
Disallow: /check_requirements.php
 
Disallow: /check_requirements.php
Disallow: /choosing.php
 
 
Disallow: /cleanup.php
 
Disallow: /cleanup.php
 
Disallow: /cmpi_popup.php
 
Disallow: /cmpi_popup.php
Disallow: /comparison.php
 
Disallow: /comparison_list.php
 
 
Disallow: /config.php
 
Disallow: /config.php
 
Disallow: /cron.php
 
Disallow: /cron.php
Disallow: /default_icon.gif
 
Disallow: /default_image.gif
 
Disallow: /default_logo.gif
 
 
Disallow: /dispatcher.php
 
Disallow: /dispatcher.php
 
Disallow: /download.php
 
Disallow: /download.php
Line 197: Line 220:
 
Disallow: /get_block.php
 
Disallow: /get_block.php
 
Disallow: /get_info.php
 
Disallow: /get_info.php
Disallow: /giftcert.php
 
 
Disallow: /giftreg_manage.php
 
Disallow: /giftreg_manage.php
Disallow: /giftregs.php
 
Disallow: /help.php
 
Disallow: /home.php
 
 
Disallow: /https.php
 
Disallow: /https.php
 
Disallow: /image.php
 
Disallow: /image.php
Disallow: /index.php
 
 
Disallow: /init.php
 
Disallow: /init.php
 
Disallow: /install.php
 
Disallow: /install.php
Disallow: /login.php
 
 
Disallow: /magnifier_xml.php
 
Disallow: /magnifier_xml.php
Disallow: /manufacturers.php
 
 
Disallow: /message.html
 
Disallow: /message.html
Disallow: /minicart.php
 
Disallow: /news.php
 
Disallow: /offers.php
 
 
Disallow: /order.php
 
Disallow: /order.php
 
Disallow: /orders.php
 
Disallow: /orders.php
 
Disallow: /pages.php
 
Disallow: /pages.php
Disallow: /pconf.php
 
 
Disallow: /permission_denied.html
 
Disallow: /permission_denied.html
 
Disallow: /popup_address.php
 
Disallow: /popup_address.php
Line 223: Line 235:
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_edit_label.php
 
Disallow: /popup_estimate_shipping.php
 
Disallow: /popup_estimate_shipping.php
Disallow: /popup_fc_products.php
 
 
Disallow: /popup_image.php
 
Disallow: /popup_image.php
 
Disallow: /popup_info.php
 
Disallow: /popup_info.php
Line 231: Line 242:
 
Disallow: /preauth.php
 
Disallow: /preauth.php
 
Disallow: /prepare.php
 
Disallow: /prepare.php
Disallow: /probe.php
 
Disallow: /preview_image.gif
 
 
Disallow: /process_order.php
 
Disallow: /process_order.php
Disallow: /product.php
 
 
Disallow: /products.php
 
Disallow: /products.php
 
Disallow: /products_map.php
 
Disallow: /products_map.php
 
Disallow: /recommends.php
 
Disallow: /recommends.php
 
Disallow: /referer.php
 
Disallow: /referer.php
Disallow: /register.php
 
 
Disallow: /returns.php
 
Disallow: /returns.php
 
Disallow: /search.php
 
Disallow: /search.php
Line 245: Line 252:
 
Disallow: /shop_closed.html
 
Disallow: /shop_closed.html
 
Disallow: /shop_registration.php
 
Disallow: /shop_registration.php
Disallow: /sitemap.php
+
Disallow: /admin/twofactor_token.php
 +
Disallow: /gak_admin.php
 +
Disallow: /provider/twofactor_token.php
 +
Disallow: /twofactor_token.php
 +
 
 
Disallow: /slabel.php
 
Disallow: /slabel.php
 
Disallow: /smarty.php
 
Disallow: /smarty.php
 
Disallow: /survey.php
 
Disallow: /survey.php
 
Disallow: /top.inc.php
 
Disallow: /top.inc.php
Disallow: /wysiwyg.php
+
Disallow: /popup_cookie_settings.php
</pre>
+
Disallow: /flash_container.swf
 
+
Disallow: /pplogin_return.php
== Blocking of the secure part ==
+
Disallow: /pplogin_popup.php
Most of e-commerce websites can be reached via both HTTP and HTTPs protocols since checkout is usually done securely. Such situation can be treated by search robots as a duplicate content issue as all HTTP pages can be accessed via HTTPs as well. However, if your web server allows using mod_rewrite, all of secure pages can easily be blocked.  
+
Disallow: /mailchimp_news.php
 
+
Disallow: /xmonitoring_api.php
First of all, you will need to create a file called .htaccess containing the following lines (or add them into the existing one):  
+
Disallow: /cloud_search_api.php
 
+
Disallow: /klarna_popup_address.php
=== .htaccess ===
+
Disallow: /xps_subscriptions.php
 
+
Disallow: /saved_cards.php
<pre>
+
Disallow: /fcommerce.php
RewriteEngine on
+
Disallow: /shop_closed_evaluation.html
RewriteCond %{HTTPS} on
+
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
RewriteRule ^robots\.txt$ robots-https.txt
+
Disallow: /xps_cron.php
 +
Disallow: /amazon_checkout.php
 +
Disallow: /VERSION.CLOUD_SEARCH
 
</pre>
 
</pre>
  
Next, create a file which will be used instead of '''robots.txt''' via the secure protocol and name it '''robots-https.txt''':  
+
'''See also:'''
 
+
* [[X-Cart:Customer_Reviews#Google_bot_can_vote.2Frate_products_when_the_.22Who_must_be_allowed_to_add_ratings.22_option_is_set_to_.22All_users.22|Google bot can vote/rate products when the "Who must be allowed to add ratings" option is set to "All users" ]]
=== robots-https.txt ===
 
 
 
<pre>
 
User-agent: *
 
Disallow: /
 
</pre>
 
 
 
Finally, you need to upload both files along with the actual version of '''robots.txt''' that you intend to use.  
 
 
 
If you ever require our assistance with the robots.txt setup for your online store, please, do not hesitate to contact our technical support for assistance. Our specialists will examine your store and perform the configuration which will be the most suitable from the search engines point of view.
 
  
 
== See also ==
 
== See also ==
  
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]  
+
* [https://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
* [http://www.google.com/support/webmasters/bin/topic.py?topic=8475 Using the robots.txt analysis tool]  
+
* [https://www.google.com/support/webmasters/bin/topic.py?topic=8475 Using the robots.txt analysis tool]
* [http://www.searchtools.com/robots/robots-txt.html#checkers Robots.txt Checkers]
+
* [https://www.searchtools.com/robots/robots-txt.html#checkers Robots.txt Checkers]
  
 
[[Category:X-Cart user manual]]
 
[[Category:X-Cart user manual]]

Latest revision as of 18:03, 22 July 2020

About robots.txt

Search engines allow webmasters to define what pages of a website are to be included in the search engines' databases and become available in their search results. For some pages, it is be better to not appear in search results, as they do not contain any information relevant for potential customers. Besides, the capacity of listings available for individual websites might be limited, so it's quite reasonable to include only content-rich parts of your website in the search engines' index.

Another important action highly recommended for successful search engine indexing is the exclusion of so-called duplicate content from search engines' databases. Those could be, for example, printable and non-JavaScript versions of content as sources of the same content in different formats.

Speaking of X-Cart based online stores, their primary sources of traffic are pages with product and industry-related materials. These pages include: home page, category listings, product details, static pages and manufacturers sections.

Below we describe how to configure an X-Cart based online store for proper search engine indexing.

Note: The description below is based on the default location of X-Cart files and folders. For customized stores, the configuration may vary slightly.

Configuring a store to make it properly visible to search engines takes placing a special text file named robots.txt into your website's root folder. Please note that the file cannot reside in a sub-folder, such as /store; see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#file-location--range-of-validity for more info. The file can be created with any plain-text editor (like Notepad) and uploaded using an FTP client or an online file manager available at your hosting control panel. If your store is located in a sub-folder (for example, http://example.com/store/home.php), the folder /store must be specified for all the Disallow directives:

Disallow: /store/admin/

The content of robots.txt may vary to meet your specific needs. For X-Cart based stores, the main differences occur in two cases:

  • Dynamic store, generated on the fly from the database.
  • Static store, based on a static HTML catalog.

Further below you will find a working example for each of the above-said cases.

Note that if your store is available via HTTPS, it is strongly recommended to add "Host: https://<your_domain>" to your robots.txt file; for example:

 
Host: https://example.com

That is needed to tell Google that the main mirror for your site is the HTTPS version, even though HTTP is still available.

Dynamic version (without HTML catalog)

If you are not using HTML catalog, only the following pages should be included in the index:

  • home and category pages (home.php)
  • product pages (product.php)
  • static pages (pages.php)
  • manufacturers (manufacturers.php)

The rest of the pages available on the website are to be hidden from search engines. Keeping this in mind, here is the configuration we suggest for a store with dynamically generated content:

User-agent: Googlebot
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Allow: *.js
Allow: *.css
Allow: *.jpg
Allow: *.gif
Allow: *.png
Disallow: /admin/
Disallow: /catalog/
Disallow: /customer/
Disallow: /files/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /choosing.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /comparison.php 
Disallow: /comparison_list.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /admin/twofactor_token.php
Disallow: /gak_admin.php
Disallow: /provider/twofactor_token.php
Disallow: /twofactor_token.php

Disallow: /giftcert.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /giftregs.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /login.php
Disallow: /magnifier_xml.php
Disallow: /mailchimp_news.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

Static version (using HTML catalog)

For the static version, only the corresponding HTML part is to be included in the index, and all the dynamic (PHP-based) part can be hidden from search engines:

User-agent: Googlebot
Disallow: /*js=*
Disallow: /*print_cat=*
Disallow: /*mode=add_vote*

User-agent: *
Allow: *.js
Allow: *.css
Allow: *.jpg
Allow: *.gif
Allow: *.png
Disallow: /admin/
Disallow: /customer/
Disallow: /files/
Disallow: /include/
Disallow: /mail/
Disallow: /modules/
Disallow: /partner/
Disallow: /payment/
Disallow: /provider/
Disallow: /schemes/
Disallow: /shipping/
Disallow: /skin/
Disallow: /skin_backup/
Disallow: /sql/
Disallow: /upgrade/
Disallow: /var/
Disallow: /404.php
Disallow: /CHANGELOG
Disallow: /COPYRIGHT
Disallow: /INSTALL
Disallow: /NEW-4.5.0
Disallow: /README
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /address_book.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /bonuses.php
Disallow: /cart.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /cleanup.php
Disallow: /cmpi_popup.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /dispatcher.php
Disallow: /download.php
Disallow: /error_message.php
Disallow: /featured_products.php
Disallow: /get_block.php
Disallow: /get_info.php
Disallow: /giftreg_manage.php
Disallow: /https.php
Disallow: /image.php
Disallow: /init.php
Disallow: /install.php
Disallow: /magnifier_xml.php
Disallow: /message.html
Disallow: /order.php
Disallow: /orders.php
Disallow: /pages.php
Disallow: /permission_denied.html
Disallow: /popup_address.php
Disallow: /popup_ask.php
Disallow: /popup_edit_label.php
Disallow: /popup_estimate_shipping.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_magnifier.php
Disallow: /popup_poptions.php
Disallow: /postauth.php
Disallow: /preauth.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /products.php
Disallow: /products_map.php
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /search.php
Disallow: /send_to_friend.php
Disallow: /shop_closed.html
Disallow: /shop_registration.php
Disallow: /admin/twofactor_token.php
Disallow: /gak_admin.php
Disallow: /provider/twofactor_token.php
Disallow: /twofactor_token.php

Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /survey.php
Disallow: /top.inc.php
Disallow: /popup_cookie_settings.php
Disallow: /flash_container.swf
Disallow: /pplogin_return.php
Disallow: /pplogin_popup.php
Disallow: /mailchimp_news.php
Disallow: /xmonitoring_api.php
Disallow: /cloud_search_api.php
Disallow: /klarna_popup_address.php
Disallow: /xps_subscriptions.php
Disallow: /saved_cards.php
Disallow: /fcommerce.php
Disallow: /shop_closed_evaluation.html
Disallow: /TERMSOFSERVICE.CLOUD_SEARCH
Disallow: /xps_cron.php
Disallow: /amazon_checkout.php
Disallow: /VERSION.CLOUD_SEARCH

See also:

See also