VWAM Help

Appearance

Allowing Our Importer ​

VWAM reads your website with an automated reader called the VWAM importer. The importer reads your product pages when you import products from a URL, when it refreshes your products each week, and when you run a Site Health audit. Some bot protection tools and robots.txt files block automated readers. This article tells you how to let the VWAM importer read your site.

Before you start ​

You need access to your site's bot protection or firewall settings, or to its robots.txt file. If a different team or agency manages your site, send them this article.

How to identify the importer ​

Every request from the importer sends a User-Agent header that contains the name VWAM-Importer. When the importer reads a product page, the full header is:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 VWAM-Importer/1.0 (+https://docs.vwam.ai/managing-product-feeds/allowing-our-importer)

The header starts with the same text that a Chrome browser sends. Because of this, a rule that matches the full header, or only its first part, does not identify the importer. Match on VWAM-Importer instead. The importer also sends VWAM-Importer when it downloads your product images, so the same rule covers them.

Allow the importer in your bot protection ​

  1. Open the bot protection or firewall settings for your site.
  2. Create a rule that allows requests when the User-Agent contains VWAM-Importer.
  3. Apply the rule to the pages that list and show your products.
  4. Save the rule.
  5. Run your import again from the VerifyProducts page.

Each bot protection tool uses different names for these settings. Look for terms like "allow rule", "skip rule", or "verified bot exception" in the documentation for your tool.

Allow the importer in robots.txt ​

The importer checks your robots.txt file when it looks for your product pages. To let the importer read all of your site, add these lines to your robots.txt file:

User-agent: VWAM-Importer
Allow: /

A group that names VWAM-Importer replaces the User-agent: * group for the importer. The importer then ignores the rules in the User-agent: * group. If you want the importer to stay out of some pages, add a Disallow line for each of those pages to the VWAM-Importer group. The importer checks these rules only when it looks for new product pages. To stop the importer reading a product that is already in your feed, remove that product from the feed.

How the importer uses your site ​

  • The importer limits how many pages it reads from your site at the same time.
  • The importer slows down when your site replies that it receives too many requests.
  • The importer stops for a period when your site refuses it or does not reply many times in a row.
  • The importer reads only public pages. It does not sign in to your site.

IP addresses ​

VWAM does not publish a list of IP addresses for the importer. Some pages need a full browser to show their content. VWAM reads those pages through a partner service, and those requests come from the partner's addresses. Those requests also do not contain the VWAM-Importer name.

If your security policy requires IP addresses, contact VWAM support.

If your products still do not import ​

  • Make sure that your bot protection rule matches VWAM-Importer and not the full header.
  • Make sure that your robots.txt file is at the root of your site, for example https://www.example.com/robots.txt.
  • Import your products from a file. For more information, see Feed Uploads.
  • Contact VWAM support and give the web address that you tried to import.