Verification and the crawler
Unagi keeps its own map of your site's pages and products, so new listings show up without any setup. That is the job of UnagiBot. Before it visits anything, you prove you control the domain.
Verifying your site
Any one of these is enough. The first is done for you the moment you install the script.
- The script tag. Unagi fetches your home page and looks for your site key in the HTML. It checks automatically every few minutes for 14 days after you add a site, and you can press Check now on the Settings → Install on your site.
- A meta tag on your home page:
<meta name="unagi-verify" content="YOUR_TOKEN"> - A text file at
/.well-known/unagi-verify.txtwhose only content is your token.
Your token is shown in Settings, under Install on your site. If your script tag is added by JavaScript (for example through a tag manager), the first method cannot see it, so use the meta tag or file.
What UnagiBot does once verified
- Reads your
robots.txtand sitemap, then follows links on your own domain. - Records each page's type, title, and, for products, price and SKU from your structured data.
- Notices new pages, removed pages and price changes.
- Revisits about every 6 hours, with at least 0.6 seconds between requests, up to 500 pages per visit. You can also press Crawl now.
What it relies on
UnagiBot reads the HTML your server sends. It does not run JavaScript, so products drawn entirely in the browser will not be seen by the crawler, though they are still discovered when visitors reach them. Prices come from structured data, so add Product markup (most shop platforms already do).
Pages are identified by their URL path. Sites that tell products apart only by a query string (for example ?p=123) will see those pages merged into one.
Keeping it out of parts of your site
UnagiBot obeys robots.txt. To exclude an area:
User-agent: UnagiBot
Disallow: /account/
Disallow: /internal/To stop it entirely, use Disallow: /. Removing a site from your account also stops crawling and deletes what was collected.