About UnagiBot

UnagiBot is the web crawler used by Unagi, a website analytics service. If you found it in your server logs, this page explains what it is doing and how to stop it.

What it does

UnagiBot reads the public pages of a website so the site's owner can see which pages and products exist and when new ones appear or prices change. It reads HTML only. It does not submit forms, log in, run scripts, or download images or media.

It only visits sites whose owner asked it to

UnagiBot never crawls a domain on its own initiative. A site owner has to add the domain to their Unagi account and prove they control it, by placing a script tag, a meta tag or a verification file on the site. Until then, the only requests made are the home page and one verification file.

How it behaves

• Identifies itself as UnagiBot/1.0 with a link to this page.
• Obeys robots.txt, including Disallow, Allow, wildcards and Crawl-delay, and reads your sitemap.
• Waits at least 0.6 seconds between requests and stops after 500 pages per visit.
• Revisits a site about every 6 hours at most.
• Never follows links to other websites, and only uses ports 80 and 443.

Stopping it

Add this to your robots.txt and UnagiBot will stop:

User-agent: UnagiBot
Disallow: /

If you did not expect it

If UnagiBot is visiting a site whose owner did not ask for it, the domain was registered by someone who does not control it, which verification should prevent. Contact us with the domain and we will remove it.