UnwovenBot

Unwoven Ai is an Australian service that helps agencies and businesses see how their websites and brands appear in AI search. It is run by Unwoven Ai (ABN 84 210 905 156) in Melbourne, Australia.

UnwovenBot reads public web pages for our customers. It does this in two situations:

• Website set-up. When a customer adds a website, the bot reads its home page and up to eight more pages linked from it, such as the about, pricing, product and help pages. That is at most nine pages per read. When a customer names a competitor's website, the bot reads only its home page.
• Website audits. For a website whose owner has proven control of the domain with a DNS record, the bot checks the site for technical issues such as broken links, redirects, robots.txt rules, sitemaps and structured data. An audit can be a crawl of the site or a check of a single page. It stays on the host names verified for that site and does not follow links to other websites.

What it requests: web pages, your robots.txt file, your sitemaps and, during audits, the images, stylesheets and script files on the same site, to check that they load.

What it does not do:

• It does not run JavaScript.
• It does not log in, fill in or submit forms. It sends only GET and HEAD requests.
• It does not read anything your robots.txt file disallows for it.

How gently it reads:

• It waits at least one second between requests to the same host, and sends one request at a time.
• While an audit of your site is running, no other UnwovenBot read reaches that host.
• If your robots.txt sets a Crawl-delay (in seconds) for UnwovenBot, the bot waits at least that long between requests. If the delay is longer than the bot can keep within a read (for audits, more than 60 seconds), it stops instead of reading faster.
• Every read has fixed limits on time, pages and data. A single response is cut off at 1 MB.
• If your server answers "Too Many Requests" (429), the bot does not retry that request. If a page answers with a block or a bot challenge, the bot records that and does not read it.

UnwovenBot sends this User-Agent header:

UnwovenBot/1.0 (+https://unwoven.ai/bot)

The name to use in robots.txt is UnwovenBot.

Our servers are currently in Sydney, Australia. We do not publish a fixed list of IP addresses at this time.

Any request that uses this name but does not come from us is not ours. If you see one, tell us at the address below.

UnwovenBot follows the robots.txt standard (RFC 9309). It reads your robots.txt before it reads a page, and obeys the group for UnwovenBot, or the * group if there is no UnwovenBot group.

To block it from your whole site, add this to https://your-site/robots.txt:

User-agent: UnwovenBot
Disallow: /

To keep it out of one folder only:

User-agent: UnwovenBot
Disallow: /private/

To slow it down (here to one request every 10 seconds):

User-agent: UnwovenBot
Crawl-delay: 10

If your robots.txt cannot be read because your server returns an error or does not answer, the bot reads no pages from that site.

You can also contact us at the address below and we will help.

The bot stores what it reads, for the customer who asked for it: copies of the pages, your robots.txt file, the response status and a few response headers (content type, robots tags, links, redirect location, retry time, server name and firewall challenge markers). This is kept in that customer's private workspace in Unwoven Ai.

• Audit results are kept for 12 months and then deleted, unless the customer chooses to keep a specific result for longer.
• Website set-up reads: kept for 12 months and then deleted.

Questions, problems or requests about UnwovenBot: bot@unwoven.ai.

Last updated: 27 October 2026