Our crawler

What it is

It is a crawler run by SEOLUTION for a client who has asked us to audit their own site. It reads public pages only. It never signs in, never submits forms, never reaches a checkout or an admin area.

How it behaves

• It obeys robots.txt. A site that tells it to stay out is left alone.
• It reads at a gentle pace: a few pages a second at most, slower when a site asks.
• It identifies itself on every request with this name:

SEOLUTION-Crawler/1.0 (+https://www.seolution.com.au/crawler)

How to allow it

Two things, both done by whoever manages the site:

1. If your robots.txt sends SEO crawlers away, add a group that lets this one in:

   User-agent: SEOLUTION-Crawler
   Allow: /

2. If a security screen challenges automated visitors, add an exception for requests whose user-agent contains SEOLUTION-Crawler. On Cloudflare that is a WAF custom rule with the action Skip, matched on the user-agent. Shopify stores instead create a crawler signature under Online Store, Preferences, Crawler access, and paste it into the SEOLUTION app.

If your rules need a fixed network address rather than a name, write to us through the app and we will give you one.

How to refuse it

User-agent: SEOLUTION-Crawler
Disallow: /

The crawler stops at the next read, and the client's SEOLUTION account shows that the site cannot be read.