ToweyaBot

ToweyaBot checks whether public websites offer machine-readable search that a search engine could query. Toweya is a search engine in development, and this check is part of a small research prototype.

What it does

• It reads robots.txt and your home page.
• If it recognises the site's platform, or the site advertises OpenSearch, it sends a few search requests to the public search URL for that platform, such as /?s=water&feed=rss2 on WordPress.
• It sends only a handful of requests per check, spaced several seconds apart, and stops if it is blocked or rate limited.
• It may send an occasional single search, about 20 a month per site, to learn which subjects your site can answer.
• When a person at Toweya reviews your site and has too little to go on, they can ask for one scan: up to 20 public pages, following robots.txt, one every few seconds.
• It stores measurements such as response time and result count, and words from result titles and scans, not your content.

How to opt out

To block the crawler, add this to robots.txt:

User-agent: ToweyaBot
Disallow: /

You can also email crawler@toweya.net from your domain, or ask us to stop or reduce requests. We will remove the domain from future checks.