Bots
1,352,529 requests refused in the last 24 hours
A trust record should be able to say who reads it. This is the automated traffic aboutus.com receives: how much we turn away, which crawlers announce themselves and what our firewall states for each, and which networks send fleets. Sourced from the log drain, updated hourly, no IP addresses. The full observatory across all our sites, with the incidents log, is on dns.pizza →
- 16,487,154
- refused in the last 7 days
- 54%
- of ~30,402,054 requests
- 16,009,136 challenged
- 478,018 denied outright
exact, from the firewall stream
allowed traffic is a 10% sample, scaled
a real browser solves a challenge; a fleet does not
Crawlers that announce themselves
Named user-agents seen on aboutus.com this week, with the policy our firewall states for each. The share refused shows whether the policy holds. Per address-hour is requests per source address per hour: a fleet on rented addresses sits near 1, a person makes several.
| Crawler | Kind | Our policy | Requests, 7 days | Refused | Per address-hour |
|---|---|---|---|---|---|
| Bingbot | Search engine | No rule | ~1,353,959 | 0.0% | 23.3 |
| ExaSearchBot | AI crawler | No rule | ~1,041,145 | 18% | 7.7 |
| Applebot | Search engine | No rule | ~201,760 | 0.0% | 1.1 |
| ClaudeBot | AI crawler | No rule | ~111,849 | 0.1% | 105.2 |
| SemrushBot | SEO crawler | Denied | ~63,758 | 99% | 7.2 |
| HeadlessChrome | Script or headless browser | No rule | ~57,708 | 2.0% | 4.5 |
| facebookexternalhit | Link preview | No rule | ~39,890 | 0.0% | 1.1 |
| reflectionbot | Self-declared bot | No rule | ~36,298 | 99% | 1.8 |
| OAI-SearchBot | AI crawler | No rule | ~33,720 | 0.2% | 2.4 |
| shapbot | Self-declared bot | No rule | ~23,230 | 0.0% | 6.1 |
| Sogou | Search engine | No rule | ~18,695 | 100% | 1.1 |
| daum | Self-declared bot | No rule | ~18,047 | 100% | 564 |
| PerplexityBot | AI crawler | No rule | ~17,420 | 0.5% | 6.1 |
| Googlebot | Search engine | No rule | ~12,722 | 1.2% | 2 |
| Unrecognised user-agent | Self-declared bot | No rule | ~12,711 | 21% | 2.3 |
| kakaobot | Self-declared bot | No rule | ~9,186 | 100% | 1 |
| DuckDuckBot impostor (wrong network) | impostor | No rule | ~9,074 | 1.6% | 2.2 |
| SeznamBot | Search engine | No rule | ~8,155 | 0.1% | 2.4 |
| serpstatbot | SEO crawler | No rule | ~7,868 | 100% | 314.8 |
| Qwantify | Search engine | No rule | ~4,930 | 0.0% | 1.4 |
| Electron | Script or headless browser | No rule | ~4,762 | 100% | 2.2 |
| UptimeRobot | Monitor | No rule | ~4,590 | 8.5% | 1.2 |
| Turnitin | Archive / research | No rule | ~4,510 | 100% | 570.5 |
| exauser | Self-declared bot | No rule | ~3,604 | 100% | 1.4 |
| (no user-agent) | No user-agent | No rule | ~3,552 | 100% | 2.2 |
Networks
Where the requests come from, by autonomous system. A residential provider with real customers is mostly allowed and shows several requests per address per hour. A cloud or hosting network sending browser-shaped requests at one per address per hour is a fleet, and is challenged.
| Network | Requests, 7 days | Refused | Per address-hour | Mostly |
|---|---|---|---|---|
BYTEPLUS-AS-AP Byteplus Pte. Ltd. AS150436 | ~10,528,938 | 100% | 37.5 | browser user-agent |
ASN-CXA-ALL-CCI-22773-RDC AS22773 | ~3,648,635 | 0.8% | 4.1 | browser user-agent |
ZEN-NET AS4229 | ~3,592,999 | 100% | 37.8 | browser user-agent |
MICROSOFT-CORP-MSN-AS-BLOCK AS8075 | ~1,531,437 | 0.2% | 12.7 | Bingbot |
AMAZON-02 AS16509 | ~1,406,987 | 0.5% | 12.5 | ClaudeBot |
AMAZON-AES AS14618 | ~1,307,381 | 0.5% | 3.7 | Node.js (fetch/undici) |
AS-WAVE-1 AS11404 | ~1,117,193 | 0.8% | 3.3 | browser user-agent |
LEASEWEB-USA-LAX AS395954 | ~1,042,663 | 18% | 7.6 | ExaSearchBot |
CDNEXT AS212238 | ~955,333 | 5.6% | 2.2 | browser user-agent |
FACEBOOK AS32934 | ~697,554 | 4.1% | 3.1 | Meta-WebIndexer |
OSL-188 AS398781 | ~592,764 | 5.6% | 2.3 | browser user-agent |
M247 AS9009 | ~534,692 | 7.2% | 2.3 | browser user-agent |
GTT-BACKBONE GTT AS3257 | ~463,299 | 3.8% | 3.2 | browser user-agent |
RCN-AS AS6079 | ~445,204 | 2.8% | 2.6 | browser user-agent |
COMCAST-7922 AS7922 | ~384,801 | 9.1% | 1.8 | browser user-agent |
Why this is on a trust site
Most of what reaches aboutus.com is not a person. Fleets wearing a stock desktop-browser user-agent arrive from cloud networks at one request per address per hour, and every one of them is billed before the firewall decides. Turning them away is what keeps the record free and fast for the people it is for. The firewall must never stop a real reader, so the one rule that outranks scraper defence is that a real browser always gets through; the observatory records the day we got that wrong.
What is not published: IP addresses (personal data), TLS fingerprints and firewall rule names (they tell an operator what to change). Method, the incidents log and the cost of refused traffic are on dns.pizza/bots. See also the methodology and status.
Feed generated 2026-10-13 04:25:03 UTC.