Bots
2,029,091 requests refused in the last 24 hours
A trust record should be able to say who reads it. This is the automated traffic aboutus.com receives: how much we turn away, which crawlers announce themselves and what our firewall states for each, and which networks send fleets. Sourced from the log drain, updated hourly, no IP addresses. The full observatory across all our sites, with the incidents log, is on dns.pizza →
- 11,074,018
- refused in the last 7 days
- 49%
- of ~22,449,928 requests
- 8,706,111 challenged
- 2,367,907 denied outright
exact, from the firewall stream
allowed traffic is a 10% sample, scaled
a real browser solves a challenge; a fleet does not
Crawlers that announce themselves
Named user-agents seen on aboutus.com this week, with the policy our firewall states for each. The share refused shows whether the policy holds. Per address-hour is requests per source address per hour: a fleet on rented addresses sits near 1, a person makes several.
| Crawler | Kind | Our policy | Requests, 7 days | Refused | Per address-hour |
|---|---|---|---|---|---|
| Applebot | Search engine | No rule | ~5,293,690 | 0.0% | 1.6 |
| Bingbot | Search engine | No rule | ~1,401,234 | 0.0% | 17.7 |
| shapbot | Self-declared bot | No rule | ~1,247,650 | 0.0% | 593 |
| ExaSearchBot | AI crawler | No rule | ~520,920 | 5.3% | 3.1 |
| ClaudeBot | AI crawler | No rule | ~488,230 | 0.0% | 85.5 |
| reflectionbot | Self-declared bot | No rule | ~47,980 | 100% | 2.7 |
| OAI-SearchBot | AI crawler | No rule | ~43,038 | 0.0% | 1.7 |
| rednote-websearch-bot | AI crawler | No rule | ~39,804 | 100% | 7.4 |
| SemrushBot | SEO crawler | Denied | ~26,009 | 100% | 3.2 |
| facebookexternalhit | Link preview | No rule | ~25,280 | 0.0% | 1.1 |
| Turnitin | Archive / research | No rule | ~15,242 | 100% | 459.9 |
| Googlebot | Search engine | No rule | ~14,525 | 0.0% | 2 |
| octensearchbot | Self-declared bot | No rule | ~9,746 | 100% | 13.8 |
| daum | Self-declared bot | No rule | ~9,666 | 100% | 420.3 |
| exauser | Self-declared bot | No rule | ~9,020 | 0.0% | 1.1 |
| facebookexternalhit impostor (wrong network) | impostor | No rule | ~8,252 | 0.3% | 1.5 |
| Unrecognised user-agent | Self-declared bot | No rule | ~7,942 | 48% | 3 |
| HeadlessChrome | Script or headless browser | No rule | ~5,807 | 2.5% | 4.7 |
| Qwantify | Search engine | No rule | ~5,480 | 0.0% | 1.2 |
| UptimeRobot | Monitor | No rule | ~4,899 | 7.9% | 1.2 |
| PerplexityBot | AI crawler | No rule | ~4,662 | 0.0% | 1.5 |
| wordpress | Self-declared bot | No rule | ~4,643 | 100% | 52.8 |
| Amazonbot | AI crawler | Challenged | ~4,355 | 100% | 1.2 |
| Python (requests/httpx/aiohttp) | Script or headless browser | No rule | ~4,173 | 100% | 3.4 |
| Electron | Script or headless browser | No rule | ~3,778 | 98% | 2.2 |
Networks
Where the requests come from, by autonomous system. A residential provider with real customers is mostly allowed and shows several requests per address per hour. A cloud or hosting network sending browser-shaped requests at one per address per hour is a fleet, and is challenged.
| Network | Requests, 7 days | Refused | Per address-hour | Mostly |
|---|---|---|---|---|
GOOGLE-CLOUD-PLATFORM AS396982 | ~11,040,519 | 2.7% | 138.1 | shapbot |
APPLE-ENGINEERING AS714 | ~5,516,438 | 0.0% | 1.6 | Applebot |
BYTEPLUS-AS-AP Byteplus Pte. Ltd. AS150436 | ~4,461,159 | 100% | 7.6 | browser user-agent |
AMAZON-02 AS16509 | ~3,193,197 | 5.3% | 17.6 | Node.js (fetch/undici) |
MICROSOFT-CORP-MSN-AS-BLOCK AS8075 | ~1,900,922 | 1.7% | 8.1 | Bingbot |
AMAZON-AES AS14618 | ~1,800,718 | 0.5% | 3.7 | Node.js (fetch/undici) |
ZEN-NET AS4229 | ~1,658,773 | 100% | 7.7 | browser user-agent |
TENCENT-NET-AP-CN Tencent Building, Kejizhongyi Avenue AS132203 | ~1,549,213 | 2.3% | 1.6 | browser user-agent |
AS-VULTR AS20473 | ~547,199 | 4.6% | 2.4 | browser user-agent |
LEASEWEB-USA-LAX AS395954 | ~502,935 | 0.0% | 2.6 | ExaSearchBot |
HWCLOUDS-AS-AP HUAWEI CLOUDS AS136907 | ~453,461 | 0.2% | 4.3 | browser user-agent |
HOSTROYALE AS203020 | ~363,393 | 49% | 1.2 | browser user-agent |
ASN-CXA-ALL-CCI-22773-RDC AS22773 | ~317,978 | 7.0% | 1.9 | browser user-agent |
HR-CUSTOMER AS207990 | ~315,824 | 24% | 1.3 | browser user-agent |
CHINA169-BACKBONE CHINA UNICOM China169 Backbone AS4837 | ~271,435 | 2.2% | 1.7 | browser user-agent |
Why this is on a trust site
Most of what reaches aboutus.com is not a person. Fleets wearing a stock desktop-browser user-agent arrive from cloud networks at one request per address per hour, and every one of them is billed before the firewall decides. Turning them away is what keeps the record free and fast for the people it is for. The firewall must never stop a real reader, so the one rule that outranks scraper defence is that a real browser always gets through; the observatory records the day we got that wrong.
What is not published: IP addresses (personal data), TLS fingerprints and firewall rule names (they tell an operator what to change). Method, the incidents log and the cost of refused traffic are on dns.pizza/bots. See also the methodology and status.
Feed generated 2026-11-22 02:25:01 UTC.