Bots
1,367,228 requests refused in the last 24 hours
A trust record should be able to say who reads it. This is the automated traffic aboutus.com receives: how much we turn away, which crawlers announce themselves and what our firewall states for each, and which networks send fleets. Sourced from the log drain, updated hourly, no IP addresses. The full observatory across all our sites, with the incidents log, is on dns.pizza →
- 9,595,825
- refused in the last 7 days
- 30%
- of ~32,342,475 requests
- 7,748,184 challenged
- 1,847,641 denied outright
exact, from the firewall stream
allowed traffic is a 10% sample, scaled
a real browser solves a challenge; a fleet does not
Crawlers that announce themselves
Named user-agents seen on aboutus.com this week, with the policy our firewall states for each. The share refused shows whether the policy holds. Per address-hour is requests per source address per hour: a fleet on rented addresses sits near 1, a person makes several.
| Crawler | Kind | Our policy | Requests, 7 days | Refused | Per address-hour |
|---|---|---|---|---|---|
| Bingbot | Search engine | No rule | ~1,969,252 | 0.0% | 31.1 |
| YandexBot impostor (wrong network) | impostor | Challenged | ~603,342 | 100% | 15.8 |
| GoogleOther | AI crawler | No rule | ~420,170 | 0.0% | 31.9 |
| ExaSearchBot | AI crawler | No rule | ~337,098 | 100% | 1.2 |
| Googlebot | Search engine | No rule | ~180,236 | 0.0% | 8 |
| serankingbacklinksbot | Self-declared bot | Denied | ~117,057 | 100% | 1045.2 |
| Applebot | Search engine | No rule | ~112,280 | 0.0% | 1.1 |
| Baiduspider | Search engine | Challenged | ~72,154 | 100% | 2.2 |
| SemrushBot | SEO crawler | Denied | ~68,805 | 99% | 7.3 |
| Bytespider (ByteDance) | AI crawler | Challenged | ~40,310 | 100% | 1.6 |
| facebookexternalhit | Link preview | No rule | ~32,540 | 0.0% | 1.1 |
| daum | Self-declared bot | No rule | ~21,953 | 100% | 448 |
| Turnitin | Archive / research | No rule | ~21,323 | 100% | 709.4 |
| reflectionbot | Self-declared bot | No rule | ~18,460 | 97% | 2.8 |
| DuckAssistBot | AI crawler | No rule | ~14,220 | 0.0% | 7.9 |
| Python (requests/httpx/aiohttp) | Script or headless browser | No rule | ~13,946 | 100% | 23.4 |
| Meta-WebIndexer | AI crawler | Challenged | ~12,954 | 100% | 1.3 |
| OAI-SearchBot | AI crawler | No rule | ~9,682 | 0.0% | 1.6 |
| Amazonbot | AI crawler | Challenged | ~8,167 | 100% | 1.1 |
| DuckDuckBot impostor (wrong network) | impostor | No rule | ~8,009 | 1.6% | 2 |
| Electron | Script or headless browser | No rule | ~7,508 | 100% | 2.5 |
| backlinksextendedbot | Self-declared bot | No rule | ~6,880 | 0.0% | 4.2 |
| UptimeRobot | Monitor | No rule | ~5,320 | 7.7% | 1.2 |
| SleepBot | SEO crawler | No rule | ~5,099 | 100% | 2.6 |
| Unrecognised user-agent | Self-declared bot | No rule | ~4,444 | 67% | 3.1 |
Networks
Where the requests come from, by autonomous system. A residential provider with real customers is mostly allowed and shows several requests per address per hour. A cloud or hosting network sending browser-shaped requests at one per address per hour is a fleet, and is challenged.
| Network | Requests, 7 days | Refused | Per address-hour | Mostly |
|---|---|---|---|---|
COMCAST-7922 AS7922 | ~4,823,981 | 1.2% | 4.9 | browser user-agent |
BYTEPLUS-AS-AP Byteplus Pte. Ltd. AS150436 | ~2,994,107 | 100% | 13.6 | browser user-agent |
RCN-AS AS6079 | ~2,964,218 | 0.8% | 5.7 | browser user-agent |
ZEN-NET AS4229 | ~2,923,663 | 100% | 13.7 | browser user-agent |
MICROSOFT-CORP-MSN-AS-BLOCK AS8075 | ~2,146,578 | 0.7% | 17.7 | Bingbot |
BT-UK-AS BTnet UK Regional network AS2856 | ~1,945,443 | 1.2% | 3.7 | browser user-agent |
LUMEN-LEGACY-L3-CUSTOMER-SHARED-USE AS10753 | ~1,559,033 | 2.9% | 2 | browser user-agent |
GTT-BACKBONE GTT AS3257 | ~1,294,160 | 11% | 4.7 | browser user-agent |
AMAZON-AES AS14618 | ~1,255,976 | 1.0% | 5.5 | Node.js (fetch/undici) |
AS-WAVE-1 AS11404 | ~1,186,219 | 2.1% | 2.3 | browser user-agent |
OSL-188 AS398781 | ~832,000 | 3.0% | 4.3 | browser user-agent |
COMCAST-33287 AS33287 | ~774,453 | 0.9% | 5.4 | browser user-agent |
GOOGLE AS15169 | ~644,608 | 0.2% | 12 | GoogleOther |
TELETECH AS208398 | ~609,090 | 99% | 15.8 | YandexBot impostor (wrong network) |
UUNET AS701 | ~557,815 | 2.4% | 3.1 | browser user-agent |
Why this is on a trust site
Most of what reaches aboutus.com is not a person. Fleets wearing a stock desktop-browser user-agent arrive from cloud networks at one request per address per hour, and every one of them is billed before the firewall decides. Turning them away is what keeps the record free and fast for the people it is for. The firewall must never stop a real reader, so the one rule that outranks scraper defence is that a real browser always gets through; the observatory records the day we got that wrong.
What is not published: IP addresses (personal data), TLS fingerprints and firewall rule names (they tell an operator what to change). Method, the incidents log and the cost of refused traffic are on dns.pizza/bots. See also the methodology and status.
Feed generated 2026-09-03 04:25:01 UTC.