AboutUsThe independent trust record for every website

Bots

2,029,091 requests refused in the last 24 hours

A trust record should be able to say who reads it. This is the automated traffic aboutus.com receives: how much we turn away, which crawlers announce themselves and what our firewall states for each, and which networks send fleets. Sourced from the log drain, updated hourly, no IP addresses. The full observatory across all our sites, with the incidents log, is on dns.pizza →

11,074,018
refused in the last 7 days

exact, from the firewall stream

49%
of ~22,449,928 requests

allowed traffic is a 10% sample, scaled

8,706,111 challenged
2,367,907 denied outright

a real browser solves a challenge; a fleet does not

Crawlers that announce themselves

Named user-agents seen on aboutus.com this week, with the policy our firewall states for each. The share refused shows whether the policy holds. Per address-hour is requests per source address per hour: a fleet on rented addresses sits near 1, a person makes several.

CrawlerKindOur policyRequests, 7 daysRefusedPer address-hour
ApplebotSearch engineNo rule~5,293,6900.0%1.6
BingbotSearch engineNo rule~1,401,2340.0%17.7
shapbotSelf-declared botNo rule~1,247,6500.0%593
ExaSearchBotAI crawlerNo rule~520,9205.3%3.1
ClaudeBotAI crawlerNo rule~488,2300.0%85.5
reflectionbotSelf-declared botNo rule~47,980100%2.7
OAI-SearchBotAI crawlerNo rule~43,0380.0%1.7
rednote-websearch-botAI crawlerNo rule~39,804100%7.4
SemrushBotSEO crawlerDenied~26,009100%3.2
facebookexternalhitLink previewNo rule~25,2800.0%1.1
TurnitinArchive / researchNo rule~15,242100%459.9
GooglebotSearch engineNo rule~14,5250.0%2
octensearchbotSelf-declared botNo rule~9,746100%13.8
daumSelf-declared botNo rule~9,666100%420.3
exauserSelf-declared botNo rule~9,0200.0%1.1
facebookexternalhit impostor (wrong network)impostorNo rule~8,2520.3%1.5
Unrecognised user-agentSelf-declared botNo rule~7,94248%3
HeadlessChromeScript or headless browserNo rule~5,8072.5%4.7
QwantifySearch engineNo rule~5,4800.0%1.2
UptimeRobotMonitorNo rule~4,8997.9%1.2
PerplexityBotAI crawlerNo rule~4,6620.0%1.5
wordpressSelf-declared botNo rule~4,643100%52.8
AmazonbotAI crawlerChallenged~4,355100%1.2
Python (requests/httpx/aiohttp)Script or headless browserNo rule~4,173100%3.4
ElectronScript or headless browserNo rule~3,77898%2.2

Networks

Where the requests come from, by autonomous system. A residential provider with real customers is mostly allowed and shows several requests per address per hour. A cloud or hosting network sending browser-shaped requests at one per address per hour is a fleet, and is challenged.

NetworkRequests, 7 daysRefusedPer address-hourMostly
GOOGLE-CLOUD-PLATFORM
AS396982
~11,040,5192.7%138.1shapbot
APPLE-ENGINEERING
AS714
~5,516,4380.0%1.6Applebot
BYTEPLUS-AS-AP Byteplus Pte. Ltd.
AS150436
~4,461,159100%7.6browser user-agent
AMAZON-02
AS16509
~3,193,1975.3%17.6Node.js (fetch/undici)
MICROSOFT-CORP-MSN-AS-BLOCK
AS8075
~1,900,9221.7%8.1Bingbot
AMAZON-AES
AS14618
~1,800,7180.5%3.7Node.js (fetch/undici)
ZEN-NET
AS4229
~1,658,773100%7.7browser user-agent
TENCENT-NET-AP-CN Tencent Building, Kejizhongyi Avenue
AS132203
~1,549,2132.3%1.6browser user-agent
AS-VULTR
AS20473
~547,1994.6%2.4browser user-agent
LEASEWEB-USA-LAX
AS395954
~502,9350.0%2.6ExaSearchBot
HWCLOUDS-AS-AP HUAWEI CLOUDS
AS136907
~453,4610.2%4.3browser user-agent
HOSTROYALE
AS203020
~363,39349%1.2browser user-agent
ASN-CXA-ALL-CCI-22773-RDC
AS22773
~317,9787.0%1.9browser user-agent
HR-CUSTOMER
AS207990
~315,82424%1.3browser user-agent
CHINA169-BACKBONE CHINA UNICOM China169 Backbone
AS4837
~271,4352.2%1.7browser user-agent

Why this is on a trust site

Most of what reaches aboutus.com is not a person. Fleets wearing a stock desktop-browser user-agent arrive from cloud networks at one request per address per hour, and every one of them is billed before the firewall decides. Turning them away is what keeps the record free and fast for the people it is for. The firewall must never stop a real reader, so the one rule that outranks scraper defence is that a real browser always gets through; the observatory records the day we got that wrong.

What is not published: IP addresses (personal data), TLS fingerprints and firewall rule names (they tell an operator what to change). Method, the incidents log and the cost of refused traffic are on dns.pizza/bots. See also the methodology and status.

Feed generated 2026-11-22 02:25:01 UTC.