User-Agent Analysis: Spotting Bots, Scanners, Scripted Clients and C2

A header that lies, usefully

The User-Agent header is a free-text string the client chooses to send. Any script can claim to be Chrome on Windows. So a UA never proves anything.

What it does do is leak habits. Attackers copy UAs from old blog posts, forget to update them for years, use library defaults, or pick a browser string that contradicts every other header in the request. Defenders who look at UAs in aggregate, rather than one at a time, find those habits quickly. That is the whole method of this article: parse, count, compare.

Anatomy of a User-Agent string

A modern desktop Chrome UA looks like this:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36

Almost every token is historical baggage. Mozilla/5.0 is claimed by nearly every browser for compatibility. AppleWebKit and Safari appear in Chrome because Blink forked from WebKit. The useful parts are:

Note the .0.0.0. Chromium's User-Agent reduction freezes the minor, build and patch numbers and the OS version in the UA string, moving detail into Client Hints instead. A UA claiming a recent Chrome major version with a full build number like Chrome/129.0.6668.89 did not come from a stock, current Chrome desktop browser. It came from something that built the string itself.

Parsing by hand is error-prone, so feed suspicious strings to a User-Agent parser to split browser, engine, OS and device cleanly before you reason about them.

The usual suspects: scripted and tool UAs

Many clients announce themselves honestly because nobody bothered to change the default:

python-requests/2.32.3
curl/8.7.1
Go-http-client/1.1
Wget/1.21.4
okhttp/4.12.0
libwww-perl/6.72

On a corporate proxy, these are not automatically bad (developers, monitoring, package managers), but each should map to a known host and purpose. A python-requests UA from a finance laptop is worth a question.

Some Windows built-ins have distinctive UAs, which is useful because they are popular LOLBins for downloading payloads:

Microsoft-CryptoAPI fetching certificate revocation lists from known CAs is normal. The same UA downloading an .exe or a .txt from a fresh domain is a well-documented download-cradle pattern. See detecting LOLBins for the process-side view of the same behaviour.

Scanners

Internet-wide scanners and vulnerability tools often leave their names in the UA, at least in default configurations: zgrab, masscan, Nmap Scripting Engine, sqlmap, Nikto. Those hits in your web logs are background radiation for anything internet-facing. They matter when they show up from inside your network, or when a scanner UA is followed minutes later by successful requests from the same source.

Technique 1: stack counting

Stack counting (least frequency of occurrence) is the core hunting move. Group all proxy or web logs over a period by UA, count requests and distinct hosts, and sort ascending.

SELECT user_agent,
       COUNT(*)                 AS requests,
       COUNT(DISTINCT src_host) AS hosts,
       MIN(ts) AS first_seen,
       MAX(ts) AS last_seen
FROM proxy_logs
WHERE ts > now() - INTERVAL 30 DAY
GROUP BY user_agent
ORDER BY hosts ASC, requests DESC
LIMIT 200;

In a managed fleet, the browser UA distribution is narrow: a few Chrome and Edge versions, maybe Firefox, plus update agents. The interesting rows are at the bottom of the curve:

• One host, one UA, many requests. A beacon with a custom UA, or a misbehaving tool.
• A browser UA that is years out of date on a fleet that auto-updates.
• A UA seen only at night or only to one destination.

The tail is where you spend your time. Twelve rare UAs is an afternoon of triage; twelve thousand alerts on "suspicious UA" is a lost week.

Technique 2: internal consistency

A UA is one claim among many in a request. Real browsers are consistent; fakes often are not.

Client Hints mismatches

Chromium-based browsers send User-Agent Client Hints over HTTPS. Low-entropy hints are sent by default:

Sec-CH-UA: "Chromium";v="129", "Not=A?Brand";v="8", "Google Chrome";v="129"
Sec-CH-UA-Mobile: ?0
Sec-CH-UA-Platform: "Windows"

Firefox and Safari do not send these headers. That gives you two simple checks wherever your proxy or WAF logs request headers:

• UA claims Chrome or Edge, request is HTTPS, no Sec-CH-UA at all. Likely a script that copied a Chrome UA string. (Some legitimate embedded or headless clients do this too, so treat it as a lead.)
• UA and hints disagree. UA says Windows, Sec-CH-UA-Platform says "Linux"; UA says Chrome 129, Sec-CH-UA says 120.

Other cheap consistency checks

• Header order and set. Browsers send Accept, Accept-Language, Accept-Encoding and, for navigations, the Sec-Fetch-* family. A "Chrome" request with no Accept-Language and no Sec-Fetch-Mode was not made by a browser.
• TLS fingerprint vs UA. If you collect JA3 or JA4 fingerprints, a Chrome UA on a TLS handshake that matches a Python or Go library is a strong mismatch.
• Host context. A macOS Safari UA from a Windows-only fleet, or a mobile UA from a server VLAN.
• Behaviour. Browsers fetch pages, then scripts, images and fonts. A "browser" that requests one URL every 60 seconds and never loads a sub-resource is not browsing.

Technique 3: C2 beacons and their UAs

C2 frameworks let operators set any UA they like, typically through a profile. In practice, you will see three patterns:

1. Stale defaults. Profiles copied from public repositories often carry old Internet Explorer or Trident-era strings (compatible; MSIE ...; Trident/...). On a modern fleet with no IE traffic, any MSIE UA is worth a look.
2. A plausible browser UA that never changes. Real browsers update; the UA on a beacon stays frozen at whatever the operator chose months ago. When the rest of the fleet has moved two major Chrome versions on and one host still reports the old one on a fixed interval, that is a lead.
3. The UA of the process that should not be talking. A beacon injected into a non-browser process may still send a browser UA, while EDR shows the connection coming from rundll32.exe or a random service. Correlating proxy UA with the process name from endpoint telemetry is one of the highest-value joins you can build.

Combine the UA with timing. A fixed interval with small jitter, small and similar response sizes, and a single destination are classic beacon traits. The Cobalt Strike hunting guide covers the network and memory side in depth, including malleable profile behaviour.

A short triage workflow

When a UA lands on your desk from a hunt or an alert:

1. Parse it. Browser, version, OS, device. Is the version plausible today? Is the format right for that browser (frozen .0.0.0 for recent Chromium)?
2. Count it. How many hosts, how many requests, since when? One host or the whole fleet?
3. Check consistency. Client hints, header set, TLS fingerprint, host OS, originating process.
4. Check the destination. Look up the domain on /domain/ and the URL on /url. A rare UA to a domain registered last week is a very different story from a rare UA to a vendor's update server.
5. Decide and document. Benign tool with an owner, add it to an allowlist with a note. Unknown, open a case on /cases and pivot from the host.

If you are new to structured hunts, your first threat hunt shows how to frame a hypothesis like "a host is beaconing with a non-browser client" and turn it into queries.

What to do Monday

• Make sure your proxy logs the UA, and ideally Sec-CH-UA and JA3/JA4. You cannot hunt on headers you drop.
• Run a 30-day UA stack count grouped by distinct hosts. Review the bottom 50.
• Build three detections: Chromium UA over HTTPS without client hints; Microsoft-CryptoAPI or Microsoft BITS fetching executables from non-vendor domains; MSIE/Trident UAs on a fleet that has no IE.
• Join proxy UA with EDR process names for outbound connections. A browser UA from a non-browser process is one of the best signals in this whole area.
• Keep an allowlist with owners for every scripted UA you accept.

The UA is a claim, and claims are cheap. Inconsistencies between claims are not, and that is what you hunt.