7 Signs You're Looking at a Bot (And How to Check in 10 Seconds)

You are looking at your server logs. There is a request from an IP you do not recognize, hitting a URL that does not exist on your site, with a User-Agent string that is either empty, suspiciously short, or a string of nonsense. Is it a real user, a malicious bot, or a search engine crawler that is just looking at the wrong URL?

Most of the time, you can tell within 10 seconds. The bot announces itself in the User-Agent string. If it is a sophisticated bot that is trying to hide, the other six signals (header order, request rate, URL pattern, IP, JavaScript, TLS fingerprint) will give it away. Here is the order to check.

Sign 1: The User-Agent string

The User-Agent header is the string a client sends to identify itself. Real browsers send something like Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15. Real bots send something like Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or AhrefsBot/7.0 or GPTBot/1.0. Empty User-Agent strings are almost always bots. Strings of random characters are always bots.

Paste any User-Agent into the Uttir User Agent Parser to see what the parser thinks it is. The parser uses a database of known bots and browser version strings, updated monthly. If the parser returns a known bot name, you have your answer.

Sign 2: The header order

Real browsers send headers in a specific order, and the order is part of the fingerprint. A Chrome browser on macOS sends headers in this order: Host, Connection, sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform, Upgrade-Insecure-Requests, User-Agent, Accept, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-User, Sec-Fetch-Dest, Accept-Encoding, Accept-Language, Cookie. A bot that is trying to fake a Chrome browser often sends the same headers but in the wrong order, or sends a subset of them.

The Uttir User Agent Parser also detects inconsistencies in the header set: a User-Agent that claims to be Chrome but is missing sec-ch-ua is not a real Chrome. A User-Agent that claims to be Safari but is missing Accept is not a real Safari.

Sign 3: The request rate

Real users make a few requests per minute, with gaps in between (reading time, scrolling time, idle time). Bots make dozens or hundreds of requests per second, with no gaps. A request rate above 60 RPM from a single IP is almost certainly a bot; a request rate above 600 RPM is definitely a bot.

The check is simple: pick an IP, count its requests in the last hour, divide by 60. If the result is above 1, it is probably a bot. If above 10, it is definitely a bot.

Sign 4: The requested URLs

Real users hit a small set of URLs (the pages they care about) in a way that looks like browsing: home page, then a content page, then another content page, then back to the home page, then a different content page. Bots hit URLs in a different pattern depending on their purpose:

• Search engine crawlers hit all URLs on the site, including the sitemap, robots.txt, and pages deep in the site hierarchy. They follow links.
• Scrapers hit a specific set of URLs (e.g. all product pages, all author pages) in a predictable order. They do not follow links; they enumerate.
• Vulnerability scanners hit URLs that should not exist (/wp-admin, /.env, /admin.php, /phpmyadmin). If your site is not WordPress and you are getting /wp-admin requests, it is a scanner.
• Comment spammers hit the form endpoint (/comment, /api/comment) with POST requests. If you are getting dozens of POSTs to a comment endpoint from one IP, it is a spammer.

Sign 5: The IP address

Real users come from residential or mobile IP ranges, assigned by ISPs to home users. Bots come from data center IP ranges (AWS, Google Cloud, DigitalOcean, Hetzner, OVH) or from known bot IP ranges (hosting providers used by scrapers). Look up the IP in a whois database — the Uttir IP Geolocation Lookup tool does this in the browser. If the IP is in a data center range and your site is consumer-facing, the visitor is almost certainly a bot.

This is not a perfect signal: a real user on a VPN will have a data center IP, and a sophisticated bot operator will route through residential proxy networks (which look exactly like real users). But for the average spammer or scraper, the IP is a giveaway.

Sign 6: The absence of JavaScript execution

Real browsers execute JavaScript. Bots that pretend to be browsers often do not. A simple test: serve a page that has a piece of JavaScript that makes a follow-up request to /js-ping with a unique token. If the initial request comes in but the /js-ping request does not, the client did not execute JavaScript. It is a bot.

This is how Cloudflare's Bot Fight Mode works: it serves a JavaScript challenge to suspicious requests. Real browsers pass the challenge; most bots do not.

Sign 7: The TLS fingerprint

Every TLS client (the thing initiating the encrypted connection) sends a "Client Hello" message that includes a list of supported cipher suites, extensions, and elliptic curves. The combination is unique to the client software: Chrome 117 on Windows has a different fingerprint from Firefox 119 on macOS, which is different from Python's requests library, which is different from Go's net/http. The JA3 hash of the Client Hello is a reliable fingerprint.

This is a more advanced signal; you usually need a server-side tool (Cloudflare, Akamai, or a custom JA3 collector) to capture the fingerprint. The other six signals catch 95% of bots; the TLS fingerprint catches the remaining 5% that are trying very hard to look like real browsers.

How to use the User-Agent parser

The Uttir User Agent Parser takes a User-Agent string and returns:

• The detected client type (browser, mobile browser, bot, library, generic)
• The browser name and version (if applicable)
• The operating system (if applicable)
• The device type (desktop, mobile, tablet, bot)
• Whether the User-Agent matches a known bot database (Googlebot, Bingbot, etc.)
• Whether the User-Agent has known inconsistencies (claiming to be Chrome but missing the sec-ch-ua header)

Paste a User-Agent from your logs, hit Parse, and you have a structured view of what the client is. From there, you can decide whether to allow, block, or challenge the request.

What to do with a confirmed bot

If the bot is a known good bot (Googlebot, Bingbot, DuckDuckBot), let it through. It is indexing your site for search engines, which is what you want.

If the bot is a known scraper (AhrefsBot, SemrushBot, MJ12bot), the right answer depends on your business. If you want your pages indexed, you probably do not need AhrefsBot. If you do not want to be scraped at all, block it. If you want to allow it but not at the rate it is going, rate-limit it.

If the bot is unknown or malicious (random User-Agent, requests to /.env, no JavaScript), block it. Most web servers and CDNs have a way to block by User-Agent, IP, or both. Cloudflare's Bot Fight Mode is a good default; it blocks most bad bots automatically and lets you whitelist the ones you want to allow.