I spent a long time believing that a blocked scraper was a headers problem. Wrong User-Agent, missing Accept-Language, something in that family. So I would copy a browser's headers field by field, hit the target again, and get the same 403. Then I would add a proxy. Same 403. Then I would add a delay. Same 403, just slower. The thing I was missing is that the server had already classified my client before my request headers arrived. It did it in the first packet of the TLS handshake, and no amount of header polishing reaches that packet. This post is about three things that, once I understood them, reorganized how I build fetchers: the handshake that identifies you, the line between what an HTTP client can present and what it structurally cannot, and the fact that a challenge page usually arrives wearing a 200. 1. The handshake is a fingerprint Before any HTTP request, your client opens a TLS connection. The first message it sends is the ClientHello. It is plaintext, it is not secret, and it is an announcement: here are the cipher suites I support, in my preferred order; here are the extensions I understand, in this order; here are the elliptic curves I accept; here are the signature algorithms; here is the ALPN list of protocols I am willing to speak afterwards. Your code chose none of this. The TLS library under your HTTP library chose it. Python's ssl module makes one set of choices. Go's crypto/tls makes another. Chrome's BoringSSL makes a third. The choices are stable per stack and different across stacks, which is exactly the property a fingerprint needs. The defense hashes the ordered set into a short identifier. The older scheme is JA3, the newer and more structured one is JA4. Then it looks the identifier up in a table. Real Chrome builds are in the table. Real Firefox builds are in the table. python-requests is also in the table, filed under automation. Here is what the two announcements look like side by side, abbreviated: real Chrome ClientHello: ciphers: [GREASE, 0x1301, 0x1302, 0x1303, 0xc02b, 0xc02f, ...] ext order: [GREASE, server_name, ..., GREASE, padding] curves: [GREASE, X25519, secp256r1, secp384r1] -> JA4 matches a known Chrome build
language-default ClientHello: ciphers: [0xc02c, 0xc030, 0x009f, ...] ext order: [server_name, ...] -> JA4 matches "known automation" Two details in there caught me out for a long time. Order is part of the signal. Two clients can support the identical set of ciphers and still fingerprint differently because they list them in a different sequence. Enabling the same algorithms as Chrome is insufficient. You have to offer them in Chrome's order, with Chrome's extensions in Chrome's positions. Browsers are deliberately messy. Those GREASE entries are meaningless values that browsers insert at randomized positions to keep the TLS ecosystem tolerant of unknown extensions. A naive imitation that omits them, or places them predictably, stands out precisely by being too tidy. A convincing handshake has to be messy in the exact way a real browser is messy. The fix: impersonate, do not reconstruct You do not rebuild Chrome's cipher order by hand. You use a library that links against a patched TLS stack and ships presets, each one a faithful copy of a specific browser version's ClientHello. In Python that library is curl_cffi : from curl_cffi.requests import AsyncSession session = AsyncSession ( impersonate = " chrome " ) # cipher order, extension order, curves, ALPN, GREASE padding
all reproduced from the named preset, not Python's ssl defaults resp = await session . get ( " https://example.com/ " ) One value names the browser. Everything below it follows. Two disciplines come with this, and both are where it breaks in the field. First, the impersonation target has to agree with the identity you claim elsewhere. If the handshake is Chrome 120's and your User-Agent says Chrome 131, the defense sees two identities that cannot both be true. That mismatch is more incriminating than an honest Python fingerprint would have been, because an honest library is merely automation while a contradiction is an attempt. Second, presets age. A preset that reproduces a three-year-old Chrome build is itself anomalous once almost no real user runs that build. Keeping the impersonation target on versions real users actually run is maintenance, not setup. What the handshake cannot do A perfect Chrome handshake from a data-center IP is still a perfect Chrome handshake from a data-center IP, and the address earns its own suspicion. The handshake also runs none of the JavaScript probes that come later if a challenge engages. It is the first thing the server reads and the cheapest thing to get right, and that is the whole of its job. 2. The twenty and the eighty This is the part that would have saved me the most time. Think about what actually leaves your machine when an HTTP client makes a request. The TLS handshake. The HTTP/2 settings and frame ordering above it. The request headers, including the whole family of client hints. The source IP. These travel as bytes, and an HTTP client can present all of them with complete fidelity, because presenting them is nothing more than writing the right bytes. Now think about what a defense's challenge script probes once it is running inside a browser. Screen dimensions and pixel density. A WebGL scene rendered and hashed, which depends on the exact GPU and driver combination. Text drawn to a canvas and measured, which exercises the font stack. A short computation pushed through the audio pipeline, where the rounding varies by CPU and OS. Hardware concurrency. Device memory. And the motion of the pointer, graded for humanity. Not one of those has a representation in an HTTP request. There is no header for the hash of a rendered scene, because the hash is the output of code running against a graphics stack that your HTTP client does not have. You can send a client hint claiming a screen width, but a hint is a claim and the script makes a measurement, and a defense that cares cross-checks the two. Roughly a fifth of the total fingerprint surface travels on the wire. The other four fifths exist only inside a running browser. Faking a WebGL hash over HTTP is a category error. It is the result of a computation your tool cannot perform. That line through the middle of the problem decides the architecture: A target that reads only the wire surfaces is served by an HTTP client, fast and cheap. Effort goes into making the wire presentation coherent. A target that probes the runtime or grades behavior cannot be served by any HTTP client. The choice narrows to running a real browser or leaving the target alone. Every system of any size I have seen ends up with both tools and a rule for choosing between them. Start cheap. Escalate only when the cheap path is refused. Keep the expensive tier idle until a target proves it needs it. There is one bridge across the line, and it is the most valuable pattern I know for the harder defenses: a browser can do the runtime work once, earn the trust token, and hand that token to the HTTP client to spend on a burst of fast requests. It works because a token, unlike a rendered-scene hash, is just bytes, and bytes travel. It comes with three conditions (same exit IP as the solve, client identity coherent with the solving browser, burst finished before the token expires) and each one is a place the naive version dies silently. That pattern gets its own chapter in the book; here I only want the line itself. 3. The challenge arrives as a 200 Everything above assumes you can tell when you were challenged. You mostly cannot, unless you build the detector on purpose. A defense rarely returns an honest error. It returns a page with a normal status code and a body of plausible size that is actually a shell: a script tag, an interstitial skeleton, and no content. A system that cannot distinguish the shell from the real page stores shells as data, counts challenges as successes, and burns its identities on the wrong signals. Two detectors, layered: Content markers. Every challenge script declares characteristic variables and identifiers that real content never contains. Keep a set per defense and scan every response body against it. False positives are rare. The cost is that markers drift as vendors revise their scripts, so a sudden rise in undetected shells is usually the first sign your marker set went stale. A size floor. The stricter defenses serve a deliberately economical challenge, a few KB with a terse marker a generic scan might miss. Real content has a weight that a shell falls far below. A body under the floor is treated as a shell regardless of status code. Here is a real trace of one work item crossing the line mid-crawl: client GET /product/48812 -> HTTP/2 200, body 9.4 KB
marker scan: "var chlgeId = ''" present size: 9.4 KB < 15 KB floor -> challenge wearing a 200 The important decision is what happens next. The naive reaction is to retry with a fresh identity. That earns the same shell, because the target is probing the runtime and no HTTP identity answers a runtime probe. The challenge is information about which tool the target requires: burn persona re-queue { url: /product/48812, force_js: True, priority: 0 } browser tier: warm, solve, serve -> 200, body 148 KB Which status codes mean what, in the retry policy I ended up with: BURN_AND_RETRY = frozenset ({ 429 , 403 , 503 }) # re-queue AND burn the identity RETRY_ONLY = frozenset ({ 500 , 502 , 504 }) # transient: re-queue, keep identity