Bot walls are a tax on automation.

The problem we kept hitting

You write a curl command in your terminal. It works. You drop it into a CI job, a cron, a monitoring check, a one-off scraper. It works for a week. Then a site upgrades its bot management, the request starts returning a 403, and you are standing in front of a choice: rewrite the pipeline around a headless browser, sign up for a paid scraping API, or pretend the failure does not matter.

None of those options are good. A headless browser is overkill for a request that is supposed to be 200 bytes of JSON. A paid scraping API is a monthly bill plus a vendor dependency for what was a single curl line. Ignoring the failure means your monitoring goes blind whenever someone else upgrades their WAF.

So we built the thing that should have existed: a curl that tries plain curl first, notices when it failed for fingerprinting reasons, and quietly escalates to whatever bypass is actually needed. Nothing more.

What recurl is

recurl is a CLI binary written in Rust. It wraps a real curl, forwards every standard flag, and intercepts the response. If the response looks blocked — 403, 429, a recognised challenge body — it escalates through two layers: TLS fingerprint impersonation via curl-impersonate, then a headless Chromium preflight that solves the challenge and replays the request through curl with the captured cookies.

That ordering matters. Plain curl is fast and has no side effects. Impersonation is slightly slower because it uses a different binary but still no browser. JS preflight is the heaviest layer and the only one that ever needs Chromium. The successful path is always the cheapest one. We do not pay browser cost on a request that did not need a browser.

What recurl refuses to be

It is not a scraping service. There is no API key, no quota, no usage reporting. The binary runs on your machine and the requests go directly from you to the destination. We do not see your traffic and have no business doing so.

It is not a paywall bypass. The bot-management vendors recurl interacts with are about distinguishing automated requests from human browsers; they are not access-control systems. If a site requires login, payment, or a ToS agreement, recurl will not get around that and is not designed to. Use it for the same scope you would use curl for.

It is not magic. There are sites whose bot detection is currently good enough to defeat recurl. When that happens, file an issue — the detector improves every time we see a new failure mode, and the project is MIT licensed precisely so that the work compounds.

Why a CLI and not a library

Because the curl-shaped hole in your workflow is at the shell. CI scripts run curl. Makefiles run curl. monitoring jobs run curl. We met you where you already were, and the upgrade is one alias.

There is a library API too, for people writing Rust, Python, or Node services that need the same behaviour inline. But the CLI is the primary interface because the CLI is where the friction was.

Who is building it

recurl is a neul labs project. The same team has spent the last several years writing scrapers, monitoring tooling, and automation pipelines that talk to other people's services — and watching every one of them eventually trip a bot wall. The codebase is small enough to read in an afternoon and architected as a state machine specifically so that contributors can wire in new bypass layers without rewriting the core.

The repo is at github.com/neul-labs/recurl. Documentation is at recurl.docs.neullabs.com. Issues are the fastest path to a fix.

How to engage

If recurl unblocks something you were going to write a Puppeteer script for, that is the win. If it does not, file the URL and we will look at the detector. If it broke a security test you were running on your own property because the escalation kicked in when you wanted the raw signal, use --recurl-strict or set RCURL_STRICT=1 and you get pure curl passthrough.

And if you want to contribute, the layers are independent and the test harness covers compliance against upstream curl flags. Pick a failing real-world URL, write a detector pattern or a new layer, send a PR.