← home · use cases · # scraping
Scraper developers and data engineersResilient scraping pipelines
# the problem
A scraping script works on your laptop and 403s in production. The difference is not the code — it is the TLS handshake and HTTP/2 behaviour the target now fingerprints. Rewriting the whole pipeline in a browser-automation framework is expensive and slow.
# what recurl does
Keep the existing curl-based pipeline and alias curl to recurl. Unprotected endpoints run at plain-curl speed with no added cost; protected ones escalate through TLS impersonation and, only if needed, a headless-browser preflight — then replay through curl so the output format your parser expects is unchanged.
# how to do it
Step by step
- 1
Alias curl=recurl (or point the pipeline at the recurl binary) — no flag changes needed.
- 2
Let recurl run plain curl on the happy path so bulk unprotected fetches stay fast.
- 3
On blocked hosts, recurl auto-escalates to impersonation, then a JS preflight, and replays via curl.
- 4
Use --recurl-strict for endpoints where you deliberately want the raw, un-escalated signal.
# authorised use
Only scrape sites you are authorised to fetch from — public APIs, your own environments, or sources you have permission to access. recurl shares curl’s ethical scope and does not bypass authentication or paywalls.
Related
See the escalation architecture, browse more use cases, or follow a guide.