Firecrawl

• Source: https://github.com/firecrawl/firecrawl
• License: AGPL-3
• Alternatives: Crawl4AI, Playwright, Scrapy, Jina Reader

Turns a URL into markdown a model can read, and searches the web.

Containers

Container	Role
firecrawl	REST API and job harness (port 3002)
firecrawl-mcp	MCP server — streamable HTTP at http://firecrawl-mcp:3003/mcp
firecrawl-playwright	Headless browser microservice (internal)
firecrawl-redis	Rate limiting and job-state cache
firecrawl-rabbitmq	Crawl job queue
firecrawl-postgres	Crawl results and job history

Access

• Containers: http://firecrawl:3002 on the exist network — unauthenticated
• Browser/LAN: https://firecrawl.EXIST_DOMAIN, which requires Authorization: Bearer $FIRECRAWL_API_KEY
• Agents: point an MCP client at http://firecrawl-mcp:3003/mcp
• Decree scripts: source workspace/.decree/lib/firecrawl.sh for web_search <query> [n] (results with each page's text) and web_read <url>. Every automation built from a note has them.

Search needs no account: with no search backend configured, Firecrawl queries DuckDuckGo itself. A search engine can rate-limit any one address, so a search can come back empty; a built automation is told to use only the sources it was given and to say when there are none.

Self-hosted Firecrawl has no authentication of its own: USE_DB_AUTHENTICATION=true needs Supabase, and with it false the server accepts any API key, including none. Caddy carries the bearer check on firecrawl.EXIST_DOMAIN instead, so that is the only path that is gated — anything already on the exist bridge can call the API without a key.

Enable

EXIST_IS_AI_FIRECRAWL=true

Then ./existential.sh && docker compose up -d from the repo root.

Notes

Its Postgres volume is volumes/firecrawl_postgres_data, declared db: true — local only, never NFS. Scraped content is a cache: if you lose it, re-crawl.