I wanted a large audit to keep running after I closed my laptop, and to be able to search the results without loading them all into memory.
How it fits together
1Redis frontierde-duplication, politeness
2Go workersfetch, render, parse
3ClickHousecrawl data
4Results APINode.js, REST, webhooks
5React appaudits and monitoring
Implementation
Crawling
Go workers pull from a Redis frontier that handles de-duplication and politeness for each host. JavaScript sites are rendered in headless Chromium through Playwright.
Storing crawl results
Crawl data lives in ClickHouse. Moving it there in October 2026 shrank the PostgreSQL database from 77 GB to 587 MB. Pages and Lighthouse reports go to object storage.
Browsing large results
Large result sets use keyset pagination, because offset pagination took about 8.6 seconds three million rows in.
Analysis
Technical audits, page monitoring, Lighthouse audits, and content-gap and topical-authority analysis.
An API
A REST API with HMAC-signed webhooks for automated workflows.
crawl-zilla.com
Results
URLs per crawl on the largest plan
5MURLs per crawl on the largest plan
PostgreSQL size after moving crawl data to ClickHouse
77 GB → 587 MBPostgreSQL size after moving crawl data to ClickHouse