Skip to content
George ProdromouAI, hosting and technical SEO
All work
Live

CrawlZilla

CrawlZilla checks large websites for technical SEO issues.

I built the crawler and the reporting interface.

Why I built it

I wanted a large audit to keep running after I closed my laptop, and to be able to search the results without loading them all into memory.

How it fits together

  1. Redis frontierde-duplication, politeness
  2. Go workersfetch, render, parse
  3. ClickHousecrawl data
  4. Results APINode.js, REST, webhooks
  5. React appaudits and monitoring

Implementation

Crawling
Go workers pull from a Redis frontier that handles de-duplication and politeness for each host. JavaScript sites are rendered in headless Chromium through Playwright.
Storing crawl results
Crawl data lives in ClickHouse. Moving it there in October 2026 shrank the PostgreSQL database from 77 GB to 587 MB. Pages and Lighthouse reports go to object storage.
Browsing large results
Large result sets use keyset pagination, because offset pagination took about 8.6 seconds three million rows in.
Analysis
Technical audits, page monitoring, Lighthouse audits, and content-gap and topical-authority analysis.
An API
A REST API with HMAC-signed webhooks for automated workflows.
The CrawlZilla home page
crawl-zilla.com

Results

URLs per crawl on the largest plan
5MURLs per crawl on the largest plan
PostgreSQL size after moving crawl data to ClickHouse
77 GB → 587 MBPostgreSQL size after moving crawl data to ClickHouse

Stack

  • Go
  • Redis
  • Node.js
  • Express
  • ClickHouse
  • PostgreSQL
  • Playwright
  • React
  • Vite
  • Tailwind CSS

More work