Taming hundreds of millions of URLs at one of America's largest retailers

Macys.com has one of the largest product catalogues on the open web. Combinatorial faceted navigation, editorial surfaces, and campaign pages pushed the URL footprint into the hundreds of millions — well past the point where crawl budget is a theoretical concern.

Google was spending crawl on near-duplicates and thin facet combinations; the pages the business actually wanted to rank weren't being discovered or refreshed often enough.

• Log-file and crawl analysis at source — identifying which URL families were consuming crawl budget, which were driving revenue, and which should never have been crawlable in the first place.
• Designed a facet policy separating indexable combinations (those with real search demand) from non-indexable ones, enforced via canonical, robots, and internal-linking rules — not by manual curation.
• Recommended an automated programmatic landing page system: demand signals (GSC queries, internal search, paid keyword data) feeding a template that generated new hub pages with clean URLs, schema, and internal links.
• Proposed a crawl-prioritisation model: XML sitemaps segmented by value and freshness, last-modified discipline across templates, and a deliberate approach to pruning long-dead legacy URLs.
• Partnered with in-house SEO and engineering to turn the recommendations into concrete architectural changesthat fit into their existing platform roadmap.

At this scale, the highest-impact decisions are the ones that remove pages from the crawl graph, not the ones that add pages to it. The facet policy and pruning plan together reshaped where Google spent its crawl budget and let the programmatic landing page system actually get discovered.

A clear architectural playbook for managing a hundreds-of-millions-of-URL inventory — with a prioritised roadmap the in-house team could execute against.

Engagement specifics remain covered by NDA; details above are generalised to the architectural approach.