A site with 40,000 URLs and a two-person SEO team cannot be optimized one page at a time. Semrush’s latest guide makes that plain: at scale, the page type—product pages, category pages, location pages, help articles—is the unit of work, not the individual URL.
When 4,000 category pages use the same template, fixing that template touches every one of them. Editing a single page remains a single page. That reframe changes how you find problems, set priorities, and ship fixes.
The guide groups large sites into three common shapes: ecommerce catalogs such as IKEA, multi-location service businesses such as Mr. Rooter Plumbing, and SaaS help centers such as Shopify’s documentation. The SEO fundamentals stay the same—crawling, indexing, content, internal links. The challenge is consistency across thousands of URLs.
Prioritize by business impact, not traffic volume
Start by organizing URLs into page types. If your URL structure is clean, folders like /products/, /collections/, /locations/ or /blog/ already map to your main types. Otherwise, your CMS classifications can be the fallback.
Next, score each type by how directly it contributes to revenue, leads, bookings or signups. A page type pulling 40,000 visits but few conversions can be less important than one bringing 3,000 visits that consistently drives sales. Use analytics, CRM or ecommerce data first; fall back to estimated organic performance per subfolder if needed.
Then add fixability. A type is easy to improve when many URLs share one template, rule or component. It is hard when problems vary page to page or require a redesign or migration. Work the high-impact, high-fixability groups first.
Recurring problems to fix once
Most large-site issues are shared-rule issues. The guide walks through the recurring ones, plus the checks that catch them.
- Taxonomy and URL structure: merge overlapping categories, remove weak hierarchy levels, and keep one consistent URL pattern per type.
- Crawling and indexation: watch access errors, robots.txt blocks, noindex mistakes, and filter or parameter URLs that waste crawl budget.
- Duplicate and near-duplicate content: product variants and filter combinations often create near-identical pages; decide whether they deserve to rank separately or need a canonical to the preferred URL.
- Hreflang: one broken tag—an invalid ‘en-UK’ code instead of ‘en-GB’—can hurt many localized pages.
- Internal linking: orphaned pages and pages buried more than three clicks from the homepage need shorter paths.
- JavaScript rendering: a shared template that fails to render links or content can hide whole sections from crawlers.
- Structured data: a template pulling the wrong price or availability field repeats the error across every product page.
Fix the rule, then monitor the type
At scale, the fix is almost always at the rule level. Update the canonical logic in the product template, correct the hreflang rule, or change the URL-generation pattern once—then rerun the audit to confirm the change stopped producing bad URLs.
Roll changes out carefully and monitor performance by page type. Large sites keep adding, changing and removing pages, so monitoring becomes the next round of priorities. The playbook is a loop: group, rank, fix the shared source, monitor, repeat.
Source: Semrush Blog
