A small SEO crawler in Node: the 20 checks worth running on every page (and a score that's easy to explain)

Most technical SEO problems are boring and findable: a template that drops the meta description, a staging noindex that shipped to production, 40 pages with the same title, links to pages that were deleted last year. A crawler finds all of them in minutes.

Here's the one I built, what it checks, and what it found on a real docs site.

Crawl politely

• Start at the home page, follow links on the same site (subdomains included), skip files (.pdf, .jpg, .zip…).
• Read robots.txt first and skip disallowed paths (longest matching rule wins).
• A few requests at a time, an honest User-Agent, and a hard page limit.

Parse with linkedom. It's a fraction of jsdom's weight and plenty for reading tags:

import { parseHTML } from 'linkedom';

const t0 = Date.now();
const res = await fetch(url, { headers: { 'User-Agent': 'my-seo-audit/1.0' }, redirect: 'follow' });
const responseTimeMs = Date.now() - t0;
const html = await res.text();
const { document } = parseHTML(html);

const meta = (k) => document.querySelector(`meta[name="${k}" i], meta[property="${k}" i]`)?.getAttribute('content')?.trim() || null;
const page = {
  title: document.querySelector('title')?.textContent.replace(/\s+/g, ' ').trim() || null,
  description: meta('description'),
  h1: [...document.querySelectorAll('h1')].map((h) => h.textContent.trim()).filter(Boolean),
  canonical: document.querySelector('link[rel="canonical" i]')?.getAttribute('href') || null,
  robots: (meta('robots') || '') + ' ' + (res.headers.get('x-robots-tag') || ''),
  imagesWithoutAlt: [...document.querySelectorAll('img')].filter((i) => !i.hasAttribute('alt')).length,
  viewport: !!meta('viewport'),
  lang: document.documentElement.getAttribute('lang'),
};

Note the X-Robots-Tag header: a noindex there doesn't show up in the HTML at all, and it's a classic leftover from staging.

The checks

Errors (the page is hurt):

• missing