v5.68.0 15 June 2026 Improvement

Search Indexing Foundations — Sitemap Architecture, Crawl Hygiene & SEO Observability

Why This Matters

SpeyBooks has been live long enough for Google to accumulate opinions about it — some of them wrong. Deleted changelog slugs, API endpoints that should never have been discoverable, a control panel indexed as public content, and a sitemap architecture that papered over its own gap with nginx redirects. None of this caused user-visible failures, but all of it eroded crawl budget, polluted Search Console coverage reports, and left the site in a worse position for the SpeyDocs scale-up that follows in the next weeks.

This release closes the debt. Every non-public property now carries a mechanically enforced noindex signal at the HTTP layer. Eight stale changelog slugs are redirected to their current canonical URLs. The sitemap is refactored into a proper index-plus-shard architecture that can accommodate thousands of documentation pages without structural change. A dedicated SEO observability log is wired across the estate. The foundations are correct; SpeyDocs can build on them.

Noindex Enforcement Across the Estate

  • Control panel noindex The React SPA at app.speybooks.com now carries a <meta name="robots" content="noindex, nofollow"> tag in the built HTML shell. A single tag covers every route in the application. The robots.txt was corrected to Allow: / so Google can crawl and read the tag rather than being blocked from doing so — the classic trap where blocking crawling prevents the noindex signal from being processed.

  • API subdomain noindex The REST API at api.speybooks.com now returns X-Robots-Tag: noindex, nofollow on every response from the /v1/ proxy block. The robots.txt was corrected to Allow: / for the same reason. Six API endpoint URLs that Google had indexed — including the Stripe webhook receiver — are now processing for removal from search results.

  • PHP endpoint noindex The contact and CSP reporting endpoints on speytech.com now return X-Robots-Tag: noindex, nofollow at the nginx location level. Neither endpoint serves HTML, so an HTTP header is the correct mechanism; a meta tag cannot live inside a non-HTML response.

Sitemap Architecture Refactored

  • Index-plus-shard structure The flat sitemap.xml (221 URLs, no index) has been replaced with a proper sitemap index at sitemap-index.xml pointing to sitemap-0.xml as the first shard. The index is the entry point submitted to Search Console. The shard carries all marketing pages, insights articles, and changelog releases with honest lastmod values derived from git history and frontmatter dates — no build timestamps.

  • Scale-ready for SpeyDocs When the SpeyDocs rebuild ships its 2000+ reference pages, a second shard (sitemap-1.xml) is added and one commented-out line in sitemap-index.xml.ts is uncommented. No structural change required. The index architecture was chosen specifically to support this without a migration.

  • @astrojs/sitemap removed The Astro sitemap integration was generating sitemap-index.xml and sitemap-0.xml at build time — files that nginx was immediately redirecting away from. The integration has been removed; the custom sitemap-0.xml.ts generator that replaced it carries honest dates and is the single source of truth.

  • Legacy continuity sitemap.xml now 301s to sitemap-index.xml for any external links or bookmarks that referenced the old URL. The two nginx redirect lines that papered over the missing sitemap-0.xml are removed.

Redirect Debt Cleared

  • Eight stale changelog slugs Eight URLs that Google indexed before the title-derived slug format settled now 301 to their current canonical pages. Two of these (5-18-0 and 5-22-0) had two stale slugs each pointing to the same current page. All eight return HTTP 301 with the correct Location header and their destination pages return 200.

  • Webhooks documentation redirect A documentation page at docs.speybooks.com/integrations/webhooks was indexed by Google from a source file that exists but was not being compiled to a built page. It now 301s to /api/guides/webhooks/ pending the SpeyDocs rebuild, which will restore the integrations section as a first-class content area.

www Certificate and Redirect Chain

  • www SSL certificate gap The www.speybooks.com subdomain was not included in the Let’s Encrypt certificate issued after the GCP to Hetzner migration. Google and other crawlers following HTTPS links to the www subdomain received an SSL certificate mismatch error before the redirect to the apex could fire. The certificate has been expanded to include www.speybooks.com. The redirect chain is now a clean single hop: https://www.speybooks.com/ → https://speybooks.com/.

SEO Observability

  • Dedicated SEO log wired across the estate speybooks.com and docs.speybooks.com now write to dedicated SEO logs using the seo_crawl format already in use on speytech.com, speysystems.com, and axilog.io. The format includes $host and $scheme, giving the crawler log analyser the context needed to distinguish apex vs www traffic and HTTP vs HTTPS on a per-request basis. The combined access log is unchanged.

Operational Impact

  • Google Search Console validation started within minutes of deploy for the API endpoint and app subdomain noindex fixes.
  • Googlebot fetched sitemap-index.xml and sitemap-0.xml within minutes of the sitemap architecture deploy.
  • No API surface changes. No breaking changes. No financial data affected.
  • All redirect chains verified as single-hop via curl.

No breaking API changes.