v5.69.0 15 June 2026 Improvement

SEO Infrastructure — IndexNow Integration, Crawl Observability & Search Hygiene

Why This Matters

Search indexing for an early-stage SaaS product is not a passive process. Without active signals, search engines crawl on their own schedule — which may be weeks after a release ships. Without structured logs, the only way to know whether Googlebot has visited is to check Search Console manually. Without proper noindex directives, API endpoints and internal PHP handlers accumulate in the coverage report as noise that erodes trust in the data.

This release closes all three gaps: search engines are notified within seconds of a deploy, crawl behaviour is observable in real time from the command line, and the coverage report is clean of endpoints that have no business being indexed.

IndexNow Integration

IndexNow is a protocol supported by Bing, Yandex, and others that allows sites to push change notifications to search engines immediately after a deploy rather than waiting for the next scheduled crawl. Googlebot does not participate in IndexNow but responds to Search Console URL inspection — the two approaches are complementary.

  • Wired into all three deploy pipelines speybooks.com, docs.speybooks.com, and axilog.io now submit changed URLs to the IndexNow API at the end of every successful deploy. The submission is diff-based: the script compares the pre-deploy and post-deploy sitemaps and submits only URLs where the lastmod value changed or a URL is new. A clean deploy with no content changes skips the API call entirely.

  • Production-grade snapshot pattern Each deploy pipeline captures a snapshot of the live sitemaps into an isolated mktemp -d temp directory before the build runs. An EXIT trap guarantees cleanup regardless of whether the deploy succeeds or fails at a gate. The snapshot directory holds all sitemap-*.xml shards alongside the index, so the Python script’s relative-path child resolution works correctly for both the old and new trees.

  • Argument-driven script interface Each site’s notify_indexnow.py receives --old-sitemap and --new-sitemap as explicit arguments from the deploy script rather than relying on hardcoded paths. This makes the contract between shell and Python explicit and testable independently.

  • Verification file deployed The IndexNow key file is now served from speybooks.com and docs.speybooks.com, confirmed reachable by Bing’s verification crawler.

Crawl Observability

The crawler log analyser (crawler-log-analyser) is a bespoke tool that parses nginx access logs and produces structured crawl health reports. Previously it ran against the combined access log, which mixes human traffic, bot traffic, and security probes into a single stream.

  • Dedicated seo_crawl log format deployed across all properties speytech.com, speybooks.com, docs.speybooks.com, and axilog.io now write a parallel SEO log alongside the standard access log. The format includes $host, $scheme, $request_time, and $upstream_response_time — fields the combined log lacks. The analyser detects this format automatically and enables latency percentiles and www-vs-apex redirect classification when present.

  • www-redirect block logging The www.speybooks.com HTTPS redirect block now writes to the SEO log, so www-to-apex redirect hops appear in the crawl timeline rather than being invisible. The first test run showed Googlebot arriving on www immediately after an IndexNow submission and being correctly redirected.

Search Console Hygiene

Nine URLs in the Search Console coverage report were returning 404. All nine are resolved.

speybooks.com — Changelog slug renames

Eight changelog entries were indexed under early-draft slugs before the title-derived slug format settled. Exact-match 301 redirects added to the nginx config map each old slug to its current canonical URL. Two slugs existed for the same release (5-18-0 and 5-22-0) — both redirect correctly.

docs.speybooks.com — Integrations section placeholder

/integrations/webhooks was indexed from a source file that exists in the repository but was not built into the static output. Both the trailing-slash and non-trailing-slash variants now redirect to /api/guides/webhooks/ until the integrations section is built out.

API endpoint noindex headers

contact.php and csp-report.php on speytech.com, and the /v1/ proxy block on api.speybooks.com, were appearing in Search Console because the X-Robots-Tag: noindex, nofollow header was absent. The header has been added to all three. The strategy is crawl-and-noindex rather than robots.txt block alone, because Google can only honour a noindex directive it can read — blocking crawl prevents it from seeing the instruction.

form-token.php returns 444 (connection close) from nginx and cannot receive a response header; the Disallow: /api/ entry in robots.txt handles it instead.

Operational Impact

  • IndexNow submission fires within seconds of a deploy completing, rather than waiting for the next Googlebot crawl cycle.
  • Nine Search Console 404s resolved; coverage report will clean up over the next 3-14 days as Google processes the reindexing requests.
  • Crawl health is now observable per-domain in real time: crawler_log_analyser.py /var/log/nginx/speybooks.com.seo.log.
  • API endpoints and PHP handlers no longer accumulate in the indexable URL set.
  • The seo_crawl log latency fields enable p95/p99 response time monitoring for crawler traffic specifically.

Files Changed

Backend: None

Infrastructure / nginx:

  • /etc/nginx/sites-available/speytech.com — X-Robots-Tag added to contact.php and csp-report.php location blocks; seo_crawl access log added to HTTP redirect, HTTPS www redirect, and canonical server blocks
  • /etc/nginx/sites-available/api.speybooks.com — X-Robots-Tag added to /v1/ proxy block
  • /etc/nginx/sites-available/speybooks.com — eight changelog slug 301 redirects added; seo_crawl access log added to www-redirect and canonical server blocks
  • /etc/nginx/sites-available/docs.speybooks.com — /integrations/webhooks redirects added; logging section added (access log, seo log, error log)

Deploy scripts:

  • monorepo.speybooks.com/deploy.sh — sitemap snapshot, mktemp -d temp dir with EXIT trap, IndexNow step after G8
  • monorepo.speybooks.com/scripts/notify_indexnow.py — new file; argument-driven IndexNow ping for speybooks.com
  • docs.speybooks.com/deploy.sh — sitemap snapshot, mktemp -d temp dir with EXIT trap, IndexNow step after G6
  • docs.speybooks.com/scripts/notify_indexnow.py — new file; argument-driven IndexNow ping for docs.speybooks.com
  • axilog.io/scripts/notify_indexnow.py — pre-existing; no changes required