feat(sitemap): add accurate lastmod from each doc's last content change

[amaan-bhati]

What has changed?

Adds an accurate <lastmod> to the docs sitemap, and fixes the TechArticle dateModified (JSON-LD) on every doc page.

Before this PR:

• The sitemap had no <lastmod> (the sitemap plugin's lastmod option was unset).
• The deploy checkout was shallow (fetch-depth: 1), so every page reported the date of the newest commit. Live check: every page I sampled had the same dateModified (2026-10-11).

What this PR does:

• Content-based last-update date (src/lastUpdate/contentLastUpdate.js): for each doc, walks git history and uses the last commit that changed the Markdown body.
  • Skips commits that only touch front matter, rendering-neutral whitespace (CRLF, trailing spaces, extra blank lines), or SEO boilerplate (the link list under ## Related / ## Related Terms, and <HowTo /> blocks marked visible={false}). Indentation still counts, since it changes Markdown structure.
  • Follows copies and renames, so cutting a docs version (e.g. v3 to v4) does not reset every page's date.
  • Commits that change bodies without changing meaning can be listed in .lastmod-ignore-revs (works like .git-blame-ignore-revs).
  • A hand-written last_update.date in front matter still wins.
  • Injected through markdown.parseFrontMatter, so no doc files change.
• Sitemap: lastmod: "date" in the sitemap plugin config.
• CI: fetch-depth: 0 in main.yml (deploy) and build_and_check.yml (PR preview).
• Ignored sweeps: perf(docs): core web vitals — self-host fonts, drop dead assets, defer pixel, add security headers #896 (media tags only) and feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897 (SEO sweep) are listed in .lastmod-ignore-revs. feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897 was squash-merged, so two pages lose a small real edit and keep an older date.
• Guards: a shallow clone fails the build on GitHub Actions and only warns elsewhere (dev, preview hosts). scripts/verify-lastmod.js (npm run verify-lastmod, run in build_and_check.yml) fails if under 90% of sitemap URLs have <lastmod>, under 10 distinct dates, or one date covers over 50%.

Why: Google and Bing use <lastmod> only when it is consistently accurate. Dates that move without real changes get ignored for the whole sitemap.

Type of change

• New feature (non-breaking change which adds functionality).
• Bug fix (non-breaking change which fixes an issue).

How Has This Been Tested?

• yarn build passes; npm run verify-lastmod and npm run verify-schema pass.
• Current docs (v4, 209 pages): every page gets a date, 46 distinct dates from 2023 to 2026, largest cluster 2026-09-30 (docs: one Keploy across the site - no editions, accurate native macOS/Windows, eBPF on Linux #918, a real rewrite) at 72 pages. No page lands on the v4 cut (2026-01-21) or on feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897 (2026-09-01).
• verify-lastmod on a sitemap rewritten to one date: exits 1.
• Shallow clone with GITHUB_ACTIONS=true: build throws.

Checklist:

• My code follows the style guidelines of this project.
• I have performed a self-review of my own code.

[amaan-bhati]

Claude Review Skill: Iteration 1

Verdict: 🔄 REQUEST CHANGES

1 blocking, 1 important, 4 nits. Reviewed at 0cf7bf1. Ran the module locally against every included doc (397 files) on a full clone to check the claims in the description.

🔴 Blocking

• DCO check is failing. The single commit has no Signed-off-by line, so CI is red and the PR cannot merge.
  • Fix: git commit --amend -s --no-edit && git push --force-with-lease.

🟡 Important

• Commit fd27e0f5 (perf(docs): core web vitals — self-host fonts, drop dead assets, defer pixel, add security headers #896) needs to be in .lastmod-ignore-revs. It swapped <img> tags for webp and gif tags for <video> on 34 v4 pages. No page says anything different, but 60 pages now carry 2026-09-18 as their last update. This is exactly the "formatting rewrite" case the ignore file was added for, and the description says no entries are needed yet. feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897 and docs: one Keploy across the site - no editions, accurate native macOS/Windows, eBPF on Linux #918 are real content changes and are fine.
  • Fix: add fd27e0f56a544fd29a59ec10a0abdee387e1cf46 to .lastmod-ignore-revs.

🟢 Nits (non blocking)

• contentLastUpdate.js:151: a file created in an ignored commit with no copy source returns null. That falls back to Docusaurus git dating, which is the newest commit touching the file, the exact thing this module exists to avoid. Creating a file is always a content change, so return change here regardless of ignored.
• contentLastUpdate.js:91: hash.startsWith(rev) with no minimum length. A stray short line in the ignore file (say a) would silently ignore most of history. Require 7+ hex chars, or full 40.
• contentLastUpdate.js:172: require("child_process") inline. spawnSync can come from the import on line 33.
• contentLastUpdate.js:223: lastUpdateFor is exported but nothing imports it. Drop it or add a one line comment saying what it is for.

✅ Verified

• Docusaurus 3.9.2: markdown.parseFrontMatter and sitemap lastmod are both supported. showLastUpdateTime is already on.
• Local run over 397 docs: 3.7s total, 0 nulls, 300 dates unchanged, 97 pages got an earlier (correct) date. Version cut handling works: concepts/general-glossary resolves to 2023-11-29, not the Jan 2026 cut.
• Hand written last_update.date wins. Files outside docs/ and versioned_docs/ are untouched.
• Front matter only and whitespace only commits are skipped (#924 on installation.md correctly skipped).
• fetch-depth: 0 is on the right checkout step in both workflows. Preview build time is unchanged (6m04s vs 6m to 8m on recent runs).
• Git process cap of 16 and memoization keep the build cost flat.

📚 Notes

• Pages that import a shared partial will not move their date when the partial changes. Same limitation as stock Docusaurus, worth a comment at the top of the module.
• No tests, but the repo has no JS test runner. A small script under scripts/ that asserts a few known file dates would catch regressions in the walk logic. Not blocking.

[amaan-bhati]

Addressed in 6a697a9, branch synced with main in bab6cc0.

Blocking

• DCO: new commits are signed off. 0cf7bf1 still needs a sign-off, which means a history rewrite and force push. Pending, will push once cleared.

Important

• perf(docs): core web vitals — self-host fonts, drop dead assets, defer pixel, add security headers #896: added fd27e0f5 to .lastmod-ignore-revs. Verified the diff is media tags only. No sitemap URL carries 2026-09-18 now.

Nits

• L151: done, a created file always returns its own date, even in an ignored commit.
• L91: done, ignore entries must be 7 to 40 hex chars, others are dropped.
• L172: done, spawnSync comes from the top import.
• L223: done, dropped the unused lastUpdateFor export.

Notes

• Shared partials: added the limitation to the module header.
• Test script: skipped. Per-file date assertions go stale on every doc edit. yarn build plus the sitemap check covers it for now.

yarn build passes, 216 of 217 sitemap URLs have <lastmod>.

[amaan-bhati]

Claude Review Skill: Iteration 2

Verdict: 🔄 REQUEST CHANGES (one blocker left, everything else resolved)

Reviewed at bab6cc0. Re-ran the module over all 397 included docs on the new head.

🔴 Blocking

• DCO still failing. 0cf7bf1 has no Signed-off-by, the two newer commits do. Needs the rewrite you mentioned: git rebase -i origin/main and add -s to that commit, or git rebase --signoff origin/main, then git push --force-with-lease. Will re-review as soon as it lands.

✅ Resolved since iteration 1

• fd27e0f5 (perf(docs): core web vitals — self-host fonts, drop dead assets, defer pixel, add security headers #896) is in .lastmod-ignore-revs. Confirmed: no page resolves to 2026-09-18 any more. 156 pages now get an earlier, correct date (was 97).
• Created file in an ignored commit returns its own date (contentLastUpdate.js:158).
• Ignore entries must be 7 to 40 hex chars and are lower cased. Bad lines are dropped, not matched.
• spawnSync comes from the top import.
• Unused lastUpdateFor export removed.
• Shared partial limitation documented in the module header.
• Test script skipped, agreed. Per-file date assertions would go stale with every doc edit.

✅ Verified

• Local run: 397 docs, 4.6s, 0 nulls, 241 unchanged, 156 earlier. Build cost is still flat.
• Hand written last_update.date still wins; non doc paths untouched.
• Merge with main brings only doc content from main, the PR diff against main is still the same 5 files.
• prettier, run-lint, deploy-preview were still pending at review time. Expect them green based on the previous run.

Next

• Push the signed off history and this goes to ✅ APPROVE.

[amaan-bhati]

Acknowledged. Only DCO is left: signing off 0cf7bf1 needs a history rewrite and force push, which lands next. No code changes in this round.

[dhananjay6561]

Reviewed by checking out bab6cc0d with full history and running withContentLastUpdate (NODE_ENV=production) over every built doc: 397 files across docs/, version-1.0.0, version-2.0.0 and version-4.0.0. That takes about 6s, so build cost isn't a concern. Copy/rename following works: none of the 209 v4 pages land on the v4 cut date (8a7d8f06, 2026-01-21).

Verdict: not mergeable as-is. The mechanism is sound, but the dates it ships aren't accurate yet. That undercuts the PR's own reason for existing: "Dates that move without real changes get ignored for the whole sitemap."

1. #897 is a sitewide SEO sweep, not a content change, and it now dates 41% of the sitemap.

• Of the 209 v4 pages (the only indexable version), 86 resolve to 2026-09-01 (80307a58, #897), and 192 of 209 (92%) are dated 2026-09-01 or later.
• The "2023-10 to 2026-10" spread in the description comes mostly from 1.0.0/2.0.0. Those are noIndex and excluded by ignorePatterns, so they never reach the sitemap.
• Details and numbers are on .lastmod-ignore-revs.

2. Collapsing all whitespace hides rendering changes. Markdown treats whitespace as meaningful. See the inline comment on L106.

3. The failure mode is silent. A shallow checkout only produces a console.warn, and the build ships the exact single-date bug this PR fixes. The repo already gates build output with verify-schema and verify-linkedin-tag, and nothing gates this. See the inline comment on L184.

4. DCO is red. 0cf7bf11 (the main feature commit) has no Signed-off-by.

Recommendation: fix #1 first, since it decides whether the sitemap is trustworthy on day one. Ignoring #897 per commit is too coarse: this repo squash-merges (259 non-merge vs 8 merge commits since 2025-01), so #897 fuses SEO boilerplate with a few real edits. Use an ignore rule that can strip known boilerplate blocks (## Related / ## Related Terms footers, <HowTo visible={false}>) from the compared body, or accept ignoring #897 entirely and lose a handful of small real edits. Then make CI fail on a shallow clone and add a verify-lastmod check alongside verify-schema.

Repro for every number here: run withContentLastUpdate over the file list above and group the resulting last_update.date by commit.

[amaan-bhati]

@dhananjay6561 thanks, all four addressed in 1a60a13 and f43c23b. Details are in the inline replies.

1. feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897: boilerplate (## Related..., <HowTo />) is stripped in contentOf(), and feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897 is ignored for the alt text, dash and prettier leftovers. v4 now has 0 pages on 2026-09-01 and 46 distinct dates. Two real feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897 edits are lost, and both pages fall back to an older date.
2. Whitespace: only CRLF, trailing spaces and extra blank lines are normalized now. Indentation and single blank lines count as changes.
3. Silent failure: a shallow clone throws on GitHub Actions. The new verify-lastmod runs in build_and_check.yml and fails a single-date sitemap (tested).
4. DCO: still pending. Signing off 0cf7bf1 needs a force push, which lands next.

The PR description now shows numbers for v4 only.

[amaan-bhati]

Claude Review Skill: Iteration 3

Verdict: 🔄 REQUEST CHANGES (DCO only, the code is approve ready)

Reviewed at f43c23b. Re-ran the module over all 397 included docs, with and without the new #897 ignore entry, and checked every workflow that builds.

🔴 Blocking

• DCO still failing. 0cf7bf1 is the only unsigned commit, the four after it are signed. Since the merge commit bab6cc0 only brought doc content from main, a plain rebase is safe:

  git rebase --signoff origin/main
  git push --force-with-lease
  

  That signs all commits and drops the merge commit. Then this goes to ✅.

✅ Verified in 1a60a13 and f43c23b

• contentOf() strips ## Related... footers (174 headings across v4) and multi line <HowTo ... /> blocks (44 files, all end with a bare /> line, so the regex matches every one). No single line <HowTo /> exists that could trip the lazy match.
• Stripping alone takes feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897 from 86 pages down to 38. The ignore entry removes the rest. Confirmed 0 pages on 2026-09-01 now.
• Whitespace: CRLF, trailing spaces and 3+ blank lines normalized, indentation and single blank lines kept. Correct for Markdown lists and fenced blocks.
• BOM written as , no raw byte.
• Shallow clone throws only when GITHUB_ACTIONS=true. Both workflows that build (main.yml, build_and_check.yml) already use fetch-depth: 0, no other workflow runs a build, so nothing else breaks. Vercel and Netlify previews still get the warning path.
• scripts/verify-lastmod.js reads build/sitemap.xml, which is where the file is at that step (the build/docs move happens later and only in main.yml). Thresholds are sane for today: 216 URLs, 46 dates, largest share 33%.
• Full run: 397 docs, 4.9s, 0 nulls, 220 pages earlier than naive git. Remaining big dates are real rewrites: docs: one Keploy across the site - no editions, accurate native macOS/Windows, eBPF on Linux #918 (72 pages), New sections to docs  #790 (21 pages).

🟢 Nits (non blocking)

• The two pages that lost a real feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897 edit (integrations-pr-agent.md, api-testing-faq.md) can keep the true date with last_update: {date: 2026-09-01} in their front matter. Hand written dates already win in withContentLastUpdate, so no code needed.
• verify-lastmod.js:14: a real sitewide rewrite larger than docs: one Keploy across the site - no editions, accurate native macOS/Windows, eBPF on Linux #918 would fail PR builds at the 50% cap. The error text could add "or raise MAX_SHARE_OF_ONE_DATE if the change is real".

Pending

• deploy-preview on f43c23b was still running at review time, so verify-lastmod has not reported yet. Will confirm on the next iteration.

[amaan-bhati]

Nits

• Error text: done in 21abd92. It now says to list the sweep in .lastmod-ignore-revs or raise MAX_SHARE_OF_ONE_DATE if the change is real.
• Front matter pin for the two pages: skipped. A hand-written last_update.date always wins, so the next real edit to either page would never move its date. An older date for two pages is the safer trade.

DCO: the author will sign off and force push this separately.

[amaan-bhati]

Claude Review Skill: Iteration 4

Verdict: ✅ APPROVE

Reviewed at 21abd92. All CI checks green, including DCO (sign-off manually approved on the DCO app) and deploy-preview with the new verify-lastmod step.

✅ Resolved since iteration 3

• Error text in verify-lastmod.js now points at both options: list the sweep, or raise the cap for a real rewrite.
• Front matter pin for the two feat(docs): site health + AI-citation fixes (A1, A2, A8 …) #897 pages skipped. Agreed, a hand written date would freeze them forever. Older date is the safer trade.
• DCO green.

✅ Final state

• verify-lastmod on CI: ok (216/217 URLs, 47 distinct dates, largest 2026-09-30 x73). Largest share is 34%, well under the 50% cap.
• Preview build time unchanged at 6m17s with full history.
• Module run on the final head: 397 docs, 0 nulls, 220 pages dated earlier than naive git, every remaining large date is a real rewrite.
• Only the five original files plus scripts/verify-lastmod.js and package.json changed against main.

Nothing blocking. Good to merge.

[dhananjay6561]

Full re-review at 21abd927. I re-ran withContentLastUpdate over all 397 built docs on a full clone (5.0s, 0 nulls) and checked each fix against real files.

Fixed and verified:

• #897: 0 v4 pages land on 2026-09-01 now. v4 has 46 distinct dates. The largest groups are #918 (72) and #790 (21), and both are real rewrites.
• Whitespace: indentation and single blank lines now count as changes. \uFEFF is an escape now, not a raw byte.
• Shallow clone: throws when GITHUB_ACTIONS=true. The only workflows that run a build (main.yml, build_and_check.yml) both use fetch-depth: 0.
• verify-lastmod: runs in deploy-preview and passes.

Not mergeable yet. The new boilerplate stripping hides real content. Two inline comments:

1. L117: the ## Related regex removes everything up to the next h1/h2 or the end of the file, not only the link list. I edited real files and passed them through the PR's own contentOf():
   • Rewriting a FAQ answer in glossary/integration-testing.md: not detected. The whole ### FAQs section (6 Q&As, 36 lines) sits under ## Related Terms.
   • Editing the closing paragraph of api-testing-adding-labels.md: not detected. The same pattern exists in api-testing-running-selective.md and api-testing-suite-settings.md.
   • \b also matches hand-written sections that #897 never added: ## Related Features / guides / pages / workflows on 8 pages, from #790, #809, #841 and #886.
2. L115: <HowTo> is visible by default (src/components/HowTo.js:38, visible = true), but the regex strips every HowTo block. Today all 44 blocks are visible={false}, so dates are unaffected. The first visible HowTo someone adds, which is the component's default, will have its step edits ignored.

Remaining debt (not blocking, flagging it so it's a deliberate choice):

• MAX_SHARE_OF_ONE_DATE = 0.5 would not have caught #897 (41%). That's a reasonable trade-off, and the error text now explains it, but the check only catches a total failure, not a sweep.
• DCO is green through a manual override on the app, and 0cf7bf11 still has no Signed-off-by. Squash-merge this PR, or the unsigned commit lands on main.

Recommendation: narrow both regexes as described inline, re-run the 397-doc pass, then this is ready to merge.

[dhananjay6561]

Full re-review, still at 21abd927 (no new commits since my last review).

Newly verified this pass: both shallow-clone safety nets work end to end. I did a real npm run build on a --depth 1 clone of this head:

Scenario	Result
GITHUB_ACTIONS=true npm run build	exit 1: [contentLastUpdate] shallow clone on CI: check out with fetch-depth: 0, … is raised from isEnabled (L200) and stops loadSite
Same clone, no GITHUB_ACTIONS	build exits 0 with the warning, then npm run verify-lastmod gives exit 1: only 1 distinct <lastmod> dates; is git history shallow?

So neither a dropped fetch-depth: 0 in main.yml nor one in build_and_check.yml can ship single-date sitemaps silently.

Walk logic re-checked: neither ignored commit (#896, #897) creates, renames or copies any doc, so the ignore-on-copy branch (L170-173) never fires today. Copy-following through the v4 cut still holds: 0 v4 pages land on 8a7d8f06.

Still open, no reply yet:

• L117 (blocking): the ## Related strip removes everything after the link list down to the next h1/h2. FAQ edits in glossary/integration-testing.md and closing-paragraph edits on 3 running-keploy pages are not detected. Repro and a tested fix are in the inline thread.
• L115: <HowTo> renders visibly by default, but the regex strips all of them. No effect today; it becomes a gap with the first visible HowTo.

Once L117 is narrowed and the 397-doc pass is re-run, I have nothing else blocking.

[amaan-bhati]

@dhananjay6561 both inline points fixed in ca10cce, replies are in the threads. yarn build, verify-lastmod and verify-schema pass. 397-doc pass: 0 nulls, 0 dates moved, v4 still has 0 pages on 2026-09-01.

On the remaining debt:

• 50% cap: deliberate. It catches total failures (shallow clone, git missing) without blocking PRs after a real sitewide rewrite. Sweeps are handled through .lastmod-ignore-revs, not this threshold.
• DCO / unsigned 0cf7bf1: noted. The author is handling the sign-off before merge.

[dhananjay6561]

Re-review at ca10cce6. I validated both fixes myself.

Verified:

Check	Result
FAQ answer edit, glossary/integration-testing.md	detected ✅
Closing paragraph edit, api-testing-adding-labels.md	detected ✅
Link edit in hand-written ## Related Features (#790)	detected ✅
Link edit inside a #897 ## Related Terms list (glossary/mocks.md)	ignored ✅ (intended)
Step edit in a hidden HowTo (keploy-explained/windows-wsl.md)	ignored ✅
Same file with the block flipped to visible={true}, step edited	detected ✅
v4 pages that still have a bare ## Related / ## Related Terms heading after stripping	0 ✅
397-doc pass vs 21abd927	0 dates moved, 0 nulls; v4: 46 distinct dates, 0 on 2026-09-01, largest 2026-09-30 x72 ✅

Moving the trailing-whitespace strip ahead of the heading match is right: it lets ## Related match, and lets the blank-line branch use a plain \n.

I accept the 50% cap and the DCO plan as the author explained them. Squash-merge so 0cf7bf11 doesn't land on main unsigned.

One non-blocking nit inline (L118): a single-line visible <HowTo … /> followed by a hidden block hides everything between them. It can't happen today, because there are 0 single-line HowTo blocks in v4.

Verdict: approve-ready from my side once deploy-preview (running at review time) goes green with verify-lastmod. Nothing blocking left.

[amaan-bhati]

@dhananjay6561 thanks. The L118 nit is fixed in a45f25e (reply in thread). yarn build and verify-lastmod pass locally. Squash-merge noted for the unsigned 0cf7bf1.

[dhananjay6561]

Manual review at a45f25e1

I tested this end to end on a full clone (gh pr checkout 928, npm install).

1. Build and sitemap

npm run build
npm run verify-lastmod
# verify-lastmod: ok (216/217 URLs, 47 distinct dates, largest 2026-09-30 x73)

dateModified on built pages now comes from the last content change, not the newest commit:

Page	dateModified	Stock git date
concepts/general-glossary	2024-02-15	2026-09-01 (#897 sweep)
running-keploy/cli-commands	2026-10-07	2026-10-08

The only sitemap URLs still on 2026-09-01 are the 6 React pages in src/pages/ (about, leadership, security, privacy-policy, application-development, glossary index). #897 really rewrote those, and they're out of scope for the module by design.

2. Each rule, tested with throwaway commits

On a scratch branch, I made one change at a time to glossary/integration-testing.md (and keploy-explained/windows-wsl.md for HowTo), committed with a fixed GIT_COMMITTER_DATE, and read the date back through withContentLastUpdate with NODE_ENV=production:

Change	Expected	Result
Front matter only	unchanged	✅
Trailing spaces / extra blank lines	unchanged	✅
Link edit inside ## Related Terms	unchanged	✅
FAQ answer edit (under Related Terms)	moves	✅
Body sentence edit	moves	✅
Re-indent a list item	moves	✅
Step edit in a hidden HowTo	unchanged	✅
Step edit in a visible HowTo	moves	✅
Body edit whose commit is in .lastmod-ignore-revs	back to old date	✅

The one-line visible HowTo followed by a hidden one is detected now as well, so the nit is fixed.

3. Shallow clone safety nets

git clone --depth 1 ...
GITHUB_ACTIONS=true npm run build   # exit 1: "shallow clone on CI: check out with fetch-depth: 0..."
npm run build                       # exit 0, warns
npm run verify-lastmod              # exit 1: "only 1 distinct <lastmod> dates"

Small note: this site has no visible "Last updated on" line. The custom DocItem only uses the date for dateModified in the JSON-LD, so the sitemap and dateModified are where to check this.

Everything passes and CI is green. LGTM from my side. Please squash-merge so the unsigned 0cf7bf11 doesn't land on main.

[amaan-bhati]

@dhananjay6561 thanks for the end-to-end pass. Good catch on the "Last updated" line: the description now says dateModified (JSON-LD) instead. Will squash-merge.