Skip to content

Apply the clusters content decisions and fix two clusters parser bugs - #1077

Open
LukasWallrich wants to merge 6 commits into
mainfrom
content-decisions-clusters
Open

LukasWallrich wants to merge 6 commits into
mainfrom
content-decisions-clusters

Conversation

@LukasWallrich

@LukasWallrich LukasWallrich commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Applies the content-review decisions for the cluster pages: corrections to the Clusters v4.1 Google Doc, regenerated with two parser fixes, and cleaner publication titles in the featured-resources generator.

Clusters Doc and regenerated pages

The Doc was corrected first. Then parse_clusters_to_sheet.py --json-only and generate_cluster_pages.py were re-run.

  • Names: clusters 5, 6 and 8 now use title case, matching the /resources/ filters ("FAIR Data and Materials", "Replication and Meta-Research", "Transparency and Reproducibility in Computation and Analysis").
  • Descriptions:
    • WEIRD now expands "Industrialized".
    • The GOLD and DIAMOND open access glosses are corrected.
    • Rewritten: the equity definition, the LGBTQ+ scope, the communication-barriers list, the Study Design and qualitative-reporting sentences, the Netherlands Code sentence and the cluster 6 grammar.
    • Removed: the repeated NHST clause and the QRP fragment.
    • Rights retention gets an explanatory sentence.
    • Decolonizing Research Practices gets a description.
  • Removed from the Doc: the "INSERT DESCRIPTION" placeholders, an editor's "To add:" note (now the Flynn, 2025 reference) and a Wikipedia note.
  • References:
    • A duplicate Bartoš et al. (2024) is gone.
    • Mozersky et al. (2020) in IASSIST Quarterly now has its own DOI, 10.29173/iq952, checked against Crossref.
    • The URL-encoded DOI 10.1037%2Fqup0000076 is normalised.

Parser fixes (scripts/parse_clusters_to_sheet.py)

  • Descriptions that cite a year: a description line that mentions a year in passing ("Clarke's (2022) piece") stays the description, so it is no longer turned into a reference. Lines that start like an author list, or that carry a DOI or URL, are still read as references. On the earlier Doc text, only the cluster 10 "Standards and guidelines for rigorous qualitative research" sub-cluster changes.
  • Group authors: when the doi.org APA text starts with the year, the Doc's author is kept. This affects Open Science Collaboration (2012, 2015) and National Academies (2017).
  • Tests are added for both fixes.

Publication titles (scripts/build_featured_data.py)

  • Titles and APA references are now stored as plain text plus <i>/<scp> markup. HTML entities are decoded, including double-escaped &lt;em&gt;; <em> becomes <i>; and whitespace is collapsed.
  • This also repairs citation buttons such as "Lakens & amp; Lieck (2022)".
  • A separate PR into rebrand-doors renders the allow-listed tags.
  • The change goes live with the next daily data-processing run.

AGENTS.md

AGENTS.md now points to --json-only. Run on its own, --export-json also clears and rewrites the sheet tabs, including the enriched Publications columns.

Checks: unit tests pass, hugo --gc --minify builds, and check_seo_html.py passes.

🤖 Generated with Claude Code


Summary by cubic

Applies the content-review decisions for the cluster pages and fixes two parser bugs that surfaced when regenerating them: clusters 5, 6, and 8 now use title case, confusing or broken descriptions and references were corrected, and the parser no longer mistakes prose that cites a year for a reference or drops group authors like Open Science Collaboration.

Parser fixes

  • A description line mentioning a year in passing ("Clarke's (2022) piece") stays a description, provided it carries no "(year)." reference date; author-list, DOI, and URL lines are still treated as references.
  • When the doi.org APA text starts with the year, the Doc's group author is kept (Open Science Collaboration, National Academies), including the cluster 8 2015 Science entry.
  • The script's --help, AGENTS.md, and the site's data-error message now recommend --json-only, since plain --export-json also clears and rewrites the Clusters, Sub-Clusters, and Publications sheet tabs.

Content

  • Corrected WEIRD's expansion, the GOLD and DIAMOND open access glosses, equity, LGBTQ+ scope, communication barriers, Study Design, and other cluster 6 descriptions.
  • Removed a duplicate Bartoš et al. reference, the "INSERT DESCRIPTION" placeholders, and the Wikipedia and "To add:" notes; added a description for Decolonizing Research Practices.
  • Publication titles and APA references now decode HTML entities and keep <i>/<scp> markup while leaving all other angle brackets escaped, preserving the full stop in "et al." and dropping it after group authors; this fixes broken citation buttons like "Lakens &amp; Lieck (2022)".

Written for commit dbec563. Summary will update on new commits.

View guided diff Turn on auto-fix

LukasWallrich and others added 4 commits November 9, 2026 17:45
The Clusters v4.1 Doc now has title-case names for clusters 5, 6 and 8,
corrected descriptions (WEIRD expansion, open access glosses, equity,
LGBTQ+, communication barriers and others), a description for
Decolonizing Research Practices, no INSERT DESCRIPTION placeholder or
editors' notes, and corrected Mozersky (IASSIST Quarterly) and Flynn
references. Re-ran parse_clusters_to_sheet.py --json-only and
generate_cluster_pages.py.

The parser now keeps a description line that cites a year in passing
("Clarke's (2022) piece") as the description, and keeps the Doc's group
author when the doi.org APA text starts with the year (Open Science
Collaboration, National Academies).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build_featured_data.py now decodes HTML entities in titles and APA
references (including double-escaped ones), turns <em> into <i> and
collapses whitespace, so card titles hold plain text plus <i>/<scp>
markup for the templates to render.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--export-json on its own also clears and rewrites the Clusters sheet tabs,
including the enriched Publications columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Open Science Collaboration. (2015)" now gives the button
"Open Science Collaboration (2015)".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LukasWallrich
LukasWallrich requested a review from a team as a code owner November 9, 2026 17:47
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

👍 All image files/references (if any) are in webp format, in line with our policy.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

✅ Spell Check Passed

No spelling issues found when checking 9 changed file(s)! 🎉

@LukasWallrich

LukasWallrich commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor Author

⚠️ Staging Deployment Status

This PR was attempted for staging deployment but had merge conflicts and was skipped.

Attempted at: 2026-11-13 12:30:41 UTC
Staging URL: https://staging.forrt.org

Please resolve conflicts with the base branch and the deployment will be retried automatically.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread AGENTS.md Resolved
Comment thread data/clusters_v4.json Resolved
Comment thread scripts/build_featured_data.py Resolved
Comment thread scripts/parse_clusters_to_sheet.py Resolved
Comment thread scripts/build_featured_data.py Resolved
- build_featured_data.py keeps any angle bracket other than <i>/<scp>
  encoded, and keeps the full stop in "et al.".
- A description line is kept only if it has no "(year)." reference date,
  so unbulleted references with group or full-name authors stay references.
- The Doc's cluster 8 entry for the 2015 Science paper now names
  Open Science Collaboration, as the cluster 1 entry does.
- AGENTS.md names all three sheet tabs that --export-json rewrites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread AGENTS.md Resolved
Comment thread scripts/build_featured_data.py Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LukasWallrich

Copy link
Copy Markdown
Contributor Author

Note

🤖 Claude Opus 5.5 responding on behalf of Lukas Wallrich

@richarddushime, could you review the code in this PR? The cluster data and page changes come from the Clusters Doc fixes Lukas approved, but these three code changes alter what the daily data run writes to forrt.org:

  1. scripts/build_featured_data.py, clean_markup_text(). Titles and APA references in featured_resources.json / pub_cards.json now decode entities up to twice, turn <em> into <i>, and keep raw <i>/<scp> while re-encoding every other < and >. Is that the right contract for the data artifact? main's current templates escape titles, so <i> still shows as literal text there until the Doors templates (Render italics and small caps in publication titles on cluster pages #1083) arrive. Do you see any consumer on main that would output these fields unescaped?

  2. scripts/parse_clusters_to_sheet.py. Two changes:

    • is_description_with_year(): a description line that cites a year in passing now stays the description.
    • keep_doc_author(): a group author such as "Open Science Collaboration." is prepended when the doi.org APA text starts with the year.

    Are the heuristics (AUTHOR_START_RE, REFERENCE_DATE_RE) narrow enough? The tests are in scripts/tests/test_parse_clusters.py.

  3. AGENTS.md and the --help / data-error text now say to regenerate with --json-only, because --export-json on its own also clears and rewrites the Clusters, Sub-Clusters and Publications sheet tabs. Is that the workflow you want documented?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant