Skip to content

fix: strip NUL bytes and invalid UTF-8 from article strings before the insert - #62

Merged
flthibaud merged 1 commit into
developfrom
fix/sanitize-article-strings
Nov 11, 2026
Merged

flthibaud merged 1 commit into
developfrom
fix/sanitize-article-strings

Conversation

@flthibaud

Copy link
Copy Markdown
Owner

PostgreSQL refuses a NUL byte or invalid UTF-8 in a text column. A feed item carrying one (a JSON Feed title with \u0000 gets through gofeed) failed its insert on every sync. A failed item keeps the feed's ETag from advancing, so the whole feed was downloaded again and the page scraped again every 15 minutes, forever.

  • New cleanArticleString applied to title, description, author, HTML and Markdown in createArticleFromItem.
  • New test TestCreateArticleFromItemCleansUnstorableStrings.

Tested with gofmt, go vet ./... and go test -race ./... against a migrated postgres:16 (no test skipped).

Closes #42

@flthibaud

Copy link
Copy Markdown
Owner Author

Review: OK to merge.

  • Code and test by Mistral. The test did not compile (gofeed.Author does not exist, the type is gofeed.Person) and a comment had a garbled escape; I fixed both.
  • The article URL and image URL are not cleaned. They come from the same feed fields, but a NUL there would already fail url.Parse or the HTTP fetch upstream for the link; the image URL is a residual gap.
  • PostgreSQL's refusal of a NUL byte was confirmed on a real database earlier (invalid byte sequence for encoding "UTF8": 0x00).

@flthibaud
flthibaud merged commit f3e351d into develop Nov 11, 2026
5 checks passed
@flthibaud
flthibaud deleted the fix/sanitize-article-strings branch November 11, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A feed item that can never be saved makes the feed re-download forever

1 participant