Repository navigation
feat(relay): record where media is shared and who uploaded it #8451
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ngoodger-block
wants to merge
19
commits into
block:main
Choose a base branch
from
ngoodger-block:feat/media-shares
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,993
−180
Open
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
e539e8d
feat(relay): record where media is shared and who uploaded it
ngoodger-block aa6ba13
test(db): upload media before posting it in share-recording tests
ngoodger-block b34e8bb
fix(db): backfill only history unchecked; lock media shares in key order
ngoodger-block d4a6a1a
fix(media): record shares safely before enforcing reads
ngoodger-block ac39fcf
fix(media): revoke grants from deleted messages
ngoodger-block 6c82228
fix(media): preserve grants through profile updates
ngoodger-block a7cdc13
docs(media): clarify backfill permission changes
ngoodger-block 3d77f74
fix(db): renumber media migration after rebase
ngoodger-block bdfb630
fix(db): bootstrap only declared legacy fences
ngoodger-block 715c972
Merge branch 'ngoodger/explicit-schema-fences' into feat/media-shares
ngoodger-block 7892f0b
test(db): audit the schema bootstrap fixture
ngoodger-block 53073b1
fix(media): bind grants to message lifetimes
ngoodger-block 6164b50
Merge branch 'ngoodger/explicit-schema-fences' into feat/media-shares
ngoodger-block 658dbbb
docs(media): require acceptance before first backfill
ngoodger-block 11f017f
fix(db): bound the media migration lock wait
ngoodger-block f67333b
fix(media): reclaim inactive grants and cover upload attribution
ngoodger-block 42e69e6
fix(media): run cleanup before telemetry collection
ngoodger-block 2a67653
fix(media): finish deferred provenance and access feedback
ngoodger-block bb44789
test(db): expect only legacy tables to carry community write fences
ngoodger-block File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| //! Which media blobs an event shares, for channel-scoped media access. | ||
|
|
||
| use std::collections::BTreeSet; | ||
|
|
||
| use crate::kind::{ | ||
| event_is_shared, is_author_only_event_kind, is_shared_gated_kind, KIND_AGENT_ENGRAM, | ||
| P_GATED_KINDS, | ||
| }; | ||
|
|
||
| /// Hashes of every blob `event` shares into its channel, or with the whole | ||
| /// community when it is channel-less: each local media path `/media/<sha256>` | ||
| /// anywhere in its content or tags (attachments, posters, thumbnails, link | ||
| /// previews, avatars, custom emoji, pasted links). | ||
| /// | ||
| /// Empty for an event only its author or tagged recipients can read, since | ||
| /// referencing a blob there shows it to no one new. | ||
| pub fn shared_blob_hashes(event: &nostr::Event) -> Vec<String> { | ||
| let kind = u32::from(event.kind.as_u16()); | ||
| if is_author_only_event_kind(event) | ||
| || P_GATED_KINDS.contains(&kind) | ||
| || kind == KIND_AGENT_ENGRAM | ||
| || (is_shared_gated_kind(kind) && !event_is_shared(event)) | ||
| { | ||
| return Vec::new(); | ||
| } | ||
| // JSON content may escape `/` as `\/`; scan the path as clients will see it. | ||
| let content = event.content.replace("\\/", "/"); | ||
| let mut hashes: BTreeSet<String> = media_path_hashes(&content).collect(); | ||
| for tag in event.tags.iter() { | ||
| hashes.extend( | ||
| tag.as_slice() | ||
| .iter() | ||
| .flat_map(|part| media_path_hashes(part)), | ||
| ); | ||
| } | ||
| hashes.into_iter().collect() | ||
| } | ||
|
|
||
| /// The media blob used as a kind:0 profile's picture. A registered agent may | ||
| /// publish this one field using its owner's upload proof; arbitrary content | ||
| /// and tags must not inherit that authority. | ||
| pub fn profile_picture_blob_hash(event: &nostr::Event) -> Option<String> { | ||
| if event.kind != nostr::Kind::Metadata { | ||
| return None; | ||
| } | ||
| let profile: serde_json::Value = serde_json::from_str(&event.content).ok()?; | ||
| let hash = media_path_hashes(profile.get("picture")?.as_str()?).next(); | ||
| hash | ||
| } | ||
|
|
||
| fn is_sha256_hex(value: &str) -> bool { | ||
| value.len() == 64 && value.chars().all(|c| matches!(c, '0'..='9' | 'a'..='f')) | ||
| } | ||
|
|
||
| /// The hash in each `/media/<sha256>` path in `text`, where the hash is not | ||
| /// followed by another hex digit. | ||
| fn media_path_hashes(text: &str) -> impl Iterator<Item = String> + '_ { | ||
| text.match_indices("/media/").filter_map(|(at, path)| { | ||
| let rest = &text[at + path.len()..]; | ||
| let hash = rest.get(..64)?; | ||
| let bounded = !rest[64..].starts_with(|c: char| matches!(c, '0'..='9' | 'a'..='f')); | ||
| (is_sha256_hex(hash) && bounded).then(|| hash.to_owned()) | ||
| }) | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
| use crate::kind::{KIND_EVENT_REMINDER, KIND_PERSONA}; | ||
| use nostr::{EventBuilder, Keys, Kind, Tag}; | ||
|
|
||
| fn media_url(sha256: &str, ext: &str) -> String { | ||
| format!("https://relay.example/media/{sha256}.{ext}") | ||
| } | ||
|
|
||
| fn signed(builder: EventBuilder) -> nostr::Event { | ||
| builder.sign_with_keys(&Keys::generate()).unwrap() | ||
| } | ||
|
|
||
| #[test] | ||
| fn shares_every_local_media_path() { | ||
| let (video, poster, embed) = ("a".repeat(64), "c".repeat(64), "d".repeat(64)); | ||
| let event = signed( | ||
| EventBuilder::new( | ||
| Kind::TextNote, | ||
| format!("look ", media_url(&embed, "png")), | ||
| ) | ||
| .tags([ | ||
| Tag::parse([ | ||
| "imeta", | ||
| &format!("url {}", media_url(&video, "mp4")), | ||
| &format!("x {video}"), | ||
| &format!("image /media/{poster}.jpg"), | ||
| ]) | ||
| .unwrap(), | ||
| Tag::parse([ | ||
| "r", | ||
| &format!("https://elsewhere.example/{}", "e".repeat(64)), | ||
| ]) | ||
| .unwrap(), | ||
| Tag::parse(["r", &format!("/media/{}0", "f".repeat(64))]).unwrap(), | ||
| ]), | ||
| ); | ||
| assert_eq!(shared_blob_hashes(&event), vec![video, poster, embed]); | ||
| } | ||
|
|
||
| #[test] | ||
| fn shares_avatars_emoji_and_shared_personas_only() { | ||
| let (avatar, emoji) = ("c".repeat(64), "d".repeat(64)); | ||
| let content = format!(r#"{{"picture":"{}"}}"#, media_url(&avatar, "jpg")); | ||
| let profile = signed(EventBuilder::new(Kind::Metadata, &content)); | ||
| assert_eq!(shared_blob_hashes(&profile), vec![avatar.clone()]); | ||
| let escaped = signed(EventBuilder::new( | ||
| Kind::Metadata, | ||
| content.replace('/', "\\/"), | ||
| )); | ||
| assert_eq!(shared_blob_hashes(&escaped), vec![avatar.clone()]); | ||
|
|
||
| let emoji_set = signed( | ||
| EventBuilder::new(Kind::from(30030), "").tags([Tag::parse([ | ||
| "emoji", | ||
| "party", | ||
| &media_url(&emoji, "png"), | ||
| ]) | ||
| .unwrap()]), | ||
| ); | ||
| assert_eq!(shared_blob_hashes(&emoji_set), vec![emoji]); | ||
|
|
||
| let persona = |shared: bool| { | ||
| let mut tags = vec![Tag::parse(["d", "helper"]).unwrap()]; | ||
| if shared { | ||
| tags.push(Tag::parse(["shared", "true"]).unwrap()); | ||
| } | ||
| signed(EventBuilder::new(Kind::from(KIND_PERSONA as u16), &content).tags(tags)) | ||
| }; | ||
| assert_eq!(shared_blob_hashes(&persona(true)), vec![avatar]); | ||
| assert!(shared_blob_hashes(&persona(false)).is_empty()); | ||
| } | ||
|
|
||
| #[test] | ||
| fn author_only_events_share_nothing() { | ||
| let reminder = signed( | ||
| EventBuilder::new(Kind::from(KIND_EVENT_REMINDER as u16), "").tags([Tag::parse([ | ||
| "imeta", | ||
| &format!("url /media/{}.jpg", "a".repeat(64)), | ||
| ]) | ||
| .unwrap()]), | ||
| ); | ||
| assert!(shared_blob_hashes(&reminder).is_empty()); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
39 changes: 39 additions & 0 deletions
39
crates/buzz-db/src/runtime/tests/schema_fence_bootstrap_postgres_tests.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| use sqlx::PgPool; | ||
|
|
||
| /// A fresh bootstrap keeps the declared backstops but never discovers new | ||
| /// community tables and silently attaches a trigger to them. | ||
| #[tokio::test] | ||
| #[ignore = "requires Postgres"] | ||
| async fn migration_schema_bootstrap_keeps_only_explicit_legacy_fences() { | ||
| let pool = PgPool::connect(&crate::test_support::database_url()) | ||
| .await | ||
| .unwrap(); | ||
| sqlx::raw_sql("CREATE TABLE new_tenant_probe (community_id UUID NOT NULL)") | ||
| .execute(&pool) | ||
| .await | ||
| .unwrap(); | ||
| // Only checked-in schema files contribute SQL to this test bootstrap. | ||
| sqlx::raw_sql(sqlx::AssertSqlSafe( | ||
| crate::test_support::desired_state_schema_sql(), | ||
| )) | ||
| .execute(&pool) | ||
| .await | ||
| .unwrap(); | ||
|
|
||
| for (table, expected) in [ | ||
| ("new_tenant_probe", false), | ||
| ("events", true), | ||
| ("users", true), | ||
| ] { | ||
| let fenced: bool = sqlx::query_scalar( | ||
| "SELECT EXISTS (SELECT 1 FROM pg_trigger \ | ||
| WHERE tgrelid = $1::regclass \ | ||
| AND tgfoid = 'enforce_community_write_fence()'::regprocedure)", | ||
| ) | ||
| .bind(table) | ||
| .fetch_one(&pool) | ||
| .await | ||
| .unwrap(); | ||
| assert_eq!(fenced, expected, "unexpected fence on {table}"); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.