Sync mechanism

[Sporiff]

Right, I know you're all probably sick of the back and forth on this, so I thought I'd open this up as a discussion here so that we can keep everything together.

Through my various experiments and trials, it's become apparent that the sync mechanism is the hard problem to solve here, and it largely comes down to two things:

1. What is the model around which we are focusing sync? Device-based or user-based?
2. What is the entry point to that sync? General sync endpoint controlled by properties such as entity_type and action or per-entity sync?

Subscriptions sync

My current thinking around subscriptions is, to summarize:

1. No central records for subscriptions. I tried this and it got far too messy, far too quickly. Instead of maintaining a record for a subscription feed, each user/feed subscription should be its own database entry. It means duplication, but it also means nobody is updating data that other people are accessing.
2. Subscriptions are very low traffic by comparison to episodes. A person will listen to several episodes in one day, while they might only subscribe to a few new feeds a week and aren't likely to rapidly un/subscribe. The expected traffic is an initial bulk sync, followed by periodic new entries and occasional unsubscribe actions.

For this kind of thing, a single sync endpoint that keeps track of actions is trivial enough, for example:

[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "create",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
      "unsubscribed_at": "2026-07-03T12:20:14+00:00",
    }
  }
]

We could make a required heading for client IDs (X-Client-ID) so that clients don't receive their own updates when they sync by default, then have a param they can pass if they want to see everything.

Episodes sync

Episodes have one real sticking point in my mind: authority over changes. We can look at this a few ways:

1. Timestamp authority: the client records a UTC timestamp of the action and sends it with each update. The action with the latest timestamp becomes the authoritative "view" of the entity.
   • The benefit here is that this feels closest to the "user-centric" view of things; whatever the user did last in real life becomes the authoritative state.
   • The big weakness here is mutability. If a user syncs their devices, but a device comes online later with a later timestamp, it could cause problems.
   • We also risk big problems if the client records the timestamp wrong, for whatever reason.
2. Last-seen authority: the server responds with the latest version of the action that it has received, based on its own authoritative timestamps.
   • The benefit here is that the server is authoritative, and the clients are all beholden to it.
   • The big drawback is that drift can easily occur. What if a device comes online later, but passes data that happened much earlier? What if the client's clock drifts?

From this, it strikes me that we need to have some sort of versioning capability (optimistic concurrency model), where the server responds with the latest version of an entity. To give a simple example:

// Server sync object
[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "create",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
	"entity_id": "31c81c29-4953-43b9-9a6e-002b854517d8"
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
	  "version": 1, // The server responds with the latest version it holds
    }
  }
]

When the client receives these data from the server, it must then refer to the latest version that it's aware of when it wants to perform an update. For example:

// Client sync request
[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "upsert",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
	"entity_id": "31c81c29-4953-43b9-9a6e-002b854517d8"
	"required_version": 1, // The expected version is listed here
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
      "unsubscribed_at": "2026-07-03T12:20:14+00:00",
    }
  }
]

If the server receives this, and the version has since incremented, this is a mismatch. The client then needs to essentially "rebase" on top of the latest change, and then send it back with an updated expectation.

// Client
[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "upsert",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
	"entity_id": "31c81c29-4953-43b9-9a6e-002b854517d8"
	"required_version": 1,
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
      "unsubscribed_at": "2026-07-03T12:20:14+00:00",
    }
  }
]

// Server
[
  {
	"status": "conflict",
	"currentVersion": 3,
	"actions": [
	  ...
	]
  }
]

// Client syncs the latest version, rebases its changes, and sends a new update
[
  {
    "event_id": "6217f6e5-9bd5-4fc7-bd8c-8d0dea32d082",
    "action": "upsert",
    "action_timestamp": "2026-07-05T14:00:45+00:00",
    "entity_type": "subscription",
	"required_version": 3, // The client updates the required version to inform the server it is aware of the changes
    "data": {
      "feed_url": "https://example.com/rss1",
      "guid": "63bdca44-e177-4b0c-8307-42e653b0ce54",
      "subscribed_at": "2023-07-04T08:23:13+00:00",
      "unsubscribed_at": "2026-07-03T12:20:14+00:00",
    }
  }
]

To summarize: I'm trying to think of ways that we can avoid having too many conflicts. As I said above, I think subscriptions are unlikely to give too many issues here, but users will likely update episodes a lot more.

I want to hear opinions, alternatives, feedback, technical challenges. Whatever you've got, basically. Working this out would basically unblock us, I feel.

[keunes]

Thanks @Sporiff for starting this discussion.

To begin with a key question:

What is the model around which we are focusing sync? Device-based or user-based?

For me this is clear: it is user-based. The user must be central, and is assumed to have multiple devices on which they listen and do other actions. And I think we concluded this already before - gpodder.net seems more device-focussed, in its UX at least.

[ByteHamster]

Definitely user based. That was one of the main confusions people had with Gpodder

[keunes]

Creating a separate sub-thread for another key point:

What is the entry point to that sync? General sync endpoint controlled by properties such as entity_type and action or per-entity sync?

I can't comment on this, really. No idea what the benefits and downsides are of either. (From the APIs that I've consumed the per-entity endpoint seems kind of standard, but that's hardly an argument.)

[ByteHamster]

The fewer requests AntennaPod needs to do (from a mobile network, etc), the easier it is. Certainly one request for each episode is not feasible for thousands of episodes. Whether it is 1 or 3 total feels like it does not matter too much

[keunes]

I was thinking just now that maybe, on a conceptual level, it's easier to understand the structure of the whole API if it's split in chunks, e.g. with PATCH /subscriptions/a388867e-ce91-54d3-a116-114b07bb84e9/episodes/65a57789-3578-4842-89e5-a221f177efb9 (body without guids) or PATCH /subscriptions/episodes (with guids in the body), than a monolithic POST /sync which contains a range of different actions and all their metadata.

[Sporiff]

@keunes There's no reason we can't support both. Conceptually, it's no more complex to map POST/PATCH/GET to individual resource endpoints in addition to a sync endpoint.

In the updated PR, I've put forward a suggestion for a centralised sync endpoint because of what @ByteHamster mentioned above: it's expensive to bulk update at multiple endpoints, especially if you're performing an initial sync of have a lot of data to send. A central sync endpoint that maintains the latest state of all items (based on client timestamp) eases this somewhat as the client can format the log of transactions into a bulk upload and fire all things at one endpoint (subscription changes, episode progress, etc.). We can always extend this to say that server developers need to support individual action endpoints, but for now I've only specified the resource endpoints as GET endpoints.

We can discuss how we want to structure episode payloads based on the actions users take on episodes, but I think the centralised endpoint works best here just because a user might perform many actions on many resources and we could possibly reduce that interaction from e.g. 5 bulk updates to the individual resources to a single call to the sync endpoint.

[keunes]

Now, a separate sub-thread on authoritativeness, looking at episodes particularly as there's the highest chance on drift:

Episodes have one real sticking point in my mind: authority over changes. We can look at this a few ways:

1. Timestamp authority: the client records a UTC timestamp of the action and sends it with each update. The action with the latest timestamp becomes the authoritative "view" of the entity.
   …
2. Last-seen authority: the server responds with the latest version of the action that it has received, based on its own authoritative timestamps.
   …

So far our approach has been this:

If a user syncs their devices, but a device comes online later with a later timestamp, it could cause problems. We also risk big problems if the client records the timestamp wrong, for whatever reason.

Our argumentation has been: we expect the client to record timestamp of when a change occurred, and submit those real timestamps -- not the timestamp of submitting this change.

What if the client's clock drifts?

Is that really a practical risk nowadays?

From this, it strikes me that we need to have some sort of versioning capability (optimistic concurrency model), where the server responds with the latest version of an entity.

In light of the above, such versioning system seems unnecessary added complexity for clients to implement. It would require additional database columns (which has an impact on app size for AntennaPod, even if the columns are empty). And it would require agreements on what exactly constitutes a version jump.

If there's drift or issues with syncing, the user will report to the server or client maintainer, who can then investigate it.

[Sporiff]

Thanks for this @keunes. Regarding

Is that really a practical risk nowadays?

Yes, sort of. I think for most people it is unlikely, but even UTC time is determined relative to the device's time (I believe). If the user has manually set their time it would affect local timestamps and could lead to drift.

I think, however, that for this kind of API it's probably reasonable to assume that most people won't hit this issue. Maybe we just want to advise in the document that any drift in the clock will affect order of operations and let client developers handle that? They might be able to implement a check against a live time server or something, but that's an implementation detail.

In light of the above, such versioning system seems unnecessary added complexity for clients to implement.

Agreed. I think we can probably stick with:

• UTC timestamp of operation as recorded by the client
• Client ID (just for diagnostics, might be useful later)

And then the server simply applies the operation if it's the latest one it's received for the target entity. I'll write up a draft and we can see if we like how it looks.

[ByteHamster]

From this, it strikes me that we need to have some sort of versioning capability (optimistic concurrency model), where the server responds with the latest version of an entity

This would be just the latest timestamp, wouldn't it? Otherwise we need to figure out whether version 25 on device A is later than version 386 on device B. The time is already a shared versioning scheme that all are (reasonably well) synchronized on.

the server responds with the latest version of the action that it has received

Sounds good