Skip to content

Add SSE events to API v2 - #1162

Draft
mxsrc wants to merge 5 commits into
mainfrom
sse
Draft

Add SSE events to API v2#1162
mxsrc wants to merge 5 commits into
mainfrom
sse

Conversation

@mxsrc

@mxsrc mxsrc commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Detail/list endpoints gain a parameter to toggle streaming. With this set, SSE is used to return the initial representation once and watch the database for future changes. The query parameter watch can be used to indicate this behavior. Watches are centrally managed, with database responses being cached. Upon a change in the database, the modified entity is returned, along with an indication of the event-type (creation, update, deletion).

An example of how this will look in practice:

[simplyblock@simplyblock-webappapi-66877b785-hv499 app]$ curl -sN $TLS -H "$AUTH" https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-pools/649d1860-40ff-47a3-8ed2-593b3f3fa594/volumes/
[{"id":"7d56579b-8cf8-4d47-9dc5-5ba402860a9c","cluster_id":"b18ba793-0fe4-4ae2-8869-190f5be5c11b","storage_node_id":"3bf373a9-0939-4fe1-94ad-5cb7f6daf422","name":"pvc-f1422b61-0171-4fe9-a650-619c925eebe0","status":"online","health_check":true,"io_error":false,"migrating":false,"nqn":"nqn.2023-02.io.simplyblock:b18ba793-0fe4-4ae2-8869-190f5be5c11b:lvol:7d56579b-8cf8-4d47-9dc5-5ba402860a9c","hostname":"vm12_4420","priority_class":0,"namespace":"","fabric":"tcp","nodes":["https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/3bf373a9-0939-4fe1-94ad-5cb7f6daf422/","https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/6968a2a2-d94b-40cb-a51f-139d151354cc/"],"port":4426,"size":10737418240,"ndcs":1,"npcs":1,"pool_uuid":"649d1860-40ff-47a3-8ed2-593b3f3fa594","pool_name":"pool","pvc_name":"simplyblock/pvc","snapshot_name":"","blobid":4294967297,"ns_id":1,"cloned_from":null,"high_availability":true,"do_replicate":false,"max_namespace_per_subsys":1,"max_rw_iops":0,"max_rw_mbytes":0,"max_r_mbytes":0,"max_w_mbytes":0,"allowed_hosts":[],"policy":"","capacity":{"date":0,"size_total":0,"size_prov":0,"size_used":0,"size_free":0,"size_util":0},"rep_info":null,"from_source":true},{"id":"fe71b591-6aff-45d6-a2fe-10733e48569e","cluster_id":"b18ba793-0fe4-4ae2-8869-190f5be5c11b","storage_node_id":"3bf373a9-0939-4fe1-94ad-5cb7f6daf422","name":"restored","status":"online","health_check":true,"io_error":false,"migrating":false,"nqn":"nqn.2023-02.io.simplyblock:b18ba793-0fe4-4ae2-8869-190f5be5c11b:lvol:fe71b591-6aff-45d6-a2fe-10733e48569e","hostname":"vm12_4420","priority_class":0,"namespace":"","fabric":"tcp","nodes":["https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/3bf373a9-0939-4fe1-94ad-5cb7f6daf422/","https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/6968a2a2-d94b-40cb-a51f-139d151354cc/"],"port":4426,"size":10737418240,"ndcs":1,"npcs":1,"pool_uuid":"649d1860-40ff-47a3-8ed2-593b3f3fa594","pool_name":"pool","pvc_name":"","snapshot_name":"","blobid":4294967307,"ns_id":1,"cloned_from":null,"high_availability":true,"do_replicate":false,"max_namespace_per_subsys":1,"max_rw_iops":0,"max_rw_mbytes":0,"max_r_mbytes":0,"max_w_mbytes":0,"allowed_hosts":[],"policy":"","capacity":{"date":0,"size_total":0,"size_prov":0,"size_used":0,"size_free":0,"size_util":0},"rep_info":null,"from_source":true}][simplyblock@simplyblock-webappapi-66877b785-hv499 app]$ curl -sN $TLS -H "$AUTH" https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-pools/649d1860-40ff-47a3-8ed2-593b3f3fa594/volumes/?watch=true
event: snapshot
data: [{"id":"7d56579b-8cf8-4d47-9dc5-5ba402860a9c","cluster_id":"b18ba793-0fe4-4ae2-8869-190f5be5c11b","storage_node_id":"3bf373a9-0939-4fe1-94ad-5cb7f6daf422","name":"pvc-f1422b61-0171-4fe9-a650-619c925eebe0","status":"online","health_check":true,"io_error":false,"migrating":false,"nqn":"nqn.2023-02.io.simplyblock:b18ba793-0fe4-4ae2-8869-190f5be5c11b:lvol:7d56579b-8cf8-4d47-9dc5-5ba402860a9c","hostname":"vm12_4420","priority_class":0,"namespace":"","fabric":"tcp","nodes":["https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/3bf373a9-0939-4fe1-94ad-5cb7f6daf422/","https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/6968a2a2-d94b-40cb-a51f-139d151354cc/"],"port":4426,"size":10737418240,"ndcs":1,"npcs":1,"pool_uuid":"649d1860-40ff-47a3-8ed2-593b3f3fa594","pool_name":"pool","pvc_name":"simplyblock/pvc","snapshot_name":"","blobid":4294967297,"ns_id":1,"cloned_from":null,"high_availability":true,"do_replicate":false,"max_namespace_per_subsys":1,"max_rw_iops":0,"max_rw_mbytes":0,"max_r_mbytes":0,"max_w_mbytes":0,"allowed_hosts":[],"policy":"","capacity":{"date":0,"size_total":0,"size_prov":0,"size_used":0,"size_free":0,"size_util":0},"rep_info":null,"from_source":true},{"id":"fe71b591-6aff-45d6-a2fe-10733e48569e","cluster_id":"b18ba793-0fe4-4ae2-8869-190f5be5c11b","storage_node_id":"3bf373a9-0939-4fe1-94ad-5cb7f6daf422","name":"restored","status":"online","health_check":true,"io_error":false,"migrating":false,"nqn":"nqn.2023-02.io.simplyblock:b18ba793-0fe4-4ae2-8869-190f5be5c11b:lvol:fe71b591-6aff-45d6-a2fe-10733e48569e","hostname":"vm12_4420","priority_class":0,"namespace":"","fabric":"tcp","nodes":["https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/3bf373a9-0939-4fe1-94ad-5cb7f6daf422/","https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/6968a2a2-d94b-40cb-a51f-139d151354cc/"],"port":4426,"size":10737418240,"ndcs":1,"npcs":1,"pool_uuid":"649d1860-40ff-47a3-8ed2-593b3f3fa594","pool_name":"pool","pvc_name":"","snapshot_name":"","blobid":4294967307,"ns_id":1,"cloned_from":null,"high_availability":true,"do_replicate":false,"max_namespace_per_subsys":1,"max_rw_iops":0,"max_rw_mbytes":0,"max_r_mbytes":0,"max_w_mbytes":0,"allowed_hosts":[],"policy":"","capacity":{"date":0,"size_total":0,"size_prov":0,"size_used":0,"size_free":0,"size_util":0},"rep_info":null,"from_source":true}]
retry: 3000

: ping - 2026-07-02 14:31:30.991114+00:00

: ping - 2026-07-02 14:31:45.994443+00:00

: ping - 2026-07-02 14:32:01.024133+00:00

event: created
data: {"id":"c0976ee5-1d1c-4c1a-8411-df92273d8d11","cluster_id":"b18ba793-0fe4-4ae2-8869-190f5be5c11b","storage_node_id":"89c7c61a-43cb-41f2-b1a7-d0f816c4c925","name":"testlvol","status":"in_creation","health_check":true,"io_error":false,"migrating":false,"nqn":"nqn.2023-02.io.simplyblock:b18ba793-0fe4-4ae2-8869-190f5be5c11b:lvol:c0976ee5-1d1c-4c1a-8411-df92273d8d11","hostname":"vm14_4424","priority_class":0,"namespace":"","fabric":"tcp","nodes":[],"port":4430,"size":1000000000,"ndcs":1,"npcs":1,"pool_uuid":"649d1860-40ff-47a3-8ed2-593b3f3fa594","pool_name":"pool","pvc_name":"","snapshot_name":"","blobid":0,"ns_id":1,"cloned_from":null,"high_availability":true,"do_replicate":false,"max_namespace_per_subsys":32,"max_rw_iops":0,"max_rw_mbytes":0,"max_r_mbytes":0,"max_w_mbytes":0,"allowed_hosts":[],"policy":"","capacity":{"date":0,"size_total":0,"size_prov":0,"size_used":0,"size_free":0,"size_util":0},"rep_info":null,"from_source":true}

event: updated
data: {"id":"c0976ee5-1d1c-4c1a-8411-df92273d8d11","cluster_id":"b18ba793-0fe4-4ae2-8869-190f5be5c11b","storage_node_id":"89c7c61a-43cb-41f2-b1a7-d0f816c4c925","name":"testlvol","status":"online","health_check":true,"io_error":false,"migrating":false,"nqn":"nqn.2023-02.io.simplyblock:b18ba793-0fe4-4ae2-8869-190f5be5c11b:lvol:c0976ee5-1d1c-4c1a-8411-df92273d8d11","hostname":"vm14_4424","priority_class":0,"namespace":"","fabric":"tcp","nodes":["https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/89c7c61a-43cb-41f2-b1a7-d0f816c4c925/","https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/3bf373a9-0939-4fe1-94ad-5cb7f6daf422/"],"port":4430,"size":1000000000,"ndcs":1,"npcs":1,"pool_uuid":"649d1860-40ff-47a3-8ed2-593b3f3fa594","pool_name":"pool","pvc_name":"","snapshot_name":"","blobid":4294967297,"ns_id":1,"cloned_from":null,"high_availability":true,"do_replicate":false,"max_namespace_per_subsys":32,"max_rw_iops":0,"max_rw_mbytes":0,"max_r_mbytes":0,"max_w_mbytes":0,"allowed_hosts":[],"policy":"","capacity":{"date":0,"size_total":0,"size_prov":0,"size_used":0,"size_free":0,"size_util":0},"rep_info":null,"from_source":true}

: ping - 2026-07-02 14:32:16.025170+00:00

event: updated
data: {"id":"c0976ee5-1d1c-4c1a-8411-df92273d8d11","cluster_id":"b18ba793-0fe4-4ae2-8869-190f5be5c11b","storage_node_id":"89c7c61a-43cb-41f2-b1a7-d0f816c4c925","name":"testlvol","status":"in_deletion","health_check":true,"io_error":false,"migrating":false,"nqn":"nqn.2023-02.io.simplyblock:b18ba793-0fe4-4ae2-8869-190f5be5c11b:lvol:c0976ee5-1d1c-4c1a-8411-df92273d8d11","hostname":"vm14_4424","priority_class":0,"namespace":"","fabric":"tcp","nodes":["https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/89c7c61a-43cb-41f2-b1a7-d0f816c4c925/","https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/3bf373a9-0939-4fe1-94ad-5cb7f6daf422/"],"port":4430,"size":1000000000,"ndcs":1,"npcs":1,"pool_uuid":"649d1860-40ff-47a3-8ed2-593b3f3fa594","pool_name":"pool","pvc_name":"","snapshot_name":"","blobid":4294967297,"ns_id":1,"cloned_from":null,"high_availability":true,"do_replicate":false,"max_namespace_per_subsys":32,"max_rw_iops":0,"max_rw_mbytes":0,"max_r_mbytes":0,"max_w_mbytes":0,"allowed_hosts":[],"policy":"","capacity":{"date":0,"size_total":0,"size_prov":0,"size_used":0,"size_free":0,"size_util":0},"rep_info":null,"from_source":true}

: ping - 2026-07-02 14:32:31.028750+00:00

: ping - 2026-07-02 14:32:46.030096+00:00

event: deleted
data: {"id":"c0976ee5-1d1c-4c1a-8411-df92273d8d11","cluster_id":"b18ba793-0fe4-4ae2-8869-190f5be5c11b","storage_node_id":"89c7c61a-43cb-41f2-b1a7-d0f816c4c925","name":"testlvol","status":"in_deletion","health_check":true,"io_error":false,"migrating":false,"nqn":"nqn.2023-02.io.simplyblock:b18ba793-0fe4-4ae2-8869-190f5be5c11b:lvol:c0976ee5-1d1c-4c1a-8411-df92273d8d11","hostname":"vm14_4424","priority_class":0,"namespace":"","fabric":"tcp","nodes":["https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/89c7c61a-43cb-41f2-b1a7-d0f816c4c925/","https://127.0.0.1:5000/api/v2/clusters/b18ba793-0fe4-4ae2-8869-190f5be5c11b/storage-nodes/3bf373a9-0939-4fe1-94ad-5cb7f6daf422/"],"port":4430,"size":1000000000,"ndcs":1,"npcs":1,"pool_uuid":"649d1860-40ff-47a3-8ed2-593b3f3fa594","pool_name":"pool","pvc_name":"","snapshot_name":"","blobid":4294967297,"ns_id":1,"cloned_from":null,"high_availability":true,"do_replicate":false,"max_namespace_per_subsys":32,"max_rw_iops":0,"max_rw_mbytes":0,"max_r_mbytes":0,"max_w_mbytes":0,"allowed_hosts":[],"policy":"","capacity":{"date":0,"size_total":0,"size_prov":0,"size_used":0,"size_free":0,"size_util":0},"rep_info":null,"from_source":true}

Comment thread simplyblock_web/api/v2/_watch.py Fixed
Detail/list endpoints gain a parameter to toggle streaming. With this
set, SSE is used to return the initial representation once and watch the
database for future changes.
_NoTxnStore.add() and the two hand-rolled transaction fakes didn't exist
when these tests were written; watched-model writes now issue an atomic
add() for the watch-index counters, which the fakes must accept.
The control plane patched StorageNodeSet.status.nodes[] from six event handlers
whenever a node's status, health, or ports changed, a device came or went, a
volume was created or deleted, or a node was removed. The operator writes every
one of those fields into the same array itself, from the same backend node
list, so this was a second writer of state it does not own.

patch_cr_node_status said as much in its own docstring: status.nodes is
"replaced wholesale by several concurrent writers", "a read can transiently
miss a node that a concurrent writer is about to (re)add", and "a blind write
can silently drop a concurrent update". Five retries with optimistic locking
existed to survive a race that only existed because of this. Removing the
writer removes the race rather than retrying it.

The operator no longer needs the mirror either. It streams storage nodes over
the control plane's own watch endpoint and writes the CRs from that, so a
status change reaches Kubernetes in about a second without anything here
reaching into the cluster.

The removal takes the per-event work with it. Each handler looked the cluster
up by id purely to test cluster.mode == "kubernetes", so five of those
lookups, and the device counting that fed the devices field, were work done on
every event to compute something nobody reads now.

Node removal goes too. Nothing here prunes an entry any more, and the
operator's own rebuild of status.nodes from the streamed node list is what
takes over -- a node the control plane no longer reports is absent from the
list rather than needing a deletion to be sent.

patch_cr_status (cluster and pool) and patch_cr_lvol_status are untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants