Skip to content

Fix races and lost state in monitoring, multiprocessing and worker - #1491

Merged
JosepSampe merged 5 commits into
lithops-cloud:masterfrom
JosepSampe:fix/monitor-multiprocessing-worker
Sep 23, 2026
Merged

JosepSampe merged 5 commits into
lithops-cloud:masterfrom
JosepSampe:fix/monitor-multiprocessing-worker

Conversation

@JosepSampe

Copy link
Copy Markdown
Member

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

- Storage monitor: keep a finished call whose init mark is not listed
  yet for the next poll, instead of dropping its worker token
- Monitor: guard status application and token release with a shared
  lock, so two threads cannot both return a token for one chunk or
  put a finished call back to running
- RLock: only the owning thread (pid, thread id) may re-enter
- Condition: a timed-out wait removes its handle from the notify list,
  or takes a notify that arrived just after the timeout
- SemLock: refresh the Redis expiry on acquire and release, since a
  released lock was recreated with no TTL
- JobRunner: report an unpickleable exception with its traceback as
  text; the traceback object itself cannot be pickled
- Handler: close the JobRunner pipe explicitly, even if the finish
  event fails
- Tests: regression test for each fix; the fake Redis now deletes
  empty lists like real Redis does; the telemetry test fakes now have
  the state flags real futures have
@JosepSampe JosepSampe self-assigned this Sep 23, 2026
@JosepSampe
JosepSampe merged commit 87e5b61 into lithops-cloud:master Sep 23, 2026
6 checks passed
@JosepSampe
JosepSampe deleted the fix/monitor-multiprocessing-worker branch September 23, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant