Repository navigation
Remove host_task object management - #2359
Merged
Merged
Conversation
ndgrigorian
requested review from
antonwolfy and
vlad-perevezentsev
as code owners
August 10, 2026 18:16
ndgrigorian
force-pushed
the
feature/drop-host-task-object-management
branch
from
August 10, 2026 18:17
c41755c to
bbf7d44
Compare
|
Deleted rendered PR docs from intelpython.github.com/dpctl, latest should be updated shortly. 🤞 |
Collaborator
|
coverage: 74.963% (+0.07%) from 74.893% — feature/drop-host-task-object-management into master |
antonwolfy
reviewed
Aug 17, 2026
antonwolfy
reviewed
Aug 19, 2026
ndgrigorian
force-pushed
the
feature/drop-host-task-object-management
branch
from
September 6, 2026 07:16
f412bf7 to
637c490
Compare
ndgrigorian
force-pushed
the
feature/drop-host-task-object-management
branch
from
September 14, 2026 21:29
ad6aa9e to
9a6f156
Compare
ndgrigorian
changed the title
host_task object management to a thread pool based approachhost_task object management
ndgrigorian
force-pushed
the
feature/drop-host-task-object-management
branch
from
September 17, 2026 22:00
84bf7e3 to
1d6e587
Compare
ndgrigorian
force-pushed
the
feature/drop-host-task-object-management
branch
from
October 29, 2026 20:02
1d6e587 to
920ff28
Compare
vlad-perevezentsev
left a comment
vlad-perevezentsev
left a comment
Collaborator
There was a problem hiding this comment.
Everything from the previous round is fixed. The forced-unwind handling also fixes an abort at exit on Python 3.12 (FATAL: exception not rethrown with the previous revision). Comments below.
Removed use of host_task that took the GIL and could free a USM allocation, so the task called sycl::free and destroyed a sycl::context inside the host_task. This is forbidden per the SYCL spec, and made dpctl unbuildable against AdaptiveCpp, which does not implement host_task. The releases are now deferred instead via a polling thread. Deprecates SyclQueue._submit_keep_args_alive in favor of SyclQueue.keep_args_alive, intended as a public method for keeping alive these Python objects in a way free of host_task. Also deprecates methods add_event_pair, num_host_task_events, and host_task_events on the SequentialOrderManager, while adding add_event, add_cleanup_event, and num_cleanup_events. _sycl_queue.pyx also gains an atexit hook to prevent crashes at process teardown.
Start dpctl's thread with the first release deferred instead of at import. Start it again in the child of a fork, through os.register_at_fork, and replace it if it ended on its own while the interpreter is still live. stop() waits at most a second for a release that blocks and then detaches the thread. It no longer holds the watcher's lock while waiting, so a concurrent start cannot deadlock. The thread stays stopped until started again, and releases it has not run yet go back on the list. collect_due queries events without holding the list's lock, so a long list does not hold up defer. The poll interval is capped at 100 ms. The interpreter's forced unwind of the thread now propagates through the watcher, run_collected and drain, and still reports the thread finished. The GIL is released if run_collected throws. A failure to start the thread after an item is pushed no longer makes async_dec_ref report a scheduled release as failed. keep_args_alive copies depends into a tuple, so the event count matches what it iterates over, and calls async_dec_ref without the GIL. dpctl4pybind11.hpp loads dpctl_capi outside the static initializer of get_deferred_releases(). Add tests for an exit and a stop with a release that blocks, a start during a stop, releases in a forked child, and a depends whose length differs from its iteration.
Keep DeferredReleases private behind a SyclQueue_DeferRelease C-API function, drop the poll interval cap, restart the thread after a timed-out stop, keep one named thread state, and drop the order manager's runtime deprecation warnings.
ndgrigorian
force-pushed
the
feature/drop-host-task-object-management
branch
from
November 4, 2026 19:10
920ff28 to
3e31cd7
Compare
vlad-perevezentsev
left a comment
vlad-perevezentsev
left a comment
Collaborator
There was a problem hiding this comment.
Thanks! I tested the new changes: everything from the previous round is fixed, including on Windows, and dpnp's test suite now passes with this branch. A few new comments below.
Declare SyclQueue.keep_args_alive last to keep the existing vtable slots, scale the poll interval by the share of items not yet due, make dpctl::utils::defer_release public, and raise import_dpctl failures from dpctl_capi.
Number each thread, and have stop let go of one it timed out on, so a start can replace it at once. A thread let go of puts back the rest of its batch after the release that blocked.
vlad-perevezentsev
previously approved these changes
Nov 5, 2026
vlad-perevezentsev
left a comment
vlad-perevezentsev
left a comment
Collaborator
There was a problem hiding this comment.
I re-reviewed the last round on Linux and Windows and everything is fixed!
The dpctl suite passes on Linux (GPU and CPU) and on Windows (3.13 and 3.14), and dpnp's suite passes with this branch on GPU.
LGTM
Thanks @ndgrigorian for such a large piece of work. It makes both dpctl and dpnp better
One nit below:
vlad-perevezentsev
approved these changes
Nov 5, 2026
6 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inspired by comment in: AdaptiveCpp/AdaptiveCpp#1915
we move away from use of
host_task, which enables compatibility with AdaptiveCpp, and fixes long-standing undefined behavior of callingsycl::freefrom within ahost_task. Thehost_taskdecrementing the refcount would cause theUSMDeleterto be called in its scope, which is undefined behavior under the SYCL specInstead, we use a polling thread which checks for event completion, then decreases references