iterate_items only covers a dataset that is already complete. There is no way to consume a run's
items as they are pushed, so anyone who wants to process output while the Actor is still working
has to hand-roll the polling loop.
Proposal
A live iterator on RunClient / RunClientAsync, which already holds both the run status and the
dataset:
async for item in apify_client.run(run_id).iterate_dataset_items():
...It polls while the run is READY or RUNNING, yields items as they appear, and returns once the
run reaches a terminal status and the dataset has been drained. Poll interval configurable, the
remaining arguments matching DatasetClient.iterate_items.
RunClient is the better home than DatasetClient: the termination condition needs the run status,
which the dataset client has no business knowing about.
Do not terminate on itemCount
Dataset.itemCount is bumped through a throttle (~5 s) while items are served fresh from S3 plus
the live buffer, so when a run finishes itemCount can still lag behind what is already readable.
Ending the final drain at read >= itemCount truncates the tail silently. Same root cause as #1058.
Drain until a page comes back short instead.
Prior art
greedyIterate in apify-orchestrator:
https://github.com/apify-professional-services/apify-orchestrator/blob/ef8b1c3feb1e08ea94ac40e1a1a6970b9c456a95/src/clients/dataset-client.ts#L54-L118
Useful for the overall shape, with two things to avoid: its final drain loop is bounded by a single
dataset.itemCount read, which is the truncation above, and itemsThreshold withholds items until
enough have accumulated, which its own TODO already marks for removal.
✍️ Drafted by Claude Code
iterate_itemsonly covers a dataset that is already complete. There is no way to consume a run'sitems as they are pushed, so anyone who wants to process output while the Actor is still working
has to hand-roll the polling loop.
Proposal
A live iterator on
RunClient/RunClientAsync, which already holds both the run status and thedataset:
It polls while the run is
READYorRUNNING, yields items as they appear, and returns once therun reaches a terminal status and the dataset has been drained. Poll interval configurable, the
remaining arguments matching
DatasetClient.iterate_items.RunClientis the better home thanDatasetClient: the termination condition needs the run status,which the dataset client has no business knowing about.
Do not terminate on
itemCountDataset.itemCountis bumped through a throttle (~5 s) while items are served fresh from S3 plusthe live buffer, so when a run finishes
itemCountcan still lag behind what is already readable.Ending the final drain at
read >= itemCounttruncates the tail silently. Same root cause as #1058.Drain until a page comes back short instead.
Prior art
greedyIteratein apify-orchestrator:https://github.com/apify-professional-services/apify-orchestrator/blob/ef8b1c3feb1e08ea94ac40e1a1a6970b9c456a95/src/clients/dataset-client.ts#L54-L118
Useful for the overall shape, with two things to avoid: its final drain loop is bounded by a single
dataset.itemCountread, which is the truncation above, anditemsThresholdwithholds items untilenough have accumulated, which its own TODO already marks for removal.
✍️ Drafted by Claude Code