Skip to content

feat: Add live iteration over a run's dataset items - #1085

Merged
vdusek merged 7 commits into
masterfrom
feat/live-dataset-iteration
Oct 5, 2026
Merged

feat: Add live iteration over a run's dataset items#1085
vdusek merged 7 commits into
masterfrom
feat/live-dataset-iteration

Conversation

@vdusek

@vdusek vdusek commented Sep 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Adds RunClient.iterateDatasetItems(). It yields a run's dataset items while the run is still pushing them and returns once the run has finished and the dataset is drained. It mirrors apify/apify-client-python#1079: same algorithm, options and tests.

Each poll reads the dataset's itemCount and fetches pages whose limit ends at it. The endpoint scans exactly limit rows, so offsets stay exact even when clean, skipEmpty or unwind change how many items a page returns. Between polls, waitForFinish() waits up to pollIntervalSecs for the run to finish.

itemCount lags about 5 s, so after the run finishes, the rows past it are read page by page until a page comes back empty. With clean, skipEmpty or unwind, one unfiltered limit: 1 read confirms the end. Polling goes on through ABORTING and TIMING-OUT, since such a run can still push items.

The options follow listItems() without desc, signature, flatten and view, plus pollIntervalSecs (default 5). unwind takes only string[], and signal reaches every request.

Docs: a new section in the Retrieve Actor data guide, and a mention on the convenience methods and pagination pages.

Closes #1073

✍️ Drafted by Claude Code

@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 29, 2026
@vdusek vdusek self-assigned this Sep 29, 2026
@github-actions

github-actions Bot commented Sep 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⚠️ There are broken links in the documentation.

See more at https://github.com/apify/apify-client-js/actions/runs/36873957439#summary-110408503694

@vdusek
vdusek marked this pull request as ready for review September 30, 2026 06:14
@vdusek
vdusek requested a review from szaganek as a code owner September 30, 2026 06:14
@vdusek vdusek changed the title feat: add live iteration over a run's dataset items feat: Add live iteration over a run's dataset items Sep 30, 2026
@vdusek
vdusek force-pushed the feat/live-dataset-iteration branch from 45f6cad to 689db0a Compare September 30, 2026 06:33
@vdusek
vdusek requested review from Pijukatel and barjin and removed request for Pijukatel September 30, 2026 06:35
@vdusek

vdusek commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Hi @barjin and @Pijukatel, FYI: I assigned Jindra to the JS PR and Pepa to the Python one. Both PRs implement the same feature.

Please also let me know WDYT about naming and whether you think adding just one method to RunClient is a good/enough way to support lazy dataset iteration, or whether it should live somewhere else too 🤔.

Thanks!

@barjin

barjin commented Sep 30, 2026

Copy link
Copy Markdown
Member

The recent force push(?) has messed up this PR's diff. Can you please merge v3 into this branch before we review this? Thank you!

@B4nan

B4nan commented Sep 30, 2026

Copy link
Copy Markdown
Member

Or just wait a bit and retarget to master, v3 will be merged there in a few minutes.

@B4nan
B4nan changed the base branch from v3 to master September 30, 2026 08:04
@B4nan B4nan mentioned this pull request Sep 30, 2026
Open
1 of 13 tasks
@vdusek
vdusek force-pushed the feat/live-dataset-iteration branch from 689db0a to c9450bd Compare September 30, 2026 08:09
@apify-service-account apify-service-account linked an issue Sep 30, 2026 that may be closed by this pull request
Live iteration over a run's dataset while the run is still running #1073
Closed
@apify-service-account apify-service-account added the tested Temporary label used only programatically for some analytics. label Sep 30, 2026
@vdusek

vdusek commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

The recent force push(?) has messed up this PR's diff. Can you please merge v3 into this branch before we review this? Thank you!

Ah, sorry. The v3 branch was rebased, so I rebased this branch as well to clean up the diff.

barjin
barjin reviewed Sep 30, 2026
View reviewed changes

@barjin barjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @vdusek ,

just one edge case regarding the /last run endpoint ⬇️

Imo the placement fits, this should be under RunClient, as it's bounded by the Run's lifetime. It's a bit weird we'll have both .dataset() and .iterateDatasetItems(), but I cannot think of a better alternative.

Comment thread src/resource_clients/run.ts Outdated
@vdusek
vdusek requested a review from barjin October 1, 2026 07:05
barjin
barjin approved these changes Oct 1, 2026
View reviewed changes

@barjin barjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have any further objections, thank you @vdusek!

@vdusek
vdusek merged commit f3ab011 into master Oct 5, 2026
9 checks passed
@vdusek
vdusek deleted the feat/live-dataset-iteration branch October 5, 2026 13:54
This was referenced Oct 6, 2026
Closed
Merged
vdusek added a commit that referenced this pull request Oct 7, 2026
@vdusek
fix: page offset iterators past the first page's total (#1125)
9cd54be 
Offset iterators (`list()`, `listItems()`) stopped at the first page's
`total`, so rows pushed during the loop were never read. They now page
until a page scans no rows or `limit` is reached, like the Python
client. Each page advances by the larger of `x-apify-pagination-count`
and `items.length`, capped at the requested limit, because the header
lags a fresh push.

Loosely related to #1085: `RunClient.iterateDatasetItems()` stays the
tool for reading a dataset a run is still writing.

Closes: #1117

*✍️ Drafted by Claude Code*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@barjin barjin barjin approved these changes
@szaganek szaganek Awaiting requested review from szaganek szaganek is a code owner

Assignees

@vdusek vdusek

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Live iteration over a run's dataset while the run is still running

4 participants

@vdusek @barjin @B4nan @apify-service-account