Skip to content

refactor: Move live dataset paging into DatasetClient.iterateItems - #1112

Closed
vdusek wants to merge 1 commit into
masterfrom
refactor/live-iteration-in-dataset-client
Closed

vdusek wants to merge 1 commit into
masterfrom
refactor/live-iteration-in-dataset-client

Conversation

@vdusek

@vdusek vdusek commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Moves the live paging behind RunClient.iterateDatasetItems() (#1085) into a new DatasetClient.iterateItems(). Without stopCondition, it iterates like listItems(). With it, the iterator follows a dataset that is still being written to: stopCondition is called before each poll, resolves to whether the writer has finished, and paces the polls. It can't be combined with desc.

iterateDatasetItems() keeps its options and sends the same requests. It now only pins the run on a lastRun() client and passes a callback that long-polls waitForFinish(). The dataset client does the paging: pages end at itemCount, the rows past the lagging itemCount are read once the run finishes, and a limit=1 read checks a page that filters emptied.

This follows Pijukatel's review on the Python twin, apify/apify-client-python#1079, which got the same change in apify/apify-client-python@fbf4192. It keeps the dataset paging in the dataset client and leaves only the end condition in the run client.

✍️ Drafted by Claude Code

@vdusek vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Oct 6, 2026
@vdusek vdusek self-assigned this Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

⚠️ There are broken links in the documentation.

See more at https://github.com/apify/apify-client-js/actions/runs/37437325785#summary-112182231331

@vdusek vdusek changed the title feat: Add DatasetClient.iterateItems for datasets still being written to refactor: Move live dataset paging into DatasetClient.iterateItems Oct 6, 2026
@vdusek
vdusek requested a review from Pijukatel October 6, 2026 08:54
@vdusek
vdusek marked this pull request as ready for review October 6, 2026 08:54
@apify-service-account apify-service-account added the tested Temporary label used only programatically for some analytics. label Oct 6, 2026
@vdusek

vdusek commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Closing: per Pijukatel's review on the Python twin, we're not adding the public stopCondition argument. The paging stays in RunClient.iterateDatasetItems (#1085) and can be refactored later if needed.

✍️ Drafted by Claude Code

@vdusek vdusek closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants