Repository navigation
[Bug][github_graphql] Collector Runs Indefinitely on Large Repos likely due to stream error: CANCEL #8614
Description
Search before asking
- I had searched in the issues and found no similar issues.
What happened
This is a follow-up to the discussion in #8576
When running a pipeline on a project that includes a very large GitHub repository (5+ years of history, 100' of commits weekly), the github_graphql task for collecting job runs hangs for days. The logs show that the process eventually fails due likely to a stream error: CANCEL received from the peer (GitHub's servers).
As identified by @klesh in the original thread, this is likely caused by the API response body size exceeding a limit on GitHub's end, which leads to the server terminating the connection.
Errors in the logs:
time="2025-10-15 13:01:52" level=warning msg=" [pipeline service] [pipeline #63] [task #16353] retry #1 graphql calling after 120s\n\tcaused by: stream error: stream ID 1; CANCEL; received from peer"
time="2025-10-15 13:04:03" level=warning msg=" [pipeline service] [pipeline #63] [task #16353] retry #2 graphql calling after 120s\n\tcaused by: non-200 OK status code: 502 Bad Gateway body: \"<html>\\r\\n<head><title>502 Bad Gateway</title></head>\\r\\n<body>\\r\\n<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx</center>\\r\\n</body>\\r\\n</html>\\r\\n\""
What do you expect to happen
The pipeline should handle large API responses either completing the data collection successfully or fail with a error message about exceeding API limits, rather than hanging indefinitely.
How to reproduce
- Configure a DevLake project with a connection to a very large GitHub repository.
- Create and run a new pipeline that includes collecting GitHub Actions data.
- Monitor the devlake container logs.
- Observe that the pipeline hangs on the
github_graphqltask, specifically "Collect Job Runs". - After a long period, the following error appears in the logs
Anything else
Attaching here a new snapshot log file.
task-16353-2-1-github_graphql.log
Version
v1.0.3-beta6@44f2db2
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
dosubot commentedon Oct 16, 2025
There isn't currently built-in handling in the github_graphql collector for stream errors like
CANCELor for large response cancellations from GitHub's servers. The collector uses pagination and retries, but if GitHub terminates the connection due to large responses, the task may hang and keep retrying without a clear error message or fail-fast behavior. This is especially likely with very large repositories, where API responses can exceed server or client limits, triggering these errors. No recent fixes or documentation address this scenario.As a workaround, you could try reducing the scope of data collection (for example, by limiting the date range or repository size), or lowering the page size if that's configurable. For long-term stability, it would be beneficial for the collector to implement clearer error handling for these cases—such as failing fast with a descriptive message when a stream error or large response cancellation is detected, and possibly exposing configuration for timeouts or retry limits.
If you need more robust handling, you may want to suggest improvements to the maintainers for better support of large repositories and clearer error reporting in the github_graphql collector.
References:
To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
klesh commentedon Oct 20, 2025
Thanks for filing the bug report.
However, as mentioned in the discussion, the connection is closed by GitHub's server, not because the pipeline failed to handle the payload.
A better approach would be to make the [page size](https://github.com/apache/incubator-devlake/blob/cb4c0d70f324ddb7302f7a469509c785fb178854/backend/plugins/gitlab/tasks/job_collector.go#L59] configurable, allowing users to reduce it when they encounter the same issue.
klesh commentedon Oct 20, 2025
#8616 has been merged, and it allows configuring GitHub PageSize of the job_collector through the Environment Variable
danielemoraschi commentedon Nov 25, 2025
@klesh I've been testing beta8 over the past few days and it seems to work. I'll close the issue for now and reach out if something else comes up. Thanks for the support!