Occasionally an ingestion job will fail with the error: "pyiceberg.exceptions.CommitFailedException: CatalogCommitConflicts: CatalogCommitConflicts => Requirement failed: Branch or tag main already exists". Rerunning the script fixes it most of the time but the pipelines need to be more robust.
Maybe write_to_table needs wrapping in a retry block?
Details:
File "/home/ubuntu/.cache/uv/environments-v2/accelerator-sharepoint-085545dc75ef208d/lib/python3.13/site-packages/pyiceberg/catalog/rest/response.py", line 111, in _handle_non_200_response
raise exception(response) from exc
pyiceberg.exceptions.CommitFailedException: CatalogCommitConflicts: CatalogCommitConflicts => Requirement failed: Branch or tag `main` already exists
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/.cache/uv/environments-v2/accelerator-sharepoint-085545dc75ef208d/lib/python3.13/site-packages/dlt/common/destination/client.py", line 424, in run_managed
self.run()
~~~~~~~~^^
File "/home/ubuntu/.cache/uv/environments-v2/accelerator-sharepoint-085545dc75ef208d/lib/python3.13/site-packages/elt_common/dlt_destinations/pyiceberg/pyiceberg.py", line 64, in run
self.iceberg_client.write_to_table(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
self.load_table_name,
^^^^^^^^^^^^^^^^^^^^^
pq.read_table(self._file_path),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
self._load_table.get("write_disposition"),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/ubuntu/.cache/uv/environments-v2/accelerator-sharepoint-085545dc75ef208d/lib/python3.13/site-packages/elt_common/dlt_destinations/pyiceberg/exceptions.py", line 33, in _wrap
raise DestinationTerminalException(status_ex) from status_ex
Occasionally an ingestion job will fail with the error: "pyiceberg.exceptions.CommitFailedException: CatalogCommitConflicts: CatalogCommitConflicts => Requirement failed: Branch or tag
mainalready exists". Rerunning the script fixes it most of the time but the pipelines need to be more robust.Maybe write_to_table needs wrapping in a retry block?
Details: