[SYNPY-1933] Updated pandas to use 3.x by andrewelamb · Pull Request #1471 · Sage-Bionetworks/synapsePythonClient

andrewelamb · 2026-10-12T18:23:26Z

Problem:

setup.cfg pins pandas>=1.5,<3.0. No pandas 2.x release has a Python 3.15 wheel. pandas 3.0.6 is the first release that supports Python 3.11 through 3.15.
On Python 3.15 (PR [SYNPY-1926] Added support for python 3.15 #1470), pip builds pandas 2.3.3 from source. pytest then stops with a segmentation fault in pd.to_datetime (synapseclient/table.py). Thus the client must support pandas 3 before it can support Python 3.15.
With pandas 3.0.6, 13 unit tests fail. The same tests pass with pandas 2.3.3. A code search found more problems that the unit tests do not cover.

Solution:

The pandas requirement changes to pandas>=1.5,<4.0

Problem	File	Change
1	`synapseclient/models/services/manifest.py`, `synapseutils/sync.py`	Change the `used` and `executed` columns to `object` dtype before lists are written into the cells. New helper `_missing_to_empty_list()` in `sync.py` changes a `None` or `NaN` value in `used` or `executed` to an empty list, so that the upload can iterate over it.
2	`manifest.py` (`_apply_synapse_store_defaults`), `sync.py` (`readManifestFile_async`)	Change the `synapseStore` column to `object` dtype before `True`/`False` is set.
3	`synapseclient/extensions/curator/schema_generation.py` (`convert_floats`)	`float_df[col] = float_df[col].fillna(...)`.
4, 8	`schema_generation.py` (`find_and_convert_ints`)	Use `dataframe.apply(lambda col: col.map(...))` for all manifests. pandarallel is removed (see below).
5	`synapseclient/table.py` (`_convert_df_date_cols_to_datetime`)	`.astype("datetime64[ns, UTC]")`.
6	`synapseutils/describe_functions.py` (`_describe_wrapper`)	Report `"object"` if `is_string_dtype()` is true, else `dtype.name`.
7	`table.py`, `synapseclient/models/mixins/table_components.py`	`pd.concat` replaces `Series.append`. Use the correct keys on all pages. Append `ROW_ETAG`. When the row ID and version are not in the index, the final DataFrame is renumbered with `reset_index(drop=True)`, so that the index is unique across all pages. In `table_components.py`, `construct_rownames` now accepts a `RowSet`. Before this change, a result with more than one page raised `AttributeError` when the row ID and version were in the index (the default).

Other changes

pandarallel is removed
Pipfile.lock is regenerated (pipenv lock). pandas 2.3.3 changes to 3.0.6. pytz, tzdata, pandarallel and dill are removed from the develop section. Other packages have small version steps.

New CI job test-pandas-versions in .github/workflows/build.yml. It runs the unit tests on ubuntu-22.04 with older pandas versions. The package job now needs this job.

Python	pandas	numpy	Reason
3.11	1.5.3	`<2`	Lower bound in `setup.cfg`.
3.12	2.1.4	`<2`	First version with `DataFrame.map` and Python 3.12 wheels.
3.13	2.2.3	no pin	First version that supports Python 3.13.

pandas < 2.2.2 is not compatible with numpy 2, thus each entry sets its own numpy requirement. This is a separate job, so that the integration tests do not run on pandas 1.5.

CLAUDE.md now has the pandas compatibility rules and a description of the new CI job.

Testing:

New unit tests:

Problem	Test
7 (new API)	`TestRowsetToPandasDf::test_multiple_pages_row_id_and_version_not_in_index`
7 (new API)	`TestRowsetToPandasDf::test_multiple_pages_row_id_and_version_not_in_index_has_unique_index`
7 (new API)	`TestRowsetToPandasDf::test_multiple_pages_row_id_and_version_in_index`
7 (legacy)	`test_rowset_asDataFrame__multiple_pages_row_id_and_version_not_in_index`
7 (legacy)	`test_rowset_asDataFrame__multiple_pages_row_id_and_version_not_in_index_has_unique_index`
1 (`sync.py`)	`test_manifest_upload_missing_provenance_builds_dependency_graph`
2 (`manifest.py`)	`TestApplySynapseStoreDefaults::test_blank_column_read_from_csv`
2 (`sync.py`)	`test_read_manifest_file_synapse_store_values_all_blank`

Changed unit tests:

TestCsvToPandasDf (4 tests): .astype("datetime64[ns, UTC]") is added to the expected date values. The client gives datetime64[ns, UTC], but pd.to_datetime(..., unit="ms") gives datetime64[ms, UTC] on pandas 3, so the old expected values conflicted with the dtype check.
TestParseDfDateColsToDatetime::test_uniform_utc_offsets_parsed_as_tz_aware: the test now checks the UTC offset, not the timezone class. pandas 1.5 gives pytz.FixedOffset, and newer versions give datetime.timezone.

Changed integration tests:

With pandas 3.0.6, 5 integration tests failed. These are test fixes only. The client code does not change.

Test	Failure on pandas 3	Change
`test_table_async.py::TestRowStorage::test_store_rows_from_df_with_datetime_columns`, `test_store_rows_from_csv_with_date_columns`, `test_store_rows_from_csv_with_tz_aware_date_columns`	`None != nan`. The query gives an `object` column with `None` for the missing date. The expected column was `float64` with `NaN`.	The expected DataFrame uses `dtype=object`, so the missing value stays `None` on all pandas versions.
`test_entityview_async.py::TestEntityView::test_update_rows_and_annotations`	`None != nan` in `integer_column`.	The expected `integer_column` and `float_column` series use `dtype=object`.
`deprecated/test_tables.py::test_tables_pandas`	`dtype` is different: `datetime64[ns, UTC]` (client) vs `datetime64[ms, UTC]` (expected).	The expected `datetime64` column is changed to `datetime64[ns, UTC]` on the Windows and non-Windows paths.

andrewelamb

Comments that explain the code changes in synapseclient/ and synapseutils/. Refer to the problem numbers in the PR description.

andrewelamb · 2026-10-13T18:03:27Z

-        ).fillna(False)
+    # Series.map is used per column because DataFrame.map needs pandas 2.1 or
+    # later, and DataFrame.applymap was removed in pandas 3
+    ints = dataframe.apply(


Problems 4 and 8. DataFrame.applymap is removed in pandas 3, and DataFrame.map is not in pandas < 2.1. Series.map is in all supported versions, thus we apply it to each column. This code path is used for all manifest sizes.

The large-manifest path used pandarallel.parallel_applymap, which does not work with pandas 3. Thus pandarallel is removed from this function, from check_curator_imports(), and from the dependencies.

andrewelamb · 2026-10-13T18:03:27Z


    # Identify cells converted to integers
-    is_int = ints.map(is_integer)  # type: ignore
+    is_int = ints.apply(lambda col: col.map(is_integer))


Same change as above: ints.map(is_integer) (DataFrame.map, pandas >= 2.1) changes to Series.map on each column, so that it works on pandas 1.5 to 3.x.

andrewelamb · 2026-10-13T18:03:27Z


        # replace values that couldn't be converted to float with the original str values
-        float_df[col].fillna(dataframe[col][float_df[col].isna()], inplace=True)
+        float_df[col] = float_df[col].fillna(dataframe[col][float_df[col].isna()])


Problem 3. In pandas 3, Copy-on-Write is always on. float_df[col].fillna(..., inplace=True) changes a temporary copy and not float_df, so the original string values were not put back. We now assign the result back to the column.

andrewelamb · 2026-10-13T18:03:27Z

    import pandas as pd

-    def construct_rownames(query_result_bundle, offset=0):
+    def construct_rownames(rowset, offset=0):


Problem 7. construct_rownames now accepts a RowSet, not a QueryResultBundle. Before this change, the function read query_result_bundle.query_result.query_results.rows for every page. Later pages are RowSet objects, so a result with more than one page raised AttributeError when the row ID and version were in the index (the default). The KeyError fallback also used rowset["rows"], but a RowSet is a dataclass, so we use rowset.rows.

andrewelamb · 2026-10-13T18:03:27Z

-            # TODO: Look into why this isn't being assigned
-            series["ROW_ID"].append(
-                pd.Series(name="ROW_ID", data=[row.id for row in rowset.rows])
+            series["ROW_ID"] = pd.concat(


Problem 7. Series.append is removed in pandas 2.0. Also, the old code did not assign the result (see the removed TODO), so the rows from later pages were not added to ROW_ID and ROW_VERSION. We now use pd.concat and assign the result.

The key also changes from row.id to row.row_id, which is the correct field name on the Row model.

andrewelamb · 2026-10-13T18:03:27Z

+                # pandas 3 reports "str" for text columns; keep "object"
+                stats[column]["dtype"] = (
+                    "object"
+                    if pd.api.types.is_string_dtype(df[column].dtype)


Problem 6. In pandas 3, dtype.name for a text column is "str", not "object". We report "object" when is_string_dtype() is true, so that the output of describe() does not change for users.

andrewelamb · 2026-10-13T18:03:27Z

 def _sortAndFixProvenance(syn, df):
    df = df.set_index("path")
+    # Lists are written into these cells, which a str dtype column cannot hold
+    for col in PROVENANCE_FIELDS:


Problem 1 (legacy path). Same fix as in models/services/manifest.py: the used and executed columns change to object dtype, so that lists can be written into the cells on pandas 3.

andrewelamb · 2026-10-13T18:03:27Z

    if "synapseStore" not in df:
        df = df.assign(synapseStore=None)
+    # A blank column is read as float64, which cannot hold bool values
+    df["synapseStore"] = df["synapseStore"].astype(object)


Problem 2 (legacy path). Same fix as in _apply_synapse_store_defaults: a blank synapseStore column is read as float64, so we change it to object dtype before True/False is set.

andrewelamb · 2026-10-13T18:03:28Z

            file,
-            row["used"] if "used" in row else [],
-            row["executed"] if "executed" in row else [],
+            _missing_to_empty_list(row["used"]) if "used" in row else [],


Problem 1. On pandas 3, iterrows() gives a str-dtype row when all values in the row are strings or missing, and a missing used/executed value becomes NaN, not None. The upload iterates over these values, so NaN caused an error. _missing_to_empty_list() changes None or NaN to [].

andrewelamb · 2026-10-13T18:03:28Z

    return True


+def _missing_to_empty_list(value):


New helper for the change above. It checks float + math.isnan instead of pd.isna, because pd.isna on a list returns an array and not a bool.

andrewelamb

Comments that explain the tests that this PR adds or changes. Refer to the problem numbers in the PR description.

andrewelamb · 2026-10-13T18:06:52Z

        assert expected_indicies == dataframe.index.values.tolist()


+def _multi_page_query_results() -> tuple[dict, list[dict]]:


New helper (problem 7, legacy API). This helper makes a mocked table query with 3 pages: a first QueryResultBundle with 2 rows, then 2 next QueryResult pages with 2 rows and 1 row. Each row has a different rowId, etag and value, so the tests can make sure that the rows of all pages are in the DataFrame, in the correct order. The rows use the REST keys rowId and versionNumber, which the fix in table.py now reads.

andrewelamb · 2026-10-13T18:06:52Z

+    return query_result, next_pages
+
+
+def test_rowset_asDataFrame__multiple_pages_row_id_and_version_not_in_index(


New test (problem 7, legacy API). With rowIdAndVersionInIndex=False, this test makes sure that ROW_ID, ROW_VERSION and ROW_ETAG have the values of all 3 pages. Before the fix, Series.append did not assign its result (and it is removed in pandas 2), and ROW_ETAG had the etags of the last page only.

andrewelamb · 2026-10-13T18:06:53Z

+    assert [f"value{i}" for i in range(5)] == list(dataframe["annotationColumn1"])
+
+
+def test_rowset_asDataFrame__multiple_pages_row_id_and_version_not_in_index_has_unique_index(


New test (problem 7, legacy API). This test makes sure that the index is unique and is 0 to 4 across all pages. Before the fix, each page numbered its rows from 0, so the index had duplicate values.

andrewelamb · 2026-10-13T18:06:53Z

            assert result.select_columns[0].id == "12345"


+class TestRowsetToPandasDf:


New test class (problem 7, new API). These tests cover _rowset_to_pandas_df with more than one page. Before this PR, no unit test covered this path. _make_bundle makes a QueryResultBundle from the dataclass models (Row, RowSet, QueryResult). _query_table_next_page is mocked to return the next pages, so no network call is made.

andrewelamb · 2026-10-13T18:06:53Z

+            )
+        )
+
+    def test_multiple_pages_row_id_and_version_not_in_index(self) -> None:


New test. With row_id_and_version_in_index=False, this test makes sure that all pages are fetched, and that ROW_ID, ROW_VERSION, ROW_ETAG and the data columns have the values of all pages. This covers the pd.concat change and the row.row_id key fix.

andrewelamb · 2026-10-13T18:06:53Z

            include_row_id_and_row_version=False,
        )

+        # dtype=object keeps None for missing values, matching the query output, on all


Changed (test fix only). In test_store_rows_from_csv_with_date_columns, the query gives an object column with None for the missing date. Without dtype=object, the expected column was float64 with NaN, and on pandas 3 the comparison failed with None != nan. With dtype=object, the missing value stays None on all pandas versions. The client code does not change.

andrewelamb · 2026-10-13T18:06:53Z

            include_row_id_and_row_version=False,
        )

+        # dtype=object keeps None for missing values, matching the query output, on all


Changed (test fix only). In test_store_rows_from_csv_with_tz_aware_date_columns, the query gives an object column with None for the missing date. Without dtype=object, the expected column was float64 with NaN, and on pandas 3 the comparison failed with None != nan. With dtype=object, the missing value stays None on all pandas versions. The client code does not change.

andrewelamb · 2026-10-13T18:06:53Z

                check_names=True,
                check_dtype=False,
            )
+            # dtype=object keeps None for missing values, matching the query output,


Changed (test fix only). In test_update_rows_and_annotations, the query gives None for missing values in integer_column and float_column. The expected series used the default dtype, so a missing value became NaN, and on pandas 3 the comparison failed with None != nan. Both series now use dtype=object. The client code does not change.

andrewelamb · 2026-10-13T18:06:53Z

    if os.name == "nt":
-        df["datetime64"] = pd.to_datetime(df["datetime64"], utc=True)
+        df["datetime64"] = pd.to_datetime(df["datetime64"], utc=True).astype(
+            "datetime64[ns, UTC]"


Changed (test fix only). The client returns DATE columns as datetime64[ns, UTC] (see the change in _convert_df_date_cols_to_datetime). On pandas 3, pd.to_datetime can give a different resolution, for example datetime64[ms, UTC], so the dtype check failed. The expected column is now cast to datetime64[ns, UTC]. This line is the Windows path.

andrewelamb · 2026-10-13T18:06:53Z

        df["datetime64"] = df["datetime64"].astype(int)
-        df["datetime64"] = pd.to_datetime(df["datetime64"], unit="ms", utc=True)
+        df["datetime64"] = pd.to_datetime(df["datetime64"], unit="ms", utc=True).astype(
+            "datetime64[ns, UTC]"


Same change as above, for the non-Windows path (unit="ms").

andrewelamb requested a review from a team as a code owner October 12, 2026 18:23

andrewelamb marked this pull request as draft October 12, 2026 18:23

andrewelamb added this pull request to stack #1472 October 12, 2026 18:30

andrewelamb changed the title ~~updated pandas to use 3.x~~ [SYNPY-1933] Updated pandas to use 3.x Oct 12, 2026

andrewelamb commented Oct 13, 2026

View reviewed changes

andrewelamb marked this pull request as ready for review October 13, 2026 18:08

thomasyu888 requested a review from BryanFauble October 19, 2026 19:13

andrewelamb added 6 commits October 23, 2026 08:06

updated pandas to use 3.x

e7c5b71

fix tests that do not work in pandas 3.x

e319fd7

fix issue with duplicated df indeces

9e1826e

fix nan to none helper

f5235a0

construct_rownames now takes a RowSet

b22c3e2

added comments to tests

dce02d2

andrewelamb force-pushed the SYNPY-1933 branch from 95d3a79 to dce02d2 Compare October 23, 2026 15:06

		assert expected_indicies == dataframe.index.values.tolist()


		def _multi_page_query_results() -> tuple[dict, list[dict]]:

		return query_result, next_pages


		def test_rowset_asDataFrame__multiple_pages_row_id_and_version_not_in_index(

		assert [f"value{i}" for i in range(5)] == list(dataframe["annotationColumn1"])


		def test_rowset_asDataFrame__multiple_pages_row_id_and_version_not_in_index_has_unique_index(

		assert result.select_columns[0].id == "12345"


		class TestRowsetToPandasDf:

Conversation

andrewelamb commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem:

Solution:

Other changes

Testing:

Uh oh!

andrewelamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewelamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

andrewelamb commented Oct 12, 2026 •

edited

Loading