[FLINK-40417][python] Add remaining basic functionality to DataFrame API - #29002
[FLINK-40417][python] Add remaining basic functionality to DataFrame API#29002auroflow wants to merge 2 commits into
Conversation
Generated-by: Codex (GPT-5)
|
|
||
| >>> import pyflink.dataframe as pf | ||
| >>> df = pf.from_records([(2, 3)], schema=["left", "right"]) | ||
| >>> result = df.with_columns( |
There was a problem hiding this comment.
We can also add an example for named arguments
There was a problem hiding this comment.
Agreed, I added two separate examples for positional and named arguments.
| >>> import pyflink.dataframe as pf | ||
| >>> df = pf.from_records([(1, "Alice")], schema=["id", "name"]) | ||
| >>> by_mapping = df.rename_columns({"id": "user_id"}) | ||
| >>> by_callable = df.rename(str.upper) |
There was a problem hiding this comment.
Could you also add an lambda function example?
There was a problem hiding this comment.
Okay, now I use a lambda function explicitly.
| def rename_columns( | ||
| self, | ||
| *args: Any, | ||
| mapping: Optional[ |
There was a problem hiding this comment.
It seems that the args isn't that necessary. Removing it will make the API more clear.
There was a problem hiding this comment.
The *args is primarily used to support passing a list of alternating old and new name pairs. For example,
by_pairs = df.rename("id", "user_id", "name", "user_name")I have updated the docstring to explain this usage more clearly.
| ) | ||
| .drop("score", "city", "destination") | ||
| .rename({"name": "customer_name"}) | ||
| .pipe( |
There was a problem hiding this comment.
I think we can call .select directly. pipe is already welled tested in test_pipe_forwards_dataframe_arguments_and_return_value.
There was a problem hiding this comment.
Makes sense, pipe does not involve runtime logic. I have removed this call.
| "non_nullable_int", | ||
| ], | ||
| ) | ||
| self.assertEqual(result.schema.get_field_names(), result.columns) |
There was a problem hiding this comment.
It's not that necessary to validate this any more in IT case. It's already covered in test_schema_exposes_ordered_metadata and test_columns_returns_defensive_ordered_list
There was a problem hiding this comment.
Agreed, I removed this validation.
Clarify usage forms in the public docstrings and remove redundant integration-test coverage. Generated-by: Codex (GPT-5)
3 participants
What is the purpose of the change
This pull request completes the remaining core PyFlink DataFrame ergonomics defined by the DataFrame API FLIP. It adds the remaining basic column transformations, functional composition support, filtering aliases, and schema metadata properties.
Brief change log
DataFrame.with_columns()for adding or replacing multiple columns.DataFrame.drop_columns()and itsdropalias.DataFrame.rename_columns()and itsrenamealias, supporting mappings, callables, and positional name pairs.DataFrame.pipe()and thewherealias forfilter.schemaandcolumnsproperties.Verifying this change
This change added tests and can be verified as follows:
with_columns, column dropping and renaming, and schema metadata properties.pipeargument forwarding, return values, and alias identity.DataFrameITTests.test_basic_functionalityso the new transformations participate in its existing single executed job.Does this pull request potentially affect one of the following parts:
@Public(Evolving): yesDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-5)