GH-45523: [R] Add bindings for Utf8View and BinaryView types by thisisnic · Pull Request #49712 · apache/arrow

thisisnic · 2026-04-11T18:15:28Z

Rationale for this change

The R package can't read or create utf8_view or binary_view arrays, so IPC files written by polars fail with "cannot handle Array of type <utf8_view>".

What changes are included in this PR?

Adds utf8_view() / string_view() and binary_view() types to the R package, with conversion to and from R vectors. Also allows view types as dictionary values in the C++ converter, which needed a small fix in the PyArrow dictionary converter.

Are these changes tested?

Yes, type and roundtrip tests for arrays and tables.

Are there any user-facing changes?

Yes, new utf8_view(), string_view() and binary_view() functions.

AI Usage

Heavily used Codex/Claude here. I'm not confident of every line of code. I read things over, and iterated on it making sure that tests pass and nothing seemed wildly incorrect.

GitHub Issue: [R] An "cannot handle Array of type <utf8_view>" error occurs when reading ipc file include strings output by r-polars #45523

thisisnic · 2026-04-11T18:17:20Z

We should rebase once #49710 is merged as changes from that PR are in this branch as they were needed to make it work.

thisisnic · 2026-05-06T20:52:43Z

A lot of .Rd updates as I used a newer roxygen2 version.

Copilot

Pull request overview

This PR adds R bindings and conversion support for Arrow utf8_view / string_view, addressing IPC/table conversion failures for data containing StringView columns.

Changes:

Adds string_view() R data type binding, export registration, and type tests.
Implements R↔Arrow conversion paths for StringView arrays and dictionary values.
Updates generated documentation and related converter support in C++/Python.

Reviewed changes

Copilot reviewed 13 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
r/R/type.R	Adds `StringView` R6 type, constructor, and canonical type aliases.
r/R/arrowExports.R	Registers R wrapper for `StringView__initialize`.
r/NAMESPACE	Exports `string_view`.
r/src/datatype.cpp	Maps Arrow `STRING_VIEW` to R `StringView` and initializes `utf8_view`.
r/src/r_to_arrow.cpp	Adds R-to-Arrow StringView conversion and dictionary StringView value handling.
r/src/array_to_vector.cpp	Adds Arrow-to-R StringView and wider dictionary index conversion support.
r/src/arrowExports.cpp	Registers native StringView initialization entry point.
cpp/src/arrow/util/converter.h	Enables dictionary converters for `StringViewType`.
python/pyarrow/src/arrow/python/python_to_arrow.cc	Adjusts Python dictionary StringView append call.
r/tests/testthat/test-Array.R	Adds StringView array round-trip tests.
r/tests/testthat/test-Table.R	Adds table/dictionary StringView and wider index tests.
r/tests/testthat/test-data-type.R	Adds StringView data type and code round-trip tests.
r/DESCRIPTION	Updates roxygen metadata.
r/man/data-type.Rd	Adds `string_view()` documentation entry.
r/man/acero.Rd	Regenerated Acero documentation.
r/man/arrow-package.Rd	Regenerated package author documentation.
r/man/csv_convert_options.Rd	Regenerated CSV conversion docs.
r/man/csv_read_options.Rd	Regenerated CSV read docs.
r/man/CsvReadOptions.Rd	Regenerated CSV read options docs.
r/man/enums.Rd	Regenerated enum documentation.
r/man/JsonFileFormat.Rd	Regenerated JSON file format docs.
r/man/reexports.Rd	Regenerated reexports docs.
r/man/vctrs_extension_array.Rd	Regenerated vctrs extension docs.

Files not reviewed (10)

r/man/CsvReadOptions.Rd: Language not supported
r/man/JsonFileFormat.Rd: Language not supported
r/man/acero.Rd: Language not supported
r/man/arrow-package.Rd: Language not supported
r/man/csv_convert_options.Rd: Language not supported
r/man/csv_read_options.Rd: Language not supported
r/man/data-type.Rd: Language not supported
r/man/enums.Rd: Language not supported
r/man/reexports.Rd: Language not supported
r/man/vctrs_extension_array.Rd: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 13 out of 23 changed files in this pull request and generated 1 comment.

Files not reviewed (10)

r/man/CsvReadOptions.Rd: Language not supported
r/man/JsonFileFormat.Rd: Language not supported
r/man/acero.Rd: Language not supported
r/man/arrow-package.Rd: Language not supported
r/man/csv_convert_options.Rd: Language not supported
r/man/csv_read_options.Rd: Language not supported
r/man/data-type.Rd: Language not supported
r/man/enums.Rd: Language not supported
r/man/reexports.Rd: Language not supported
r/man/vctrs_extension_array.Rd: Language not supported

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Files not reviewed (1)

r/man/data-type.Rd: Language not supported

thisisnic · 2026-06-18T15:00:31Z

Current CI failure is unrelated; #50219

…ings

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

…nary The offset-buffer fast path in Converter_String::Ingest_some_nulls() hardcoded GetValues<int32_t>, which is incorrect for LargeStringArray (int64_t offsets). Use StringArrayType::offset_type so the correct width is resolved at compile time. Also switch the view-type guard from a runtime is_binary_view_like() check to if constexpr, since StringViewArray has no offset_type member and the template would fail to instantiate otherwise. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Exercise out-of-line storage (>12 bytes) in the dictionary string_view test and add NULL + mixed lengths to the binary_view table test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

thisisnic · 2026-11-10T12:15:14Z

@jonkeane - Got time to look this over some time?

Copilot

🟢 Approval recommended

No active review issues were identified for the added R string-view and binary-view support.

No open issues 🎉

1 resolved since last review

Converter_BinaryView::Ingest_some_nulls() allocates a raw vector using value.size() without…

3 files not reviewed

r/man/data-type.Rd: Generated file
r/man/read_json_arrow.Rd: Generated file
r/man/schema.Rd: Generated file

_{Review effort: Balanced}

jonkeane · 2026-11-11T02:01:43Z

-  Status UnsafeAppendUtf8Strings(const cpp11::strings& s, int64_t size, int64_t offset) {
-    RETURN_NOT_OK(this->primitive_builder_->Reserve(s.size()));
+  Status AppendUtf8Strings(const cpp11::strings& s, int64_t size, int64_t offset) {


We change the name here from Unsafe... to ... is that because this is no longer unsafe? What did we change in here to do that? We still use UnsafeAppend in the non-stringview branch. Maybe it's that this no longer is always unsafe, just sometimes (for data types most people use?). I won't die on this hill, but it seems if it's important to flag safety in the name at all, we would want it still here if this is sometimes unsafe.

jonkeane

Do we need dplyr tests for this? When I explored this branch with Claude, it seems to think there are issues here:

dplyr treats string_view columns as not character (r/R/dplyr-funcs-type.R:184). The is.character() binding only checks STRING and LARGE_STRING, so a string column from a Polars IPC file (the case this PR is for) quietly returns FALSE. The same gap is in the string-to-date/time cast logic at r/R/dplyr-funcs-simple.R:218.

Also, do we need to test stringviews with our stringr bindings too? I suspect yes, there might be funny bits in there.

Another thing that Claude found that I didn't — this seems to be true (and we should fix), but I might be misunderstanding:

A blob can't be converted back to binary_view (r/R/array.R:328). as_arrow_array.blob only accepts binary() and large_binary(). The branch now turns binary_view arrays into arrow_binary objects, which are blobs, so converting one back with type = binary_view() fails with "Can't create Array from object…". The fix is to add binary_view() to that check.

github-actions Bot added Component: R Component: C++ awaiting committer review labels Apr 11, 2026

thisisnic mentioned this pull request Apr 30, 2026

GH-39603: [R] Error: Cannot convert Dictionary Array of type dictionary<values=large_string, indices=uint32, ordered=0> to R #49710

Merged

thisisnic force-pushed the GH-45523-ipc-polars branch 2 times, most recently from e8768d0 to bb711a7 Compare May 5, 2026 16:41

github-actions Bot added Component: Python awaiting changes and removed awaiting committer review labels May 5, 2026

thisisnic commented May 6, 2026

View reviewed changes

Comment thread python/pyarrow/src/arrow/python/python_to_arrow.cc Resolved

thisisnic commented May 6, 2026

View reviewed changes

Comment thread r/src/r_to_arrow.cpp Resolved

Copilot AI review requested due to automatic review settings May 28, 2026 09:37

thisisnic force-pushed the GH-45523-ipc-polars branch from de59365 to cf9ad10 Compare May 28, 2026 09:37

Copilot started reviewing on behalf of thisisnic May 28, 2026 09:37

View session

github-actions Bot added awaiting change review and removed awaiting changes labels May 28, 2026

Copilot AI reviewed May 28, 2026

View reviewed changes

Comment thread r/src/r_to_arrow.cpp Resolved

Comment thread r/man/acero.Rd Resolved

Copilot AI review requested due to automatic review settings May 28, 2026 12:11

Copilot started reviewing on behalf of thisisnic May 28, 2026 12:11

View session

Copilot AI reviewed May 28, 2026

View reviewed changes

Comment thread r/src/r_to_arrow.cpp Resolved

thisisnic force-pushed the GH-45523-ipc-polars branch from a74c9c1 to 1842e7e Compare June 10, 2026 15:10

Copilot AI review requested due to automatic review settings June 10, 2026 15:18

Copilot started reviewing on behalf of thisisnic June 10, 2026 15:18

View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

thisisnic marked this pull request as ready for review June 18, 2026 15:01

thisisnic requested a review from jonkeane as a code owner June 18, 2026 15:01

Copilot AI review requested due to automatic review settings June 18, 2026 15:01

thisisnic and others added 21 commits November 10, 2026 09:48

Implement string view

3bf01bb

add context

ecc997f

expand tests

0e95dd8

Fix offset bug

47547d9

Fix DictionaryBuilder<StringViewType> Append signature in Python bind…

027badd

…ings

Fix DictionaryBuilder::Append for StringViewType in Python bindings

f19995d

Simplify tests

a83c185

Simplify tests further

46c430c

appease linter

7f07db1

Simplify code as per Codex feedback

fac813e

Potential fix for pull request finding

48a48ee

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

5898260

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Update r/tests/testthat/test-Array.R

9c1215f

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Implement binary view

a2a7041

Add binaryview

88626cd

ditch errant test

3ffb288

Revert formatting frama

30c5538

Check R_XLEN_T_MAX guard

0b16620

Add longer strings to Table string_view/binary_view tests

b2e77bd

Exercise out-of-line storage (>12 bytes) in the dictionary string_view test and add NULL + mixed lengths to the binary_view table test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Respond to PR comments

7636e43

thisisnic force-pushed the GH-45523-ipc-polars branch from 8878bfa to 7636e43 Compare November 10, 2026 09:50

thisisnic changed the title ~~GH-45523: [R] Implement Utf8View type bindings~~ GH-45523: [R] Add bindings for Utf8View and BinaryView types Nov 10, 2026

thisisnic marked this pull request as ready for review November 10, 2026 12:06

Copilot AI balanced review requested due to automatic review settings November 10, 2026 12:06

Copilot started reviewing on behalf of thisisnic November 10, 2026 12:21

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

jonkeane reviewed Nov 11, 2026

View reviewed changes

Navigation Menu

Repository navigation

GH-45523: [R] Add bindings for Utf8View and BinaryView types - #49712

GH-45523: [R] Add bindings for Utf8View and BinaryView types#49712
thisisnic wants to merge 22 commits into
apache:mainfrom
thisisnic:GH-45523-ipc-polars

thisisnic commented Apr 11, 2026 •

edited

Loading

Uh oh!

thisisnic commented Apr 11, 2026

Uh oh!

Uh oh!

Uh oh!

thisisnic commented May 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

thisisnic commented Jun 18, 2026

Uh oh!

thisisnic commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jonkeane Nov 11, 2026

Uh oh!

jonkeane left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

thisisnic commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

AI Usage

Uh oh!

thisisnic commented Apr 11, 2026

Uh oh!

Uh oh!

Uh oh!

thisisnic commented May 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

thisisnic commented Jun 18, 2026

Uh oh!

thisisnic commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

jonkeane Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

jonkeane left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

thisisnic commented Apr 11, 2026 •

edited

Loading