From 10153b41a25de322e854269ca0f20e7bf433eeda Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Fri, 23 Oct 2026 22:42:08 +0700 Subject: [PATCH 1/3] feat(linux-browse): paste rows from the clipboard into a table tab --- linux/ROADMAP.md | 5 +- .../app/src/application/accelerators.rs | 8 +- .../crates/app/src/services/change_tracker.rs | 16 +- .../app/src/services/column_relayout.rs | 4 +- .../app/src/ui/browse_tab/browse_actions.rs | 11 +- .../app/src/ui/browse_tab/browse_tab_input.rs | 2 + .../src/ui/browse_tab/clipboard_handlers.rs | 10 +- .../app/src/ui/browse_tab/clipboard_input.rs | 14 +- .../app/src/ui/browse_tab/confirming_paste.rs | 12 + .../app/src/ui/browse_tab/edit_handlers.rs | 7 +- .../app/src/ui/browse_tab/edit_input.rs | 2 + linux/crates/app/src/ui/browse_tab/mod.rs | 137 ++- .../crates/app/src/ui/browse_tab/paste_at.rs | 9 + .../app/src/ui/browse_tab/paste_handlers.rs | 285 ++++++ .../app/src/ui/browse_tab/paste_read.rs | 77 ++ .../crates/app/src/ui/browse_tab/tests/mod.rs | 1 + .../app/src/ui/browse_tab/tests/paste.rs | 449 ++++++++++ .../app/src/ui/clipboard/clipboard_format.rs | 4 + .../app/src/ui/clipboard/clipboard_payload.rs | 4 + .../app/src/ui/clipboard/clipboard_writer.rs | 35 +- .../app/src/ui/clipboard/copy_renderer.rs | 58 +- linux/crates/app/src/ui/clipboard/mod.rs | 2 +- linux/crates/app/src/ui/error_text/edit.rs | 50 +- linux/crates/app/src/ui/error_text/mod.rs | 2 +- linux/crates/app/src/ui/file_read.rs | 5 +- linux/crates/app/src/ui/grid/cell_focus.rs | 15 + linux/crates/app/src/ui/grid/cell_lock.rs | 18 + linux/crates/app/src/ui/grid/grid_menus.rs | 15 + linux/crates/app/src/ui/grid/grid_msg.rs | 5 + .../crates/app/src/ui/grid/grid_row_index.rs | 3 +- linux/crates/app/src/ui/grid/mod.rs | 4 +- .../crates/app/src/ui/grid/row_json_dialog.rs | 1 + linux/crates/app/src/ui/workspace/import.rs | 12 +- linux/crates/core/src/edit/mod.rs | 2 + linux/crates/core/src/edit/new_draft_cell.rs | 13 + .../core/src/export/encoder/cells_encoder.rs | 75 ++ linux/crates/core/src/export/encoder/mod.rs | 2 + .../core/src/export/encoder/tsv_encoder.rs | 4 +- linux/crates/core/src/export/mod.rs | 6 +- linux/crates/core/src/export/tsv.rs | 3 + linux/crates/core/src/import/csv_import.rs | 4 +- linux/crates/core/src/import/mod.rs | 8 +- linux/crates/core/src/import/pasted_rows.rs | 10 + linux/crates/core/src/import/pasted_table.rs | 829 ++++++++++++++++++ .../src/conformance/cases/round_trip.rs | 204 ++++- .../src/conformance/round_trip_suite_macro.rs | 6 +- linux/data/resources/shortcuts-dialog.ui | 6 + linux/docs/testing.md | 3 +- linux/po/POTFILES.in | 5 + 49 files changed, 2368 insertions(+), 94 deletions(-) create mode 100644 linux/crates/app/src/ui/browse_tab/confirming_paste.rs create mode 100644 linux/crates/app/src/ui/browse_tab/paste_at.rs create mode 100644 linux/crates/app/src/ui/browse_tab/paste_handlers.rs create mode 100644 linux/crates/app/src/ui/browse_tab/paste_read.rs create mode 100644 linux/crates/app/src/ui/browse_tab/tests/paste.rs create mode 100644 linux/crates/core/src/edit/new_draft_cell.rs create mode 100644 linux/crates/core/src/export/encoder/cells_encoder.rs create mode 100644 linux/crates/core/src/import/pasted_rows.rs create mode 100644 linux/crates/core/src/import/pasted_table.rs diff --git a/linux/ROADMAP.md b/linux/ROADMAP.md index 20b8d90072..c4ff91c3c1 100644 --- a/linux/ROADMAP.md +++ b/linux/ROADMAP.md @@ -46,7 +46,7 @@ The app runs on GNOME 50 ([ADR 0015](docs/decisions/0015-gnome-50-platform-floor | Filters | Per-column operators, a raw WHERE fragment checked as one expression, match any or all; binary columns compare to `0x` hex. Kept per table, and per tab across a restart or a reopen | | Sorting | A header click sorts tables on the server and query results in the grid, one column at a time | | Cell editing | Editors by column kind: inline text, a multi-line editor, a calendar for dates, a toggle for booleans, JSON checked before saving; values are edited from their exact text, never from what the cell displays. Table tabs only, on tables with a primary key; key, generated and auto-increment columns, and binary, spatial, array and composite values, are read-only | -| Rows | Insert, duplicate and delete; Set NULL and Empty, and DEFAULT on a new row; undo and redo; save or discard pending changes, and stop a save. On ClickHouse a save is not atomic, and Stop takes effect between statements | +| Rows | Insert, duplicate and delete; Paste (Ctrl+V) adds tab-separated rows from the clipboard, filling the columns from the focused one or by a header's names, or edits the focused cell with one value, and asks first above 1,000 rows, up to 100,000; Set NULL and Empty, and DEFAULT on a new row; undo and redo; save or discard pending changes, and stop a save. On ClickHouse a save is not atomic, and Stop takes effect between statements | | Bulk delete | Deleting 10 rows or more asks first, unless Confirm Before Deleting Rows is off in Preferences | | Copy | Copy; Copy As rows, with headers, JSON, CSV with or without headers, Markdown or IN clause, and INSERT in an editable table tab; Copy Column Name; Show Row as JSON | | Foreign keys | Open Referenced Row (Ctrl+J) on a key cell opens the table it points at in a new tab, filtered on every column of the key; Show Referencing Rows on a saved row opens a table whose key points at it. A tab already showing those rows comes to the front | @@ -123,7 +123,7 @@ The app runs on GNOME 50 ([ADR 0015](docs/decisions/0015-gnome-50-platform-floor | Desktop integration | Desktop file with Open With for SQLite files, D-Bus activation, a man page, metainfo with screenshots, and SVG icons | | Translations | Vietnamese ships. The app's own strings go through gettext, and CI fails when `POTFILES.in` is stale or a translation loses a placeholder. CI builds the template with gettext 1.0 and publishes it from main and with each release; translators send pull requests against `po/*.po` ([po/README.md](po/README.md#where-translators-work)) | | Logs | stderr, which the session journals, with JSON lines on request. A panic also writes a crash report under `$XDG_STATE_HOME`, offered with Show Report on the next launch | -| Tests | Unit, gtk (with an accessibility audit of the picker, the connect pages, the workspace, the connect dialog and every preferences page), data and packaging suites; connection, DDL and TLS conformance, and a CSV and SQL INSERT export round trip, on every engine; Docker jobs for PostgreSQL with PgBouncer, MySQL and MariaDB, SQL Server, ClickHouse, SSH, the session and the grid save ([testing.md](docs/testing.md)); every critical audit finding has named guards ([regression-tests.md](docs/regression-tests.md)) | +| Tests | Unit, gtk (with an accessibility audit of the picker, the connect pages, the workspace, the connect dialog and every preferences page), data and packaging suites; connection, DDL and TLS conformance, and a CSV and SQL INSERT export round trip and a copy and paste round trip, on every engine; Docker jobs for PostgreSQL with PgBouncer, MySQL and MariaDB, SQL Server, ClickHouse, SSH, the session and the grid save ([testing.md](docs/testing.md)); every critical audit finding has named guards ([regression-tests.md](docs/regression-tests.md)) | ## Backlog @@ -136,7 +136,6 @@ Not built yet, in no order. ### Browse, edit and structure -- Paste rows from the clipboard. Ctrl+V in the grid says it is not supported yet. - Set a cell to `NOW()` or `CURRENT_TIMESTAMP`, which needs SQL expressions as cell values. - Edit binary columns from and to a file. They are read-only today, as are array, spatial, composite and variant columns. - Time and timestamp pickers. Dates have a calendar; times and timestamps are typed. diff --git a/linux/crates/app/src/application/accelerators.rs b/linux/crates/app/src/application/accelerators.rs index d8b794f0c0..007cd1cb55 100644 --- a/linux/crates/app/src/application/accelerators.rs +++ b/linux/crates/app/src/application/accelerators.rs @@ -190,7 +190,13 @@ pub const ACCELERATORS: &[Accelerator] = &[ &["c"], Listed(DataGrid), ), - local(BrowseGrid, BUBBLE, "browse.paste-rows", &["v"], Unlisted), + local( + BrowseGrid, + BUBBLE, + "browse.paste-rows", + &["v"], + Listed(DataGrid), + ), local( BrowseGrid, BUBBLE, diff --git a/linux/crates/app/src/services/change_tracker.rs b/linux/crates/app/src/services/change_tracker.rs index b667048ee3..d31f21eae9 100644 --- a/linux/crates/app/src/services/change_tracker.rs +++ b/linux/crates/app/src/services/change_tracker.rs @@ -264,7 +264,7 @@ impl TabChangeTracker { if rows.is_empty() { return self.delta(Vec::new()); } - let drafts: Vec = rows + let mut drafts: Vec = rows .into_iter() .map(|cells| { let draft_id = self.next_draft_id; @@ -276,6 +276,10 @@ impl TabChangeTracker { } }) .collect(); + // Drafts are kept bottom up, the last one shown on top, so the + // rows go in last first to read down in their own order. Their + // touches keep that order for the save. + drafts.reverse(); self.push_undo(UndoOp::Insert { drafts: drafts.clone() }); self.put_drafts(drafts) } @@ -1031,15 +1035,6 @@ fn insert_cells(columns: &[ColumnInfo], draft: &DraftRow) -> Vec { .collect() } -/// What a new row holds in `column` until the user types: the server's -/// value where it has one, else NULL. -pub fn new_draft_cell(column: &ColumnInfo) -> CellInput { - match column.is_auto_increment || column.is_generated || !column.default.is_none() { - true => CellInput::Default, - false => CellInput::Value(Value::Null), - } -} - #[cfg(test)] mod tests { /// A column carrying the server's spelling and the kind it means, @@ -1073,6 +1068,7 @@ mod tests { use super::*; use tablepro_core::ColumnInfo; + use tablepro_core::edit::new_draft_cell; fn pk_col(name: &str) -> ColumnInfo { ColumnInfo { diff --git a/linux/crates/app/src/services/column_relayout.rs b/linux/crates/app/src/services/column_relayout.rs index daa4c34870..d9a6ac8e20 100644 --- a/linux/crates/app/src/services/column_relayout.rs +++ b/linux/crates/app/src/services/column_relayout.rs @@ -1,10 +1,10 @@ use tablepro_core::column::ColumnType; -use tablepro_core::edit::{CellInput, parse_literal_text}; +use tablepro_core::edit::{CellInput, new_draft_cell, parse_literal_text}; use tablepro_core::export::value_text; use tablepro_core::value::same_value; use tablepro_core::{ColumnInfo, Value}; -use super::change_tracker::{RowKey, new_draft_cell}; +use super::change_tracker::RowKey; /// How the columns a tab's pending changes were made against line up /// with the columns it read since. Columns are matched by name: once a diff --git a/linux/crates/app/src/ui/browse_tab/browse_actions.rs b/linux/crates/app/src/ui/browse_tab/browse_actions.rs index 7903a7f6cc..a63bc9df60 100644 --- a/linux/crates/app/src/ui/browse_tab/browse_actions.rs +++ b/linux/crates/app/src/ui/browse_tab/browse_actions.rs @@ -73,7 +73,16 @@ pub(super) fn install_browse_actions(tab: &BrowseTab, page_size: u64) -> BrowseA }); group.add_action(&page_size); add("refresh", || PagingInput::RequestRefresh.into()); - add("paste-rows", || ClipboardInput::PasteNotSupported.into()); + let paste_rows = gio::SimpleAction::new("paste-rows", None); + let weak = tab.downgrade(); + paste_rows.connect_activate(move |_, _| { + let Some(tab) = weak.upgrade() else { + return; + }; + let at = tab.read(BrowseModel::focused_cell).flatten(); + tab.send(ClipboardInput::Paste(at)); + }); + group.add_action(&paste_rows); let previous = add("previous-page", || PagingInput::PrevPage.into()); let next = add("next-page", || PagingInput::NextPage.into()); tab.imp().pager_bar.follow_page_actions(ACTION_GROUP, &previous, &next); diff --git a/linux/crates/app/src/ui/browse_tab/browse_tab_input.rs b/linux/crates/app/src/ui/browse_tab/browse_tab_input.rs index 870dd93490..69a63a1176 100644 --- a/linux/crates/app/src/ui/browse_tab/browse_tab_input.rs +++ b/linux/crates/app/src/ui/browse_tab/browse_tab_input.rs @@ -2,6 +2,7 @@ use super::clipboard_input::ClipboardInput; use super::data_input::DataInput; use super::edit_input::EditInput; use super::paging_input::PagingInput; +use super::paste_at::PasteAt; use super::save_input::SaveInput; use super::selection_input::SelectionInput; use crate::ui::grid::GridMsg; @@ -70,6 +71,7 @@ impl From for BrowseTabInput { GridMsg::DeleteRows { rows } => EditInput::DeleteRows(rows).into(), GridMsg::InsertRow => EditInput::InsertRow.into(), GridMsg::DuplicateRows { rows } => EditInput::DuplicateRows(rows).into(), + GridMsg::Paste { at } => ClipboardInput::Paste(at.map(|(row, column)| PasteAt { row, column })).into(), GridMsg::FollowReference(reference) => ClipboardInput::GridFollowReference(reference).into(), } } diff --git a/linux/crates/app/src/ui/browse_tab/clipboard_handlers.rs b/linux/crates/app/src/ui/browse_tab/clipboard_handlers.rs index 56fc4b9c72..dfbdabe5d5 100644 --- a/linux/crates/app/src/ui/browse_tab/clipboard_handlers.rs +++ b/linux/crates/app/src/ui/browse_tab/clipboard_handlers.rs @@ -6,7 +6,7 @@ use crate::ui::export::{ExportOffer, ExportOrigin, ExportQuery, ExportSource}; use crate::ui::grid::{GridRowId, RowObject, RowSnapshot, export_snapshot}; impl BrowseModel { - pub(super) fn on_clipboard(&self, input: ClipboardInput, fx: &mut Effects) { + pub(super) fn on_clipboard(&mut self, input: ClipboardInput, fx: &mut Effects) { match input { ClipboardInput::Export(page) => self.offer_export(page, fx), ClipboardInput::GridCopy(request) => fx.output(BrowseTabOutput::Copy(request)), @@ -22,11 +22,9 @@ impl BrowseModel { ))), }, ClipboardInput::CopySelectedRowsAsTsv => self.copy_selected_rows(fx), - ClipboardInput::PasteNotSupported => { - fx.output(BrowseTabOutput::ShowToast(crate::i18n::gettext( - "Pasting rows isn't supported yet", - ))); - } + ClipboardInput::Paste(at) => self.paste(at, fx), + ClipboardInput::PasteRead(read) => self.paste_read(*read, fx), + ClipboardInput::PasteAnswered { ticket, confirmed } => self.paste_answered(ticket, confirmed, fx), ClipboardInput::GridShowRowJson(request) => fx.output(BrowseTabOutput::ShowRowJson(request)), ClipboardInput::GridShowToast(message) => fx.output(BrowseTabOutput::ShowToast(message)), ClipboardInput::GridFollowReference(reference) => fx.output(BrowseTabOutput::FollowReference(reference)), diff --git a/linux/crates/app/src/ui/browse_tab/clipboard_input.rs b/linux/crates/app/src/ui/browse_tab/clipboard_input.rs index a6093bc3b9..eb1d654177 100644 --- a/linux/crates/app/src/ui/browse_tab/clipboard_input.rs +++ b/linux/crates/app/src/ui/browse_tab/clipboard_input.rs @@ -1,11 +1,12 @@ use tablepro_core::browse::RowReference; +use super::paste_at::PasteAt; +use super::paste_read::PasteRead; use crate::ui::clipboard::{CopyRequest, InClauseRows}; use crate::ui::export::ExportSource; use crate::ui::grid::{RowJsonRequest, RowObject}; -/// What the tab hands on and keeps no state for: copies in each format, -/// exports and the grid's notices. +/// Copies in each format, pastes, exports and the grid's notices. #[derive(Debug)] pub(crate) enum ClipboardInput { /// The paginator's Export button, or the grid's Export with the page @@ -15,7 +16,14 @@ pub(crate) enum ClipboardInput { GridCopyRowsAsInsert(Vec), GridCopyInClause(InClauseRows), CopySelectedRowsAsTsv, - PasteNotSupported, + /// Ctrl+V or a menu's Paste, at the cell it names. + Paste(Option), + PasteRead(Box), + /// The user answered the question a large paste asked. + PasteAnswered { + ticket: u64, + confirmed: bool, + }, GridShowRowJson(RowJsonRequest), GridShowToast(String), GridFollowReference(RowReference), diff --git a/linux/crates/app/src/ui/browse_tab/confirming_paste.rs b/linux/crates/app/src/ui/browse_tab/confirming_paste.rs new file mode 100644 index 0000000000..8a92ff467a --- /dev/null +++ b/linux/crates/app/src/ui/browse_tab/confirming_paste.rs @@ -0,0 +1,12 @@ +use libadwaita as adw; +use tablepro_core::ColumnInfo; +use tablepro_core::import::PastedRows; + +/// A paste large enough to ask about, waiting for the answer. Its cells +/// sit in the order of the columns they were read against. +pub(super) struct ConfirmingPaste { + pub(super) ticket: u64, + pub(super) rows: PastedRows, + pub(super) columns: Vec, + pub(super) dialog: adw::AlertDialog, +} diff --git a/linux/crates/app/src/ui/browse_tab/edit_handlers.rs b/linux/crates/app/src/ui/browse_tab/edit_handlers.rs index bfa6e0aa81..4dc8cd4ee4 100644 --- a/linux/crates/app/src/ui/browse_tab/edit_handlers.rs +++ b/linux/crates/app/src/ui/browse_tab/edit_handlers.rs @@ -5,12 +5,12 @@ use gtk4::prelude::*; use libadwaita as adw; use libadwaita::prelude::*; use tablepro_core::Value; -use tablepro_core::edit::{CellInput, EditParseError, parse_edit_text}; +use tablepro_core::edit::{CellInput, EditParseError, new_draft_cell, parse_edit_text}; use super::confirming_delete::ConfirmingDelete; use super::row_deletion::{RowDeletion, plan_row_deletion}; use super::{BULK_DELETE_CONFIRM_THRESHOLD, BrowseModel, BrowseTabOutput, EditInput, Effects, apply_draft_changes}; -use crate::services::change_tracker::{RowKey, new_draft_cell, pk_identity}; +use crate::services::change_tracker::{RowKey, pk_identity}; use crate::services::column_relayout::ColumnRelayout; use crate::services::tracker_delta::TrackerDelta; use crate::services::tracker_edit::TrackerEdit; @@ -28,6 +28,7 @@ impl BrowseModel { fx.input(EditInput::FocusInsertedDraft); } EditInput::FocusInsertedDraft => self.focus_inserted_draft(), + EditInput::SelectPastedDrafts(count) => self.select_pasted_drafts(count), EditInput::DeleteRows(rows) => self.delete_rows(&rows, fx), EditInput::GridEdit { row, column, edit } => { // The cell shows what was typed until its row is drawn again. @@ -189,7 +190,7 @@ impl BrowseModel { self.sync_delete_action(); } - fn apply_edit(&mut self, edit: TrackerEdit, fx: &mut Effects) { + pub(super) fn apply_edit(&mut self, edit: TrackerEdit, fx: &mut Effects) { if self.tracker.refuses(&edit) && !std::mem::replace(&mut fx.told_edit_held, true) { fx.output(BrowseTabOutput::ShowToast(crate::i18n::gettext( "This row is being saved. Change it again once the save finishes.", diff --git a/linux/crates/app/src/ui/browse_tab/edit_input.rs b/linux/crates/app/src/ui/browse_tab/edit_input.rs index 8b1f5ae13d..81446c9ed7 100644 --- a/linux/crates/app/src/ui/browse_tab/edit_input.rs +++ b/linux/crates/app/src/ui/browse_tab/edit_input.rs @@ -10,6 +10,8 @@ pub(crate) enum EditInput { /// Queued after an insert, so the focus lands on the column view /// as it is once the insert is drawn. FocusInsertedDraft, + /// Queued after a paste, as `FocusInsertedDraft` is after an insert. + SelectPastedDrafts(u32), DeleteRows(Vec), /// Checked against the cell's lock before the tracker takes it. GridEdit { diff --git a/linux/crates/app/src/ui/browse_tab/mod.rs b/linux/crates/app/src/ui/browse_tab/mod.rs index 70843497ad..36b48042f8 100644 --- a/linux/crates/app/src/ui/browse_tab/mod.rs +++ b/linux/crates/app/src/ui/browse_tab/mod.rs @@ -21,6 +21,7 @@ mod column_recheck; mod columns_fetch; mod command_handlers; mod confirming_delete; +mod confirming_paste; mod count_failure_text; mod count_policy; mod data_handlers; @@ -36,6 +37,9 @@ mod page_state; mod pager_bar; mod paging_handlers; mod paging_input; +mod paste_at; +mod paste_handlers; +mod paste_read; mod pending_changes_bar; mod row_count; mod row_deletion; @@ -77,6 +81,7 @@ use self::browse_tab_widgets::BrowseTabWidgets; pub(crate) use self::clipboard_input::ClipboardInput; use self::column_recheck::ColumnRecheck; use self::confirming_delete::ConfirmingDelete; +use self::confirming_paste::ConfirmingPaste; pub(crate) use self::data_input::DataInput; pub(crate) use self::edit_input::EditInput; pub(crate) use self::page_reply::PageReply; @@ -84,6 +89,7 @@ use self::page_state::PageState; #[cfg(test)] pub(super) use self::pager_bar::PagerBar; pub(crate) use self::paging_input::PagingInput; +use self::paste_read::{PasteOutcome, PasteRead, PasteRequest, PasteSource, read_pasted}; #[cfg(test)] pub(super) use self::pending_changes_bar::PendingChangesBar; pub(crate) use self::row_total::RowTotal; @@ -210,6 +216,8 @@ impl BrowseTab { suppress_combo_emit: Cell::new(true), confirming_delete: None, next_confirmation: 0, + next_paste: 0, + confirming_paste: None, widgets, }; imp.model.replace(Some(model)); @@ -440,9 +448,43 @@ impl BrowseTab { BrowseTabOutput::FollowReference(reference) => { self.emit_by_name::<()>("follow-reference", &[&glib::BoxedAnyObject::new(reference)]); } + BrowseTabOutput::ReadClipboard(request) => self.read_paste(request), } } + /// Read the clipboard and turn what it holds into rows off the GTK + /// thread, then hand the tab what it came to. + fn read_paste(&self, request: PasteRequest) { + let Some(tasks) = self.read(|model| model.tasks.clone()) else { + return; + }; + let clipboard = self.clipboard(); + let tab = self.downgrade(); + glib::spawn_future_local(async move { + let outcome = match read_clipboard(&clipboard).await { + Ok(source) => { + let columns = request.columns.clone(); + let in_order = request.in_order.clone(); + let at_cell = request.at.is_some(); + match tasks + .spawn_blocking_task(move || read_pasted(&source, &columns, &in_order, at_cell)) + .await + { + Ok(outcome) => outcome, + Err(failure) => { + tracing::warn!(%failure, "pasted text could not be read"); + PasteOutcome::Failed + } + } + } + Err(outcome) => outcome, + }; + if let Some(tab) = tab.upgrade() { + tab.send(ClipboardInput::PasteRead(Box::new(PasteRead { request, outcome }))); + } + }); + } + /// Render a copy off the GTK thread, and toast how it went. fn copy(&self, request: CopyRequest) { let Some(tasks) = self.read(|model| model.tasks.clone()) else { @@ -572,6 +614,9 @@ pub(crate) struct BrowseModel { suppress_combo_emit: Cell, confirming_delete: Option, next_confirmation: u64, + /// The ticket of the latest paste, the only one that may land. + next_paste: u64, + confirming_paste: Option, widgets: BrowseTabWidgets, } @@ -597,6 +642,8 @@ pub(crate) enum BrowseTabOutput { RefreshRequested, /// The grid asked to open the rows a foreign key leads to. FollowReference(RowReference), + /// A paste the tab can take, for the tab to read the clipboard for. + ReadClipboard(PasteRequest), } impl BrowseModel { @@ -672,6 +719,44 @@ impl BrowseModel { } } +/// What the clipboard holds: the cells a grid copy offers, which keep +/// each value and NULL as they were, or else its text. Either is read +/// only up to the import's limit, since another program's clipboard can +/// hold anything. +async fn read_clipboard(clipboard: >k::gdk::Clipboard) -> Result { + use crate::ui::clipboard::CELLS_MIME_TYPE; + use crate::ui::file_read::{FileReadError, read_capped}; + + let cells = clipboard.formats().contain_mime_type(CELLS_MIME_TYPE); + let wanted: &[&str] = match cells { + true => &[CELLS_MIME_TYPE], + false => &["text/plain;charset=utf-8", "text/plain"], + }; + let stream = match clipboard.read_future(wanted, glib::Priority::DEFAULT).await { + Ok((stream, _)) => stream, + Err(error) => { + tracing::debug!(%error, "the clipboard holds nothing to paste"); + return Err(PasteOutcome::Nothing); + } + }; + let bytes = match read_capped(&stream, tablepro_core::import::MAX_CSV_FILE_BYTES).await { + Ok(bytes) => bytes, + Err(FileReadError::TooLarge) => return Err(PasteOutcome::TooLarge), + Err(error) => { + tracing::warn!(?error, "the clipboard could not be read"); + return Err(PasteOutcome::Failed); + } + }; + match cells { + true => Ok(PasteSource::Cells(bytes)), + false => String::from_utf8(bytes).map(PasteSource::Text).map_err(|error| { + PasteOutcome::Unreadable(tablepro_core::import::ImportError::NotUtf8 { + offset: error.utf8_error().valid_up_to(), + }) + }), + } +} + /// A status page reads its description as markup, and a table's name /// can hold an `&` or a `<`. fn loading_description(table: &str) -> glib::GString { @@ -688,18 +773,50 @@ fn qualified(schema: Option<&str>, table: &str) -> String { } } -/// The one place the store gains or loses draft rows. +/// The one place the store gains or loses draft rows. A paste can add +/// or an undo remove thousands at once, so each run of rows is one +/// change to the store rather than one per row. fn apply_draft_changes(store: >k::gio::ListStore, index: &mut GridRowIndex, delta: &TrackerDelta) { - for draft_id in &delta.removed_drafts { - let key = RowKey::Draft(*draft_id); - if let Some(position) = index.position_in(store, &key) { - store.remove(position); + if !delta.removed_drafts.is_empty() { + remove_drafts(store, &delta.removed_drafts); + for draft_id in &delta.removed_drafts { + index.remove(&RowKey::Draft(*draft_id)); } - index.remove(&key); } - for draft_id in &delta.inserted_drafts { - let row = RowObject::draft(*draft_id); - store.insert(0, &row); - index.insert(&row); + if !delta.inserted_drafts.is_empty() { + // The tracker lists drafts bottom up: the last one goes on top. + let rows: Vec = delta + .inserted_drafts + .iter() + .rev() + .map(|draft_id| RowObject::draft(*draft_id)) + .collect(); + for row in &rows { + index.insert(row); + } + store.splice(0, 0, &rows); + } +} + +fn remove_drafts(store: >k::gio::ListStore, removed: &[u64]) { + let removed: std::collections::HashSet = removed.iter().copied().collect(); + let is_removed = |position: u32| { + store + .item(position) + .and_downcast::() + .and_then(|row| row.draft_id()) + .is_some_and(|draft_id| removed.contains(&draft_id)) + }; + let mut position = store.n_items(); + while position > 0 { + position -= 1; + if !is_removed(position) { + continue; + } + let end = position + 1; + while position > 0 && is_removed(position - 1) { + position -= 1; + } + store.splice(position, end - position, &[] as &[RowObject]); } } diff --git a/linux/crates/app/src/ui/browse_tab/paste_at.rs b/linux/crates/app/src/ui/browse_tab/paste_at.rs new file mode 100644 index 0000000000..b44898906e --- /dev/null +++ b/linux/crates/app/src/ui/browse_tab/paste_at.rs @@ -0,0 +1,9 @@ +use crate::ui::grid::RowObject; + +/// The cell a paste starts at: one value goes into it, and rows fill +/// the table's columns from its column on. +#[derive(Debug, Clone)] +pub(crate) struct PasteAt { + pub(crate) row: RowObject, + pub(crate) column: usize, +} diff --git a/linux/crates/app/src/ui/browse_tab/paste_handlers.rs b/linux/crates/app/src/ui/browse_tab/paste_handlers.rs new file mode 100644 index 0000000000..d77374f301 --- /dev/null +++ b/linux/crates/app/src/ui/browse_tab/paste_handlers.rs @@ -0,0 +1,285 @@ +use gtk4 as gtk; +use gtk4::glib; +use gtk4::prelude::*; +use libadwaita as adw; +use libadwaita::prelude::*; +use tablepro_core::edit::parse_edit_text; +use tablepro_core::import::{ImportError, PastedRows}; + +use super::confirming_paste::ConfirmingPaste; +use super::paste_at::PasteAt; +use super::paste_read::{PasteOutcome, PasteRead, PasteRequest}; +use super::{BrowseModel, BrowseTabOutput, ClipboardInput, EditInput, Effects}; +use crate::i18n::{gettext, gettext_f, ngettext_f}; +use crate::services::change_tracker::pk_identity; +use crate::services::tracker_edit::TrackerEdit; +use crate::ui::format::{format_thousands, plural_count}; +use crate::ui::grid::{ + CellPreset, CommitResult, EditSnapshot, GridCell, cell_editability, commit_input, data_columns, edit_context, + select_top_rows, +}; + +/// A paste of more rows asks first: each becomes a pending row, and +/// Import CSV is the tool for a lot of data. +pub(crate) const PASTE_CONFIRM_ROWS: usize = 1_000; + +impl BrowseModel { + pub(super) fn paste(&mut self, at: Option, fx: &mut Effects) { + if self.read_only { + fx.output(BrowseTabOutput::ShowToast(gettext("This connection is read-only."))); + return; + } + if self.current_columns.is_empty() || self.grid_is_stale() { + return; + } + if !self.has_primary_key() { + fx.output(BrowseTabOutput::ShowToast(gettext( + "Rows can only be pasted into a table with a primary key.", + ))); + return; + } + self.next_paste += 1; + let in_order = self.columns_from(at.as_ref().map(|at| at.column)); + fx.output(BrowseTabOutput::ReadClipboard(PasteRequest { + ticket: self.next_paste, + at, + columns: self.current_columns.clone(), + in_order, + })); + } + + /// The columns as the grid shows them, left to right after any drag + /// of a header, from `start` on, each by its place in a row. + fn columns_from(&self, start: Option) -> Vec { + let Some(view) = self.current_column_view.as_ref() else { + return Vec::new(); + }; + let shown: Vec = data_columns(view) + .iter() + .filter(|column| column.is_visible()) + .filter_map(|column| self.view_columns.iter().position(|cell_column| cell_column == column)) + .collect(); + let first = start + .and_then(|start| shown.iter().position(|index| *index == start)) + .unwrap_or_default(); + shown.get(first..).unwrap_or_default().to_vec() + } + + pub(super) fn focused_cell(&self) -> Option { + let view = self.current_column_view.as_ref()?; + let window = view.root().and_downcast::()?; + let focus = gtk::prelude::GtkWindowExt::focus(&window)?; + if !focus.is_ancestor(view) { + return None; + } + let cell = GridCell::from_focus(&focus)?; + Some(PasteAt { + row: cell.row()?, + column: cell.column(), + }) + } + + pub(super) fn paste_read(&mut self, read: PasteRead, fx: &mut Effects) { + let PasteRead { request, outcome } = read; + if request.ticket != self.next_paste { + return; + } + if request.columns != self.current_columns || self.grid_is_stale() { + fx.output(BrowseTabOutput::ShowToast(gettext( + "The table changed while pasting. Paste again.", + ))); + return; + } + match outcome { + PasteOutcome::Nothing => { + fx.output(BrowseTabOutput::ShowToast(gettext( + "The clipboard has no rows to paste.", + ))); + } + PasteOutcome::Failed => fx.output(BrowseTabOutput::ShowToast(gettext("Couldn't read the clipboard."))), + PasteOutcome::TooLarge => fx.output(BrowseTabOutput::ShowToast(gettext( + "The clipboard holds too much to paste. Import CSV takes large amounts of data.", + ))), + PasteOutcome::Unreadable(error) => paste_alert(unreadable_text(&error), fx), + PasteOutcome::Refused(error) => paste_alert(crate::ui::error_text::row_error_message(&error), fx), + PasteOutcome::Value(text) => match request.at { + Some(at) => self.paste_value(at, text.as_deref(), fx), + None => tracing::error!("a single value came back for a paste with no cell"), + }, + PasteOutcome::Rows(rows) if rows.rows.len() > PASTE_CONFIRM_ROWS => self.confirm_paste(rows), + PasteOutcome::Rows(rows) => self.insert_pasted(rows, fx), + } + } + + /// One value goes into the cell, as if typed there, and a NULL as + /// Set to NULL puts it. + fn paste_value(&mut self, at: PasteAt, text: Option<&str>, fx: &mut Effects) { + let PasteAt { row, column } = at; + // A page read while the clipboard answered replaces the row's + // object, and the old one holds the values it had then. + let Some(row) = row.id().tracker_key().and_then(|key| self.row_index.get(&key)) else { + fx.output(BrowseTabOutput::ShowToast(gettext( + "The table changed while pasting. Paste again.", + ))); + return; + }; + let Some(info) = self.current_columns.get(column) else { + return; + }; + if let Err(lock) = cell_editability(info, row.id(), &pk_identity(&self.current_columns), self.read_only) { + fx.output(BrowseTabOutput::ShowToast(lock.explanation())); + return; + } + let Some(text) = text else { + fx.input(EditInput::GridSetCellValue { + row, + column, + preset: CellPreset::Null, + }); + return; + }; + let input = match parse_edit_text(text, info, edit_context(row.id())) { + Ok(input) => input, + Err(error) => { + return fx.output(BrowseTabOutput::ShowToast(crate::ui::error_text::edit_parse_message( + &error, + ))); + } + }; + // As a typed edit commits, so a value equal to the cell's, in + // the spelling the column keeps, changes nothing. + let snapshot = EditSnapshot { + original: self.grid_context().edit_input(&row, column), + row, + column, + }; + match commit_input(info, &snapshot, input) { + CommitResult::Edit(edit) => self.apply_edit(edit, fx), + CommitResult::Unchanged => {} + CommitResult::Rejected(message) => fx.output(BrowseTabOutput::ShowToast(message)), + } + } + + fn insert_pasted(&mut self, pasted: PastedRows, fx: &mut Effects) { + let PastedRows { rows, skipped } = pasted; + let count = rows.len(); + if count == 0 { + fx.output(BrowseTabOutput::ShowToast(gettext( + "The clipboard has no rows to paste.", + ))); + return; + } + self.apply_edit(TrackerEdit::InsertDrafts { rows }, fx); + fx.input(EditInput::SelectPastedDrafts(u32::try_from(count).unwrap_or(u32::MAX))); + let n = count as u64; + fx.output(BrowseTabOutput::ShowToast(ngettext_f( + "Pasted {n} row", + "Pasted {n} rows", + plural_count(n), + &[("n", &format_thousands(n))], + ))); + if skipped > 0 { + let n = skipped as u64; + fx.output(BrowseTabOutput::ShowToast(ngettext_f( + "Left out {n} value no column could take", + "Left out {n} values no column could take", + plural_count(n), + &[("n", &format_thousands(n))], + ))); + } + } + + /// Select the rows a paste added, which sit on top, once the grid + /// has drawn them. + pub(super) fn select_pasted_drafts(&self, count: u32) { + let Some(view) = self.current_column_view.clone() else { + return; + }; + glib::idle_add_local_once(move || { + if view.is_mapped() { + select_top_rows(&view, count); + } + }); + } + + fn confirm_paste(&mut self, rows: PastedRows) { + let count = rows.rows.len() as u64; + let title = ngettext_f( + "Paste {n} Row?", + "Paste {n} Rows?", + plural_count(count), + &[("n", &format_thousands(count))], + ); + let body = gettext( + "Each one is added as a new row that waits until you save. Import CSV, in the table's menu in the sidebar, writes large amounts of data straight to the table.", + ); + let dialog = adw::AlertDialog::new(Some(&title), Some(&body)); + dialog.add_response("cancel", &gettext("Cancel")); + dialog.add_response("paste", &gettext("Paste")); + dialog.set_response_appearance("paste", adw::ResponseAppearance::Suggested); + dialog.set_default_response(Some("paste")); + dialog.set_close_response("cancel"); + if let Some(pending) = self.confirming_paste.take() { + pending.dialog.force_close(); + } + let ticket = self.next_paste; + let tab = self.this.clone(); + dialog.connect_response(None, move |_, response| { + if let Some(tab) = tab.upgrade() { + tab.send(ClipboardInput::PasteAnswered { + ticket, + confirmed: response == "paste", + }); + } + }); + dialog.present(Some(&self.widgets.inner_stack)); + self.confirming_paste = Some(ConfirmingPaste { + ticket, + rows, + columns: self.current_columns.clone(), + dialog, + }); + } + + pub(super) fn paste_answered(&mut self, ticket: u64, confirmed: bool, fx: &mut Effects) { + if self + .confirming_paste + .as_ref() + .is_none_or(|pending| pending.ticket != ticket) + { + return; + } + let Some(pending) = self.confirming_paste.take() else { + return; + }; + if !confirmed { + return; + } + if pending.columns != self.current_columns { + fx.output(BrowseTabOutput::ShowToast(gettext( + "The table changed while pasting. Paste again.", + ))); + return; + } + self.insert_pasted(pending.rows, fx); + } +} + +fn paste_alert(body: String, fx: &mut Effects) { + fx.output(BrowseTabOutput::ShowSelectionAlert { + title: gettext("Couldn't Paste"), + body, + }); +} + +fn unreadable_text(error: &ImportError) -> String { + match error { + ImportError::TooManyFields { line, limit } => gettext_f( + "Line {line} of the pasted text has more than {limit} values, more than any table holds.", + &[("line", &line.to_string()), ("limit", &limit.to_string())], + ), + ImportError::NotUtf8 { .. } | ImportError::Malformed { .. } | ImportError::NoColumns => { + gettext("The pasted text couldn't be read as rows.") + } + } +} diff --git a/linux/crates/app/src/ui/browse_tab/paste_read.rs b/linux/crates/app/src/ui/browse_tab/paste_read.rs new file mode 100644 index 0000000000..c80186fbde --- /dev/null +++ b/linux/crates/app/src/ui/browse_tab/paste_read.rs @@ -0,0 +1,77 @@ +use tablepro_core::ColumnInfo; +use tablepro_core::import::{CsvRowError, ImportError, PastedRows, PastedTable}; + +use super::paste_at::PasteAt; + +#[derive(Debug)] +pub(crate) struct PasteRequest { + /// Only the latest paste lands: the clipboard of another program can + /// take any time to answer, or never. + pub(crate) ticket: u64, + pub(crate) at: Option, + pub(crate) columns: Vec, + /// The columns values fill one after another, by their place in a + /// row: as the grid shows them, from the one the paste starts at. + pub(crate) in_order: Vec, +} + +#[derive(Debug)] +pub(crate) enum PasteOutcome { + /// No text, or text with no rows in it. + Nothing, + /// One value alone, for the cell the paste started at. `None` is a + /// NULL. + Value(Option), + Rows(PastedRows), + Unreadable(ImportError), + Refused(CsvRowError), + /// Reading the text failed in a way the user can only retry. + Failed, + /// More than a paste reads at once. + TooLarge, +} + +/// What the clipboard held: the cells a grid copy offers beside its +/// text, or text alone. +#[derive(Debug)] +pub(crate) enum PasteSource { + Cells(Vec), + Text(String), +} + +#[derive(Debug)] +pub(crate) struct PasteRead { + pub(crate) request: PasteRequest, + pub(crate) outcome: PasteOutcome, +} + +/// Read what the clipboard held against `columns`. With a cell to start +/// at, one value alone is that cell's. +pub(super) fn read_pasted( + source: &PasteSource, + columns: &[ColumnInfo], + in_order: &[usize], + at_cell: bool, +) -> PasteOutcome { + let read = match source { + PasteSource::Cells(json) => PastedTable::from_cells(json), + PasteSource::Text(text) => PastedTable::read(text), + }; + let table = match read { + Ok(table) => table, + Err(error) => return PasteOutcome::Unreadable(error), + }; + if table.is_empty() { + return PasteOutcome::Nothing; + } + if table.truncated() { + return PasteOutcome::TooLarge; + } + if at_cell && let Some(value) = table.single_value() { + return PasteOutcome::Value(value.map(str::to_owned)); + } + match table.rows(columns, in_order) { + Ok(rows) => PasteOutcome::Rows(rows), + Err(error) => PasteOutcome::Refused(error), + } +} diff --git a/linux/crates/app/src/ui/browse_tab/tests/mod.rs b/linux/crates/app/src/ui/browse_tab/tests/mod.rs index c8280d60d6..2b3dd76d1b 100644 --- a/linux/crates/app/src/ui/browse_tab/tests/mod.rs +++ b/linux/crates/app/src/ui/browse_tab/tests/mod.rs @@ -1,4 +1,5 @@ mod load; +mod paste; mod references; mod total; diff --git a/linux/crates/app/src/ui/browse_tab/tests/paste.rs b/linux/crates/app/src/ui/browse_tab/tests/paste.rs new file mode 100644 index 0000000000..d41380ea6c --- /dev/null +++ b/linux/crates/app/src/ui/browse_tab/tests/paste.rs @@ -0,0 +1,449 @@ +use std::cell::RefCell; +use std::rc::Rc; + +use gtk4::glib; +use gtk4::prelude::*; +use gtk4::subclass::prelude::ObjectSubclassIsExt; +use tablepro_core::edit::CellInput; +use tablepro_core::meta::RowIdentity; +use tablepro_core::{ColumnInfo, QueryResult, Value}; +use tablepro_test_support::{CallCounters, ConnectionScript, FakeReply}; + +use super::{ + Harness, Shown, WAIT, column, column_view, copied_by, draft_rows, grid_cell_of, key, menu_gesture, named_col, rows, + serving_table, toasts, +}; +use crate::test_support::{drain_main_context, wait_until}; +use crate::ui::browse_tab::paste_at::PasteAt; +use crate::ui::browse_tab::paste_handlers::PASTE_CONFIRM_ROWS; +use crate::ui::browse_tab::{BrowseTab, ClipboardInput, EditInput, PagingInput}; +use crate::ui::clipboard::{CopyKind, CopyRequest}; +use crate::ui::grid::GridCell; +use crate::ui::grid::RowSnapshot; + +/// An `id` key, a name and an age, with Ada and Grace saved. +fn people() -> Vec { + let mut id = named_col("id", "integer", false); + id.primary_key = true; + vec![id, named_col("name", "text", true), named_col("age", "integer", true)] +} + +fn people_table() -> Harness { + people_holding(vec![ + vec![Value::Int(0), Value::Text("Ada".into()), Value::Int(36)], + vec![Value::Int(1), Value::Text("Grace".into()), Value::Int(85)], + ]) +} + +fn people_holding(saved: Vec>) -> Harness { + let columns = people(); + let mut script = ConnectionScript::default(); + script.fetch_columns.always(FakeReply::ok(columns.clone())); + script.row_identity.always(FakeReply::ok(RowIdentity::UniqueKey { + columns: vec!["id".to_owned()], + })); + script.count_rows.always(FakeReply::ok(2)); + script.probe.always(FakeReply::ok(())); + script.fetch_outbound_foreign_keys.always(FakeReply::ok(Vec::new())); + script.fetch_inbound_foreign_keys.always(FakeReply::ok(Vec::new())); + let count = saved.len(); + script.query_bound.always(FakeReply::ok(QueryResult::new( + columns.iter().map(ColumnInfo::result_column).collect::>(), + saved, + ))); + let harness = Harness::open(script, []); + wait_until(WAIT, || harness.saved_rows().len() == count).expect("the first page"); + harness +} + +/// Put `text` on the tab's clipboard and paste it at `at`. The read +/// comes back later, so each test waits for what it should do. +fn paste(harness: &Harness, text: &str, at: Option) { + harness.tab().clipboard().set_text(text); + harness.tab().send(ClipboardInput::Paste(at)); +} + +fn alerts(tab: &BrowseTab) -> Rc>> { + let alerts = Rc::new(RefCell::new(Vec::new())); + let sink = alerts.clone(); + tab.connect_alert(move |_, title, body| sink.borrow_mut().push(format!("{title}: {body}"))); + alerts +} + +fn confirming_paste(tab: &BrowseTab) -> Option { + tab.imp() + .model + .borrow() + .as_ref() + .and_then(|model| model.confirming_paste.as_ref().map(|pending| pending.dialog.clone())) +} + +fn draft_cells(harness: &Harness, position: u32) -> Vec { + let draft_id = harness.grid_row(position).draft_id().expect("a new row"); + harness.tracker.draft_inputs(draft_id).expect("the draft's cells") +} + +fn int(value: i64) -> CellInput { + CellInput::Value(Value::Int(value)) +} + +fn name(value: &str) -> CellInput { + CellInput::Value(Value::Text(value.to_owned())) +} + +fn null() -> CellInput { + CellInput::Value(Value::Null) +} + +#[gtk4::test] +fn pasted_rows_are_new_rows_in_the_order_they_were_copied() { + let harness = people_table(); + let toasts = toasts(harness.tab()); + + paste(&harness, "7\tLinus\t54\n8\tKen\t\n", None); + + wait_until(WAIT, || draft_rows(&harness.store()) == 2).expect("the rows were pasted"); + assert_eq!(draft_cells(&harness, 0), vec![int(7), name("Linus"), int(54)]); + assert_eq!(draft_cells(&harness, 1), vec![int(8), name("Ken"), null()]); + assert_eq!( + harness.grid_row(2).key(), + Some(&key(0)), + "the saved rows moved above the new ones" + ); + assert_eq!(toasts.borrow().as_slice(), ["Pasted 2 rows"]); +} + +/// The copy offers the cells beside its text, which says the same of a +/// NULL and the text `NULL`, and quotes what it then cannot unquote. +#[gtk4::test] +fn rows_copied_from_the_grid_paste_back_exactly() { + let harness = people_holding(vec![ + vec![Value::Int(0), Value::Text("NULL".into()), Value::Null], + vec![Value::Int(1), Value::Text("\"two\nlines\"".into()), Value::Int(3)], + ]); + harness.selection().select_all(); + copied_by(harness.tab(), || { + harness.tab().send(ClipboardInput::CopySelectedRowsAsTsv) + }); + + harness.tab().send(ClipboardInput::Paste(None)); + + wait_until(WAIT, || draft_rows(&harness.store()) == 2).expect("the rows were pasted"); + assert_eq!(draft_cells(&harness, 0), vec![int(0), name("NULL"), null()]); + assert_eq!(draft_cells(&harness, 1), vec![int(1), name("\"two\nlines\""), int(3)]); +} + +#[gtk4::test] +fn a_copied_cell_pastes_into_another_cell_as_it_was() { + let harness = people_holding(vec![ + vec![Value::Int(0), Value::Text("\"two\nlines\"".into()), Value::Null], + vec![Value::Int(1), Value::Text("Grace".into()), Value::Int(85)], + ]); + let request = CopyRequest::Rows { + columns: people().iter().map(ColumnInfo::result_column).collect(), + rows: vec![RowSnapshot::new( + harness.grid_row(0).cells().expect("the row's cells"), + Vec::new(), + )], + kind: CopyKind::Value(1), + }; + copied_by(harness.tab(), || harness.tab().send(ClipboardInput::GridCopy(request))); + + harness.tab().send(ClipboardInput::Paste(Some(PasteAt { + row: harness.grid_row(1), + column: 1, + }))); + + wait_until(WAIT, || harness.tracker.pending_count() == 1).expect("the cell was edited"); + assert_eq!( + harness.tracker.pending_values(&key(1), 3), + vec![(1, Value::Text("\"two\nlines\"".into()))] + ); +} + +#[gtk4::test] +fn one_undo_takes_back_the_whole_paste() { + let harness = people_table(); + paste(&harness, "7\tLinus\t54\n8\tKen\t1\n9\tDennis\t70\n", None); + wait_until(WAIT, || draft_rows(&harness.store()) == 3).expect("the rows were pasted"); + + harness.tab().send(EditInput::Undo); + + assert_eq!(draft_rows(&harness.store()), 0); + assert_eq!(harness.tracker.pending_count(), 0); + harness.tab().send(EditInput::Redo); + assert_eq!(draft_rows(&harness.store()), 3); + assert_eq!( + draft_cells(&harness, 0)[1], + name("Linus"), + "the redone rows came back in another order" + ); +} + +#[gtk4::test] +fn rows_fill_the_columns_from_the_one_the_paste_starts_at() { + let harness = people_table(); + let toasts = toasts(harness.tab()); + let at = PasteAt { + row: harness.grid_row(0), + column: 1, + }; + + paste(&harness, "Linus\t54\textra\n", Some(at)); + + wait_until(WAIT, || draft_rows(&harness.store()) == 1).expect("the row was pasted"); + assert_eq!(draft_cells(&harness, 0), vec![null(), name("Linus"), int(54)]); + assert_eq!( + toasts.borrow().as_slice(), + ["Pasted 1 row", "Left out 1 value no column could take"] + ); +} + +#[gtk4::test] +fn values_fill_the_columns_in_the_order_the_grid_shows_them() { + let harness = people_table(); + let view = column_view(harness.tab()); + let shown = crate::ui::grid::data_columns(&view); + let (name_column, age_column) = (shown[1].clone(), shown[2].clone()); + let columns = view.columns(); + let name_at = (0..columns.n_items()) + .find(|position| columns.item(*position).as_ref() == Some(name_column.upcast_ref::())) + .expect("the name column"); + view.insert_column(name_at, &age_column); + + paste( + &harness, + "7\t54\tLinus\n", + Some(PasteAt { + row: harness.grid_row(0), + column: 0, + }), + ); + + wait_until(WAIT, || draft_rows(&harness.store()) == 1).expect("the row was pasted"); + assert_eq!(draft_cells(&harness, 0), vec![int(7), name("Linus"), int(54)]); +} + +#[gtk4::test] +fn a_header_line_maps_the_values_by_name() { + let harness = people_table(); + + paste(&harness, "age\tname\tid\n54\tLinus\t7\n", None); + + wait_until(WAIT, || draft_rows(&harness.store()) == 1).expect("the row was pasted"); + assert_eq!(draft_cells(&harness, 0), vec![int(7), name("Linus"), int(54)]); +} + +#[gtk4::test] +fn one_value_goes_into_the_cell_the_paste_starts_at() { + let harness = people_table(); + let ada = harness.grid_row(0); + + paste( + &harness, + "Lovelace", + Some(PasteAt { + row: ada.clone(), + column: 1, + }), + ); + + wait_until(WAIT, || harness.tracker.pending_count() == 1).expect("the cell was edited"); + assert_eq!(draft_rows(&harness.store()), 0, "one value made a new row"); + assert_eq!( + harness.tracker.pending_values(&key(0), 3), + vec![(1, Value::Text("Lovelace".into()))] + ); +} + +#[gtk4::test] +fn one_value_a_cell_cannot_take_says_why() { + let harness = people_table(); + let toasts = toasts(harness.tab()); + let ada = harness.grid_row(0); + + paste( + &harness, + "5", + Some(PasteAt { + row: ada.clone(), + column: 0, + }), + ); + wait_until(WAIT, || toasts.borrow().len() == 1).expect("the key said why"); + paste(&harness, "old", Some(PasteAt { row: ada, column: 2 })); + wait_until(WAIT, || toasts.borrow().len() == 2).expect("the number said why"); + + assert_eq!(harness.tracker.pending_count(), 0); + assert_eq!( + toasts.borrow().as_slice(), + ["A saved row's key can't be changed here", "Enter a whole number"] + ); +} + +#[gtk4::test] +fn ctrl_v_pastes_at_the_focused_cell() { + let harness = people_table(); + let shown = Shown::new(harness.tab()); + let view = column_view(harness.tab()); + let age = crate::ui::grid::data_columns(&view) + .into_iter() + .nth(2) + .expect("the age column"); + view.scroll_to(1, Some(&age), gtk4::ListScrollFlags::FOCUS, None); + view.grab_focus(); + wait_until(WAIT, || { + gtk4::prelude::GtkWindowExt::focus(&shown.0) + .and_then(|focus| GridCell::from_focus(&focus)) + .is_some_and(|cell| cell.position() == Some(1) && cell.column() == 2) + }) + .expect("Grace's age takes the focus"); + harness.tab().clipboard().set_text("90"); + + harness + .tab() + .activate_action("browse.paste-rows", None) + .expect("the tab's Paste"); + + wait_until(WAIT, || harness.tracker.pending_count() == 1).expect("the focused cell was edited"); + assert_eq!(harness.tracker.pending_values(&key(1), 3), vec![(2, Value::Int(90))]); +} + +#[gtk4::test] +fn the_cell_menu_pastes_rows_from_its_column() { + let harness = people_table(); + let _shown = Shown::new(harness.tab()); + let cell = grid_cell_of(harness.tab(), 1, 1); + harness.tab().clipboard().set_text("Linus\t54\nKen\t81\n"); + + menu_gesture(&cell).emit_by_name::<()>("pressed", &[&1i32, &1.0f64, &1.0f64]); + column_view(harness.tab()) + .activate_action("cell.paste", None) + .expect("the menu's Paste"); + + wait_until(WAIT, || draft_rows(&harness.store()) == 2).expect("the rows were pasted"); + assert_eq!(draft_cells(&harness, 1), vec![null(), name("Ken"), int(81)]); + wait_until(WAIT, || { + crate::ui::grid::selected_positions(&harness.selection()) == vec![0, 1] + }) + .expect("the pasted rows are the selection"); +} + +#[gtk4::test] +fn a_value_the_column_refuses_stops_the_paste_and_names_it() { + let harness = people_table(); + let alerts = alerts(harness.tab()); + + paste(&harness, "7\tLinus\t54\n8\tKen\tforty\n", None); + + wait_until(WAIT, || !alerts.borrow().is_empty()).expect("the paste said why it stopped"); + assert_eq!(draft_rows(&harness.store()), 0, "part of the paste landed"); + assert_eq!( + alerts.borrow().as_slice(), + ["Couldn't Paste: Line 2, column age: Enter a whole number (the value was “forty”)"] + ); +} + +#[gtk4::test] +fn an_empty_clipboard_says_there_is_nothing_to_paste() { + let harness = people_table(); + let toasts = toasts(harness.tab()); + + paste(&harness, "\n\n", None); + + wait_until(WAIT, || !toasts.borrow().is_empty()).expect("the paste said why it stopped"); + assert_eq!(toasts.borrow().as_slice(), ["The clipboard has no rows to paste."]); +} + +#[gtk4::test] +fn a_read_only_connection_refuses_to_paste() { + let mut script = serving_table(); + script.query_bound.always(FakeReply::ok(rows(&["Ada"]))); + let harness = Harness::open_read_only(script); + wait_until(WAIT, || harness.shown().len() == 1).expect("the first page"); + let toasts = toasts(harness.tab()); + + paste(&harness, "7\tLinus\n", None); + + assert_eq!(harness.tracker.pending_count(), 0); + assert_eq!(toasts.borrow().as_slice(), ["This connection is read-only."]); +} + +/// A view has no key either: rows pasted there could not be named once +/// saved, as the toolbar's Insert says. +#[gtk4::test] +fn a_table_without_a_key_refuses_to_paste() { + let columns = [column("id", false), column("name", false)]; + let mut script = serving_table(); + script.fetch_columns.always(FakeReply::ok(columns.to_vec())); + script.query_bound.always(FakeReply::ok(QueryResult::new( + columns.iter().map(ColumnInfo::result_column).collect::>(), + vec![vec![Value::Int(0), Value::Text("Ada".into())]], + ))); + let harness = Harness::open(script, []); + wait_until(WAIT, || harness.shown().len() == 1).expect("the first page"); + let toasts = toasts(harness.tab()); + + paste(&harness, "7\tLinus\n", None); + + assert_eq!(harness.tracker.pending_count(), 0); + assert_eq!( + toasts.borrow().as_slice(), + ["Rows can only be pasted into a table with a primary key."] + ); +} + +#[gtk4::test] +fn a_large_paste_asks_first() { + let harness = people_table(); + let text: String = (0..=PASTE_CONFIRM_ROWS) + .map(|n| format!("{}\tuser {n}\t\n", n + 10)) + .collect(); + + paste(&harness, &text, None); + + wait_until(WAIT, || confirming_paste(harness.tab()).is_some()).expect("a paste this large asks first"); + let dialog = confirming_paste(harness.tab()).expect("the question"); + assert_eq!(harness.tracker.pending_count(), 0, "the rows landed before the answer"); + dialog.emit_by_name::<()>("response", &[&"paste"]); + drain_main_context(); + + assert_eq!(draft_rows(&harness.store()), PASTE_CONFIRM_ROWS + 1); + assert_eq!(draft_cells(&harness, 0)[1], name("user 0")); +} + +#[gtk4::test] +fn more_lines_than_a_paste_takes_point_to_import_csv() { + let harness = people_table(); + let toasts = toasts(harness.tab()); + let text = "x\n".repeat(tablepro_core::import::MAX_PASTED_ROWS + 1); + + paste(&harness, &text, None); + + wait_until(WAIT, || !toasts.borrow().is_empty()).expect("the paste said why it stopped"); + assert_eq!( + toasts.borrow().as_slice(), + ["The clipboard holds too much to paste. Import CSV takes large amounts of data."] + ); + assert_eq!(harness.tracker.pending_count(), 0); +} + +#[gtk4::test] +fn duplicated_rows_keep_their_order_on_screen_and_after_a_refresh() { + let harness = people_table(); + let (ada, grace) = (harness.grid_row(0), harness.grid_row(1)); + + harness.tab().send(EditInput::DuplicateRows(vec![ada, grace])); + + let names = |harness: &Harness| (draft_cells(harness, 0)[1].clone(), draft_cells(harness, 1)[1].clone()); + assert_eq!(names(&harness), (name("Ada"), name("Grace"))); + let reads = || CallCounters::get(&harness.connection.counters().query_bound); + let before = reads(); + harness.tab().send(PagingInput::Refresh); + wait_until(WAIT, || reads() > before && harness.saved_rows().len() == 2).expect("the page read again"); + assert_eq!( + names(&harness), + (name("Ada"), name("Grace")), + "a refresh turned them around" + ); +} diff --git a/linux/crates/app/src/ui/clipboard/clipboard_format.rs b/linux/crates/app/src/ui/clipboard/clipboard_format.rs index 973d4e6b36..c71aa2c44b 100644 --- a/linux/crates/app/src/ui/clipboard/clipboard_format.rs +++ b/linux/crates/app/src/ui/clipboard/clipboard_format.rs @@ -1,3 +1,7 @@ +/// The type a grid copy offers its cells as, beside the text: what +/// `CellsEncoder` writes, which a paste prefers to the text. +pub(crate) const CELLS_MIME_TYPE: &str = "application/x-tablepro-cells+json"; + /// What a copy holds, so an application that reads that type can take /// it as such: a spreadsheet pastes tab-separated values into cells. #[derive(Debug, Clone, Copy, PartialEq, Eq)] diff --git a/linux/crates/app/src/ui/clipboard/clipboard_payload.rs b/linux/crates/app/src/ui/clipboard/clipboard_payload.rs index f747bc9401..887b9fc382 100644 --- a/linux/crates/app/src/ui/clipboard/clipboard_payload.rs +++ b/linux/crates/app/src/ui/clipboard/clipboard_payload.rs @@ -9,6 +9,9 @@ pub(crate) struct ClipboardPayload { pub rows: usize, /// Values an IN list had to leave out. pub skipped: usize, + /// The copied cells as `CellsEncoder` writes them, offered beside the + /// text so a paste reads them back exactly. + pub cells: Option, } impl ClipboardPayload { @@ -18,6 +21,7 @@ impl ClipboardPayload { format: ClipboardFormat::PlainText, rows: 1, skipped: 0, + cells: None, } } } diff --git a/linux/crates/app/src/ui/clipboard/clipboard_writer.rs b/linux/crates/app/src/ui/clipboard/clipboard_writer.rs index 237d9a41cc..31f2756040 100644 --- a/linux/crates/app/src/ui/clipboard/clipboard_writer.rs +++ b/linux/crates/app/src/ui/clipboard/clipboard_writer.rs @@ -2,7 +2,7 @@ use gtk4::prelude::*; use gtk4::{gdk, glib}; use thiserror::Error; -use super::ClipboardPayload; +use super::{CELLS_MIME_TYPE, ClipboardPayload}; #[derive(Debug, Error)] pub(crate) enum ClipboardError { @@ -27,20 +27,32 @@ pub(super) fn holds_nul(text: &str) -> bool { text.contains('\0') } -/// Put `payload` on the clipboard as its own type and as plain text, so -/// an application that knows the type takes it, and any other gets the -/// text. The caller has already refused text that `holds_nul`. +/// Put `payload` on the clipboard as its own type, as the cells it +/// holds and as plain text, so an application that knows a type takes +/// it, and any other gets the text. The caller has already refused text +/// that `holds_nul`. pub(super) fn write(clipboard: &gdk::Clipboard, payload: ClipboardPayload) -> Result<(), ClipboardError> { - let Some(mime_type) = payload.format.mime_type() else { + let mime_type = payload.format.mime_type(); + if mime_type.is_none() && payload.cells.is_none() { clipboard.set_text(&payload.text); return Ok(()); - }; + } + let mut providers = Vec::with_capacity(3); + if let Some(cells) = payload.cells { + providers.push(gdk::ContentProvider::for_bytes( + CELLS_MIME_TYPE, + &glib::Bytes::from_owned(cells.into_bytes()), + )); + } let text = payload.text.to_value(); - let bytes = glib::Bytes::from_owned(payload.text.into_bytes()); - let provider = gdk::ContentProvider::new_union(&[ - gdk::ContentProvider::for_bytes(mime_type, &bytes), - gdk::ContentProvider::for_value(&text), - ]); + if let Some(mime_type) = mime_type { + providers.push(gdk::ContentProvider::for_bytes( + mime_type, + &glib::Bytes::from_owned(payload.text.into_bytes()), + )); + } + providers.push(gdk::ContentProvider::for_value(&text)); + let provider = gdk::ContentProvider::new_union(&providers); clipboard.set_content(Some(&provider)).map_err(|error| { tracing::warn!(%error, "the clipboard refused a copy"); ClipboardError::Refused(error) @@ -66,6 +78,7 @@ mod tests { format: ClipboardFormat::Csv, rows: 1, skipped: 0, + cells: None, }; write(&clipboard, payload).expect("written"); diff --git a/linux/crates/app/src/ui/clipboard/copy_renderer.rs b/linux/crates/app/src/ui/clipboard/copy_renderer.rs index 2468210b44..3d502f042f 100644 --- a/linux/crates/app/src/ui/clipboard/copy_renderer.rs +++ b/linux/crates/app/src/ui/clipboard/copy_renderer.rs @@ -4,8 +4,8 @@ use tablepro_core::column::{ColumnInfo, ColumnType, ResultColumn}; use tablepro_core::dialect::{LiteralError, SqlDialect}; use tablepro_core::edit::CellInput; use tablepro_core::export::{ - self, CsvEncoder, CsvOptions, EncodeError, JsonArrayEncoder, MarkdownEncoder, RenderInsertError, RowEncoder, - TsvEncoder, + self, CellsEncoder, CsvEncoder, CsvOptions, EncodeError, JsonArrayEncoder, MarkdownEncoder, RenderInsertError, + RowEncoder, TsvEncoder, }; use tablepro_core::meta::TableRef; use tablepro_core::value::Value; @@ -110,7 +110,8 @@ fn render_rows( rows: &[RowSnapshot], kind: CopyKind, ) -> Result { - match kind { + let cells = copied_cells(&columns, rows, kind)?; + let payload = match kind { CopyKind::Value(column) => value_text(&columns, rows, column).map(ClipboardPayload::plain), CopyKind::Rows => encode(TsvEncoder::new(columns, false), rows, ClipboardFormat::Tsv), CopyKind::RowsWithHeaders => encode(TsvEncoder::new(columns, true), rows, ClipboardFormat::Tsv), @@ -126,7 +127,31 @@ fn render_rows( ), CopyKind::Json => encode(JsonArrayEncoder::new(columns), rows, ClipboardFormat::Json), CopyKind::Markdown => encode(MarkdownEncoder::new(columns), rows, ClipboardFormat::Markdown), + }?; + Ok(ClipboardPayload { + cells: Some(cells), + ..payload + }) +} + +/// The cells beside the text, which a paste reads back as they were. +/// A copy that names its columns lets the paste place values by name. +fn copied_cells(columns: &Arc<[ResultColumn]>, rows: &[RowSnapshot], kind: CopyKind) -> Result { + let with_headers = match kind { + CopyKind::Value(_) | CopyKind::Rows | CopyKind::Csv => false, + CopyKind::RowsWithHeaders | CopyKind::CsvWithHeaders | CopyKind::Json | CopyKind::Markdown => true, + }; + let CopyKind::Value(column) = kind else { + return encoded(CellsEncoder::new(columns.clone(), with_headers), rows); + }; + let mut encoder = CellsEncoder::new(columns.get(column).cloned().into_iter().collect(), with_headers); + let mut out = Vec::new(); + encoder.begin(&mut out); + if let Some(row) = rows.first() { + encoder.row(&mut out, &mut std::iter::once(row.value(column)))?; } + encoder.finish(&mut out); + Ok(utf8(out)) } /// CSV as the clipboard carries it. Copied text goes as the user sees @@ -154,6 +179,7 @@ fn render_in_clause(dialect: &dyn SqlDialect, values: &InClauseRows) -> Result } fn encode( - mut encoder: impl RowEncoder, + encoder: impl RowEncoder, rows: &[RowSnapshot], format: ClipboardFormat, ) -> Result { + Ok(ClipboardPayload { + text: encoded(encoder, rows)?, + format, + rows: rows.len(), + skipped: 0, + cells: None, + }) +} + +fn encoded(mut encoder: impl RowEncoder, rows: &[RowSnapshot]) -> Result { let mut out = Vec::new(); encoder.begin(&mut out); for row in rows { encoder.row(&mut out, &mut row.values())?; } encoder.finish(&mut out); - Ok(ClipboardPayload { - // Every encoder writes UTF-8, so the lossy branch is never taken. - text: String::from_utf8(out).unwrap_or_else(|error| String::from_utf8_lossy(error.as_bytes()).into_owned()), - format, - rows: rows.len(), - skipped: 0, - }) + Ok(utf8(out)) +} + +/// Every encoder writes UTF-8, so the lossy branch is never taken. +fn utf8(out: Vec) -> String { + String::from_utf8(out).unwrap_or_else(|error| String::from_utf8_lossy(error.as_bytes()).into_owned()) } fn render_insert(insert: InsertCopy) -> Result { @@ -223,5 +258,6 @@ fn render_insert(insert: InsertCopy) -> Result { format: ClipboardFormat::Sql, rows: rows.len(), skipped: 0, + cells: None, }) } diff --git a/linux/crates/app/src/ui/clipboard/mod.rs b/linux/crates/app/src/ui/clipboard/mod.rs index cc8a6a31dd..26a11047d2 100644 --- a/linux/crates/app/src/ui/clipboard/mod.rs +++ b/linux/crates/app/src/ui/clipboard/mod.rs @@ -8,7 +8,7 @@ mod clipboard_writer; mod copy_renderer; pub(crate) use clipboard_copy::{copy_payload, copy_rows, copy_text}; -pub(crate) use clipboard_format::ClipboardFormat; +pub(crate) use clipboard_format::{CELLS_MIME_TYPE, ClipboardFormat}; pub(crate) use clipboard_payload::ClipboardPayload; pub(crate) use clipboard_writer::ClipboardError; #[cfg(test)] diff --git a/linux/crates/app/src/ui/error_text/edit.rs b/linux/crates/app/src/ui/error_text/edit.rs index 040e6de896..52ad45b43f 100644 --- a/linux/crates/app/src/ui/error_text/edit.rs +++ b/linux/crates/app/src/ui/error_text/edit.rs @@ -1,5 +1,6 @@ use tablepro_core::column::IntegerKind; use tablepro_core::edit::{EditParseError, ReadOnlyReason}; +use tablepro_core::import::CsvRowError; use tablepro_core::value::JsonTextError; use crate::i18n::{gettext, gettext_f, ngettext_f}; @@ -58,6 +59,28 @@ pub fn edit_parse_message(error: &EditParseError) -> String { } } +/// How much of a refused value an alert quotes: enough to find it in +/// the source, never a whole document laid out in a dialog. +const QUOTED_VALUE_CHARS: usize = 80; + +/// Where an imported or pasted row holds a value its column refused, +/// and why. +pub fn row_error_message(error: &CsvRowError) -> String { + let text = match error.text.char_indices().nth(QUOTED_VALUE_CHARS) { + Some((end, _)) => format!("{}…", &error.text[..end]), + None => error.text.clone(), + }; + gettext_f( + "Line {line}, column {column}: {reason} (the value was “{text}”)", + &[ + ("line", &error.line.to_string()), + ("column", &error.column), + ("reason", &edit_parse_message(&error.reason)), + ("text", &text), + ], + ) +} + fn not_editable_message(reason: ReadOnlyReason) -> String { match reason { ReadOnlyReason::Bytes => gettext("Binary values can't be edited here"), @@ -92,7 +115,9 @@ mod tests { use tablepro_core::edit::{EditParseError, ReadOnlyReason}; use tablepro_core::value::{DecimalParseError, JsonTextError}; - use super::{edit_parse_message, integer_range}; + use tablepro_core::import::CsvRowError; + + use super::{QUOTED_VALUE_CHARS, edit_parse_message, integer_range, row_error_message}; #[test] fn every_edit_parse_error_variant_has_a_distinct_message() { @@ -152,6 +177,29 @@ mod tests { assert_eq!(messages.len(), reasons.len()); } + #[test] + fn a_refused_value_is_quoted_short() { + let error = |text: String| CsvRowError { + line: 3, + column: "age".to_owned(), + text, + reason: EditParseError::InvalidInteger, + }; + + let short = row_error_message(&error("forty".to_owned())); + let long = row_error_message(&error("é".repeat(10_000))); + + assert_eq!( + short, + "Line 3, column age: Enter a whole number (the value was “forty”)" + ); + assert!( + long.contains(&format!("“{}…”", "é".repeat(QUOTED_VALUE_CHARS))), + "{long}" + ); + assert!(long.chars().count() < 200, "{long}"); + } + #[test] fn integer_ranges_follow_width_and_sign() { let range = |width, signed| integer_range(IntegerKind { width, signed }); diff --git a/linux/crates/app/src/ui/error_text/mod.rs b/linux/crates/app/src/ui/error_text/mod.rs index 7f43fa9b7f..9a91db9fca 100644 --- a/linux/crates/app/src/ui/error_text/mod.rs +++ b/linux/crates/app/src/ui/error_text/mod.rs @@ -25,7 +25,7 @@ mod unsupported; pub use ddl::{build_error_text, refusal_text, save_blocker_text}; pub use driver::{config_message, driver_message}; -pub use edit::edit_parse_message; +pub use edit::{edit_parse_message, row_error_message}; pub use files::storage_io_message; pub use operation::{connect_error_message, operation_error_message, task_failure_message}; pub use reference::reference_message; diff --git a/linux/crates/app/src/ui/file_read.rs b/linux/crates/app/src/ui/file_read.rs index fc007e6cab..0913e50234 100644 --- a/linux/crates/app/src/ui/file_read.rs +++ b/linux/crates/app/src/ui/file_read.rs @@ -45,11 +45,12 @@ pub(crate) async fn read_capped_file(file: &gio::File, limit: usize) -> Result Result, FileReadError> { +pub(crate) async fn read_capped(stream: &gio::InputStream, limit: usize) -> Result, FileReadError> { let mut contents = Vec::new(); loop { let chunk = stream diff --git a/linux/crates/app/src/ui/grid/cell_focus.rs b/linux/crates/app/src/ui/grid/cell_focus.rs index e385abcbb4..98b1523c4c 100644 --- a/linux/crates/app/src/ui/grid/cell_focus.rs +++ b/linux/crates/app/src/ui/grid/cell_focus.rs @@ -133,6 +133,21 @@ pub(crate) fn edit_row(view: >k::ColumnView, position: u32) { } } +/// Select the first `count` rows, where new rows land, and give the +/// first of them the focus. +pub(crate) fn select_top_rows(view: >k::ColumnView, count: u32) { + let Some(model) = view.model() else { + return; + }; + let count = count.min(model.n_items()); + if count == 0 { + return; + } + model.select_range(0, count, true); + view.scroll_to(0, None, gtk::ListScrollFlags::FOCUS, None); + view.grab_focus(); +} + #[cfg(test)] mod tests { use std::time::Duration; diff --git a/linux/crates/app/src/ui/grid/cell_lock.rs b/linux/crates/app/src/ui/grid/cell_lock.rs index 5ad3165190..5657569ece 100644 --- a/linux/crates/app/src/ui/grid/cell_lock.rs +++ b/linux/crates/app/src/ui/grid/cell_lock.rs @@ -13,3 +13,21 @@ pub enum CellLock { ResultRow, ReadOnlySession, } + +impl CellLock { + /// Why the cell takes no value, for an edit that cannot open an + /// editor to show it, such as a paste. + pub fn explanation(self) -> String { + match self { + Self::Value(reason) => { + crate::ui::error_text::edit_parse_message(&tablepro_core::edit::EditParseError::NotEditable(reason)) + } + Self::KeyColumn => crate::i18n::gettext("A saved row's key can't be changed here"), + Self::Generated => crate::i18n::gettext("The database computes this column's values"), + Self::AutoIncrement => crate::i18n::gettext("The database numbers this column's values"), + Self::UnkeyedRow => crate::i18n::gettext("Rows of a table without a primary key can't be edited"), + Self::ResultRow => crate::i18n::gettext("Query results can't be edited here"), + Self::ReadOnlySession => crate::i18n::gettext("This connection is read-only."), + } + } +} diff --git a/linux/crates/app/src/ui/grid/grid_menus.rs b/linux/crates/app/src/ui/grid/grid_menus.rs index 3c93c3214c..6dc4dc1d70 100644 --- a/linux/crates/app/src/ui/grid/grid_menus.rs +++ b/linux/crates/app/src/ui/grid/grid_menus.rs @@ -166,6 +166,9 @@ fn build_cell_menu(shape: MenuShape, references: &gio::Menu) -> gio::Menu { Some(&crate::i18n::gettext("Copy Column Name")), Some("cell.copy-column-name"), ); + if shape.row_ops { + copy_section.append(Some(&crate::i18n::gettext("Paste")), Some("cell.paste")); + } menu.append_section(None, ©_section); let json_section = gio::Menu::new(); @@ -534,6 +537,13 @@ impl CellActionKit { .activate(move |_, _, _| s(GridMsg::InsertRow)) .build() }; + // Below the rows there is no cell to paste at. + let paste_rows = { + let s = self.sink.clone(); + gio::ActionEntry::builder("paste-rows") + .activate(move |_, _, _| s(GridMsg::Paste { at: None })) + .build() + }; vec![ // Editing opens on the cell, which must still show the row. self.on_target("edit", |cell, row, _rows| { @@ -550,6 +560,10 @@ impl CellActionKit { insert_row, self.send("delete-row", |_cell, _row, rows| GridMsg::DeleteRows { rows }), self.send("duplicate-row", |_cell, _row, rows| GridMsg::DuplicateRows { rows }), + self.send("paste", |cell, row, _rows| GridMsg::Paste { + at: Some((row, cell.column())), + }), + paste_rows, ] } } @@ -585,6 +599,7 @@ fn eager_popover(column_view: >k::ColumnView, model: &gio::Menu) -> gtk::Popov fn install_empty_space_menu(column_view: >k::ColumnView) -> gtk::PopoverMenu { let empty_menu = gio::Menu::new(); empty_menu.append(Some(&crate::i18n::gettext("Insert Row")), Some("cell.insert-row")); + empty_menu.append(Some(&crate::i18n::gettext("Paste")), Some("cell.paste-rows")); let empty_popover = gtk::PopoverMenu::from_model_full(&empty_menu, gtk::PopoverMenuFlags::NESTED); empty_popover.set_has_arrow(true); empty_popover.set_parent(column_view); diff --git a/linux/crates/app/src/ui/grid/grid_msg.rs b/linux/crates/app/src/ui/grid/grid_msg.rs index 184327ee74..50e9343e6a 100644 --- a/linux/crates/app/src/ui/grid/grid_msg.rs +++ b/linux/crates/app/src/ui/grid/grid_msg.rs @@ -48,6 +48,11 @@ pub enum GridMsg { DuplicateRows { rows: Vec, }, + /// Paste at the cell `at` names, a row and a column, or as new rows + /// from the first column with no cell. + Paste { + at: Option<(RowObject, usize)>, + }, /// Open the rows a foreign key leads to from the row. FollowReference(RowReference), } diff --git a/linux/crates/app/src/ui/grid/grid_row_index.rs b/linux/crates/app/src/ui/grid/grid_row_index.rs index 54b6019a00..120421267b 100644 --- a/linux/crates/app/src/ui/grid/grid_row_index.rs +++ b/linux/crates/app/src/ui/grid/grid_row_index.rs @@ -41,7 +41,8 @@ impl GridRowIndex { row } - pub fn position_in(&mut self, store: &gio::ListStore, key: &RowKey) -> Option { + #[cfg(test)] + fn position_in(&mut self, store: &gio::ListStore, key: &RowKey) -> Option { store.find(&self.get(key)?) } diff --git a/linux/crates/app/src/ui/grid/mod.rs b/linux/crates/app/src/ui/grid/mod.rs index 8698d35c7c..7dbbccf8d8 100644 --- a/linux/crates/app/src/ui/grid/mod.rs +++ b/linux/crates/app/src/ui/grid/mod.rs @@ -53,13 +53,13 @@ mod value_order; #[cfg(test)] pub(crate) use self::cell_edit::begin_edit; pub(crate) use self::cell_editability::cell_editability; -pub(crate) use self::cell_focus::{commit_open_edit, edit_row}; +pub(crate) use self::cell_focus::{commit_open_edit, edit_row, select_top_rows}; pub(crate) use self::cell_lock::CellLock; pub(crate) use self::cell_preset::{CellPreset, cell_presets}; pub(crate) use self::column_factory::{GridSorting, build_column_view}; pub(crate) use self::column_width_binding::ColumnWidthBinding; pub(crate) use self::delete_availability::{DeleteAvailability, delete_availability}; -pub(crate) use self::edit_commit::edit_context; +pub(crate) use self::edit_commit::{CommitResult, commit_input, edit_context}; pub(crate) use self::edit_snapshot::EditSnapshot; pub(crate) use self::grid_cell::GridCell; pub(crate) use self::grid_content::GridContent; diff --git a/linux/crates/app/src/ui/grid/row_json_dialog.rs b/linux/crates/app/src/ui/grid/row_json_dialog.rs index 9f60d40cb7..8343847491 100644 --- a/linux/crates/app/src/ui/grid/row_json_dialog.rs +++ b/linux/crates/app/src/ui/grid/row_json_dialog.rs @@ -174,6 +174,7 @@ fn row_json_dialog(json: RowJson) -> adw::Dialog { format: ClipboardFormat::Json, rows: 1, skipped: 0, + cells: None, }; let toasts = adw::ToastOverlay::new(); toasts.set_child(Some(&scrolled)); diff --git a/linux/crates/app/src/ui/workspace/import.rs b/linux/crates/app/src/ui/workspace/import.rs index 55ef8778a2..d185304f33 100644 --- a/linux/crates/app/src/ui/workspace/import.rs +++ b/linux/crates/app/src/ui/workspace/import.rs @@ -499,17 +499,7 @@ fn import_rows(import: &CsvImport) -> Result>, String> { &import.options, first_line + index, ) - .map_err(|error| { - crate::i18n::gettext_f( - "Line {line}, column {column}: {reason} (the value was “{text}”)", - &[ - ("line", &error.line.to_string()), - ("column", &error.column), - ("reason", &crate::ui::error_text::edit_parse_message(&error.reason)), - ("text", &error.text), - ], - ) - }) + .map_err(|error| crate::ui::error_text::row_error_message(&error)) }) .collect() } diff --git a/linux/crates/core/src/edit/mod.rs b/linux/crates/core/src/edit/mod.rs index 25201079c5..715989f0f4 100644 --- a/linux/crates/core/src/edit/mod.rs +++ b/linux/crates/core/src/edit/mod.rs @@ -7,6 +7,7 @@ mod edit_parse_error; mod edit_text; mod empty_input_meaning; mod is_unchanged; +mod new_draft_cell; mod parse_edit_text; mod parse_literal_text; @@ -16,6 +17,7 @@ pub use edit_parse_error::{EditParseError, ReadOnlyReason}; pub use edit_text::{EditText, edit_text, read_only_reason}; pub use empty_input_meaning::{EmptyInputMeaning, empty_input_meaning}; pub use is_unchanged::is_unchanged; +pub use new_draft_cell::new_draft_cell; pub use parse_edit_text::parse_edit_text; pub use parse_literal_text::{ parse_filter_text, parse_literal_text, parse_time_with_offset, parse_timestamp_with_offset, diff --git a/linux/crates/core/src/edit/new_draft_cell.rs b/linux/crates/core/src/edit/new_draft_cell.rs new file mode 100644 index 0000000000..1b42944319 --- /dev/null +++ b/linux/crates/core/src/edit/new_draft_cell.rs @@ -0,0 +1,13 @@ +use crate::column::ColumnInfo; +use crate::value::Value; + +use super::CellInput; + +/// What a new row holds in `column` until the user types: the server's +/// value where it has one, else NULL. +pub fn new_draft_cell(column: &ColumnInfo) -> CellInput { + match column.is_auto_increment || column.is_generated || !column.default.is_none() { + true => CellInput::Default, + false => CellInput::Value(Value::Null), + } +} diff --git a/linux/crates/core/src/export/encoder/cells_encoder.rs b/linux/crates/core/src/export/encoder/cells_encoder.rs new file mode 100644 index 0000000000..e4d45dd905 --- /dev/null +++ b/linux/crates/core/src/export/encoder/cells_encoder.rs @@ -0,0 +1,75 @@ +use std::sync::Arc; + +use crate::column::ResultColumn; +use crate::value::Value; + +use super::super::column_type; +use super::super::value_text::value_text; +use super::{EncodeError, RowEncoder, readable_cells}; + +/// The cells a grid copy holds, as JSON a paste reads back exactly: +/// each value's full text, a NULL as `null`, and the names of the +/// columns they came from, with `headers` set when the copy showed them. +/// Text loses that: TSV writes a NULL and the text `NULL` alike, and +/// quotes a value it then cannot tell from one that was quoted. +pub struct CellsEncoder { + columns: Arc<[ResultColumn]>, + with_headers: bool, + rows: usize, +} + +impl CellsEncoder { + pub fn new(columns: Arc<[ResultColumn]>, with_headers: bool) -> Self { + Self { + columns, + with_headers, + rows: 0, + } + } +} + +impl RowEncoder for CellsEncoder { + fn begin(&mut self, out: &mut Vec) { + out.extend_from_slice(b"{\"columns\":"); + write_list(out, self.columns.iter().map(|column| Some(column.name.as_str()))); + out.extend_from_slice(b",\"headers\":"); + match self.with_headers { + true => write_list(out, self.columns.iter().map(|column| Some(column.name.as_str()))), + false => out.extend_from_slice(b"null"), + } + out.extend_from_slice(b",\"rows\":["); + } + + fn row(&mut self, out: &mut Vec, cells: &mut dyn Iterator) -> Result<(), EncodeError> { + let cells = readable_cells(&self.columns, cells)?; + if self.rows > 0 { + out.push(b','); + } + let texts: Vec> = cells + .iter() + .enumerate() + .map(|(index, value)| value_text(value, column_type(&self.columns, index))) + .collect(); + write_list(out, texts.iter().map(Option::as_deref)); + self.rows += 1; + Ok(()) + } + + fn finish(&mut self, out: &mut Vec) { + out.extend_from_slice(b"]}"); + } +} + +fn write_list<'a>(out: &mut Vec, items: impl Iterator>) { + out.push(b'['); + for (index, item) in items.enumerate() { + if index > 0 { + out.push(b','); + } + match item { + Some(text) => out.extend_from_slice(serde_json::Value::from(text).to_string().as_bytes()), + None => out.extend_from_slice(b"null"), + } + } + out.push(b']'); +} diff --git a/linux/crates/core/src/export/encoder/mod.rs b/linux/crates/core/src/export/encoder/mod.rs index 830910ea51..fc1e790ef6 100644 --- a/linux/crates/core/src/export/encoder/mod.rs +++ b/linux/crates/core/src/export/encoder/mod.rs @@ -1,3 +1,4 @@ +mod cells_encoder; mod csv_encoder; mod encode_error; mod html_encoder; @@ -14,6 +15,7 @@ use std::io; use crate::column::ResultColumn; use crate::value::Value; +pub use cells_encoder::CellsEncoder; pub use csv_encoder::CsvEncoder; pub use encode_error::EncodeError; pub use html_encoder::HtmlEncoder; diff --git a/linux/crates/core/src/export/encoder/tsv_encoder.rs b/linux/crates/core/src/export/encoder/tsv_encoder.rs index f4f2d66b69..245012465e 100644 --- a/linux/crates/core/src/export/encoder/tsv_encoder.rs +++ b/linux/crates/core/src/export/encoder/tsv_encoder.rs @@ -4,7 +4,7 @@ use crate::column::ResultColumn; use crate::value::Value; use super::super::column_type; -use super::super::tsv::tsv_writer_builder; +use super::super::tsv::{TSV_NULL, tsv_writer_builder}; use super::super::value_text::value_text; use super::{EncodeError, RecordWriter, RowEncoder, readable_cells}; @@ -42,7 +42,7 @@ impl RowEncoder for TsvEncoder { self.writer.write( out, cells.iter().enumerate().map(|(index, value)| { - value_text(value, column_type(&self.columns, index)).unwrap_or_else(|| "NULL".to_owned()) + value_text(value, column_type(&self.columns, index)).unwrap_or_else(|| TSV_NULL.to_owned()) }), ); Ok(()) diff --git a/linux/crates/core/src/export/mod.rs b/linux/crates/core/src/export/mod.rs index 97091039c2..e33c8c2b21 100644 --- a/linux/crates/core/src/export/mod.rs +++ b/linux/crates/core/src/export/mod.rs @@ -21,8 +21,8 @@ use crate::value::Value; pub use csv::{FORMULA_LEADS, csv_writer_builder, neutralise_formula, render_csv, render_text_csv}; pub use csv_options::{CsvDecimal, CsvDelimiter, CsvLineBreak, CsvOptions, CsvQuote}; pub use encoder::{ - CsvEncoder, EncodeError, HtmlEncoder, JsonArrayEncoder, MarkdownEncoder, RowEncoder, SqlInsertEncoder, TsvEncoder, - XmlEncoder, + CellsEncoder, CsvEncoder, EncodeError, HtmlEncoder, JsonArrayEncoder, MarkdownEncoder, RowEncoder, + SqlInsertEncoder, TsvEncoder, XmlEncoder, }; pub use html::render_html; pub use in_clause::{InClause, render_in_clause}; @@ -31,7 +31,7 @@ pub use json_field_names::json_field_names; pub use markdown::render_markdown; pub use render_insert_error::RenderInsertError; pub use sql::{literal, render_insert, render_sql_insert}; -pub use tsv::{render_tsv, tsv_writer_builder}; +pub use tsv::{TSV_NULL, render_tsv, tsv_writer_builder}; pub use value_text::{value_text, value_to_text}; pub use xlsx::{XLSX_MAX_DATA_ROWS, XlsxSheetWriter, render_xlsx}; pub use xml::render_xml; diff --git a/linux/crates/core/src/export/tsv.rs b/linux/crates/core/src/export/tsv.rs index ef92f68b7c..5a967c1189 100644 --- a/linux/crates/core/src/export/tsv.rs +++ b/linux/crates/core/src/export/tsv.rs @@ -3,6 +3,9 @@ use crate::value::Value; use super::encoder::{EncodeError, TsvEncoder, encode_rows}; +/// What a copy writes for a NULL, and what a paste reads back as one. +pub const TSV_NULL: &str = "NULL"; + /// A tab, a line break or a quote inside a value would move the following /// text into the next column or the next row, so such a value is quoted /// and its own quotes doubled. That is what a spreadsheet puts on the diff --git a/linux/crates/core/src/import/csv_import.rs b/linux/crates/core/src/import/csv_import.rs index 38113ec003..eabd92a3bb 100644 --- a/linux/crates/core/src/import/csv_import.rs +++ b/linux/crates/core/src/import/csv_import.rs @@ -150,14 +150,14 @@ pub fn read_csv(bytes: &[u8], options: &CsvImportOptions, limit: Option) }) } -fn malformed(error: csv::Error) -> ImportError { +pub(super) fn malformed(error: csv::Error) -> ImportError { ImportError::Malformed { line: error.position().map(csv::Position::line), } } /// Refuse a record wider than `MAX_FIELDS` before its fields are copied. -fn within_width(record: &csv::StringRecord) -> Result<(), ImportError> { +pub(super) fn within_width(record: &csv::StringRecord) -> Result<(), ImportError> { match record.len() > MAX_FIELDS { true => Err(ImportError::TooManyFields { line: record.position().map_or(1, csv::Position::line), diff --git a/linux/crates/core/src/import/mod.rs b/linux/crates/core/src/import/mod.rs index 80bca8b77f..80b9439d07 100644 --- a/linux/crates/core/src/import/mod.rs +++ b/linux/crates/core/src/import/mod.rs @@ -1,9 +1,11 @@ -//! Reading a file into a table, which is the other direction from -//! `export`. +//! Reading a file or pasted text into a table, which is the other +//! direction from `export`. mod csv_import; mod import_column; mod import_plan; +mod pasted_rows; +mod pasted_table; pub use csv_import::{ CsvImportOptions, CsvRowError, CsvSheet, ImportError, MAX_CSV_FILE_BYTES, MAX_FIELDS, MAX_PREVIEW_ROWS, read_csv, @@ -11,3 +13,5 @@ pub use csv_import::{ }; pub use import_column::ImportColumn; pub use import_plan::{ImportPlan, plan_import}; +pub use pasted_rows::PastedRows; +pub use pasted_table::{MAX_PASTED_ROWS, PastedTable}; diff --git a/linux/crates/core/src/import/pasted_rows.rs b/linux/crates/core/src/import/pasted_rows.rs new file mode 100644 index 0000000000..36d9dd83e4 --- /dev/null +++ b/linux/crates/core/src/import/pasted_rows.rs @@ -0,0 +1,10 @@ +use crate::edit::CellInput; + +#[derive(Debug, Clone, PartialEq, Default)] +pub struct PastedRows { + pub rows: Vec>, + /// Values the paste found no column to put in: past the last + /// column, under a header no column has, or in a column the server + /// fills or the grid cannot edit. + pub skipped: usize, +} diff --git a/linux/crates/core/src/import/pasted_table.rs b/linux/crates/core/src/import/pasted_table.rs new file mode 100644 index 0000000000..ebec29f6cb --- /dev/null +++ b/linux/crates/core/src/import/pasted_table.rs @@ -0,0 +1,829 @@ +use std::fmt; + +use serde::Deserialize; +use serde::de::{self, Deserializer, IgnoredAny, SeqAccess, Visitor}; + +use crate::column::ColumnInfo; +use crate::edit::{CellInput, EditContext, EditParseError, new_draft_cell, parse_edit_text, read_only_reason}; +use crate::export::TSV_NULL; +use crate::value::Value; + +use super::csv_import::{malformed, within_width}; +use super::{CsvRowError, ImportError, MAX_FIELDS, PastedRows}; + +/// The most lines a paste reads, a header among them. Each becomes a +/// pending row the grid holds, and Import CSV is the tool for more. +pub const MAX_PASTED_ROWS: usize = 100_000; + +/// Rows from the clipboard, before they meet a table: the cells a grid +/// copy wrote, or text read as tab-separated rows, which is what the +/// copy also writes and what a spreadsheet puts on the clipboard. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct PastedTable { + header: Header, + records: Vec, + /// More lines followed the last one kept than a paste reads. + truncated: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq, Default)] +enum Header { + /// Text says nothing, so its first line may name them. + #[default] + Unknown, + /// A grid copy that showed no names: every record is a row. It still + /// says which columns the cells came from. + Unnamed(Vec), + Named(Vec), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct PastedRecord { + /// Where the record starts, from 1, which an error names. + line: usize, + /// `None` is a NULL. + fields: Vec>, +} + +/// The column each field of a record goes to, by position or by the +/// names a header gave them. +enum Placement<'a> { + InOrder(&'a [usize]), + Named(Vec>), +} + +/// What `CellsEncoder` writes. Another program can put anything under +/// its type, so the rows are counted as they are read. +#[derive(Deserialize)] +struct CopiedCells { + #[serde(default)] + columns: Vec, + headers: Option>, + rows: CopiedRows, +} + +struct CopiedRows { + rows: Vec>>, + truncated: bool, +} + +impl<'de> Deserialize<'de> for CopiedRows { + fn deserialize>(deserializer: D) -> Result { + deserializer.deserialize_seq(RowsVisitor) + } +} + +struct RowsVisitor; + +impl<'de> Visitor<'de> for RowsVisitor { + type Value = CopiedRows; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("a list of rows") + } + + fn visit_seq>(self, mut seq: A) -> Result { + let mut rows = Vec::new(); + while rows.len() < MAX_PASTED_ROWS { + match seq.next_element::()? { + Some(CopiedRow(fields)) => rows.push(fields), + None => return Ok(CopiedRows { rows, truncated: false }), + } + } + let truncated = seq.next_element::()?.is_some(); + while seq.next_element::()?.is_some() {} + Ok(CopiedRows { rows, truncated }) + } +} + +/// One row's fields, refused past the widest a table can be before they +/// are all held. +struct CopiedRow(Vec>); + +impl<'de> Deserialize<'de> for CopiedRow { + fn deserialize>(deserializer: D) -> Result { + deserializer.deserialize_seq(RowVisitor) + } +} + +struct RowVisitor; + +impl<'de> Visitor<'de> for RowVisitor { + type Value = CopiedRow; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("a list of values") + } + + fn visit_seq>(self, mut seq: A) -> Result { + let mut fields = Vec::new(); + while let Some(field) = seq.next_element::>()? { + if fields.len() == MAX_FIELDS { + return Err(de::Error::custom(format!("a row has more than {MAX_FIELDS} values"))); + } + fields.push(field); + } + Ok(CopiedRow(fields)) + } +} + +impl PastedTable { + /// Text as tab-separated rows. A quoted field keeps its tabs, line + /// breaks and doubled quotes, as the copy wrote them, and `NULL`, + /// which is how the copy writes a NULL, is one. + pub fn read(text: &str) -> Result { + let mut reader = csv::ReaderBuilder::new() + .delimiter(b'\t') + .has_headers(false) + .flexible(true) + .from_reader(text.as_bytes()); + let mut records = Vec::new(); + let mut truncated = false; + for record in reader.records() { + let record = record.map_err(malformed)?; + if records.len() == MAX_PASTED_ROWS { + truncated = true; + break; + } + within_width(&record)?; + let line = record + .position() + .map_or(1, |position| usize::try_from(position.line()).unwrap_or(usize::MAX)); + let fields = record + .iter() + .map(|field| (field != TSV_NULL).then(|| field.to_owned())) + .collect(); + records.push(PastedRecord { line, fields }); + } + Ok(Self { + header: Header::Unknown, + records, + truncated, + }) + } + + /// The cells a grid copy wrote with `CellsEncoder`, each value and + /// NULL as it was. + pub fn from_cells(json: &[u8]) -> Result { + let cells: CopiedCells = serde_json::from_slice(json).map_err(|error| ImportError::Malformed { + line: u64::try_from(error.line()).ok(), + })?; + let first = 1 + usize::from(cells.headers.is_some()); + let records = cells + .rows + .rows + .into_iter() + .enumerate() + .map(|(index, fields)| PastedRecord { + line: first + index, + fields, + }) + .collect(); + Ok(Self { + header: match cells.headers { + Some(headers) => Header::Named(headers), + None => Header::Unnamed(cells.columns), + }, + records, + truncated: cells.rows.truncated, + }) + } + + pub fn is_empty(&self) -> bool { + self.records.is_empty() + } + + /// Whether the clipboard held more lines than a paste reads, which + /// were left unread. + pub fn truncated(&self) -> bool { + self.truncated + } + + /// The one value the clipboard holds, when it holds one and nothing + /// else: `Some(None)` is a NULL. + pub fn single_value(&self) -> Option> { + match self.records.as_slice() { + [record] => match record.fields.as_slice() { + [field] => Some(field.as_deref()), + _ => None, + }, + _ => None, + } + } + + /// The new rows the cells make in a table of `columns`. The fields + /// of a record go to the columns `in_order` lists, the first field to + /// the first, or by name under a header. Cells a grid copied go to + /// the columns they came from when the table has each of them, so a + /// copy between tables alike lands whatever order either grid shows. + /// + /// Each value is read as a cell edit of a new row reads it, and a + /// NULL is NULL. A column left out holds what a new row holds. A + /// column the server fills, and one whose values the grid cannot + /// edit, keep that too, and a value meant for one counts as skipped. + /// A key the server cannot fill takes the value: the row cannot be + /// saved without it. + pub fn rows(&self, columns: &[ColumnInfo], in_order: &[usize]) -> Result { + let (placement, records) = match &self.header { + Header::Named(names) => (Placement::Named(named_columns(columns, names)), self.records.as_slice()), + Header::Unnamed(names) => match every_column_named(columns, names) { + Some(named) => (Placement::Named(named), self.records.as_slice()), + None => (Placement::InOrder(in_order), self.records.as_slice()), + }, + Header::Unknown => match self.first_line_names(columns) { + Some(named) => (Placement::Named(named), self.records.get(1..).unwrap_or_default()), + None => (Placement::InOrder(in_order), self.records.as_slice()), + }, + }; + let mut pasted = PastedRows::default(); + for record in records { + let mut cells: Vec = columns.iter().map(new_draft_cell).collect(); + for (field, text) in record.fields.iter().enumerate() { + let text = text.as_deref(); + let target = match &placement { + Placement::InOrder(in_order) => in_order.get(field).copied(), + Placement::Named(named) => named.get(field).copied().flatten(), + }; + let (Some(index), Some(column)) = (target, target.and_then(|index| columns.get(index))) else { + pasted.skipped += usize::from(carries_value(text)); + continue; + }; + if left_to_the_row(column) { + pasted.skipped += usize::from(carries_value(text)); + continue; + } + cells[index] = pasted_cell(text, column).map_err(|reason| CsvRowError { + line: record.line, + column: column.name.clone(), + text: text.unwrap_or(TSV_NULL).to_owned(), + reason, + })?; + } + pasted.rows.push(cells); + } + Ok(pasted) + } + + /// The column each field of the first record names, when that + /// record is a header: a row follows it, and more than half its + /// fields name a column, no two the same. + fn first_line_names(&self, columns: &[ColumnInfo]) -> Option>> { + let [first, _, ..] = self.records.as_slice() else { + return None; + }; + let names: Vec<&str> = first + .fields + .iter() + .map(|field| field.as_deref().unwrap_or("")) + .collect(); + let named = named_columns(columns, &names); + let matched = named.iter().flatten().count(); + (matched * 2 > names.len()).then_some(named) + } +} + +/// The column each name names, when every one names a column of its own. +fn every_column_named(columns: &[ColumnInfo], names: &[String]) -> Option>> { + let named = named_columns(columns, names); + (!named.is_empty() && named.iter().all(Option::is_some)).then_some(named) +} + +/// The column each name names, each column taken once. +fn named_columns(columns: &[ColumnInfo], names: &[impl AsRef]) -> Vec> { + let mut taken = vec![false; columns.len()]; + names + .iter() + .map(|name| { + let index = column_named(columns, name.as_ref())?; + let seen = taken.get_mut(index)?; + (!std::mem::replace(seen, true)).then_some(index) + }) + .collect() +} + +/// The column a header field names: its exact name first, so a name +/// with spaces or capitals of its own finds its column, then the name in +/// any case with the field's surrounding space ignored, as the import's +/// mapping matches. +fn column_named(columns: &[ColumnInfo], field: &str) -> Option { + if let Some(exact) = columns.iter().position(|column| column.name == field) { + return Some(exact); + } + let wanted = field.trim().to_lowercase(); + if wanted.is_empty() { + return None; + } + columns + .iter() + .position(|column| column.name.trim().to_lowercase() == wanted) +} + +/// A column a pasted row leaves as a new row has it: the server fills +/// it, or the grid has no text form to edit its values in. +fn left_to_the_row(column: &ColumnInfo) -> bool { + column.is_generated + || column.is_auto_increment + || (column.primary_key && !column.default.is_none()) + || read_only_reason(&column.column_type.kind).is_some() +} + +/// Whether leaving the field out loses anything. +fn carries_value(text: Option<&str>) -> bool { + text.is_some_and(|text| !text.is_empty()) +} + +fn pasted_cell(text: Option<&str>, column: &ColumnInfo) -> Result { + if let Some(text) = text { + return parse_edit_text(text, column, EditContext::InsertDraft); + } + if column.nullable { + return Ok(CellInput::Value(Value::Null)); + } + // A NOT NULL column with a default is better served by it than by + // an insert the server refuses. + match column.default.is_none() { + true => Err(EditParseError::Required), + false => Ok(CellInput::Default), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::column::{ + ColumnDefault, ColumnKind, ColumnType, IntegerKind, IntegerWidth, ReadForm, SqlExpression, TextLength, + }; + use crate::export::{CellsEncoder, RowEncoder, render_tsv}; + + fn column(name: &str, kind: ColumnKind) -> ColumnInfo { + ColumnInfo { + name: name.to_owned(), + column_type: ColumnType { + name: "t".to_owned(), + kind, + catalog: None, + dynamic_storage: false, + read_form: ReadForm::Native, + }, + nullable: true, + primary_key: false, + is_auto_increment: false, + is_generated: false, + default: ColumnDefault::None, + comment: None, + } + } + + fn int(name: &str) -> ColumnInfo { + column( + name, + ColumnKind::Integer(IntegerKind { + width: IntegerWidth::W32, + signed: true, + }), + ) + } + + fn text(name: &str) -> ColumnInfo { + column( + name, + ColumnKind::Text { + length: TextLength::Unbounded, + }, + ) + } + + fn key(name: &str) -> ColumnInfo { + let mut key = int(name); + key.primary_key = true; + key.nullable = false; + key + } + + fn serial(name: &str) -> ColumnInfo { + let mut serial = key(name); + serial.is_auto_increment = true; + serial + } + + fn value(text: &str) -> CellInput { + CellInput::Value(Value::Text(text.to_owned())) + } + + fn number(number: i64) -> CellInput { + CellInput::Value(Value::Int(number)) + } + + fn null() -> CellInput { + CellInput::Value(Value::Null) + } + + /// The columns from `start` on, left to right. + fn from(start: usize, columns: &[ColumnInfo]) -> Vec { + (start..columns.len()).collect() + } + + fn pasted(text: &str, columns: &[ColumnInfo], start: usize) -> PastedRows { + PastedTable::read(text) + .expect("read") + .rows(columns, &from(start, columns)) + .expect("rows") + } + + #[test] + fn rows_the_copy_writes_paste_back_as_the_same_values() { + let columns = [text("name"), text("note"), int("age")]; + let result: Vec<_> = columns.iter().map(ColumnInfo::result_column).collect(); + let copied = render_tsv( + &result, + &[ + vec![ + Value::Text("tab\there".into()), + Value::Text("two\nlines".into()), + Value::Int(7), + ], + vec![ + Value::Text("say \"hi\"".into()), + Value::Text(String::new()), + Value::Null, + ], + ], + false, + ) + .expect("the copy"); + + let rows = pasted(&copied, &columns, 0); + + assert_eq!( + rows.rows, + vec![ + vec![value("tab\there"), value("two\nlines"), number(7)], + vec![value("say \"hi\""), value(""), null()], + ] + ); + assert_eq!(rows.skipped, 0); + } + + #[test] + fn values_fill_the_columns_from_the_start_column_on() { + let columns = [key("id"), text("name"), text("city")]; + + let rows = pasted("Ada\tLondon\n", &columns, 1); + + assert_eq!(rows.rows, vec![vec![null(), value("Ada"), value("London")]]); + } + + #[test] + fn values_follow_the_columns_in_the_order_given() { + let columns = [key("id"), text("name"), text("city")]; + + let rows = PastedTable::read("London\tAda\n") + .expect("read") + .rows(&columns, &[2, 1]) + .expect("rows"); + + assert_eq!(rows.rows, vec![vec![null(), value("Ada"), value("London")]]); + } + + #[test] + fn values_past_the_last_column_are_counted_as_skipped() { + let columns = [text("name"), text("city")]; + + let rows = pasted("Ada\tLondon\textra\t\nGrace\tNew York\n", &columns, 1); + + assert_eq!(rows.rows[0], vec![null(), value("Ada")]); + assert_eq!(rows.skipped, 3, "London, extra and New York had no column"); + } + + #[test] + fn a_header_maps_the_values_by_name() { + let columns = [key("id"), text("name"), text("city")]; + + let rows = pasted("City\tname\tshoe size\nLondon\tAda\t38\n", &columns, 2); + + assert_eq!(rows.rows, vec![vec![null(), value("Ada"), value("London")]]); + assert_eq!(rows.skipped, 1, "the shoe size has no column"); + } + + #[test] + fn a_first_line_naming_too_few_columns_is_data() { + let columns = [text("name"), text("city")]; + + let rows = pasted("name\tParis\tFrance\nAda\tLondon\tUK\n", &columns, 0); + + assert_eq!(rows.rows.len(), 2); + assert_eq!(rows.rows[0], vec![value("name"), value("Paris")]); + } + + #[test] + fn a_lone_line_naming_the_columns_is_a_row() { + let columns = [text("name"), text("city")]; + + let rows = pasted("name\tcity\n", &columns, 0); + + assert_eq!(rows.rows, vec![vec![value("name"), value("city")]]); + } + + #[test] + fn a_column_the_server_fills_keeps_its_default_and_counts_the_value() { + let mut stamped = text("created"); + stamped.is_generated = true; + let columns = [serial("id"), text("name"), stamped]; + + let rows = pasted("41\tAda\t2024-01-01\n42\tGrace\t\n", &columns, 0); + + assert_eq!(rows.rows[0], vec![CellInput::Default, value("Ada"), CellInput::Default]); + assert_eq!(rows.skipped, 3, "an empty field loses nothing"); + } + + #[test] + fn a_key_takes_the_value_unless_the_server_fills_it() { + let mut generated_uuid = column("id", ColumnKind::Uuid); + generated_uuid.primary_key = true; + generated_uuid.default = ColumnDefault::Expression(SqlExpression::from_catalog_text("gen_random_uuid()")); + + let natural = pasted("33\tFrance\n", &[key("code"), text("name")], 0); + let filled = pasted( + "8a5f2c4e-0000-4000-8000-000000000000\tFrance\n", + &[generated_uuid, text("name")], + 0, + ); + + assert_eq!(natural.rows, vec![vec![number(33), value("France")]]); + assert_eq!(filled.rows[0][0], CellInput::Default); + assert_eq!(filled.skipped, 1); + } + + #[test] + fn a_bad_value_names_its_line_its_column_and_why() { + let columns = [text("name"), int("age")]; + + let error = PastedTable::read("Ada\t36\n\"Grace\nHopper\"\t\nLinus\tforty\n") + .expect("read") + .rows(&columns, &from(0, &columns)) + .expect_err("forty is no number"); + + assert_eq!( + error, + CsvRowError { + line: 4, + column: "age".to_owned(), + text: "forty".to_owned(), + reason: EditParseError::InvalidInteger, + } + ); + } + + #[test] + fn null_is_null_and_an_empty_field_is_read_as_an_empty_cell() { + let mut level = int("level"); + level.nullable = false; + level.default = ColumnDefault::Literal(Value::Int(1)); + let mut rank = int("rank"); + rank.nullable = false; + let columns = [text("name"), int("age"), level, rank]; + + let rows = pasted("NULL\t\tNULL\t3\n", &columns, 0); + let refused = PastedTable::read("Ada\t1\t1\tNULL\n") + .expect("read") + .rows(&columns, &from(0, &columns)) + .expect_err("rank needs a value"); + + assert_eq!(rows.rows[0], vec![null(), null(), CellInput::Default, number(3)]); + assert_eq!(refused.reason, EditParseError::Required); + } + + #[test] + fn a_column_the_grid_cannot_edit_is_left_alone() { + let blob = column( + "data", + ColumnKind::Bytes { + length: TextLength::Unbounded, + }, + ); + + let rows = pasted("Ada\t0x01ff\n", &[text("name"), blob], 0); + + assert_eq!(rows.rows[0], vec![value("Ada"), null()]); + assert_eq!(rows.skipped, 1); + } + + #[test] + fn windows_line_breaks_and_blank_lines_make_no_rows_of_their_own() { + let rows = pasted("Ada\r\n\r\nGrace\r\n", &[text("name")], 0); + + assert_eq!(rows.rows, vec![vec![value("Ada")], vec![value("Grace")]]); + } + + #[test] + fn one_value_alone_is_a_single_value() { + let one = PastedTable::read("Ada\n").expect("read"); + let quoted = PastedTable::read("\"two\nlines\"").expect("read"); + let row = PastedTable::read("Ada\tLondon\n").expect("read"); + let lines = PastedTable::read("Ada\nGrace\n").expect("read"); + + assert_eq!(one.single_value(), Some(Some("Ada"))); + assert_eq!(quoted.single_value(), Some(Some("two\nlines"))); + assert_eq!(PastedTable::read("NULL").expect("read").single_value(), Some(None)); + assert_eq!(row.single_value(), None); + assert_eq!(lines.single_value(), None); + } + + #[test] + fn empty_text_holds_no_rows() { + assert!(PastedTable::read("").expect("read").is_empty()); + assert!(PastedTable::read("\n\n").expect("read").is_empty()); + } + + fn copied_cells(columns: &[ColumnInfo], rows: &[Vec], with_headers: bool) -> Vec { + let result: Vec<_> = columns.iter().map(ColumnInfo::result_column).collect(); + let mut encoder = CellsEncoder::new(result.into(), with_headers); + let mut out = Vec::new(); + encoder.begin(&mut out); + for row in rows { + encoder.row(&mut out, &mut row.iter()).expect("a row"); + } + encoder.finish(&mut out); + out + } + + #[test] + fn copied_cells_keep_every_value_and_null_as_it_was() { + let columns = [text("name"), text("note"), int("age")]; + let copied = copied_cells( + &columns, + &[ + vec![ + Value::Text("NULL".into()), + Value::Text("\"quoted\"".into()), + Value::Null, + ], + vec![ + Value::Text("two\nlines".into()), + Value::Text("tab\there".into()), + Value::Int(3), + ], + ], + false, + ); + + let rows = PastedTable::from_cells(&copied) + .expect("read") + .rows(&columns, &from(0, &columns)) + .expect("rows"); + + assert_eq!( + rows.rows, + vec![ + vec![value("NULL"), value("\"quoted\""), null()], + vec![value("two\nlines"), value("tab\there"), number(3)], + ] + ); + } + + #[test] + fn copied_cells_with_names_map_by_name_from_the_first_row() { + let source = [text("city"), text("name")]; + let copied = copied_cells( + &source, + &[vec![Value::Text("London".into()), Value::Text("Ada".into())]], + true, + ); + + let rows = PastedTable::from_cells(&copied) + .expect("read") + .rows(&[key("id"), text("name"), text("city")], &[2]) + .expect("rows"); + + assert_eq!(rows.rows, vec![vec![null(), value("Ada"), value("London")]]); + } + + #[test] + fn copied_cells_without_names_never_take_a_row_for_a_header() { + let columns = [text("name"), text("city")]; + let copied = copied_cells( + &columns, + &[ + vec![Value::Text("name".into()), Value::Text("city".into())], + vec![Value::Text("Ada".into()), Value::Text("London".into())], + ], + false, + ); + + let rows = PastedTable::from_cells(&copied) + .expect("read") + .rows(&columns, &from(0, &columns)) + .expect("rows"); + + assert_eq!(rows.rows.len(), 2, "the first row was read as a header"); + } + + #[test] + fn a_name_with_spaces_of_its_own_finds_its_column() { + let columns = [text("name"), text(" name")]; + let copied = copied_cells(&[text(" name")], &[vec![Value::Text("Ada".into())]], true); + + let rows = PastedTable::from_cells(&copied) + .expect("read") + .rows(&columns, &from(0, &columns)) + .expect("rows"); + + assert_eq!(rows.rows, vec![vec![null(), value("Ada")]]); + } + + #[test] + fn copied_cells_go_to_the_columns_they_came_from() { + let source = [text("name"), text("city")]; + let copied = copied_cells( + &source, + &[vec![Value::Text("Ada".into()), Value::Text("London".into())]], + false, + ); + + let rows = PastedTable::from_cells(&copied) + .expect("read") + .rows(&[key("id"), text("city"), text("name")], &[0, 1, 2]) + .expect("rows"); + + assert_eq!(rows.rows, vec![vec![null(), value("London"), value("Ada")]]); + } + + #[test] + fn copied_cells_from_columns_the_table_lacks_fill_the_columns_in_order() { + let source = [text("person"), text("town")]; + let copied = copied_cells( + &source, + &[vec![Value::Text("Ada".into()), Value::Text("London".into())]], + false, + ); + + let rows = PastedTable::from_cells(&copied) + .expect("read") + .rows(&[key("id"), text("name"), text("city")], &[1, 2]) + .expect("rows"); + + assert_eq!(rows.rows, vec![vec![null(), value("Ada"), value("London")]]); + } + + #[test] + fn one_copied_cell_is_a_single_value() { + let copied = copied_cells(&[text("name")], &[vec![Value::Text("\"Ada\"".into())]], false); + let nothing = copied_cells(&[text("name")], &[vec![Value::Null]], false); + + assert_eq!( + PastedTable::from_cells(&copied).expect("read").single_value(), + Some(Some("\"Ada\"")) + ); + assert_eq!( + PastedTable::from_cells(¬hing).expect("read").single_value(), + Some(None) + ); + } + + #[test] + fn cells_that_are_not_a_copy_are_refused() { + assert!(matches!( + PastedTable::from_cells(b"{\"rows\": 3}"), + Err(ImportError::Malformed { .. }) + )); + } + + #[test] + fn more_lines_than_a_paste_reads_are_left_unread() { + let text = "x\n".repeat(MAX_PASTED_ROWS + 1); + let at_the_limit = "x\n".repeat(MAX_PASTED_ROWS); + let cells = format!( + "{{\"headers\":null,\"rows\":[{}[\"x\"]]}}", + "[\"x\"],".repeat(MAX_PASTED_ROWS) + ); + + let over = PastedTable::read(&text).expect("read"); + let copied = PastedTable::from_cells(cells.as_bytes()).expect("read"); + + assert!(over.truncated()); + assert_eq!(over.records.len(), MAX_PASTED_ROWS); + assert!(copied.truncated()); + assert_eq!(copied.records.len(), MAX_PASTED_ROWS); + assert!(!PastedTable::read(&at_the_limit).expect("read").truncated()); + } + + #[test] + fn copied_cells_wider_than_any_table_are_refused() { + let wide = format!("{{\"headers\":null,\"rows\":[[{}null]]}}", "null,".repeat(MAX_FIELDS)); + + assert!(matches!( + PastedTable::from_cells(wide.as_bytes()), + Err(ImportError::Malformed { .. }) + )); + } + + #[test] + fn a_line_wider_than_any_table_is_refused() { + let wide = "x\t".repeat(super::super::MAX_FIELDS); + + assert!(matches!( + PastedTable::read(&wide), + Err(ImportError::TooManyFields { line: 1, .. }) + )); + } +} diff --git a/linux/crates/test-support/src/conformance/cases/round_trip.rs b/linux/crates/test-support/src/conformance/cases/round_trip.rs index 9f22c387b3..cdd8794323 100644 --- a/linux/crates/test-support/src/conformance/cases/round_trip.rs +++ b/linux/crates/test-support/src/conformance/cases/round_trip.rs @@ -9,9 +9,12 @@ use tablepro_core::column::{ColumnInfo, ColumnType, ResultColumn}; use tablepro_core::dialect::SqlDialect; use tablepro_core::edit::{EditText, edit_text, read_only_reason}; use tablepro_core::export::{ - CsvDelimiter, CsvEncoder, CsvLineBreak, CsvOptions, CsvQuote, RowEncoder, SqlInsertEncoder, + CellsEncoder, CsvDelimiter, CsvEncoder, CsvLineBreak, CsvOptions, CsvQuote, RowEncoder, SqlInsertEncoder, + TsvEncoder, }; -use tablepro_core::import::{CsvImportOptions, MAX_CSV_FILE_BYTES}; +use tablepro_core::import::{CsvImportOptions, MAX_CSV_FILE_BYTES, PastedTable}; +use tablepro_core::meta::RowIdentity; +use tablepro_core::save::{ChangeSet, RowChange, plan_save}; use tablepro_core::sql_syntax::script::{LexicalSettings, ScriptPlan}; use tablepro_core::{ Connection, EditorSession, EditorSessionOptions, FilterSet, ReadStatement, ResultEvent, ResultSink, SinkFlow, @@ -30,6 +33,7 @@ use crate::{import_csv, lock}; const CSV_CASE: &str = "csv_export_imports_back_as_the_same_rows"; const SQL_CASE: &str = "sql_insert_export_runs_back_as_the_same_rows"; const NEAR_LIMIT_CASE: &str = "csv_export_near_the_import_limit_imports_back"; +const PASTE_CASE: &str = "tsv_copy_pastes_back_as_the_same_rows"; /// What the import reads as NULL in every pairing, and what the export /// writes for one. @@ -128,6 +132,202 @@ pub async fn csv_export_imports_back_as_the_same_rows( }) } +/// Every corpus value with a text form, copied from the grid as rows and +/// pasted into an empty copy of its table, saves as the same rows. A +/// copy offers the cells and their text: each is pasted on its own, read +/// as a paste reads it, and saved as a table tab saves new rows. +/// +/// The rows are compared in the order of their `id`, without it: a +/// paste leaves an id the server numbers, such as SQLite's row id, to +/// the server. +pub async fn tsv_copy_pastes_back_as_the_same_rows(engine: E) -> Result<(), ConformanceFailure> { + let corpus = engine.value_corpus(); + expect(PASTE_CASE, !corpus.is_empty(), || { + "the engine lists no corpus".to_owned() + })?; + let connection = connect(PASTE_CASE, &engine, writable()).await?; + let mut wrong = Vec::new(); + let mut round_trips = 0; + for (index, column) in corpus.iter().enumerate() { + let source = match load(&engine, connection.as_ref(), &format!("tsv_{index}"), column).await { + Ok(table) => table, + Err(failure) => { + wrong.push(failure.to_string()); + continue; + } + }; + let columns = connection.fetch_columns(&source, within(CALL_DEADLINE)).await?; + if columns + .iter() + .any(|info| read_only_reason(&info.column_type.kind).is_some()) + { + continue; + } + let mut expected = read_rows(connection.as_ref(), &source, texts, KeepValues::Yes).await?; + let read_columns = expected.values.columns.clone(); + expected + .values + .rows + .retain(|row| has_text(read_columns.as_deref(), row)); + let text = String::from_utf8(encode(&expected.values, &tsv_encoder())?).map_err(ConformanceFailure::setup)?; + let cells = encode(&expected.values, &cells_encoder())?; + let pastes = [ + ("text", PastedTable::read(&text)), + ("cells", PastedTable::from_cells(&cells)), + ]; + for (copy_index, (form, pasted)) in pastes.into_iter().enumerate() { + let what = format!("`{}` copied and pasted as {form}", column.declaration); + let copy = corpus_table( + &engine, + connection.as_ref(), + &format!("tsv_{index}_copy_{copy_index}"), + column, + &[], + ) + .await?; + let pasted = match pasted { + Ok(pasted) => pasted, + Err(error) => { + wrong.push(format!("{what}: the copy did not read as rows: {error}")); + continue; + } + }; + if let Some(failure) = paste(connection.as_ref(), ©, &pasted).await? { + wrong.push(format!("{what}: {failure}")); + continue; + } + let actual = read_rows(connection.as_ref(), ©, texts, KeepValues::No).await?; + wrong.extend(order_differences( + &what, + &in_id_order(&expected, &expected.without_text), + &in_id_order(&actual, &BTreeSet::new()), + )); + round_trips += 1; + } + } + connection.close().await?; + expect(PASTE_CASE, round_trips > 0, || { + "no corpus column could travel as copied rows".to_owned() + })?; + expect(PASTE_CASE, wrong.is_empty(), || { + format!( + "{} differences after a copy and paste:\n{}", + wrong.len(), + wrong.join("\n") + ) + }) +} + +/// Paste `pasted` into `table` and save the rows, as a table tab does +/// with the rows a paste gives it. Returns why that stopped short. +async fn paste( + connection: &dyn Connection, + table: &TableRef, + pasted: &PastedTable, +) -> Result, ConformanceFailure> { + let columns = connection.fetch_columns(table, within(CALL_DEADLINE)).await?; + let in_order: Vec = (0..columns.len()).collect(); + let rows = match pasted.rows(&columns, &in_order) { + Ok(rows) => rows, + Err(error) => return Ok(Some(format!("the paste refused a value: {error:?}"))), + }; + let identity = RowIdentity::UniqueKey { + columns: columns + .iter() + .filter(|column| column.primary_key) + .map(|column| column.name.clone()) + .collect(), + }; + let changes = ChangeSet { + changes: rows.rows.into_iter().map(|cells| RowChange::Insert { cells }).collect(), + }; + let plan = match plan_save(&*connection.dialect(), table, &columns, &identity, &changes) { + Ok(plan) => plan, + Err(error) => return Ok(Some(format!("the save could not be planned: {error}"))), + }; + Ok(connection + .write_batch(&plan.batch, within(IMPORT_DEADLINE)) + .await + .err() + .map(|error| format!("the save failed: {error}"))) +} + +/// Rows as the grid's Copy writes them: tab-separated, no header. +fn tsv_encoder() -> EncoderFor { + Box::new(|columns| Box::new(TsvEncoder::new(columns, false))) +} + +/// The cells the grid's Copy offers beside that text. +fn cells_encoder() -> EncoderFor { + Box::new(|columns| Box::new(CellsEncoder::new(columns, false))) +} + +/// Whether every value of `row` has a text form, which a copy needs to +/// carry it. +fn has_text(columns: Option<&[ResultColumn]>, row: &[Value]) -> bool { + let untyped = ColumnType::unknown(); + row.iter().enumerate().all(|(index, value)| { + let column_type = columns + .and_then(|columns| columns.get(index)) + .map_or(&untyped, |column| &column.column_type); + matches!(edit_text(value, column_type), EditText::Editable(_)) + }) +} + +/// The texts of each row in the order of its numeric `id`, without the +/// `id`, leaving out the rows `skipped` names. +fn in_id_order(rows: &Rows, skipped: &BTreeSet) -> Vec>> { + let id = rows.columns.iter().position(|name| name == "id"); + let mut ordered: Vec<(i64, Vec>)> = rows + .by_id + .iter() + .filter(|(key, _)| !skipped.contains(*key)) + .filter_map(|(key, kept)| { + let Kept::Texts(texts) = kept else { + return None; + }; + let texts = texts + .iter() + .enumerate() + .filter(|(index, _)| Some(*index) != id) + .map(|(_, text)| text.clone()) + .collect(); + Some((key.parse().ok()?, texts)) + }) + .collect(); + ordered.sort_by_key(|(id, _)| *id); + ordered.into_iter().map(|(_, texts)| texts).collect() +} + +fn order_differences(what: &str, source: &[Vec>], copy: &[Vec>]) -> Vec { + let mut wrong = Vec::new(); + if source.len() != copy.len() { + wrong.push(format!( + "{what}: {} rows came back, {} were copied", + copy.len(), + source.len() + )); + } + for (position, (expected, actual)) in source.iter().zip(copy).enumerate() { + if expected != actual { + let quote = |texts: &[Option]| { + texts + .iter() + .map(|text| quoted(text.as_deref())) + .collect::>() + .join(", ") + }; + wrong.push(format!( + "{what}: row {} came back as ({}), copied as ({})", + position + 1, + quote(actual), + quote(expected) + )); + } + } + wrong +} + /// Every corpus value exports as SQL INSERT, and the file, run in an /// editor against an empty copy of its table, gives back the same rows, /// each value of the same kind. diff --git a/linux/crates/test-support/src/conformance/round_trip_suite_macro.rs b/linux/crates/test-support/src/conformance/round_trip_suite_macro.rs index b9c281a980..9b44d91626 100644 --- a/linux/crates/test-support/src/conformance/round_trip_suite_macro.rs +++ b/linux/crates/test-support/src/conformance/round_trip_suite_macro.rs @@ -3,8 +3,9 @@ /// marks them `#[ignore = "requires docker"]`. /// /// The value corpus goes out as CSV and comes back through the import, -/// and goes out as SQL INSERT and comes back through an editor run, each -/// into an empty copy of its table. `near_limit: true` adds a case that +/// goes out as SQL INSERT and comes back through an editor run, and goes +/// out as the grid's Copy and comes back as a paste saved, each into an +/// empty copy of its table. `near_limit: true` adds a case that /// exports a table whose CSV lands just under the import's size limit /// and imports it back. It describes how fast the engine takes an /// import, so no capability stands behind it. @@ -13,6 +14,7 @@ macro_rules! round_trip_suite { (engine: $engine:ty, docker: $docker:tt, near_limit: $near_limit:tt $(,)?) => { $crate::__round_trip_case! { $docker, $engine, csv_export_imports_back_as_the_same_rows } $crate::__round_trip_case! { $docker, $engine, sql_insert_export_runs_back_as_the_same_rows } + $crate::__round_trip_case! { $docker, $engine, tsv_copy_pastes_back_as_the_same_rows } $crate::__near_limit_case! { $near_limit, $docker, $engine } }; } diff --git a/linux/data/resources/shortcuts-dialog.ui b/linux/data/resources/shortcuts-dialog.ui index 9437cc425d..fc9aab84ba 100644 --- a/linux/data/resources/shortcuts-dialog.ui +++ b/linux/data/resources/shortcuts-dialog.ui @@ -282,6 +282,12 @@ <Primary>c + + + Paste Rows + <Primary>v + + Clear the Multi-Row Selection diff --git a/linux/docs/testing.md b/linux/docs/testing.md index bb032d9c1b..aca18b5bf4 100644 --- a/linux/docs/testing.md +++ b/linux/docs/testing.md @@ -216,7 +216,7 @@ Every target runs it, the MySQL target on MariaDB as well, and SQLite's runs in ### Export round trip -`round_trip_suite!` checks that a file the export writes loads back with the same rows. Each driver's `crates/drivers//tests/integration/round_trip.rs` invokes it with an engine that implements `ValueCorpusEngine` and `LargeTableEngine`: +`round_trip_suite!` checks that a file the export writes, and rows the grid copies, load back with the same rows. Each driver's `crates/drivers//tests/integration/round_trip.rs` invokes it with an engine that implements `ValueCorpusEngine` and `LargeTableEngine`: ```rust tablepro_test_support::round_trip_suite! { @@ -228,6 +228,7 @@ tablepro_test_support::round_trip_suite! { - `csv_export_imports_back_as_the_same_rows` loads each corpus column into a table, exports it as CSV and imports the file into an empty copy, through `import_csv`, which runs the plan the import dialog runs. It does this under four pairings of export and import options that are meant to lose nothing: every delimiter, quoting always or where needed, LF, CRLF and CR line breaks, NULL written and read as `NULL`, and formulas left as they are. Row count and the text of every row have to match. A column of a kind `read_only_reason` names (bytes, arrays, spatial, variant and composite values) and a value `edit_text` gives no text for, such as a blob in a SQLite column typed by its values, have no CSV text the import reads back, so the SQL case covers them. - `sql_insert_export_runs_back_as_the_same_rows` exports each corpus column as SQL INSERT into an empty copy and runs the file as the editor runs one: split by the engine's grammar and sent a batch at a time. Every value has to come back with the same text and of the same kind. +- `tsv_copy_pastes_back_as_the_same_rows` writes each corpus column's rows as the grid's Copy does, as tab-separated text and as the cells it offers beside the text (`CellsEncoder`), reads each as a paste does (`PastedTable`) and saves the rows into an empty copy as a table tab saves new ones (`plan_save`). Rows are compared in `id` order without the `id`, since a paste leaves an id the server numbers, such as SQLite's row id, to the server. The columns and values the CSV case skips are skipped here too. - `csv_export_near_the_import_limit_imports_back`, with `near_limit: true`, fills a table through `LargeTableEngine` so its CSV under the export defaults lands within 1 MiB under `MAX_CSV_FILE_BYTES`, imports it into an empty copy, and compares a hash of every row. ClickHouse passes `false`: its import sends values in the request URL, 8 KiB a statement, so such a file takes some 8,000 statements and longer than a case may run. Each case has four minutes (`ROUND_TRIP_CASE_LIMIT`). Every target runs the suite, the MySQL target on MariaDB as well, and SQLite's runs in the unit suite. diff --git a/linux/po/POTFILES.in b/linux/po/POTFILES.in index 6e39521c10..b951f0578b 100644 --- a/linux/po/POTFILES.in +++ b/linux/po/POTFILES.in @@ -140,6 +140,7 @@ crates/app/src/ui/browse_tab/column_recheck.rs crates/app/src/ui/browse_tab/columns_fetch.rs crates/app/src/ui/browse_tab/command_handlers.rs crates/app/src/ui/browse_tab/confirming_delete.rs +crates/app/src/ui/browse_tab/confirming_paste.rs crates/app/src/ui/browse_tab/count_failure_text.rs crates/app/src/ui/browse_tab/count_policy.rs crates/app/src/ui/browse_tab/data_handlers.rs @@ -156,6 +157,9 @@ crates/app/src/ui/browse_tab/page_state.rs crates/app/src/ui/browse_tab/pager_bar.rs crates/app/src/ui/browse_tab/paging_handlers.rs crates/app/src/ui/browse_tab/paging_input.rs +crates/app/src/ui/browse_tab/paste_at.rs +crates/app/src/ui/browse_tab/paste_handlers.rs +crates/app/src/ui/browse_tab/paste_read.rs crates/app/src/ui/browse_tab/pending_changes_bar.rs crates/app/src/ui/browse_tab/row_count.rs crates/app/src/ui/browse_tab/row_deletion.rs @@ -168,6 +172,7 @@ crates/app/src/ui/browse_tab/selection_snapshot.rs crates/app/src/ui/browse_tab/table_filter.rs crates/app/src/ui/browse_tab/tests/load.rs crates/app/src/ui/browse_tab/tests/mod.rs +crates/app/src/ui/browse_tab/tests/paste.rs crates/app/src/ui/browse_tab/tests/references.rs crates/app/src/ui/browse_tab/tests/total.rs crates/app/src/ui/clipboard/clipboard_copy.rs From 4f15923f830d3cc8d5322bca6fc9e805d9b38b37 Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Fri, 23 Oct 2026 22:43:03 +0700 Subject: [PATCH 2/3] docs(linux): changelog fragments for pasting rows --- linux/changelog.d/3815.added.md | 1 + linux/changelog.d/3815.fixed.1.md | 1 + linux/changelog.d/3815.fixed.md | 1 + 3 files changed, 3 insertions(+) create mode 100644 linux/changelog.d/3815.added.md create mode 100644 linux/changelog.d/3815.fixed.1.md create mode 100644 linux/changelog.d/3815.fixed.md diff --git a/linux/changelog.d/3815.added.md b/linux/changelog.d/3815.added.md new file mode 100644 index 0000000000..4667393293 --- /dev/null +++ b/linux/changelog.d/3815.added.md @@ -0,0 +1 @@ +Paste rows into a table with Ctrl+V or Paste in the cell menu, copied from a TablePro grid or a spreadsheet. diff --git a/linux/changelog.d/3815.fixed.1.md b/linux/changelog.d/3815.fixed.1.md new file mode 100644 index 0000000000..86ad40eed1 --- /dev/null +++ b/linux/changelog.d/3815.fixed.1.md @@ -0,0 +1 @@ +A CSV import error quoted the whole refused value, however long it was. diff --git a/linux/changelog.d/3815.fixed.md b/linux/changelog.d/3815.fixed.md new file mode 100644 index 0000000000..4fd760e3e9 --- /dev/null +++ b/linux/changelog.d/3815.fixed.md @@ -0,0 +1 @@ +Duplicating several rows showed the copies in reverse order. From 8cb18b37bb51fa14833a06088746622086c86b02 Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Fri, 23 Oct 2026 23:32:00 +0700 Subject: [PATCH 3/3] fix(linux-browse): keep untyped values' storage and cap the cells a paste makes --- .../app/src/ui/browse_tab/paste_read.rs | 2 +- .../core/src/export/encoder/cells_encoder.rs | 37 +++- linux/crates/core/src/import/mod.rs | 2 +- linux/crates/core/src/import/pasted_table.rs | 206 ++++++++++++++++-- .../src/conformance/cases/round_trip.rs | 51 +++-- linux/docs/testing.md | 2 +- 6 files changed, 246 insertions(+), 54 deletions(-) diff --git a/linux/crates/app/src/ui/browse_tab/paste_read.rs b/linux/crates/app/src/ui/browse_tab/paste_read.rs index c80186fbde..8442105708 100644 --- a/linux/crates/app/src/ui/browse_tab/paste_read.rs +++ b/linux/crates/app/src/ui/browse_tab/paste_read.rs @@ -64,7 +64,7 @@ pub(super) fn read_pasted( if table.is_empty() { return PasteOutcome::Nothing; } - if table.truncated() { + if table.too_large_for(columns.len()) { return PasteOutcome::TooLarge; } if at_cell && let Some(value) = table.single_value() { diff --git a/linux/crates/core/src/export/encoder/cells_encoder.rs b/linux/crates/core/src/export/encoder/cells_encoder.rs index e4d45dd905..d7d8272f6c 100644 --- a/linux/crates/core/src/export/encoder/cells_encoder.rs +++ b/linux/crates/core/src/export/encoder/cells_encoder.rs @@ -12,6 +12,10 @@ use super::{EncodeError, RowEncoder, readable_cells}; /// columns they came from, with `headers` set when the copy showed them. /// Text loses that: TSV writes a NULL and the text `NULL` alike, and /// quotes a value it then cannot tell from one that was quoted. +/// +/// A number or bytes says so, as `{"int": "42"}`, `{"real": "1.5"}` or +/// `{"bytes": "0x01"}`: a column of no declared type, such as SQLite +/// keeps, stores the value as it comes, and its text alone is text. pub struct CellsEncoder { columns: Arc<[ResultColumn]>, with_headers: bool, @@ -45,12 +49,22 @@ impl RowEncoder for CellsEncoder { if self.rows > 0 { out.push(b','); } - let texts: Vec> = cells - .iter() - .enumerate() - .map(|(index, value)| value_text(value, column_type(&self.columns, index))) - .collect(); - write_list(out, texts.iter().map(Option::as_deref)); + out.push(b'['); + for (index, value) in cells.iter().enumerate() { + if index > 0 { + out.push(b','); + } + let Some(text) = value_text(value, column_type(&self.columns, index)) else { + out.extend_from_slice(b"null"); + continue; + }; + let text = serde_json::Value::from(text).to_string(); + match stored_as(value) { + Some(kind) => out.extend_from_slice(format!("{{\"{kind}\":{text}}}").as_bytes()), + None => out.extend_from_slice(text.as_bytes()), + } + } + out.push(b']'); self.rows += 1; Ok(()) } @@ -60,6 +74,17 @@ impl RowEncoder for CellsEncoder { } } +/// The storage a value's text does not carry. +fn stored_as(value: &Value) -> Option<&'static str> { + match value { + Value::Int(_) | Value::UInt(_) | Value::WideInt(_) => Some("int"), + // A decimal's text is exact, which a binary float is not. + Value::Float32(_) | Value::Float64(_) => Some("real"), + Value::Bytes(_) => Some("bytes"), + _ => None, + } +} + fn write_list<'a>(out: &mut Vec, items: impl Iterator>) { out.push(b'['); for (index, item) in items.enumerate() { diff --git a/linux/crates/core/src/import/mod.rs b/linux/crates/core/src/import/mod.rs index 80b9439d07..756d3da261 100644 --- a/linux/crates/core/src/import/mod.rs +++ b/linux/crates/core/src/import/mod.rs @@ -14,4 +14,4 @@ pub use csv_import::{ pub use import_column::ImportColumn; pub use import_plan::{ImportPlan, plan_import}; pub use pasted_rows::PastedRows; -pub use pasted_table::{MAX_PASTED_ROWS, PastedTable}; +pub use pasted_table::{MAX_PASTED_CELLS, MAX_PASTED_ROWS, PastedTable}; diff --git a/linux/crates/core/src/import/pasted_table.rs b/linux/crates/core/src/import/pasted_table.rs index ebec29f6cb..0210cdc22f 100644 --- a/linux/crates/core/src/import/pasted_table.rs +++ b/linux/crates/core/src/import/pasted_table.rs @@ -3,7 +3,7 @@ use std::fmt; use serde::Deserialize; use serde::de::{self, Deserializer, IgnoredAny, SeqAccess, Visitor}; -use crate::column::ColumnInfo; +use crate::column::{ColumnInfo, ColumnKind, ReadForm}; use crate::edit::{CellInput, EditContext, EditParseError, new_draft_cell, parse_edit_text, read_only_reason}; use crate::export::TSV_NULL; use crate::value::Value; @@ -15,6 +15,11 @@ use super::{CsvRowError, ImportError, MAX_FIELDS, PastedRows}; /// pending row the grid holds, and Import CSV is the tool for more. pub const MAX_PASTED_ROWS: usize = 100_000; +/// The most cells the new rows of a paste may hold, every column of the +/// table counted: a short line into a wide table makes a whole row, and +/// the undo history keeps a second copy. +pub const MAX_PASTED_CELLS: usize = 2_000_000; + /// Rows from the clipboard, before they meet a table: the cells a grid /// copy wrote, or text read as tab-separated rows, which is what the /// copy also writes and what a spreadsheet puts on the clipboard. @@ -42,7 +47,76 @@ struct PastedRecord { /// Where the record starts, from 1, which an error names. line: usize, /// `None` is a NULL. - fields: Vec>, + fields: Vec>, +} + +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +#[serde(from = "CopiedCell")] +struct PastedField { + text: String, + stored: Stored, +} + +/// How the copied value was stored, which its text does not say. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Stored { + Text, + Integer, + Real, + Bytes, +} + +/// A cell as `CellsEncoder` writes it. +#[derive(Deserialize)] +#[serde(untagged)] +enum CopiedCell { + Text(String), + Integer { int: String }, + Real { real: String }, + Bytes { bytes: String }, +} + +impl From for PastedField { + fn from(cell: CopiedCell) -> Self { + let (text, stored) = match cell { + CopiedCell::Text(text) => (text, Stored::Text), + CopiedCell::Integer { int } => (int, Stored::Integer), + CopiedCell::Real { real } => (real, Stored::Real), + CopiedCell::Bytes { bytes } => (bytes, Stored::Bytes), + }; + Self { text, stored } + } +} + +impl PastedField { + fn plain(text: &str) -> Self { + Self { + text: text.to_owned(), + stored: Stored::Text, + } + } + + /// The value as it was stored, for a column that keeps whatever it is + /// given. + fn stored_value(&self) -> Option { + match self.stored { + Stored::Text => None, + Stored::Integer => self + .text + .parse() + .map(Value::Int) + .or_else(|_| self.text.parse().map(Value::UInt)) + .ok(), + // SQLite stores a NaN as NULL, which its text keeps. + Stored::Real => self + .text + .parse() + .ok() + .filter(|real: &f64| real.is_finite()) + .map(Value::Float64), + Stored::Bytes => crate::hex::decode(self.text.strip_prefix("0x")?).map(Value::Bytes), + } + } } /// The column each field of a record goes to, by position or by the @@ -63,7 +137,7 @@ struct CopiedCells { } struct CopiedRows { - rows: Vec>>, + rows: Vec>>, truncated: bool, } @@ -98,7 +172,7 @@ impl<'de> Visitor<'de> for RowsVisitor { /// One row's fields, refused past the widest a table can be before they /// are all held. -struct CopiedRow(Vec>); +struct CopiedRow(Vec>); impl<'de> Deserialize<'de> for CopiedRow { fn deserialize>(deserializer: D) -> Result { @@ -117,7 +191,7 @@ impl<'de> Visitor<'de> for RowVisitor { fn visit_seq>(self, mut seq: A) -> Result { let mut fields = Vec::new(); - while let Some(field) = seq.next_element::>()? { + while let Some(field) = seq.next_element::>()? { if fields.len() == MAX_FIELDS { return Err(de::Error::custom(format!("a row has more than {MAX_FIELDS} values"))); } @@ -151,7 +225,7 @@ impl PastedTable { .map_or(1, |position| usize::try_from(position.line()).unwrap_or(usize::MAX)); let fields = record .iter() - .map(|field| (field != TSV_NULL).then(|| field.to_owned())) + .map(|field| (field != TSV_NULL).then(|| PastedField::plain(field))) .collect(); records.push(PastedRecord { line, fields }); } @@ -193,10 +267,11 @@ impl PastedTable { self.records.is_empty() } - /// Whether the clipboard held more lines than a paste reads, which - /// were left unread. - pub fn truncated(&self) -> bool { - self.truncated + /// Whether the rows are more than a paste takes into a table of + /// `width` columns: more lines than it reads, which were left unread, + /// or more cells than the new rows may hold. + pub fn too_large_for(&self, width: usize) -> bool { + self.truncated || self.records.len().saturating_mul(width) > MAX_PASTED_CELLS } /// The one value the clipboard holds, when it holds one and nothing @@ -204,7 +279,7 @@ impl PastedTable { pub fn single_value(&self) -> Option> { match self.records.as_slice() { [record] => match record.fields.as_slice() { - [field] => Some(field.as_deref()), + [field] => Some(field.as_ref().map(|field| field.text.as_str())), _ => None, }, _ => None, @@ -238,8 +313,8 @@ impl PastedTable { let mut pasted = PastedRows::default(); for record in records { let mut cells: Vec = columns.iter().map(new_draft_cell).collect(); - for (field, text) in record.fields.iter().enumerate() { - let text = text.as_deref(); + for (field, value) in record.fields.iter().enumerate() { + let text = value.as_ref().map(|value| value.text.as_str()); let target = match &placement { Placement::InOrder(in_order) => in_order.get(field).copied(), Placement::Named(named) => named.get(field).copied().flatten(), @@ -252,7 +327,7 @@ impl PastedTable { pasted.skipped += usize::from(carries_value(text)); continue; } - cells[index] = pasted_cell(text, column).map_err(|reason| CsvRowError { + cells[index] = pasted_cell(value.as_ref(), column).map_err(|reason| CsvRowError { line: record.line, column: column.name.clone(), text: text.unwrap_or(TSV_NULL).to_owned(), @@ -274,7 +349,7 @@ impl PastedTable { let names: Vec<&str> = first .fields .iter() - .map(|field| field.as_deref().unwrap_or("")) + .map(|field| field.as_ref().map_or("", |field| field.text.as_str())) .collect(); let named = named_columns(columns, &names); let matched = named.iter().flatten().count(); @@ -327,14 +402,30 @@ fn left_to_the_row(column: &ColumnInfo) -> bool { || read_only_reason(&column.column_type.kind).is_some() } +/// A column of no type the app knows whose values are read as the +/// engine stored them, such as SQLite's untyped columns, which keep each +/// value as it comes. A type only the server knows, such as PostgreSQL's +/// `tsvector`, is read and written as its text. +fn stores_as_given(column: &ColumnInfo) -> bool { + let column_type = &column.column_type; + matches!(column_type.kind, ColumnKind::Other) + && column_type.read_form == ReadForm::Native + && column_type.dynamic_storage +} + /// Whether leaving the field out loses anything. fn carries_value(text: Option<&str>) -> bool { text.is_some_and(|text| !text.is_empty()) } -fn pasted_cell(text: Option<&str>, column: &ColumnInfo) -> Result { - if let Some(text) = text { - return parse_edit_text(text, column, EditContext::InsertDraft); +fn pasted_cell(field: Option<&PastedField>, column: &ColumnInfo) -> Result { + if let Some(field) = field { + if stores_as_given(column) + && let Some(value) = field.stored_value() + { + return Ok(CellInput::Value(value)); + } + return parse_edit_text(&field.text, column, EditContext::InsertDraft); } if column.nullable { return Ok(CellInput::Value(Value::Null)); @@ -800,11 +891,84 @@ mod tests { let over = PastedTable::read(&text).expect("read"); let copied = PastedTable::from_cells(cells.as_bytes()).expect("read"); - assert!(over.truncated()); + assert!(over.too_large_for(1)); assert_eq!(over.records.len(), MAX_PASTED_ROWS); - assert!(copied.truncated()); + assert!(copied.too_large_for(1)); assert_eq!(copied.records.len(), MAX_PASTED_ROWS); - assert!(!PastedTable::read(&at_the_limit).expect("read").truncated()); + assert!(!PastedTable::read(&at_the_limit).expect("read").too_large_for(1)); + } + + #[test] + fn a_short_paste_into_a_wide_table_counts_every_cell_it_makes() { + let lines = PastedTable::read(&"x\n".repeat(1_000)).expect("read"); + + assert!(!lines.too_large_for(MAX_PASTED_CELLS / 1_000)); + assert!(lines.too_large_for(MAX_PASTED_CELLS / 1_000 + 1)); + } + + #[test] + fn an_untyped_column_keeps_how_the_copied_value_was_stored() { + let mut untyped = column("v", ColumnKind::Other); + untyped.column_type.dynamic_storage = true; + let copied = copied_cells( + std::slice::from_ref(&untyped), + &[ + vec![Value::Int(42)], + vec![Value::Text("42".into())], + vec![Value::Float64(1.5)], + vec![Value::Bytes(vec![1, 255])], + ], + false, + ); + + let rows = PastedTable::from_cells(&copied) + .expect("read") + .rows(std::slice::from_ref(&untyped), &[0]) + .expect("rows"); + let typed = PastedTable::from_cells(&copied).expect("read").rows(&[int("v")], &[0]); + + assert_eq!(rows.rows[0], vec![number(42)]); + assert!( + matches!(&rows.rows[1][0], CellInput::Value(Value::Other(other)) if other.text == "42"), + "{:?}", + rows.rows[1] + ); + assert_eq!(rows.rows[2], vec![CellInput::Value(Value::Float64(1.5))]); + assert_eq!(rows.rows[3], vec![CellInput::Value(Value::Bytes(vec![1, 255]))]); + assert!(typed.is_err(), "an integer column took 1.5"); + } + + #[test] + fn a_type_only_the_server_knows_takes_the_text() { + let mut server_typed = column("v", ColumnKind::Other); + server_typed.column_type.dynamic_storage = true; + server_typed.column_type.read_form = ReadForm::ServerText; + let mut untyped = server_typed.clone(); + untyped.column_type.read_form = ReadForm::Native; + let copied = copied_cells( + &[server_typed.clone()], + &[ + vec![Value::Int(42)], + vec![Value::Decimal("9007199254740993".parse().expect("a decimal"))], + vec![Value::Float64(f64::NAN)], + ], + false, + ); + + let read = |column: &ColumnInfo| { + PastedTable::from_cells(&copied) + .expect("read") + .rows(std::slice::from_ref(column), &[0]) + .expect("rows") + .rows + }; + + for rows in [read(&server_typed), read(&untyped).split_off(1)] { + for row in rows { + assert!(matches!(&row[0], CellInput::Value(Value::Other(_))), "{row:?}"); + } + } + assert_eq!(read(&untyped)[0], vec![number(42)]); } #[test] diff --git a/linux/crates/test-support/src/conformance/cases/round_trip.rs b/linux/crates/test-support/src/conformance/cases/round_trip.rs index cdd8794323..dac88ac1b4 100644 --- a/linux/crates/test-support/src/conformance/cases/round_trip.rs +++ b/linux/crates/test-support/src/conformance/cases/round_trip.rs @@ -169,13 +169,16 @@ pub async fn tsv_copy_pastes_back_as_the_same_rows(engine: .values .rows .retain(|row| has_text(read_columns.as_deref(), row)); + let typed = read_rows(connection.as_ref(), &source, typed_texts, KeepValues::No).await?; let text = String::from_utf8(encode(&expected.values, &tsv_encoder())?).map_err(ConformanceFailure::setup)?; let cells = encode(&expected.values, &cells_encoder())?; + // Text carries no kinds, as a column SQLite types by its values + // shows; the cells keep how each value was stored. let pastes = [ - ("text", PastedTable::read(&text)), - ("cells", PastedTable::from_cells(&cells)), + ("text", PastedTable::read(&text), texts as Keep, &expected), + ("cells", PastedTable::from_cells(&cells), typed_texts as Keep, &typed), ]; - for (copy_index, (form, pasted)) in pastes.into_iter().enumerate() { + for (copy_index, (form, pasted, keep, source_rows)) in pastes.into_iter().enumerate() { let what = format!("`{}` copied and pasted as {form}", column.declaration); let copy = corpus_table( &engine, @@ -196,10 +199,10 @@ pub async fn tsv_copy_pastes_back_as_the_same_rows(engine: wrong.push(format!("{what}: {failure}")); continue; } - let actual = read_rows(connection.as_ref(), ©, texts, KeepValues::No).await?; + let actual = read_rows(connection.as_ref(), ©, keep, KeepValues::No).await?; wrong.extend(order_differences( &what, - &in_id_order(&expected, &expected.without_text), + &in_id_order(source_rows, &source_rows.without_text), &in_id_order(&actual, &BTreeSet::new()), )); round_trips += 1; @@ -276,30 +279,34 @@ fn has_text(columns: Option<&[ResultColumn]>, row: &[Value]) -> bool { /// The texts of each row in the order of its numeric `id`, without the /// `id`, leaving out the rows `skipped` names. -fn in_id_order(rows: &Rows, skipped: &BTreeSet) -> Vec>> { +fn in_id_order(rows: &Rows, skipped: &BTreeSet) -> Vec> { let id = rows.columns.iter().position(|name| name == "id"); - let mut ordered: Vec<(i64, Vec>)> = rows + let mut ordered: Vec<(i64, Vec)> = rows .by_id .iter() .filter(|(key, _)| !skipped.contains(*key)) .filter_map(|(key, kept)| { - let Kept::Texts(texts) = kept else { - return None; + let cells: Vec = match kept { + Kept::Texts(texts) => texts.iter().map(|text| quoted(text.as_deref())).collect(), + Kept::Typed(cells) => cells + .iter() + .map(|cell| format!("{} {}", cell.kind, quoted(cell.text.as_deref()))) + .collect(), }; - let texts = texts - .iter() + let cells = cells + .into_iter() .enumerate() .filter(|(index, _)| Some(*index) != id) - .map(|(_, text)| text.clone()) + .map(|(_, cell)| cell) .collect(); - Some((key.parse().ok()?, texts)) + Some((key.parse().ok()?, cells)) }) .collect(); ordered.sort_by_key(|(id, _)| *id); - ordered.into_iter().map(|(_, texts)| texts).collect() + ordered.into_iter().map(|(_, cells)| cells).collect() } -fn order_differences(what: &str, source: &[Vec>], copy: &[Vec>]) -> Vec { +fn order_differences(what: &str, source: &[Vec], copy: &[Vec]) -> Vec { let mut wrong = Vec::new(); if source.len() != copy.len() { wrong.push(format!( @@ -310,18 +317,11 @@ fn order_differences(what: &str, source: &[Vec>], copy: &[Vec]| { - texts - .iter() - .map(|text| quoted(text.as_deref())) - .collect::>() - .join(", ") - }; wrong.push(format!( "{what}: row {} came back as ({}), copied as ({})", position + 1, - quote(actual), - quote(expected) + actual.join(", "), + expected.join(", ") )); } } @@ -640,6 +640,9 @@ enum Kept { Typed(Vec), } +/// How a comparison keeps a row's cells. +type Keep = fn(Vec) -> Kept; + fn texts(cells: Vec) -> Kept { Kept::Texts(cells.into_iter().map(|cell| cell.text).collect()) } diff --git a/linux/docs/testing.md b/linux/docs/testing.md index e1fa43ac8d..1320cd44a9 100644 --- a/linux/docs/testing.md +++ b/linux/docs/testing.md @@ -228,7 +228,7 @@ tablepro_test_support::round_trip_suite! { - `csv_export_imports_back_as_the_same_rows` loads each corpus column into a table, exports it as CSV and imports the file into an empty copy, through `import_csv`, which runs the plan the import dialog runs. It does this under four pairings of export and import options that are meant to lose nothing: every delimiter, quoting always or where needed, LF, CRLF and CR line breaks, NULL written and read as `NULL`, and formulas left as they are. Row count and the text of every row have to match. A column of a kind `read_only_reason` names (bytes, arrays, spatial, variant and composite values) and a value `edit_text` gives no text for, such as a blob in a SQLite column typed by its values, have no CSV text the import reads back, so the SQL case covers them. - `sql_insert_export_runs_back_as_the_same_rows` exports each corpus column as SQL INSERT into an empty copy and runs the file as the editor runs one: split by the engine's grammar and sent a batch at a time. Every value has to come back with the same text and of the same kind. -- `tsv_copy_pastes_back_as_the_same_rows` writes each corpus column's rows as the grid's Copy does, as tab-separated text and as the cells it offers beside the text (`CellsEncoder`), reads each as a paste does (`PastedTable`) and saves the rows into an empty copy as a table tab saves new ones (`plan_save`). Rows are compared in `id` order without the `id`, since a paste leaves an id the server numbers, such as SQLite's row id, to the server. The columns and values the CSV case skips are skipped here too. +- `tsv_copy_pastes_back_as_the_same_rows` writes each corpus column's rows as the grid's Copy does, as tab-separated text and as the cells it offers beside the text (`CellsEncoder`), reads each as a paste does (`PastedTable`) and saves the rows into an empty copy as a table tab saves new ones (`plan_save`). Rows are compared in `id` order without the `id`, since a paste leaves an id the server numbers, such as SQLite's row id, to the server: by text for the text, and by text and kind for the cells, which keep how each value was stored. The columns and values the CSV case skips are skipped here too. - `csv_export_near_the_import_limit_imports_back`, with `near_limit: true`, fills a table through `LargeTableEngine` so its CSV under the export defaults lands within 1 MiB under `MAX_CSV_FILE_BYTES`, imports it into an empty copy, and compares a hash of every row. ClickHouse passes `false`: its import sends values in the request URL, 8 KiB a statement, so such a file takes some 8,000 statements and longer than a case may run. Each case has four minutes (`ROUND_TRIP_CASE_LIMIT`). Every target runs the suite, the MySQL target on MariaDB as well, and SQLite's runs in the unit suite.