all.equal.data.table throws "Dataset 'current' has rows not present in 'target'" when double precision values are numerically, but not exactly, equal

[kjellpk]

When ignore.row.order = TRUE, all.equal.data.table throws' "Dataset 'current' has rows not present in 'target'" when there are multiple double precision columns and any but the last one are not perfectly equal.

> library(data.table)
data.table 1.18.4 using 5 threads (see ?getDTthreads).  Latest news: r-datatable.com

Attaching package: ‘data.table’

The following object is masked from ‘package:base’:

    %notin%

> eps <- sqrt(.Machine$double.eps) / 3.0
>
> x <- data.table(a = "a", ione = 1L, done = 1.0, dtwo = 2.0)
> y <- copy(x)
> y[, done := done - eps]
> all.equal(x, y, ignore.row.order = TRUE)
[1] "Dataset 'current' has rows not present in 'target'"

[kjellpk]

The test is at line 244 of setops.R:

ans1 = target[current, roll=tolerance, rollends=TRUE, which=TRUE, on=jn.on]

[MichaelChirico]

Importantly, all.equal.data.frame() finds them equal:

x <- data.frame(v = 1.0)
y <- transform(x, v=v - sqrt(.Machine$double.eps) / 3.0)
all.equal(x,y)
# [1] TRUE

[jangorecki]

There is related issue: #7434

I have a wip branch to rewrite data.table's method to follow "data.frame way" (although it doesn't have ignore.row.order feature), so instead of rolling joins, to sort the data and compare using all.equal.list.
I believe it will break a lot of downstream code so I somehow haven't made it ready enough to open PR.
If we agree on replacing rolling joins with sort + all.equal.list then I am happy to submit it.
Optionally we could have a new function for that, as there are few other useful things that could be plugged in.