Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions client.go
Original file line number Diff line number Diff line change
Expand Up @@ -1559,12 +1559,23 @@ func (f *File) WriteTo(w io.Writer) (written int64, err error) {

// Reduce: serialize the results from the reads into sequential writes.
cur := writeCh
shortRead := false
for {
packet, ok := <-cur
if !ok {
return written, errors.New("sftp.File.WriteTo: unexpectedly closed channel")
}

// The reads are dispatched at fixed offsets (off, off+chunkSize, ...), so a
// server returning fewer bytes than requested mid-stream leaves a gap that the
// following chunk cannot fill. If a short read is followed by another chunk that
// still has data, the stream can no longer be reassembled, so fail loudly rather
// than silently drop the skipped bytes. A short final chunk is fine: it is
// followed only by the EOF packet, which carries no data.
Comment on lines +1569 to +1574

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[style] Do not break lines just to keep a specific line length: https://go.dev/wiki/CodeReviewComments#line-length

If you’re looking for guidance on how I recommend breaking up comments into multiple lines, I recommend Semantic Breaking. The additional advantage of semantic breaking of comments is that editing of a single idea in the comment will not cause a cascade formatting change of the rest of the comment, which generates unnecessary PR churn.

if shortRead && len(packet.b) > 0 {
return written, errors.New("sftp: server returned a short read mid-stream, cannot reassemble concurrent WriteTo")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest better "concurrent WriteTo was aborted" rather than “cannot reassemble concurrent WriteTo” The later says nothing especially useful about what happened, the former says definitively: I refused to continue under these conditions.

}

// Because writes are serialized, this will always be the last successfully read byte.
f.offset = packet.off + int64(len(packet.b))

Expand All @@ -1584,6 +1595,10 @@ func (f *File) WriteTo(w io.Writer) (written int64, err error) {
return written, packet.err
}

if len(packet.b) < chunkSize {
shortRead = true
}

pool.Put(packet.b)
cur = packet.next
}
Expand Down
67 changes: 67 additions & 0 deletions client_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -198,3 +198,70 @@ func TestClientNoSid(t *testing.T) {
t.Fatal("expected ErrSSHFxConnectionLost, got", err)
}
}

// Issue #658: the concurrent File.WriteTo path (used by io.Copy) must not
// silently drop data when the server returns short reads. A server is free to
// return fewer bytes than asked for, which it will whenever the client's max
// packet size is larger than the server's. Because the concurrent path
// dispatches reads at fixed offsets, it cannot reassemble the stream across a
// mid-stream short read, so it must fail loudly rather than return a truncated
// copy that looks successful.
func TestClientWriteToShortReads(t *testing.T) {
cr, sw := io.Pipe()
sr, cw := io.Pipe()

// The default server max packet size is 32768, so a bigger client packet
// size makes every read come back short.
server, err := NewServer(struct {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this test runs against a server, it is properly an integration test, not a quick test. Since it only runs against our own server implementation, it should also not run when we’re running with -testserver.

io.Reader
io.WriteCloser
}{sr, sw})
if err != nil {
t.Fatal(err)
}
go server.Serve()

client, err := NewClientPipe(cr, cw, MaxPacketUnchecked(128*1024))
if err != nil {
t.Fatal(err)
}
// Close the client first (LIFO), so its receive loop sees the server go away.
defer client.Close()
defer server.Close()
Comment on lines +228 to +230

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is misleading. defer statements are executed as noted in LIFO order. So, we are not closing the client first, we are setting up the client to close after the server.

Rework this to a defer func() { … }() so that the proper intended ordering off the instruction is both clear, and identical to the natural order a human would expect events to happen in.


// Bigger than the client packet size so WriteTo takes the concurrent path,
// and not a multiple of it so the last chunk is partial as well.
want := make([]byte, 5*128*1024+123)
for i := range want {
want[i] = byte(i)
}

tmp, err := os.CreateTemp("", "sftp-writeto-shortread")
if err != nil {
t.Fatal(err)
}
defer os.Remove(tmp.Name())
if _, err := tmp.Write(want); err != nil {
t.Fatal(err)
}
if err := tmp.Close(); err != nil {
t.Fatal(err)
}

f, err := client.Open(tmp.Name())
if err != nil {
t.Fatal(err)
}
defer f.Close()

var buf bytes.Buffer
n, err := f.WriteTo(&buf)
if err == nil {
t.Fatalf("WriteTo succeeded but should have reported the short read; wrote %d of %d bytes", n, len(want))
}
Comment on lines +259 to +261

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We’ve setup the current conditions that would cause this error to arise, but we’re inherently relying here upon the implementation of the server.

Reasonably, there should be a fix to the server that ensures we do not generally return short packets. In such a case, this test would then fail, because the underlying issue was addressed on both ends: the client returns an error on a short read, but the server also does not return short reads in the first place.

// Whatever was written must be a correct prefix of the source: we may stop
// early, but we must never emit misaligned or skipped bytes.
if !bytes.Equal(buf.Bytes(), want[:buf.Len()]) {
t.Errorf("WriteTo produced %d bytes that are not a prefix of the source", buf.Len())
}
}