Repository navigation
TextIOWrapper.truncate() truncates at the read-ahead position instead of the current position #161275
Description
Bug report
Bug description
:meth:IOBase.truncate is documented to "Resize the stream to the given size in bytes
(or the current position if size is not specified)". On a text stream neither
implementation does that:
_io.TextIOWrapper.truncate()forwardsposstraight to the buffer
(_io_TextIOWrapper_truncate_impl(),Modules/_io/textio.c), so with no argument the
file is cut wherever read-ahead left the binary stream — at a chunk boundary, i.e. at
a position that depends only on an internal buffer size._pyio.TextIOWrapper.truncate()passesself.tell()to the buffer, but a text-stream
cookie is an opaque number, not a byte offset. It happens to equal the byte offset in
the simple case, so the pure implementation usually does the right thing; as soon as
the decoder has state the cookie is huge andtruncate()raisesOverflowError.
import os
for n in (10, 20000):
with open("x", "w") as f:
f.write("x" * n)
with open("x", "r+") as f:
f.read(3)
f.truncate()
print(n, "->", os.path.getsize("x"))10 -> 10 # expected 3; nothing was truncated at all
20000 -> 8192 # expected 3; cut at the read-ahead chunk size
| file | operation | _io |
_pyio |
expected |
|---|---|---|---|---|
| 10 ascii chars | read(3); truncate() |
10 | 3 | 3 |
| 20000 ascii chars | read(3); truncate() |
8192 | 3 | 3 |
| 10 × U+1234, utf-8 | read(3); truncate() |
30 | 9 | 9 |
| 10 ascii chars, utf-16 | read(3); truncate() |
22 | 8 | 8 |
"a\rb\rc", universal newlines |
read(2); truncate() |
5 | OverflowError | 3 |
The last row is the cookie case: tell() returns
6277101735386680764176071790128604879584176795969512275970 there, and _pyio hands
that to buffer.truncate(). I found 145 such (encoding, newline, text, count)
combinations just among \r-containing inputs.
For comparison, on a binary stream the same sequence truncates at 3, f.truncate(f.tell())
on the text stream also truncates at 3, and the write path is unaffected
(f.write("abc"); f.truncate() → 3). So only the no-argument form after a read is wrong.
Suggested fix
Synchronise the binary buffer with the logical position first, then let the buffer
truncate at its own position. In Lib/_pyio.py:
def truncate(self, pos=None):
self.flush()
if pos is None:
- pos = self.tell()
+ # Synchronise the buffer with our logical position, then let it
+ # truncate there: a cookie from tell() is not a byte offset.
+ self.seek(self.tell())
+ return self.buffer.truncate()
return self.buffer.truncate(pos)With that, the five rows above give 3 / 3 / 9 / 8 / 3. Modules/_io/textio.c needs the
same two steps when pos is Py_None.
I can put up a PR for both implementations; I have only verified the pure-Python side so
far (I don't have a local build yet to run test_io against a C change).
CPython versions tested on
3.11, 3.13 — _io_TextIOWrapper_truncate_impl() and _pyio.TextIOWrapper.truncate() are
unchanged on main.
Operating systems tested on
macOS, but the code is platform independent.
Activity
bedevere-app commentedon Nov 3, 2026
Thank you for taking the time to open this issue. Python is a community-driven project, and we want it to be a welcoming place to learn and collaborate.
We expect issues to be explained in your own words. We believe this issue does not meet our guidelines for using AI tools, so we're closing it. If you think we've misunderstood, you're welcome to clarify the issue or ask questions about the guidelines.
Please read our guidelines for using AI tools.
tsayin4 commentedon Nov 3, 2026
Sorry, I let a tool polish the text and it came out reading like slop. Straight version:
I was comparing io and _pyio behaviour and hit this:
I expected 3. 8192 is just the read-ahead chunk size. With a 10 byte file I get 10, so nothing is cut at all. In binary mode the same code gives me 3, and
f.truncate(f.tell())on the text file also gives 3, so it's onlytruncate()with no argument after a read.I tried ascii, utf-8 with 3 byte chars, utf-16, and
\runder universal newlines. io gives 10, 8192, 30, 22, 5 and I expected 3, 3, 9, 8, 3. _pyio gives 3, 3, 9, 8 and then raises OverflowError on the\rone, which was odd enough that I went and read the C code:_io_TextIOWrapper_truncate_implinModules/_io/textio.cforwards pos to the buffer as is, so when pos is None the buffer truncates wherever it happens to be.English isn't my first language and I used an LLM to write the issue up, that part is on me. The numbers above are from my own runs. Could you reopen it?
tsayin4 commentedon Nov 4, 2026
I dug into why this happens and it turns out not to be a _pyio thing at all.
In the C version truncate() passes nothing down, so the file gets cut at wherever the
buffer happened to read ahead to instead of at the stream position. On a 5 byte file it
does nothing at all:
_pyio passes tell() to the buffer and gets 2, so the two implementations disagree. I put
up #161280 for the C side; the test in it fails on _io and passes on _pyio as is, so it
needs no _pyio change. Could this be reopened?
pythongh-161275: Fix TextIOWrapper.truncate() with no argument