Skip to content

Commit 6cdd4d7

Browse files
authored
Merge pull request #299 from thisisthepy/feat/tp-buffer
TypedPython: memoryview buffer parameters, BufferError on resize (#149)
2 parents 2103f93 + c3a74c1 commit 6cdd4d7

9 files changed

Lines changed: 635 additions & 11 deletions

File tree

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
import array
2+
import sys
3+
4+
5+
def bump_list(a: list[float], i: int) -> None:
6+
a[i] += 1.0
7+
8+
9+
def bump_array(a: array.array[float], i: int) -> None:
10+
a[i] += 1.0
11+
12+
13+
def bump_view(a: memoryview[float], i: int) -> None:
14+
a[i] += 1.0
15+
16+
17+
def run_list(a: list[float], calls: int) -> None:
18+
n: int = len(a)
19+
for k in range(calls):
20+
bump_list(a, (k * 7919) % n)
21+
22+
23+
def run_array(a: array.array[float], calls: int) -> None:
24+
n: int = len(a)
25+
for k in range(calls):
26+
bump_array(a, (k * 7919) % n)
27+
28+
29+
def run_view(a: memoryview[float], calls: int) -> None:
30+
n: int = len(a)
31+
for k in range(calls):
32+
bump_view(a, (k * 7919) % n)
33+
34+
35+
def main() -> None:
36+
# argv[1] = number of bump calls, argv[2] = array size (default 1000000). Each call touches one element
37+
# of a large buffer: a list argument is copied in and written back on every call (O(n)), an exact
38+
# array.array / memoryview argument is used in place (SPEC N-8, #149). Output: one checksum per variant.
39+
calls: int = int(sys.argv[1])
40+
size: int = int(sys.argv[2]) if len(sys.argv) > 2 else 1000000
41+
as_list: list[float] = [0.0] * size
42+
as_array: array.array[float] = array.array("d", [0.0]) * size
43+
backing: array.array[float] = array.array("d", [0.0]) * size
44+
as_view: memoryview[float] = memoryview(backing)
45+
run_list(as_list, calls)
46+
run_array(as_array, calls)
47+
run_view(as_view, calls)
48+
print(f"{sum(as_list)} {sum(as_array)} {sum(backing)}")
49+
50+
51+
if __name__ == "__main__":
52+
main()

‎docs/SPEC.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -688,7 +688,7 @@ All in `PM/commonTest`, so they run wherever the interpreter loads.
688688
| Array access in bounds, or a runtime bounds check kept | the check stays (never an unchecked access) |
689689
| Array parameters only indexed, stored, `len`-ed; no aliasing (entry guard) | not compiled; diagnostic |
690690
| Integer ops proven safe by the verifier's intervals (constants, `len`, `range` variables, guards such as `if x < e` / `while lo < hi`, joined local assignments, `//` and `%` by constants) | emitted unchecked (a shift/mask for `//`/`%` by a power of two of a non-negative value); the sanitizer CI run builds with `TP_CHECK_PROVEN` and turns a wrong proof into a SystemError (#148) |
691-
| `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. Open (#149): while the export is held, another thread's resize of that array raises BufferError. numpy arrays: see the next row | anything else: object path |
691+
| `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. While the export is held, a resize of that array (`append`, `extend`, `del a[i]`, ... from another thread) raises BufferError, exactly as CPython does for any held export; the compiled call never sees a resize. Concurrent writes from another thread to the same elements are not synchronised: as in CPython, no ordering between the two threads' element accesses is guaranteed, only that each access is whole (no torn element). `memoryview[float]` / `memoryview[int]` parameters (#149: exact `memoryview`, 1-D, C-contiguous, format `d` or `q`/`l`, 8-byte aligned, writable when stored to; any exporter: an `array.array`, a numpy array, a `bytearray` cast with `.cast('d')`) take the same path with array.array's element semantics (an element is a Python float / int); the export is taken on the memoryview, so while the call runs `release()` on it raises BufferError, as CPython does for an exported view. An out-of-range index raises memoryview's `IndexError("index out of bounds on dimension 1")`. A released, read-only (when stored to), non-contiguous, multi-dimensional or other-format view deopts: the interpreter raises or runs it. numpy arrays: see the next row | anything else: object path |
692692
| numpy `npt.NDArray[np.float64]` / `NDArray[np.int64]` parameters (exact `numpy.ndarray`, 1-D, C-contiguous, native byte order, writable when stored to) take the same buffer path (#220, step 1) when the function accesses the array purely by element (`x[i]`, `len(x)`); a function that uses it as a whole object (`np.sum(x)`, `x.T`, passing `x` on, a slice) compiles as before with the parameter an opaque object. `NDArray[np.bool_]` (#261) is the same path with 1-byte items (buffer format `?`; a `uint8` or other array deopts): an element read is a native bool, boxed as `np.bool` when it escapes; a store compiles only for a bool value (`x[i] = v`, `v: bool`), because numpy converts an int or float by truthiness and that function stays interpreted. Other dtypes are opaque objects. Results follow the interpreter: a value derived from an element is a numpy scalar there, so it is boxed as `np.float64` / `np.int64` / `np.bool` when it escapes (return, `Box`), decided per value at run time by a taint flag; a function declared `-> float` / `int` / `bool` that holds a numpy array returns the interpreter's object. `float(v)` of such a value is a plain float. An out-of-range index raises numpy's `IndexError("index i is out of bounds for axis 0 with size n")`. Arithmetic on a value derived from an element follows numpy's scalar semantics (#220, step 2): float `/`, `//`, `%` never raise ZeroDivisionError (inf / nan, with numpy's result bits and sign rules, `//` and `%` as numpy's `divmod`), int64 `//` and `%` by zero give 0, int64 `/` converts both operands to double. numpy's RuntimeWarning (category and text exactly as numpy's, e.g. `overflow encountered in scalar add`, `divide by zero encountered in scalar divide`, `invalid value encountered in scalar floor_divide`) is emitted once per flagged operation, with the source file and line of the operation. A pure function does not emit it itself: on any floating-point exception (overflow, invalid, divide by zero) or int64 wrap or division by zero it deopts, so the interpreter emits it; an impure function cannot deopt after an effect, so the compiled code emits the warning (`PyErr_WarnExplicitObject`, honouring `warnings` filters and the module's `__warningregistry__`). Float `+ - * / // %` on such values is compiled in both kinds of function, including a store of the result into the array (`x[i] -= d * m`); int64 `/ // %` is compiled in a pure function only. Int64 `+ - *` and `// %` of an element in an impure function (a native INT would have to be a numpy scalar) stay on the object path | anything else: object path |
693693
| A function holding an array parameter runs no user code: closed, and no slot read or `C(args)` whose slow path runs user code, in its whole call tree (#203) | not compiled; diagnostic |
694694
| OBJ references follow one ownership rule; no IR form for free, borrowed alias or NULL read | not compiled; diagnostic |

‎python-multiplatform-ksp/src/main/python/typedpython/cgen.py‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@ class CGenError(Exception):
186186
_ARRAY_CT = {Type.F64_ARRAY: "tp_f64_array", Type.I64_ARRAY: "tp_i64_array"}
187187
# Buffer arrays (#149): `ArrayParam.buffer` -> the exact type the guard compares against (the index
188188
# into the module state's `buf_types`), and the format strings the export must carry.
189-
_BUFFER_KINDS = {None: None, "array": 0, "numpy": 1}
189+
_BUFFER_KINDS = {None: None, "array": 0, "numpy": 1, "memoryview": None}
190190
# buf_types slots 2..4: the numpy scalar types a Box.np makes (np.float64, np.int64, np.bool_)
191191
_NP_SLOT = {Type.F64: 2, Type.I64: 3, Type.BOOL: 4}
192192
_BUFFER_FMT = {Type.F64_ARRAY: ('"d"', "NULL"), Type.I64_ARRAY: ('"q"', '"l"'), Type.BOOL_ARRAY: ('"?"', "NULL")}
@@ -2784,9 +2784,10 @@ def wrapper(self) -> str:
27842784
for i, p in buffers:
27852785
fmt1, fmt2 = _BUFFER_FMT[p.type]
27862786
kind = _BUFFER_KINDS[p.buffer]
2787-
enter = "tp_buf_enter_np" if p.buffer == "numpy" else "tp_buf_enter"
2788-
L.append(f" tp_s = {enter}(tp_args[{i}], "
2789-
f"((tp_state *)PyModule_GetState(tp_module))->buf_types[{kind}], "
2787+
enter = {"numpy": "tp_buf_enter_np", "memoryview": "tp_buf_enter_mv"}.get(p.buffer, "tp_buf_enter")
2788+
exact = ("(PyObject *)&PyMemoryView_Type" if p.buffer == "memoryview"
2789+
else f"((tp_state *)PyModule_GetState(tp_module))->buf_types[{kind}]")
2790+
L.append(f" tp_s = {enter}(tp_args[{i}], {exact}, "
27902791
f"{fmt1}, {fmt2}, {int(p.stored)}, &tp_a{i});")
27912792
L.append(" if (tp_s < 0) goto tp_error;")
27922793
L.append(" if (tp_s != 0) goto tp_deopt;")

‎python-multiplatform-ksp/src/main/python/typedpython/frontend.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -326,7 +326,7 @@ def _qualified(node: ast.expr, info: _ModuleInfo) -> str | None:
326326
def _buffer_kind(annotation: ast.expr, info: _ModuleInfo) -> tuple[Type, str] | None:
327327
"""The buffer-array reading of an annotation (#149), else None (an ordinary object parameter):
328328
`array.array[float]` / `array.array[int]` (any spelling that resolves, through this module's
329-
own imports, to `array.array`). A bare `array.array`, another element type, a numpy
329+
own imports, to `array.array`), and the builtin `memoryview[float]` / `memoryview[int]`. A bare `array.array`, another element type, a numpy
330330
`NDArray[...]` (not supported: its element arithmetic is numpy's), or a name bound more than
331331
once is not one."""
332332
if not isinstance(annotation, ast.Subscript):
@@ -336,6 +336,10 @@ def _buffer_kind(annotation: ast.expr, info: _ModuleInfo) -> tuple[Type, str] |
336336
if base == "array.array" and isinstance(arg, ast.Name) and arg.id in ("float", "int") \
337337
and not info.bound(arg.id):
338338
return ARRAY_KIND[f"list[{arg.id}]"], "array"
339+
if isinstance(annotation.value, ast.Name) and annotation.value.id == "memoryview" \
340+
and not info.bound("memoryview") and isinstance(arg, ast.Name) \
341+
and arg.id in ("float", "int") and not info.bound(arg.id):
342+
return ARRAY_KIND[f"list[{arg.id}]"], "memoryview"
339343
if base == "numpy.typing.NDArray":
340344
dtype = _qualified(arg, info)
341345
if dtype in _NUMPY_DTYPE:

‎python-multiplatform-ksp/src/main/python/typedpython/runtime/API.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,8 +95,9 @@ An exact `array.array` argument (`array.array[float]` / `array.array[int]`; the
9595
through the module's own top-level imports) is not copied.
9696
typedef struct { Py_buffer view; Py_ssize_t len; void *data; int held; } tp_buf;
9797
int tp_buf_enter(PyObject *obj, PyObject *exact, const char *fmt1, const char *fmt2, int writable, tp_buf *b);
98+
int tp_buf_enter_mv(PyObject *obj, PyObject *exact, const char *fmt1, const char *fmt2, int writable, tp_buf *b); /* memoryview (#149): memoryview's IndexError text */
9899
void tp_buf_exit(tp_buf *b); /* PyBuffer_Release if held; zeroes b; idempotent */
99-
int tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t *slot); /* 0 / -1, array.array's IndexError text */
100+
int tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t *slot); /* 0 / -1, array.array's, numpy's or memoryview's IndexError text */
100101
double tp_buf_ld_f64(const tp_buf *b, Py_ssize_t i); void tp_buf_st_f64(const tp_buf *b, Py_ssize_t i, double v);
101102
int64_t tp_buf_ld_i64(const tp_buf *b, Py_ssize_t i); void tp_buf_st_i64(const tp_buf *b, Py_ssize_t i, int64_t v);
102103
int tp_import_attr(const char *mod, const char *attr, PyObject **out); /* exec slot: new ref or NULL (cleared); -1 MemoryError */

‎python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1224,8 +1224,22 @@ static inline int tp_buf_enter_np(PyObject *obj, PyObject *exact, const char *fm
12241224
return rc;
12251225
}
12261226

1227+
/* A memoryview (#149): the same guard on the exact builtin memoryview (the export is taken on the
1228+
* view itself, so its release() raises BufferError while the call runs); `np` 2 selects
1229+
* memoryview's IndexError text. Elements are Python floats / ints, as for array.array. */
1230+
static inline int tp_buf_enter_mv(PyObject *obj, PyObject *exact, const char *fmt1, const char *fmt2,
1231+
int writable, tp_buf *b)
1232+
{
1233+
int rc = tp_buf_enter(obj, exact, fmt1, fmt2, writable, b);
1234+
if (rc == 0) {
1235+
b->np = 2;
1236+
}
1237+
return rc;
1238+
}
1239+
12271240
/* index: a negative index counts from the end; out of range -> IndexError with the text CPython's
1228-
* own array.array uses ("array index out of range" / "array assignment index out of range") */
1241+
* own array.array uses ("array index out of range" / "array assignment index out of range"),
1242+
* numpy's (np 1) or memoryview's (np 2) */
12291243
static inline int tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t *slot)
12301244
{
12311245
int64_t orig = i;
@@ -1239,6 +1253,10 @@ static inline int tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t
12391253
*slot = (Py_ssize_t)i;
12401254
return 0;
12411255
out_of_range:
1256+
if (b->np == 2) {
1257+
PyErr_SetString(PyExc_IndexError, "index out of bounds on dimension 1");
1258+
return -1;
1259+
}
12421260
if (b->np) {
12431261
PyErr_Format(PyExc_IndexError, "index %lld is out of bounds for axis 0 with size %zd",
12441262
(long long)orig, b->len);

‎python-multiplatform-ksp/src/main/python/typedpython/verify.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@
4545
verify/redo `Call.redo` where it is not legal (see below)
4646
verify/entry-global-open a function with entry_globals is not closed
4747
verify/array-open a function with an ArrayParam runs user code (#203)
48-
verify/buffer a malformed `ArrayParam.buffer` (only None and "array"), or a
48+
verify/buffer a malformed `ArrayParam.buffer` (None, "array", "numpy", "memoryview"), or a
4949
CopyArray of a buffer array (#149)
5050
verify/par-loop a `ForRange.par` (SPEC N-15, #277) that the verifier does not recompute to
5151
the same value: the function is not marked `parallel`, or the loop is
@@ -450,7 +450,7 @@ def declarations(self) -> frozenset[str]:
450450
self.error(TYPE, f"array parameter '{p.name}' has non-array type {p.type}")
451451
elif not isinstance(p.stored, bool):
452452
self.error(TYPE, f"array parameter '{p.name}' has a non-bool `stored`")
453-
elif p.buffer not in (None, "array", "numpy"):
453+
elif p.buffer not in (None, "array", "numpy", "memoryview"):
454454
self.error(BUFFER, f"array parameter '{p.name}' has buffer kind {p.buffer!r}")
455455
else:
456456
self.arrays[p.name] = p.type

‎python-multiplatform-ksp/src/test/python/test_buffer_arrays.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -472,7 +472,7 @@ def test_verifier_buffer_kind_and_copy_rules():
472472
from test_verify import F64 as VF64, F64A as VF64A, assert_rejected, fn
473473
from typedpython.ir import ArrayParam, Assign, Const, CopyArray, Index, Return, Type
474474

475-
for kind in ("ndarray", "memoryview"): # "numpy" is a buffer kind since #220
475+
for kind in ("ndarray", "bytes"): # "numpy" (#220) and "memoryview" (#149) are buffer kinds
476476
assert_rejected("verify/buffer", fn("f", params=[ArrayParam("a", VF64A, False, kind)],
477477
returns=VF64, pure=True,
478478
body=[Return(Index(VF64, "a", Const(Type.I64, 0)))]))
@@ -483,6 +483,8 @@ def test_verifier_buffer_kind_and_copy_rules():
483483
import dataclasses
484484
assert_rejected("verify/buffer", dataclasses.replace(
485485
copy, params=[ArrayParam("a", VF64A, False, "numpy")]))
486+
assert_rejected("verify/buffer", dataclasses.replace(
487+
copy, params=[ArrayParam("a", VF64A, False, "memoryview")]))
486488

487489

488490
def test_verifier_buffer_params_are_not_limited_but_lists_still_are():

0 commit comments

Comments
 (0)