Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 6 additions & 24 deletions docs/api/elementwise.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Elementwise Operators

Every op on this page is used the same way: construct it once, then call it. The
constructor takes what the kernel is compiled with; the call takes the tensors.
Both are documented under each op — `__init__` and `forward`, where `forward` is
what runs when you call `op(...)`.
constructor takes what the kernel is compiled with; the call takes the inputs, tensors
or numbers. Both are documented under each op — `__init__` and `forward`, where
`forward` is what runs when you call `op(...)`.

## Arithmetic

Expand Down Expand Up @@ -61,13 +61,7 @@ what runs when you call `op(...)`.
heading_level: 3
members: ["__init__", "forward"]

::: tileops.elementwise.LerpScalarFwdOp
options:
show_root_heading: true
heading_level: 3
members: ["__init__", "forward"]

::: tileops.elementwise.LerpTensorFwdOp
::: tileops.elementwise.LerpFwdOp
options:
show_root_heading: true
heading_level: 3
Expand Down Expand Up @@ -391,25 +385,13 @@ what runs when you call `op(...)`.
heading_level: 3
members: ["__init__", "forward"]

::: tileops.elementwise.ClampTensorFwdOp
options:
show_root_heading: true
heading_level: 3
members: ["__init__", "forward"]

::: tileops.elementwise.ClampScalarFwdOp
options:
show_root_heading: true
heading_level: 3
members: ["__init__", "forward"]

::: tileops.elementwise.MaskedFillTensorFwdOp
::: tileops.elementwise.ClampFwdOp
options:
show_root_heading: true
heading_level: 3
members: ["__init__", "forward"]

::: tileops.elementwise.MaskedFillScalarFwdOp
::: tileops.elementwise.MaskedFillFwdOp
options:
show_root_heading: true
heading_level: 3
Expand Down
2 changes: 1 addition & 1 deletion docs/api/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ a tool rather than an op, comes last. The Benchmarks pages use the same order.
| Page | What it covers |
| --- | --- |
| [Elementwise](elementwise.md) | unary and binary maps, activations, dropout, and the in-place forms |
| [RoPE](rope.md) | rotary position embedding — NeoX and interleaved layouts, Llama 3.1, YaRN, LongRoPE |
| [RoPE](rope.md) | rotary position embedding — NeoX and interleaved rotation, Llama 3.1, YaRN, LongRoPE |
| [Reduction](reduction.md) | sums, extrema, arg-reductions, cumulative scans, softmax |
| [Normalization](normalization.md) | RMSNorm, LayerNorm, GroupNorm, BatchNorm and the fused variants, including RMSNorm with per-token quantization and the q/k RMSNorm with RoPE |
| [Pooling](pool.md) | average, max and adaptive pooling, with and without indices, plus the chunked sequence mean |
Expand Down
12 changes: 3 additions & 9 deletions docs/api/rope.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,22 +7,16 @@ what runs when you call `op(...)`.

## Base frequencies

One op serves both rotation conventions: pass `rope_layout` as `"neox"` to
rotate the two halves of a head against each other, or `"interleaved"` to rotate
each adjacent pair.
One op serves both rotation conventions: pass `is_neox_style=True` to rotate the
two halves of a head against each other, or `is_neox_style=False` to rotate each
adjacent pair.

::: tileops.rope.RoPEFwdOp
options:
show_root_heading: true
heading_level: 3
members: ["__init__", "forward"]

::: tileops.rope.RoPENeoxPositionIdsFwdOp
options:
show_root_heading: true
heading_level: 3
members: ["__init__", "forward"]

## Scaled frequencies

::: tileops.rope.RoPELlama31FwdOp
Expand Down
19 changes: 13 additions & 6 deletions docs/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,14 @@ def build_kernel(*inputs: "TensorSpec | None", **params) -> Callable[..., Kernel

Implemented by the backend, one per `(op, target)`. Its signature is the op's manifest
signature: `inputs` correspond one-to-one to `signature.inputs` in declaration order, and
`params` are named after `signature.params`. An optional input that was not passed on
this call arrives as `None`.
`params` are named after `signature.params`. Each input arrives by what it holds on this
call:

- a tensor arrives as its `TensorSpec`;
- a tensor sequence (`type: "list[Tensor]"`) arrives as a tuple of `TensorSpec`s;
- a call-time scalar, such as `ClampFwdOp`'s `min` passed as a number, arrives as its
Python type (`int`, `float` or `bool`), never its value;
- an input that holds `None`, passed or left at its default, arrives as `None`.

```python
# GroupNormFwdOp's spec: weight and bias are optional, and arrive as None when absent
Expand Down Expand Up @@ -325,7 +331,8 @@ This call path implies four things:
decide which in-tree kernel is fetched, what it is looked up by and how it is built.
Once the op settles on a target, that target serves the whole op, and none of them runs.
- **Tensors arrive positionally, params by name.** `build_kernel(*inputs, **params)`: the
positional arguments are `TensorSpec`s (`None` for an optional input the call omitted),
positional arguments describe the inputs (`TensorSpec`, a tuple of them for a tensor
sequence, the Python type of a call-time scalar, `None` for an input that holds `None`),
and the keyword arguments are the manifest's `params` names with the values settled for
this call.
- **One builder per `(op, target)`.** The backend is not told which kernel the in-tree
Expand Down Expand Up @@ -494,7 +501,7 @@ signature:
eps: {type: "float | None", default: null}
inputs: # declaration order is call order
x: {dtype: T, shape: "[*B, *normalized_shape]"}
weight: {dtype: T, shape: "[*normalized_shape]", optional: true}
weight: {dtype: T, shape: "[*normalized_shape]", type: "Tensor | None", default: null}
```

The corresponding builder signature:
Expand Down Expand Up @@ -773,8 +780,8 @@ backend serves an op only when its `detect` claims the device, or when it is nam
TileOPs memoises a builder's return value by **device plus input signature**:

> the call's device (see [Call device](user-guide/manifest/calls.md#device)), plus `(dtype, shape)` taken per input in
> `signature.inputs` order; an optional input not passed on this call is recorded
> as `None`.
> `signature.inputs` order; an input that holds `None` on this call is recorded as `None`,
> and a call-time scalar is recorded by its Python type, never its value.

**Two calls with the same device and input signature get the same kernel**, with no
further call to `build_kernel`. A second card under the same target builds again, because
Expand Down
13 changes: 9 additions & 4 deletions docs/backends.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,12 @@ def detect(device: torch.device) -> bool:
def build_kernel(*inputs: "TensorSpec | None", **params) -> Callable[..., KernelResult]: ...
```

由 backend 实现,每组 `(op, target)` 一个。它的签名就是该 op 的 manifest 签名:`inputs` 按声明顺序与 `signature.inputs` 的条目一一对应,`params` 按 `signature.params` 命名。声明为 optional 的输入在本次调用中没有传入时,对应的实参是 `None`。
由 backend 实现,每组 `(op, target)` 一个。它的签名就是该 op 的 manifest 签名:`inputs` 按声明顺序与 `signature.inputs` 的条目一一对应,`params` 按 `signature.params` 命名。每个输入按它在本次调用中持有的值传入:

- 张量传入它的 `TensorSpec`;
- 张量序列(`type: "list[Tensor]"`)传入由 `TensorSpec` 组成的 tuple;
- 调用期标量(例如以数值传入的 `ClampFwdOp` 的 `min`)传入它的 Python 类型(`int`、`float` 或 `bool`),不传入它的值;
- 持有 `None` 的输入,无论是显式传入还是取了默认值,对应的实参是 `None`。

```python
# GroupNormFwdOp 的 spec:weight、bias 是可选输入,没传时实参是 None
Expand Down Expand Up @@ -263,7 +268,7 @@ register_kernel_builder(op="GEMMFwdOp", target="acme", build_kernel=build_gemm)
这条调用路径对应以下四点:

- **`kernel_for` 与各实现的 `entry_for` 只服务 in-tree 实现。** 它们决定取哪个 in-tree kernel、按什么查表以及如何构造。op 选中某个 target 之后,整个 op 由这个 target 服务,这几处都不会执行。
- **张量按位置传入,参数按名字传入。** 在 `build_kernel(*inputs, **params)` 中,位置实参是 `TensorSpec`(没有传入的可选输入是 `None`),关键字实参是 manifest 中 `params` 的名字与本次调用的确定值。
- **张量按位置传入,参数按名字传入。** 在 `build_kernel(*inputs, **params)` 中,位置实参描述各输入(`TensorSpec`;张量序列是由它组成的 tuple;调用期标量是它的 Python 类型;持有 `None` 的输入是 `None`),关键字实参是 manifest 中 `params` 的名字与本次调用的确定值。
- **一组 `(op, target)` 只注册一个 builder。** in-tree 路径会选用哪个实现(例如 GEMM 在 `kernel_types` 中声明的多个实现之一)不会告诉 backend,`build_kernel` 从 `TensorSpec` 自行判断应返回哪个 kernel。
- **backend 不必自己缓存构建结果。** 设备与输入签名相同时,op 层不会再次调用 `build_kernel`;需要更细的区分或更少的重建时,backend 在 `build_kernel` 内部另加一层缓存。专为外部 backend 编写的 op 既不声明 `kernel_types` 也不声明 `interfaces`,没有 target 认领设备时,调用直接抛出 `OpNotAvailableError`。

Expand Down Expand Up @@ -400,7 +405,7 @@ signature:
eps: {type: "float | None", default: null}
inputs: # 声明顺序即传入顺序
x: {dtype: T, shape: "[*B, *normalized_shape]"}
weight: {dtype: T, shape: "[*normalized_shape]", optional: true}
weight: {dtype: T, shape: "[*normalized_shape]", type: "Tensor | None", default: null}
```

对应的 builder 签名:
Expand Down Expand Up @@ -618,7 +623,7 @@ TileOPs 的 in-tree kernel([`src/tileops/_kernels/`](https://github.com/tile-a

TileOPs 按**设备加输入签名**记住 `build_kernel` 的返回值。这个键的构成是:

> 调用设备(见[调用设备](user-guide/manifest/calls.md#device)),加上按 `signature.inputs` 顺序逐项取出的 `(dtype, shape)`;声明为 optional 而本次没有传入的输入,这一项记 `None`。
> 调用设备(见[调用设备](user-guide/manifest/calls.md#device)),加上按 `signature.inputs` 顺序逐项取出的 `(dtype, shape)`;本次持有 `None` 的输入,这一项记 `None`;调用期标量按它的 Python 类型记录,不记录它的值。

因此,**设备与输入签名都相同的两次调用,TileOPs 使用同一个 kernel**,不再调用 `build_kernel`。同一个 target 的第二块卡会重新构造一次,因为为一块卡编译出的产物不一定能在另一块卡上启动。参数不进入这个键,因为它们对一个 op 实例而言是固定的。

Expand Down
2 changes: 1 addition & 1 deletion docs/new-op.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ class GEMMFwdOp(Op):
| 1 | `__init__` | the names, order and defaults in `signature.params`, then the keyword-only `target`; it assigns each manifest parameter to the attribute of the same name, then calls `super().__init__(target=target)` |
| 2 | `kernel_types` | the Kernel classes that can serve the op, each under a key |
| 3 | `interfaces` | one entry per kernel call the op makes, mapping the name `kernel_for` uses to the `KernelInterface` class that the implementations serving that call inherit |
| 4 | `forward` | the parameters: the order of `signature.inputs`, with optional inputs last and defaulting to `None`; the body: making the inputs contiguous, writing the function that builds the call spec, fetching the kernel and calling it |
| 4 | `forward` | the parameters: the order of `signature.inputs`, an input with a manifest `default` taking it as its default and following the inputs without one; the body: making the inputs contiguous, writing the function that builds the call spec, fetching the kernel and calling it |
| 5 | `roof_key` | optional: the hardware unit whose peak prices the op's FLOPs; the default is fp32 on CUDA cores, and the member is written only when the op uses another unit |

`_infer_output_shapes` and `eval_roofline` are generated from the spec and are not
Expand Down
2 changes: 1 addition & 1 deletion docs/new-op.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ class GEMMFwdOp(Op):
| 1 | `__init__` | `signature.params` 的名字、顺序与默认值,再加上仅限关键字的 `target`;先把每个 manifest 参数赋给同名属性,再调用 `super().__init__(target=target)` |
| 2 | `kernel_types` | 能服务这个 op 的 Kernel 类,各对应一个 key |
| 3 | `interfaces` | op 发出的每一个 kernel 调用各占一条,从 `kernel_for` 使用的名字映射到服务这个调用的各实现所继承的 `KernelInterface` 类 |
| 4 | `forward` | 参数按 `signature.inputs` 的顺序,可选输入排在最后,默认值为 `None`;方法体把输入转为连续张量,写出构造 call spec 的函数,取得 kernel,再调用它 |
| 4 | `forward` | 参数按 `signature.inputs` 的顺序,有 manifest `default` 的输入以它为默认值,排在没有 `default` 的输入之后;方法体把输入转为连续张量,写出构造 call spec 的函数,取得 kernel,再调用它 |
| 5 | `roof_key` | 可选。表示 op 的 FLOPs 按哪个硬件单元的峰值计算,默认是 CUDA core 上的 fp32,只在使用其他单元时编写 |

`_infer_output_shapes` 与 `eval_roofline` 都依照 spec 生成,不需要编写。
Expand Down
Loading
Loading