diff --git a/README.md b/README.md index ec9d522..5bd1f43 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,9 @@ Everything in them is real. The tokens are what `tokenize` returns and the instr |---|---|---| | a01 | [One line of Python, seven stages](anim/rendered/a01-seven-stages.gif) | T01 | | a02 | [A name is a label, not a box](anim/rendered/a02-a-name-is-a-label.gif) | T08 | +| a03 | [The stack machine](anim/rendered/a03-the-stack-machine.gif) | T07 | +| a04 | [How a dict finds a key](anim/rendered/a04-how-a-dict-finds-a-key.gif) | T08 | +| a05 | [A cycle, and what frees it](anim/rendered/a05-a-cycle-and-the-collector.gif) | T09 | They are all drawn from the same fifteen shapes, listed with what each one means in [xraymanim/VISUAL-SYSTEM.md](xraymanim/VISUAL-SYSTEM.md), and adding a sixteenth needs an amendment to that document first. That rule is not a convention somebody has to remember: the shape list is code, a storyboard that names a shape outside it fails, and a shape that is drawable but undescribed fails too. [anim/README.md](anim/README.md) has the index and how to render them. diff --git a/anim/README.md b/anim/README.md index e8e0a99..d8deb76 100644 --- a/anim/README.md +++ b/anim/README.md @@ -10,6 +10,9 @@ Everything in them is real. The tokens are what `tokenize` returns, the instruct |---|---|---|---| | a01 | [One line of Python, seven stages](rendered/a01-seven-stages.gif) | T01 | 40s | | a02 | [A name is a label, not a box](rendered/a02-a-name-is-a-label.gif) | T08 | 34s | +| a03 | [The stack machine](rendered/a03-the-stack-machine.gif) | T07 | 38s | +| a04 | [How a dict finds a key](rendered/a04-how-a-dict-finds-a-key.gif) | T08 | 41s | +| a05 | [A cycle, and what frees it](rendered/a05-a-cycle-and-the-collector.gif) | T09 | 36s | ## a01, one line of Python, seven stages @@ -23,6 +26,24 @@ Everything in them is real. The tokens are what `tokenize` returns, the instruct `a = []`, then `b = a`, then `del a`. Two names, one list, and a count on the object that says how many names are pointing at it. When the count reaches zero the memory goes back. This is the single most useful thing to understand about Python values, and it is much easier to see moving than described. +## a03, the stack machine + +![one call to a small function, run one instruction at a time, with the value stack drawn beside it](https://raw.githubusercontent.com/tamnd/cpython-internals/main/anim/rendered/a03-the-stack-machine.gif) + +`def area(w): return w * 2 + 1`, called with 6. A disassembly listing tells you what each instruction is called and nothing about what it does to the pile of values underneath, which is the part that is actually hard. So the pile is on screen: seven instructions, the stack going up to two and back down to one twice, and the frame disappearing when the answer is handed back. The instructions and the depths are from `pyxray.stack.walk` on 3.15.0rc1, which is why `LOAD_FAST_BORROW` is there instead of the `LOAD_FAST` you might expect. + +## a04, how a dict finds a key + +![a dict drawn as an eight slot index array beside three entries, with a lookup that collides and probes again](https://raw.githubusercontent.com/tamnd/cpython-internals/main/anim/rendered/a04-how-a-dict-finds-a-key.gif) + +A dict is not one table. Since 3.6 it has been a small array of slot numbers and a separate list of entries in the order you wrote them, and both of the questions people ask about dicts follow from that split: why the order is kept, and what a collision costs. Here `d[9]` lands on slot 1, finds the wrong key, probes again, and finds the right one on the second look. The eight slot array on screen was read out of a live interpreter, and there is a test that reads it again and fails if it ever differs. + +## a05, a cycle, and what frees it + +![two objects pointing at each other, their counts falling to one and staying there, and the collector taking them](https://raw.githubusercontent.com/tamnd/cpython-internals/main/anim/rendered/a05-a-cycle-and-the-collector.gif) + +Reference counting is right nearly all of the time, so the interesting question is when it is not. Two objects that hold each other: take the names away, both counts drop to one, and one is not zero, so nothing is freed and nothing can reach them either. The second half is what the collector does about it, which is smaller than people expect. It copies each count, subtracts one for every reference it finds inside the group, and anything whose copy reaches zero was only ever being kept alive from inside. + ## How they are made Each `aNN_*.py` here is a manim scene built out of the shared library in [xraymanim](../xraymanim), which is what stops a hundred animations from each inventing their own way to draw a pointer. The shapes and what they mean are in [VISUAL-SYSTEM.md](../xraymanim/VISUAL-SYSTEM.md). diff --git a/anim/a03_the_stack_machine.py b/anim/a03_the_stack_machine.py new file mode 100644 index 0000000..e5de9e6 --- /dev/null +++ b/anim/a03_the_stack_machine.py @@ -0,0 +1,120 @@ +"""One call to a two line function, one instruction at a time, with the stack on screen. + +The hard part of the eval loop for a beginner is not the loop, it is the stack. Reading a +disassembly listing tells you nothing about it, because the listing shows what each +instruction is called and not what it does to the pile of values underneath. So this shows +the pile, and lets the listing sit above it with a pointer walking along. + +The function is `def area(w): return w * 2 + 1`, called with 6, which gives 13. It was +chosen for its stack profile rather than for what it computes: the depth goes 0, 1, 2, 1, 2, +1, and those two climbs and two drops are the whole idea. + +The instructions and the depths came from `pyxray.stack.walk` on 3.15.0rc1. `LOAD_FAST_BORROW` +really is what the compiler emits for `w` here rather than `LOAD_FAST`, and it is left as it +is, because a reader who disassembles this function themselves should see what we showed them. +""" + +from __future__ import annotations + +from manim import DOWN, RIGHT, FadeIn, FadeOut, Transform + +from xraymanim.catalogue import THE_STACK_MACHINE +from xraymanim.grammar import UNIT +from xraymanim.mobjects import CodeStrip, Frame +from xraymanim.primitives import box +from xraymanim.scene import Explainer + +SOURCE = "def area(w): return w * 2 + 1" + +#: The instructions as `dis` prints them, in order, from a real 3.15.0rc1 run. +INSTRUCTIONS = ( + "RESUME 0", + "LOAD_FAST_BORROW w", + "LOAD_SMALL_INT 2", + "BINARY_OP *", + "LOAD_SMALL_INT 1", + "BINARY_OP +", + "RETURN_VALUE", +) + +#: What the value stack holds after each of those instructions, first item at the bottom of +#: the pile. The depths are from `pyxray.stack.walk`, so they are the compiler's own numbers +#: rather than a guess, and the values are what a call with w = 6 actually pushes. +STACKS = ( + (), + ("6",), + ("6", "2"), + ("12",), + ("12", "1"), + ("13",), + ("13",), +) + +SOURCE_Y = 2.35 +CODE_Y = 0.95 +FRAME_X = -0.4 +RETURN_X = 4.4 +FLOOR_Y = -2.7 + + +class A03TheStackMachine(Explainer): + storyboard = THE_STACK_MACHINE + + def construct(self) -> None: + source = box(SOURCE, tone="input", width=7.4, height=0.7, mono=True) + source.move_to([0, SOURCE_Y, 0]) + + self.strip = CodeStrip(INSTRUCTIONS, at=0, rows=2) + self.strip.scale_to_fit_width(11.0).move_to([0, CODE_Y, 0]) + self.play(FadeIn(source), FadeIn(self.strip), run_time=0.6) + + # The frame arrives on the first beat rather than with the code, because the first + # caption is about the call happening. Before the call there is no frame, and a + # picture that shows one anyway has answered a question nobody asked yet. + self.frame = self.frame_at(0) + seconds = self.beat() + self.play(FadeIn(self.frame), run_time=0.8) + self.wait(max(seconds - 0.8, 0.4)) + + # One beat per instruction, except the last. RETURN_VALUE gets its own ending, + # because what it does is take the frame away, and that is not a change to the + # picture the loop knows how to make. + for index in range(len(INSTRUCTIONS) - 1): + self.step(index) + + returned = box( + "13", tone="focus", width=2.4, height=0.8, mono=True, note="back to the caller" + ) + returned.move_to([RETURN_X, FLOOR_Y + 1.2, 0]) + seconds = self.beat() + self.play( + Transform(self.strip.pointer, self.strip.at(len(INSTRUCTIONS) - 1)), + FadeOut(self.frame, shift=DOWN * UNIT), + FadeIn(returned, shift=RIGHT * UNIT), + run_time=1.0, + ) + self.wait(max(seconds - 1.0, 0.6)) + + def step(self, index: int) -> None: + """Run one instruction: move the pointer, and redraw the frame with the new stack.""" + seconds = self.beat() + wanted = self.frame_at(index) + self.play( + Transform(self.strip.pointer, self.strip.at(index)), + FadeOut(self.frame), + run_time=min(seconds * 0.3, 0.6), + ) + self.frame = wanted + self.play(FadeIn(self.frame), run_time=0.4) + self.wait(max(seconds - 1.0, 0.4)) + + def frame_at(self, index: int) -> Frame: + """The frame as it stands after instruction `index`, sitting on a floor that stays put. + + A new one each time rather than one that is edited, because the frame gets taller as + the stack does, and cross fading two of them keeps the floor still while the top of + the box moves. That is the right way round: a value stack grows upward. + """ + made = Frame("area", {"w": "6"}, STACKS[index], width=6.2) + made.scale(0.8).move_to([FRAME_X, FLOOR_Y, 0], aligned_edge=DOWN) + return made diff --git a/anim/a04_how_a_dict_finds_a_key.py b/anim/a04_how_a_dict_finds_a_key.py new file mode 100644 index 0000000..c2238f2 --- /dev/null +++ b/anim/a04_how_a_dict_finds_a_key.py @@ -0,0 +1,111 @@ +"""What happens between `d[9]` and the answer, in a dict that has a collision in it. + +Most pictures of a dict draw one table of key and value pairs, and that picture cannot +explain either of the two things people actually ask about dicts: why they keep insertion +order, and what a collision costs. CPython has not stored a dict that way since 3.6. There +is a small array of slot numbers, and there are the entries in the order they went in, and +almost everything interesting follows from the split. + +The dict is `{1: "one", 5: "five", 9: "nine"}`, chosen because integers hash to themselves, +so nothing here depends on the hash randomization that would make string keys land in a +different slot on every run. With eight slots the mask is 7, key 1 wants slot 1, key 5 wants +slot 5, and key 9 wants slot 1 as well, which is the collision. + +The index array on screen is `[-1, 0, -1, -1, -1, 1, 2, -1]`, and that is not derived from +the probing rule, it was read out of a live 3.15.0rc1 interpreter through `ctypes` at the +address of the dict. The tests next to this file do the same read and would fail if CPython +ever laid it out differently. +""" + +from __future__ import annotations + +from manim import DOWN, LEFT, RIGHT, FadeIn, FadeOut, Transform, VGroup + +from xraymanim.catalogue import HOW_A_DICT_FINDS_A_KEY +from xraymanim.grammar import UNIT +from xraymanim.mobjects import DictTable +from xraymanim.primitives import arrow, box, highlight +from xraymanim.scene import Explainer + +SOURCE = 'd = {1: "one", 5: "five", 9: "nine"}' + +#: The real index array, read from a live interpreter. -1 is DKIX_EMPTY, and a number is the +#: position of the entry in the entries array, not the key and not the value. +INDICES = (-1, 0, -1, -1, -1, 1, 2, -1) + +#: The entries, in the order they were written, which is the order the dict will hand back. +ENTRIES = (("1", "'one'"), ("5", "'five'"), ("9", "'nine'")) + +#: How each slot is written on screen. An empty slot says so rather than showing -1, because +#: -1 is a detail of how CPython stores "nothing here" and is not what the slot means. +SLOT_LABELS = tuple("-" if index < 0 else str(index) for index in INDICES) + +#: The table is tall, because eight slots stacked up is eight slots stacked up. So the +#: sentence, the question and the answer all live in the column to the left of it rather +#: than above and below, which is the only way this fits in a wide frame. +TABLE_X = 2.4 +TABLE_Y = -0.3 +SIDE_X = -3.6 + + +class A04HowADictFindsAKey(Explainer): + storyboard = HOW_A_DICT_FINDS_A_KEY + + def construct(self) -> None: + source = box(SOURCE, tone="input", width=6.6, height=0.8, mono=True) + source.move_to([SIDE_X, 2.3, 0]) + + self.table = DictTable(ENTRIES, index=SLOT_LABELS, tone="durable") + self.table.scale(0.84).move_to([TABLE_X, TABLE_Y, 0]) + + self.play(FadeIn(source), run_time=0.5) + self.show(FadeIn(self.table)) + + self.spot = highlight(self.table.entries, tone="durable") + self.show(FadeIn(self.spot)) + + self.show(Transform(self.spot, highlight(self.table.index, tone="quiet"))) + + # The hash is the number itself for a small int, and the mask is seven, so this is + # one bitwise and rather than anything the reader has to take on trust. + hashed = box("d[9]", tone="focus", width=2.4, height=0.8, mono=True, note="hash(9) & 7 = 1") + hashed.move_to([SIDE_X, 0.7, 0]) + self.show( + FadeIn(hashed, shift=DOWN * UNIT), + Transform(self.spot, highlight(self.slot(1), tone="focus")), + ) + + # No words on the arrow. It leaves at an angle, so a label beside it lands on top + # of the index array, and the caption underneath is already saying which key it is. + wrong = arrow(self.slot(1), self.entry(0), tone="warning", direction=RIGHT) + self.show(FadeIn(wrong)) + + self.show( + FadeOut(wrong), + Transform(self.spot, highlight(self.slot(6), tone="focus")), + ) + + right = arrow(self.slot(6), self.entry(2), tone="focus", direction=RIGHT) + found = box("'nine'", tone="focus", width=2.4, height=0.8, mono=True, note="the answer") + found.move_to([SIDE_X, -1.2, 0]) + self.show(FadeIn(right), FadeIn(found, shift=LEFT * UNIT)) + + both = VGroup( + highlight(self.slot(1), tone="warning"), + highlight(self.slot(6), tone="focus"), + ) + self.show(FadeOut(self.spot), FadeIn(both)) + + def slot(self, index: int) -> VGroup: + """One cell of the index array, which is the thing a lookup actually reads first.""" + return self.table.index.submobjects[index] + + def entry(self, index: int) -> VGroup: + """One entry, in insertion order, which is where the key and the value really live.""" + return self.table.entries.submobjects[index] + + def show(self, *animations: object) -> None: + """Play one beat's worth of change, then hold while the reader takes it in.""" + seconds = self.beat() + self.play(*animations, run_time=min(seconds * 0.4, 1.0)) + self.wait(max(seconds - 1.0, 0.5)) diff --git a/anim/a05_a_cycle_and_the_collector.py b/anim/a05_a_cycle_and_the_collector.py new file mode 100644 index 0000000..ec08b0f --- /dev/null +++ b/anim/a05_a_cycle_and_the_collector.py @@ -0,0 +1,137 @@ +"""Two objects that keep each other alive, and the pass that notices. + +Reference counting is easy to explain and it is right nearly all of the time, so the honest +next question is when it is wrong. This is the case: two objects that hold each other. Take +the names away and both counts drop, but neither drops to zero, because each object is still +being pointed at by the other one. Nothing is left that can reach them, and nothing will free +them either. + +The numbers here are from a real 3.15.0rc1 run. With both names bound each object's count is +2, one from the name and one from the other object's attribute. After `del a, b` each count +is 1, read straight out of the header with `ctypes` so that nothing in the measurement is +itself holding a reference. `gc.collect()` then reports 2, which is these two and nothing +else. + +The second half is the collector's trick, and it is smaller than people expect. It copies +each count into a scratch field, then walks every reference from one tracked object to +another and takes one off the copy at the far end. A copy that reaches zero means every +reference to that object came from inside the group, so nothing outside can reach it. +""" + +from __future__ import annotations + +import numpy as np +from manim import LEFT, FadeIn, FadeOut, Transform, VGroup + +from xraymanim.catalogue import A_CYCLE_AND_THE_COLLECTOR +from xraymanim.mobjects import ArenaMap, PyObjectBox, RefArrow +from xraymanim.primitives import box, counter, graph, highlight +from xraymanim.scene import Explainer + +SOURCE = "a = Node(); b = Node(); a.other = b; b.other = a" + +#: One row of a pool. The two stars are the pair this animation is about, and the rest is +#: there so that freeing them reads as two blocks going back to a pool rather than as the +#: whole of memory being handed back, which is not what happens. +POOL_BEFORE = "##*###*#....#..." +POOL_AFTER = "##.###.#....#..." +POOL_LABEL = "the pool these two live in" + +OBJECT_Y = 0.9 +#: Wide enough that `ob_type: Node` and the count are not fighting over the same space, +#: then scaled down so that the two of them plus the arrows between them still fit. +OBJECT_WIDTH = 4.4 +OBJECT_SCALE = 0.88 +OBJECT_X = 3.75 +NAME_X = 6.9 +GC_Y = -0.7 +POOL_Y = -2.2 + +#: How far the two arrows between the objects curve, in radians. Straight they would be one +#: line drawn twice, because the pair points both ways along the same gap. +BEND = 0.9 + + +class A05ACycleAndTheCollector(Explainer): + storyboard = A_CYCLE_AND_THE_COLLECTOR + + def construct(self) -> None: + source = box(SOURCE, tone="input", width=9.4, height=0.75, mono=True) + source.move_to([0, 2.2, 0]) + + self.pool = ArenaMap(POOL_BEFORE, columns=16, label_text=POOL_LABEL) + self.pool.move_to([0, POOL_Y, 0]) + self.play(FadeIn(source), FadeIn(self.pool), run_time=0.6) + + # The picture people already have, first. Two things, two arrows, no counts. Beat 2 + # replaces it with what CPython actually holds, and putting them one after the other + # is the point: the familiar picture is not wrong, it is just missing the field that + # decides when either of them is freed. + sketch = graph( + {"a": (-2.4, OBJECT_Y), "b": (2.4, OBJECT_Y)}, + [("a", "b"), ("b", "a")], + tone="quiet", + bend=BEND, + ) + self.hold(FadeIn(sketch)) + + self.left = PyObjectBox("Node", refcount=2, width=OBJECT_WIDTH) + self.left.scale(OBJECT_SCALE).move_to([-OBJECT_X, OBJECT_Y, 0]) + self.right = PyObjectBox("Node", refcount=2, width=OBJECT_WIDTH) + self.right.scale(OBJECT_SCALE).move_to([OBJECT_X, OBJECT_Y, 0]) + self.cycle = VGroup( + RefArrow(self.left.shell, self.right.shell, text="a.other", bend=BEND), + RefArrow(self.right.shell, self.left.shell, text="b.other", bend=BEND, direction=LEFT), + ) + self.names = VGroup( + RefArrow(np.array([-NAME_X, OBJECT_Y, 0.0]), self.left.shell, text="a"), + RefArrow(np.array([NAME_X, OBJECT_Y, 0.0]), self.right.shell, text="b", direction=LEFT), + ) + self.hold( + FadeOut(sketch), + FadeIn(self.left), + FadeIn(self.right), + FadeIn(self.cycle), + FadeIn(self.names), + ) + + self.hold(FadeOut(self.names)) + + on_the_count = highlight(self.left.count, tone="warning") + self.hold( + Transform(self.left.count, self.left.refcount(1)), + Transform(self.right.count, self.right.refcount(1)), + FadeIn(on_the_count), + ) + + # The scratch counts go under each object rather than inside it, because gc_refs is + # not a field of the object. It lives in the header the collector keeps in front of + # every tracked object, and drawing it inside would be teaching a struct that does + # not exist. + # No highlight on these. They arrive in the warning tone and they are the only thing + # that moves for the next two beats, which is enough. A box around a pair of counters + # standing on their own is a border round mostly empty space. + self.copies = VGroup(self.gc_refs(-OBJECT_X, 1), self.gc_refs(OBJECT_X, 1)) + self.hold(FadeOut(on_the_count), FadeIn(self.copies)) + + zeroed = VGroup(self.gc_refs(-OBJECT_X, 0), self.gc_refs(OBJECT_X, 0)) + self.hold(Transform(self.copies, zeroed)) + + freed = ArenaMap(POOL_AFTER, columns=16, label_text=POOL_LABEL) + freed.move_to([0, POOL_Y, 0]) + self.hold( + FadeOut(self.left, self.right, self.cycle, self.copies), + Transform(self.pool, freed), + ) + + def gc_refs(self, x: float, value: int) -> VGroup: + """The collector's scratch copy of one object's count, sitting under that object.""" + made = counter(value, name="gc_refs", tone="warning") + made.scale(0.85).move_to([x, GC_Y, 0]) + return made + + def hold(self, *animations: object) -> None: + """Play one beat's worth of change, then stop moving while the reader reads it.""" + seconds = self.beat() + self.play(*animations, run_time=min(seconds * 0.4, 1.0)) + self.wait(max(seconds - 1.0, 0.5)) diff --git a/anim/rendered/a03-the-stack-machine.gif b/anim/rendered/a03-the-stack-machine.gif new file mode 100644 index 0000000..c1ef2ff Binary files /dev/null and b/anim/rendered/a03-the-stack-machine.gif differ diff --git a/anim/rendered/a04-how-a-dict-finds-a-key.gif b/anim/rendered/a04-how-a-dict-finds-a-key.gif new file mode 100644 index 0000000..7de34d5 Binary files /dev/null and b/anim/rendered/a04-how-a-dict-finds-a-key.gif differ diff --git a/anim/rendered/a05-a-cycle-and-the-collector.gif b/anim/rendered/a05-a-cycle-and-the-collector.gif new file mode 100644 index 0000000..07861c1 Binary files /dev/null and b/anim/rendered/a05-a-cycle-and-the-collector.gif differ diff --git a/xraymanim/VISUAL-SYSTEM.md b/xraymanim/VISUAL-SYSTEM.md index 28a32ad..a9dadf5 100644 --- a/xraymanim/VISUAL-SYSTEM.md +++ b/xraymanim/VISUAL-SYSTEM.md @@ -17,12 +17,12 @@ If a scene needs something the system has no name for, that is worth stopping ov | Shape | What it means | When it is wrong | |---|---|---| | `box` | Something with a memory address. If it is drawn as a box you could take a pointer to it | A count, an index or a flag. Those are text, not boxes, and the difference is the point | -| `arrow` | A reference from one thing to another. Thick is owned and counted, thin is borrowed and not | Anything that is not a pointer. Time, data flow and causation get other treatments | +| `arrow` | A reference from one thing to another. Thick is owned and counted, thin is borrowed and not. It may be curved, which is for the one case a straight line cannot draw: two objects pointing at each other | Anything that is not a pointer. Time, data flow and causation get other treatments | | `slots` | An array or a table, indexed by number. The cells touch, because they are next to each other in memory | A list of unrelated things that happen to be near each other on the page | | `column` | A stack, growing upward, in every lesson without exception | Anything that does not push and pop | | `stream` | Tokens, instructions, bytes: things consumed in order, one at a time, by something with a position | A collection that is looked at all at once. That is `slots` | | `tree` | The syntax tree, the type hierarchy, anything where a node has one parent | A graph that happens to look like a tree in this example | -| `graph` | The control flow graph, the object graph, anything with cycles. Positions are always given by hand | Nothing, but the layout is never automatic. An automatically placed control flow graph is unreadable | +| `graph` | The control flow graph, the object graph, anything with cycles. Positions are always given by hand, and which side of a box an edge leaves from follows from them | Nothing, but the layout is never automatic. An automatically placed control flow graph is unreadable | | `counter` | A reference count, a version tag, anything that goes up and down and is worth watching | A number that never changes. That is a label | | `highlight` | The thing happening right now. One at a time | Two at once. Two highlights tell the reader to look in two places, which tells them nothing | @@ -34,7 +34,7 @@ If a scene needs something the system has no name for, that is worth stopping ov | `RefArrow` | One reference, with the owning or borrowing weight and a label saying what is doing the pointing | | `Frame` | One `_PyInterpreterFrame`: a name, the locals as slots, and the value stack on the right growing upward | | `CodeStrip` | A run of bytecode as a stream, with the instruction pointer as a separate mobject that moves along it | -| `DictTable` | A dict as CPython stores one: a small index array on the left, the entries in insertion order on the right | +| `DictTable` | A dict as CPython stores one: a small index array on the left, the entries in insertion order on the right. The slot numbers are written outside the array, because a slot number is a position and not something CPython stores | | `ArenaMap` | The allocator's memory, one cell per block. Free is an outline with nothing in it, used is filled | ## Colour diff --git a/xraymanim/src/xraymanim/catalogue.py b/xraymanim/src/xraymanim/catalogue.py index 30608f3..cee1ea9 100644 --- a/xraymanim/src/xraymanim/catalogue.py +++ b/xraymanim/src/xraymanim/catalogue.py @@ -48,10 +48,72 @@ ), ) +THE_STACK_MACHINE = Storyboard( + slug="a03-the-stack-machine", + title="The stack machine", + lesson="T07", + shapes=("box", "column", "slots", "highlight", "Frame", "CodeStrip"), + beats=( + Beat("Calling area(6) makes a frame. The argument goes in a slot.", 5.0), + Beat("RESUME is where the interpreter checks whether anything wants a turn.", 4.5), + Beat("Now w goes on the stack. Nothing is copied, only pointed at.", 5.0), + Beat("A 2 goes on top of it. That 2 was written down when the code compiled.", 5.0), + Beat("BINARY_OP takes two off, multiplies them, and puts one back.", 5.0), + Beat("Another small integer, and the stack is two deep again.", 4.5), + Beat("The second BINARY_OP adds them, and one value is left.", 4.5), + Beat("RETURN_VALUE hands it back, and the frame is gone.", 4.5), + ), +) + +HOW_A_DICT_FINDS_A_KEY = Storyboard( + slug="a04-how-a-dict-finds-a-key", + title="How a dict finds a key", + lesson="T08", + shapes=("box", "arrow", "slots", "highlight", "DictTable"), + beats=( + Beat("Three keys. CPython keeps this as two pieces rather than one table.", 5.0), + Beat("On the right, the entries, in the order you wrote them.", 4.5), + Beat("On the left, eight slots. Each one names an entry, or nothing.", 5.0), + Beat("To find key 9, take its hash and keep the last three bits. Slot 1.", 5.5), + Beat("Slot 1 points at entry 0, whose key is 1. Not the key we asked for.", 5.5), + Beat("So it tries again, by a rule that reaches every slot eventually.", 5.0), + Beat("Slot 6 points at entry 2, and that key is 9. Found, in two looks.", 5.5), + Beat("Two keys wanted slot 1 and both still fit. That is all a collision is.", 5.0), + ), +) + +A_CYCLE_AND_THE_COLLECTOR = Storyboard( + slug="a05-a-cycle-and-the-collector", + title="A cycle, and what frees it", + lesson="T09", + shapes=( + "box", + "arrow", + "graph", + "counter", + "highlight", + "PyObjectBox", + "RefArrow", + "ArenaMap", + ), + beats=( + Beat("Two objects, and each one is pointing at the other.", 4.5), + Beat("Each count is two: one from the name, one from the other object.", 5.0), + Beat("del a and del b take the names away. The objects still point at each other.", 5.5), + Beat("Both counts drop to one. One is not zero, so neither gets freed.", 5.0), + Beat("The collector copies the counts, then looks at every arrow inside.", 5.0), + Beat("It subtracts one per arrow. Both copies reach zero.", 5.0), + Beat("Zero means nothing outside is pointing in, so the whole group goes.", 5.5), + ), +) + #: In course order, which is also render order and the order they are listed in anim/README. ANIMATIONS: tuple[Storyboard, ...] = ( SEVEN_STAGES, A_NAME_IS_A_LABEL, + THE_STACK_MACHINE, + HOW_A_DICT_FINDS_A_KEY, + A_CYCLE_AND_THE_COLLECTOR, ) diff --git a/xraymanim/src/xraymanim/mobjects.py b/xraymanim/src/xraymanim/mobjects.py index d5d68bc..741c0ae 100644 --- a/xraymanim/src/xraymanim/mobjects.py +++ b/xraymanim/src/xraymanim/mobjects.py @@ -112,9 +112,12 @@ def __init__( text: str = "", tone: str = "input", direction: object = RIGHT, + bend: float = 0.0, ) -> None: super().__init__() - self.body = arrow(source, target, owned=owned, tone=tone, text=text, direction=direction) + self.body = arrow( + source, target, owned=owned, tone=tone, text=text, direction=direction, bend=bend + ) self.owned = owned self.add(self.body) @@ -165,13 +168,24 @@ def __init__( self.locals.next_to(rule, DOWN, buff=UNIT).align_to(self.shell, LEFT).shift(RIGHT * GAP) self.add(self.locals) + # The stack sits on the floor of the frame and grows upward, which is the rule for + # every stack in the project. Hanging it from the rule instead makes it grow + # downward as values are pushed, and a reader who sees that once reads every later + # picture wrong. An empty stack is the floor with nothing on it, because the stack + # is still there between two instructions. self.stack = column(list(stack) or [""], tone="focus", width=1.9) - self.stack.next_to(rule, DOWN, buff=UNIT * 2).align_to(self.shell, RIGHT).shift(LEFT * GAP) + if not stack: + self.stack.submobjects[0].set_opacity(0.0) + self.stack.align_to(self.shell, RIGHT).shift(LEFT * GAP) + self.stack.align_to(self.shell, DOWN).shift(UP * (UNIT + 0.2)) self.add(self.stack) self.add( - label("value stack", size=CAPTION_SIZE, colour=MUTED).next_to( - self.stack, DOWN, buff=UNIT / 2 - ) + label("locals", size=CAPTION_SIZE, colour=MUTED) + .next_to(self.shell, DOWN, buff=UNIT / 2) + .align_to(self.locals, LEFT), + label("value stack", size=CAPTION_SIZE, colour=MUTED) + .next_to(self.shell, DOWN, buff=UNIT / 2) + .align_to(self.stack, RIGHT), ) @@ -239,8 +253,24 @@ def __init__( height=0.62, columns=1, ) - self.entries.next_to(self.index, RIGHT, buff=GAP).align_to(self.index, UP) - self.add(self.index, self.entries) + # A wide gap between the two arrays, not a hairline. They are two separate pieces + # of memory, and every picture in this animation draws an arrow from one to the + # other, which needs somewhere to be drawn. + self.entries.next_to(self.index, RIGHT, buff=GAP * 3).align_to(self.index, UP) + + # The slot numbers, outside the array rather than in it. Without them a reader can + # count cells to work out which one is slot 6, and a picture that has to be counted + # is a picture that gets read wrong. They sit outside because a slot number is not + # stored anywhere: it is the position, in the same way a list index is. + self.numbers = VGroup( + *( + label(str(number), size=CAPTION_SIZE, colour=MUTED).next_to( + cell, LEFT, buff=GAP / 2 + ) + for number, cell in enumerate(self.index.submobjects) + ) + ) + self.add(self.index, self.entries, self.numbers) self.add( label("indices", size=CAPTION_SIZE, colour=MUTED).next_to(self.index, UP, buff=UNIT), label("entries", size=CAPTION_SIZE, colour=MUTED).next_to(self.entries, UP, buff=UNIT), diff --git a/xraymanim/src/xraymanim/primitives.py b/xraymanim/src/xraymanim/primitives.py index 8a46d80..887fb5f 100644 --- a/xraymanim/src/xraymanim/primitives.py +++ b/xraymanim/src/xraymanim/primitives.py @@ -22,6 +22,7 @@ RIGHT, UP, Arrow, + CurvedArrow, Line, Rectangle, RoundedRectangle, @@ -102,6 +103,7 @@ def arrow( tone: str = "quiet", text: str = "", direction: object = RIGHT, + bend: float = 0.0, ) -> VGroup: """Primitive 2, the arrow: a reference from one thing to another. @@ -109,31 +111,59 @@ def arrow( and is counted in the refcount. A thin one is borrowed and is not. That single distinction is most of what goes wrong when somebody writes C against CPython, so it gets a visual difference rather than a footnote. + + `bend` is an angle in radians, and it is for the one case a straight line cannot draw: + two objects that point at each other. Straight, the two arrows are one line drawn twice. + Bent by the same angle they come out as the lens everybody draws a cycle as, because the + two arrows have their ends swapped, so the same angle curves them apart. """ ink = pen(tone) tail = start.get_edge_center(direction) if hasattr(start, "get_edge_center") else start head = end.get_edge_center(-direction) if hasattr(end, "get_edge_center") else end - line = Arrow( - tail, - head, - buff=UNIT / 2, - color=ink.stroke, - stroke_width=OWNED_STROKE if owned else BORROWED_STROKE, - max_tip_length_to_length_ratio=0.18, - ) + tail = np.asarray(tail, dtype=float) + head = np.asarray(head, dtype=float) + stroke = OWNED_STROKE if owned else BORROWED_STROKE + if bend: + # An arc has no `buff` of its own, so the gap between the arrow and the boxes it + # runs between is taken off the ends here, to keep it the same gap a straight arrow + # leaves. + along = head - tail + span = float(np.linalg.norm(along)) + if span: + step = along / span * (UNIT / 2) + tail, head = tail + step, head - step + line = CurvedArrow( + tail, head, angle=bend, color=ink.stroke, stroke_width=stroke, tip_length=0.2 + ) + else: + line = Arrow( + tail, + head, + buff=UNIT / 2, + color=ink.stroke, + stroke_width=stroke, + max_tip_length_to_length_ratio=0.18, + ) group = VGroup(line) if text: # The label goes beside the middle of the arrow, offset at a right angle to it, # rather than above its bounding box. For a diagonal arrow those are not the same # place, and the bounding box answer puts the words on top of the line. - along = line.get_end() - line.get_start() - sideways = np.array([-along[1], along[0], 0.0]) + if bend: + # For an arc, beside means outside the curve, so the offset is measured from + # the straight line between the ends out to the middle of the arc itself. + anchor = line.point_from_proportion(0.5) + sideways = anchor - (tail + head) / 2 + else: + anchor = line.get_center() + along = line.get_end() - line.get_start() + sideways = np.array([-along[1], along[0], 0.0]) length = float(np.linalg.norm(sideways)) sideways = np.array([0.0, 1.0, 0.0]) if length == 0 else sideways / length - if sideways[1] < 0: + if not bend and sideways[1] < 0: sideways = -sideways caption = label(text, size=CAPTION_SIZE, colour=MUTED) - caption.move_to(line.get_center() + sideways * (caption.height / 2 + UNIT)) + caption.move_to(anchor + sideways * (caption.height / 2 + UNIT)) group.add(caption) group.line = line return group @@ -276,24 +306,51 @@ def graph( *, tone: str = "durable", width: float = 1.6, + bend: float = 0.0, ) -> VGroup: """Primitive 7, the graph: the control flow graph, the object graph, anything cyclic. Positions are given, never computed. A control flow graph laid out automatically is a control flow graph nobody can read, and the graphs here are small enough that placing them by hand takes a minute and is worth it every time. + + What is computed is which side of a box each edge leaves from, because that follows from + the positions and there is only one sensible answer. `bend` is passed on to every arrow, + and is what a graph with a two node cycle in it needs. """ drawn = {name: box(name, tone=tone, width=width, height=0.7, size=LABEL_SIZE) for name in nodes} for name, position in nodes.items(): drawn[name].move_to([position[0], position[1], 0]) group = VGroup() for source, target in edges: - group.add(arrow(drawn[source], drawn[target], owned=False, tone=tone, direction=DOWN)) + group.add( + arrow( + drawn[source], + drawn[target], + owned=False, + tone=tone, + direction=_facing(drawn[source], drawn[target]), + bend=bend, + ) + ) group.add(*drawn.values()) group.nodes = drawn return group +def _facing(source: object, target: object) -> object: + """Which side of `source` an edge to `target` should leave from. + + The axis the target mostly lies along, so a box below is left from the bottom and a box + to the right is left from the right hand side. Always leaving from the bottom is fine + for a tree and wrong for everything else. + """ + along = target.get_center() - source.get_center() + if abs(along[0]) >= abs(along[1]): + return RIGHT if along[0] >= 0 else LEFT + return UP if along[1] >= 0 else DOWN + + def counter(value: int, *, name: str = "refcount", tone: str = "focus") -> VGroup: """Primitive 8, the counter: a refcount, a version tag, anything that goes up and down. diff --git a/xraymanim/src/xraymanim/scene.py b/xraymanim/src/xraymanim/scene.py index 776f562..1d9a73d 100644 --- a/xraymanim/src/xraymanim/scene.py +++ b/xraymanim/src/xraymanim/scene.py @@ -11,6 +11,8 @@ from __future__ import annotations +import math + from manim import DOWN, UP, FadeIn, FadeOut, Scene, VGroup, config from .grammar import CAPTION_SIZE, FADE, INK, MUTED, PAPER, TITLE_SIZE, UNIT @@ -66,9 +68,7 @@ def beat(self, index: int | None = None) -> float: return max(self.storyboard.beats[wanted].seconds - FADE, FADE) def tear_down(self) -> None: - if config.from_animation_number > 0 or config.upto_animation_number >= 0: - # Somebody is rendering a slice of the scene to look at one moment of it, so - # manim stopped `construct` early on purpose and the tally means nothing. + if self.rendering_a_slice(): return expected = len(self.storyboard.beats) if self.played != expected: @@ -76,3 +76,15 @@ def tear_down(self) -> None: f"{self.storyboard.slug} played {self.played} beat(s) but its storyboard " f"has {expected}; the plan and the picture have come apart" ) + + @staticmethod + def rendering_a_slice() -> bool: + """Whether manim was asked for part of the scene rather than all of it. + + `manim render -n 4,4` stops `construct` early on purpose, to look at one moment of + the animation, so the beat tally would be wrong every time and would fail every + probe render. Note what "all of it" looks like: the default upper bound is infinity + and not -1, which is worth spelling out because reading it as a sentinel is how the + tally ends up switched off in every render including the real ones. + """ + return config.from_animation_number > 0 or config.upto_animation_number != math.inf diff --git a/xraymanim/tests/test_xraymanim_catalogue.py b/xraymanim/tests/test_xraymanim_catalogue.py index 1a452bc..9cc0d84 100644 --- a/xraymanim/tests/test_xraymanim_catalogue.py +++ b/xraymanim/tests/test_xraymanim_catalogue.py @@ -4,7 +4,7 @@ import pytest -from xraymanim import catalogue +from xraymanim import catalogue, grammar @pytest.mark.parametrize("storyboard", catalogue.ANIMATIONS, ids=lambda item: item.slug) @@ -42,3 +42,16 @@ def test_the_file_and_the_class_are_derived_from_the_slug(): def test_the_whole_set_is_a_sitting_worth_of_watching(): """Not a correctness rule, a design one. If this ever gets large the set needs splitting.""" assert catalogue.seconds() == sum(item.seconds for item in catalogue.ANIMATIONS) + + +def test_the_set_so_far_draws_every_shape_there_is(): + """A shape nothing draws is a shape nobody has had to make readable yet. + + The nine primitives and the six named objects are the whole visual grammar, and the + grammar earns its keep by being used. One that no animation reaches is a description in + a document rather than a drawing anybody has looked at, and it will be wrong the first + time somebody needs it. This is not a rule the project can keep forever, but while there + are fifteen shapes and five animations it is worth holding. + """ + drawn = {shape for storyboard in catalogue.ANIMATIONS for shape in storyboard.shapes} + assert sorted(set(grammar.SHAPES) - drawn) == [] diff --git a/xraymanim/tests/test_xraymanim_facts.py b/xraymanim/tests/test_xraymanim_facts.py new file mode 100644 index 0000000..c6e71f9 --- /dev/null +++ b/xraymanim/tests/test_xraymanim_facts.py @@ -0,0 +1,161 @@ +"""The numbers the animations put on screen, checked against the interpreter running them. + +An animation is a claim about CPython that nobody can run. A reader watching a05 cannot +pause it and check that the count really is 2, and a reader watching a04 has no way to know +whether the eight slots on screen are the eight slots CPython made or eight slots somebody +drew because the picture looked better that way. So the claims are checked here instead, and +the constants are read straight out of the scene files rather than copied, because a copy is +a second place to be wrong. + +The scene files are parsed rather than imported. Importing one needs manim, these facts do +not, and a check on CPython that only runs when an optional drawing library is installed is +a check that will be quietly skipped in exactly the job where it matters. +""" + +from __future__ import annotations + +import ast +import ctypes +import gc +import sys +from pathlib import Path + +import pytest + +ANIM = Path(__file__).resolve().parents[2] / "anim" + + +def constant(module: str, name: str) -> object: + """One module level constant from a scene file, without importing the scene file.""" + tree = ast.parse((ANIM / f"{module}.py").read_text(encoding="utf-8")) + for node in tree.body: + if isinstance(node, ast.Assign) and any( + isinstance(target, ast.Name) and target.id == name for target in node.targets + ): + return ast.literal_eval(node.value) + raise AssertionError(f"{module}.py has no constant called {name}") + + +class DictKeys(ctypes.Structure): + """The header of `_dictkeysobject`, up to the point where the index array starts. + + Read against `Objects/dict.c` and `Include/internal/pycore_dict.h`. The three one byte + fields are followed by a pad byte, which is why `dk_version` lands where it does, and + `dk_indices` is the flexible array that follows this header in memory. + """ + + _fields_ = ( + ("dk_refcnt", ctypes.c_ssize_t), + ("dk_log2_size", ctypes.c_uint8), + ("dk_log2_index_bytes", ctypes.c_uint8), + ("dk_kind", ctypes.c_uint8), + ("dk_version", ctypes.c_uint32), + ("dk_usable", ctypes.c_ssize_t), + ("dk_nentries", ctypes.c_ssize_t), + ) + + +class DictObject(ctypes.Structure): + """`PyDictObject`, which is the object header and then four fields.""" + + _fields_ = ( + ("ob_refcnt", ctypes.c_ssize_t), + ("ob_type", ctypes.c_void_p), + ("ma_used", ctypes.c_ssize_t), + ("_ma_watcher_tag", ctypes.c_uint64), + ("ma_keys", ctypes.c_void_p), + ("ma_values", ctypes.c_void_p), + ) + + +def index_array(mapping: dict) -> list[int]: + """The slot array of a real dict, one number per slot, with -1 for an empty slot. + + This is the array a lookup reads first, and it is the thing a04 draws. There is no + supported way to see it from Python, so this walks the structs, which is the same thing + the animation's source note says it did. + """ + header = DictObject.from_address(id(mapping)) + keys = DictKeys.from_address(header.ma_keys) + slots = 1 << keys.dk_log2_size + width = (1 << keys.dk_log2_index_bytes) // slots + signed = {1: ctypes.c_int8, 2: ctypes.c_int16, 4: ctypes.c_int32, 8: ctypes.c_int64}[width] + start = header.ma_keys + ctypes.sizeof(DictKeys) + return [signed.from_address(start + position * width).value for position in range(slots)] + + +@pytest.fixture(scope="module") +def sample(): + """The dict a04 draws. Integer keys, so nothing here moves when the hash seed changes.""" + return {1: "one", 5: "five", 9: "nine"} + + +def test_the_slot_array_a04_draws_is_the_one_cpython_made(sample): + """The eight slots on screen, against the eight slots in memory. + + If CPython ever changes how it lays a small dict out, this fails and the animation gets + fixed, rather than going on showing a picture of a version nobody runs. + """ + assert index_array(sample) == list(constant("a04_how_a_dict_finds_a_key", "INDICES")) + + +def test_the_entries_a04_draws_are_in_insertion_order(sample): + drawn = constant("a04_how_a_dict_finds_a_key", "ENTRIES") + assert [(str(key), repr(value)) for key, value in sample.items()] == [ + tuple(pair) for pair in drawn + ] + + +def test_the_collision_a04_is_about_is_a_real_collision(sample): + """Two of the three keys want the same slot, which is the reason for the animation.""" + mask = len(index_array(sample)) - 1 + wanted = [hash(key) & mask for key in sample] + assert wanted == [1, 5, 1] + + +def test_a05_has_its_refcounts_right(): + """2 while both names are bound, 1 after `del`, and the pair still there either way. + + The counts are read out of the header rather than through `sys.getrefcount`, because + passing an object to a function is itself a reference and the animation is about counting + references exactly. + """ + + class Node: + __slots__ = ("other",) + + a = Node() + b = Node() + a.other = b + b.other = a + both = (id(a), id(b)) + assert [ctypes.c_ssize_t.from_address(at).value for at in both] == [2, 2] + + was = gc.isenabled() + gc.disable() + try: + del a, b + assert [ctypes.c_ssize_t.from_address(at).value for at in both] == [1, 1] + finally: + if was: + gc.enable() + + +def test_a05_needs_the_collector_to_free_the_pair(): + """One count each and nothing that can reach them, so only the collector gets them.""" + + class Node: + __slots__ = ("other",) + + gc.collect() + a = Node() + b = Node() + a.other = b + b.other = a + del a, b + assert gc.collect() == 2 + + +def test_this_is_the_interpreter_the_animations_are_about(): + """A guard on the struct layouts above, which are read from a particular source tree.""" + assert sys.version_info[:2] >= (3, 14) diff --git a/xraymanim/tests/test_xraymanim_scene.py b/xraymanim/tests/test_xraymanim_scene.py new file mode 100644 index 0000000..278fb47 --- /dev/null +++ b/xraymanim/tests/test_xraymanim_scene.py @@ -0,0 +1,75 @@ +"""The beat tally, which is the thing that makes a storyboard a plan rather than a comment. + +Needs manim, because `Explainer` is a manim `Scene`. It does not render anything: `tear_down` +is called directly, which is the whole point of keeping the check in a method with no +drawing in it. +""" + +from __future__ import annotations + +import math + +import pytest + +pytest.importorskip("manim", reason="the drawing half of xraymanim, installed by --extra anim") + +from manim import config + +from xraymanim.scene import Explainer +from xraymanim.storyboard import Beat, Storyboard + +DEMO = Storyboard( + slug="a99-a-demo", + title="A demo that only exists in the tests", + lesson="T99", + shapes=("box",), + beats=(Beat("One.", 4.0), Beat("Two.", 4.0), Beat("Three.", 4.0)), +) + + +class Demo(Explainer): + storyboard = DEMO + + def construct(self) -> None: # pragma: no cover - nothing here renders + pass + + +@pytest.fixture +def scene(): + made = Demo() + made.played = 0 + return made + + +def test_a_scene_that_played_its_beats_is_fine(scene): + scene.played = 3 + scene.tear_down() + + +def test_a_scene_that_skipped_a_beat_fails_the_render(scene): + scene.played = 2 + with pytest.raises(AssertionError) as raised: + scene.tear_down() + assert "played 2 beat(s) but its storyboard has 3" in str(raised.value) + + +def test_the_whole_scene_is_not_a_slice(): + """The default upper bound is infinity, not -1. + + Reading it as a sentinel is not a hypothetical mistake, it is the one that was made + here: `upto_animation_number >= 0` is true for infinity, so the tally was switched off + in every render including the real ones, and a scene could quietly stop playing its + plan. This test is the reason that cannot come back. + """ + assert config.upto_animation_number == math.inf + assert not Explainer.rendering_a_slice() + + +@pytest.mark.parametrize( + ("field", "value"), + [("from_animation_number", 4), ("upto_animation_number", 4)], +) +def test_a_slice_switches_the_tally_off(field, value, monkeypatch): + """`manim render -n 4,4` stops construct early on purpose, so the tally means nothing.""" + monkeypatch.setattr(config, field, value) + assert Explainer.rendering_a_slice()