Three more animations: the stack machine, a dict lookup and a cycle

README.md

@@ -96,6 +96,9 @@ Everything in them is real. The tokens are what `tokenize` returns and the instr
 |---|---|---|
 | a01 | [One line of Python, seven stages](anim/rendered/a01-seven-stages.gif) | T01 |
 | a02 | [A name is a label, not a box](anim/rendered/a02-a-name-is-a-label.gif) | T08 |
+| a03 | [The stack machine](anim/rendered/a03-the-stack-machine.gif) | T07 |
+| a04 | [How a dict finds a key](anim/rendered/a04-how-a-dict-finds-a-key.gif) | T08 |
+| a05 | [A cycle, and what frees it](anim/rendered/a05-a-cycle-and-the-collector.gif) | T09 |
 
 They are all drawn from the same fifteen shapes, listed with what each one means in [xraymanim/VISUAL-SYSTEM.md](xraymanim/VISUAL-SYSTEM.md), and adding a sixteenth needs an amendment to that document first. That rule is not a convention somebody has to remember: the shape list is code, a storyboard that names a shape outside it fails, and a shape that is drawable but undescribed fails too. [anim/README.md](anim/README.md) has the index and how to render them.
 

anim/README.md

@@ -10,6 +10,9 @@ Everything in them is real. The tokens are what `tokenize` returns, the instruct
 |---|---|---|---|
 | a01 | [One line of Python, seven stages](rendered/a01-seven-stages.gif) | T01 | 40s |
 | a02 | [A name is a label, not a box](rendered/a02-a-name-is-a-label.gif) | T08 | 34s |
+| a03 | [The stack machine](rendered/a03-the-stack-machine.gif) | T07 | 38s |
+| a04 | [How a dict finds a key](rendered/a04-how-a-dict-finds-a-key.gif) | T08 | 41s |
+| a05 | [A cycle, and what frees it](rendered/a05-a-cycle-and-the-collector.gif) | T09 | 36s |
 
 ## a01, one line of Python, seven stages
 
@@ -23,6 +26,24 @@ Everything in them is real. The tokens are what `tokenize` returns, the instruct
 
 `a = []`, then `b = a`, then `del a`. Two names, one list, and a count on the object that says how many names are pointing at it. When the count reaches zero the memory goes back. This is the single most useful thing to understand about Python values, and it is much easier to see moving than described.
 
+## a03, the stack machine
+
+![one call to a small function, run one instruction at a time, with the value stack drawn beside it](https://raw.githubusercontent.com/tamnd/cpython-internals/main/anim/rendered/a03-the-stack-machine.gif)
+
+`def area(w): return w * 2 + 1`, called with 6. A disassembly listing tells you what each instruction is called and nothing about what it does to the pile of values underneath, which is the part that is actually hard. So the pile is on screen: seven instructions, the stack going up to two and back down to one twice, and the frame disappearing when the answer is handed back. The instructions and the depths are from `pyxray.stack.walk` on 3.15.0rc1, which is why `LOAD_FAST_BORROW` is there instead of the `LOAD_FAST` you might expect.
+
+## a04, how a dict finds a key
+
+![a dict drawn as an eight slot index array beside three entries, with a lookup that collides and probes again](https://raw.githubusercontent.com/tamnd/cpython-internals/main/anim/rendered/a04-how-a-dict-finds-a-key.gif)
+
+A dict is not one table. Since 3.6 it has been a small array of slot numbers and a separate list of entries in the order you wrote them, and both of the questions people ask about dicts follow from that split: why the order is kept, and what a collision costs. Here `d[9]` lands on slot 1, finds the wrong key, probes again, and finds the right one on the second look. The eight slot array on screen was read out of a live interpreter, and there is a test that reads it again and fails if it ever differs.
+
+## a05, a cycle, and what frees it
+
+![two objects pointing at each other, their counts falling to one and staying there, and the collector taking them](https://raw.githubusercontent.com/tamnd/cpython-internals/main/anim/rendered/a05-a-cycle-and-the-collector.gif)
+
+Reference counting is right nearly all of the time, so the interesting question is when it is not. Two objects that hold each other: take the names away, both counts drop to one, and one is not zero, so nothing is freed and nothing can reach them either. The second half is what the collector does about it, which is smaller than people expect. It copies each count, subtracts one for every reference it finds inside the group, and anything whose copy reaches zero was only ever being kept alive from inside.
+
 ## How they are made
 
 Each `aNN_*.py` here is a manim scene built out of the shared library in [xraymanim](../xraymanim), which is what stops a hundred animations from each inventing their own way to draw a pointer. The shapes and what they mean are in [VISUAL-SYSTEM.md](../xraymanim/VISUAL-SYSTEM.md).

anim/a03_the_stack_machine.py

@@ -0,0 +1,120 @@
+"""One call to a two line function, one instruction at a time, with the stack on screen.
+
+The hard part of the eval loop for a beginner is not the loop, it is the stack. Reading a
+disassembly listing tells you nothing about it, because the listing shows what each
+instruction is called and not what it does to the pile of values underneath. So this shows
+the pile, and lets the listing sit above it with a pointer walking along.
+
+The function is `def area(w): return w * 2 + 1`, called with 6, which gives 13. It was
+chosen for its stack profile rather than for what it computes: the depth goes 0, 1, 2, 1, 2,
+1, and those two climbs and two drops are the whole idea.
+
+The instructions and the depths came from `pyxray.stack.walk` on 3.15.0rc1. `LOAD_FAST_BORROW`
+really is what the compiler emits for `w` here rather than `LOAD_FAST`, and it is left as it
+is, because a reader who disassembles this function themselves should see what we showed them.
+"""
+
+from __future__ import annotations
+
+from manim import DOWN, RIGHT, FadeIn, FadeOut, Transform
+
+from xraymanim.catalogue import THE_STACK_MACHINE
+from xraymanim.grammar import UNIT
+from xraymanim.mobjects import CodeStrip, Frame
+from xraymanim.primitives import box
+from xraymanim.scene import Explainer
+
+SOURCE = "def area(w): return w * 2 + 1"
+
+#: The instructions as `dis` prints them, in order, from a real 3.15.0rc1 run.
+INSTRUCTIONS = (
+    "RESUME 0",
+    "LOAD_FAST_BORROW w",
+    "LOAD_SMALL_INT 2",
+    "BINARY_OP *",
+    "LOAD_SMALL_INT 1",
+    "BINARY_OP +",
+    "RETURN_VALUE",
+)
+
+#: What the value stack holds after each of those instructions, first item at the bottom of
+#: the pile. The depths are from `pyxray.stack.walk`, so they are the compiler's own numbers
+#: rather than a guess, and the values are what a call with w = 6 actually pushes.
+STACKS = (
+    (),
+    ("6",),
+    ("6", "2"),
+    ("12",),
+    ("12", "1"),
+    ("13",),
+    ("13",),
+)
+
+SOURCE_Y = 2.35
+CODE_Y = 0.95
+FRAME_X = -0.4
+RETURN_X = 4.4
+FLOOR_Y = -2.7
+
+
+class A03TheStackMachine(Explainer):
+    storyboard = THE_STACK_MACHINE
+
+    def construct(self) -> None:
+        source = box(SOURCE, tone="input", width=7.4, height=0.7, mono=True)
+        source.move_to([0, SOURCE_Y, 0])
+
+        self.strip = CodeStrip(INSTRUCTIONS, at=0, rows=2)
+        self.strip.scale_to_fit_width(11.0).move_to([0, CODE_Y, 0])
+        self.play(FadeIn(source), FadeIn(self.strip), run_time=0.6)
+
+        # The frame arrives on the first beat rather than with the code, because the first
+        # caption is about the call happening. Before the call there is no frame, and a
+        # picture that shows one anyway has answered a question nobody asked yet.
+        self.frame = self.frame_at(0)
+        seconds = self.beat()
+        self.play(FadeIn(self.frame), run_time=0.8)
+        self.wait(max(seconds - 0.8, 0.4))
+
+        # One beat per instruction, except the last. RETURN_VALUE gets its own ending,
+        # because what it does is take the frame away, and that is not a change to the
+        # picture the loop knows how to make.
+        for index in range(len(INSTRUCTIONS) - 1):
+            self.step(index)
+
+        returned = box(
+            "13", tone="focus", width=2.4, height=0.8, mono=True, note="back to the caller"
+        )
+        returned.move_to([RETURN_X, FLOOR_Y + 1.2, 0])
+        seconds = self.beat()
+        self.play(
+            Transform(self.strip.pointer, self.strip.at(len(INSTRUCTIONS) - 1)),
+            FadeOut(self.frame, shift=DOWN * UNIT),
+            FadeIn(returned, shift=RIGHT * UNIT),
+            run_time=1.0,
+        )
+        self.wait(max(seconds - 1.0, 0.6))
+
+    def step(self, index: int) -> None:
+        """Run one instruction: move the pointer, and redraw the frame with the new stack."""
+        seconds = self.beat()
+        wanted = self.frame_at(index)
+        self.play(
+            Transform(self.strip.pointer, self.strip.at(index)),
+            FadeOut(self.frame),
+            run_time=min(seconds * 0.3, 0.6),
+        )
+        self.frame = wanted
+        self.play(FadeIn(self.frame), run_time=0.4)
+        self.wait(max(seconds - 1.0, 0.4))
+
+    def frame_at(self, index: int) -> Frame:
+        """The frame as it stands after instruction `index`, sitting on a floor that stays put.
+
+        A new one each time rather than one that is edited, because the frame gets taller as
+        the stack does, and cross fading two of them keeps the floor still while the top of
+        the box moves. That is the right way round: a value stack grows upward.
+        """
+        made = Frame("area", {"w": "6"}, STACKS[index], width=6.2)
+        made.scale(0.8).move_to([FRAME_X, FLOOR_Y, 0], aligned_edge=DOWN)
+        return made

anim/a04_how_a_dict_finds_a_key.py

@@ -0,0 +1,111 @@
+"""What happens between `d[9]` and the answer, in a dict that has a collision in it.
+
+Most pictures of a dict draw one table of key and value pairs, and that picture cannot
+explain either of the two things people actually ask about dicts: why they keep insertion
+order, and what a collision costs. CPython has not stored a dict that way since 3.6. There
+is a small array of slot numbers, and there are the entries in the order they went in, and
+almost everything interesting follows from the split.
+
+The dict is `{1: "one", 5: "five", 9: "nine"}`, chosen because integers hash to themselves,
+so nothing here depends on the hash randomization that would make string keys land in a
+different slot on every run. With eight slots the mask is 7, key 1 wants slot 1, key 5 wants
+slot 5, and key 9 wants slot 1 as well, which is the collision.
+
+The index array on screen is `[-1, 0, -1, -1, -1, 1, 2, -1]`, and that is not derived from
+the probing rule, it was read out of a live 3.15.0rc1 interpreter through `ctypes` at the
+address of the dict. The tests next to this file do the same read and would fail if CPython
+ever laid it out differently.
+"""
+
+from __future__ import annotations
+
+from manim import DOWN, LEFT, RIGHT, FadeIn, FadeOut, Transform, VGroup
+
+from xraymanim.catalogue import HOW_A_DICT_FINDS_A_KEY
+from xraymanim.grammar import UNIT
+from xraymanim.mobjects import DictTable
+from xraymanim.primitives import arrow, box, highlight
+from xraymanim.scene import Explainer
+
+SOURCE = 'd = {1: "one", 5: "five", 9: "nine"}'
+
+#: The real index array, read from a live interpreter. -1 is DKIX_EMPTY, and a number is the
+#: position of the entry in the entries array, not the key and not the value.
+INDICES = (-1, 0, -1, -1, -1, 1, 2, -1)
+
+#: The entries, in the order they were written, which is the order the dict will hand back.
+ENTRIES = (("1", "'one'"), ("5", "'five'"), ("9", "'nine'"))
+
+#: How each slot is written on screen. An empty slot says so rather than showing -1, because
+#: -1 is a detail of how CPython stores "nothing here" and is not what the slot means.
+SLOT_LABELS = tuple("-" if index < 0 else str(index) for index in INDICES)
+
+#: The table is tall, because eight slots stacked up is eight slots stacked up. So the
+#: sentence, the question and the answer all live in the column to the left of it rather
+#: than above and below, which is the only way this fits in a wide frame.
+TABLE_X = 2.4
+TABLE_Y = -0.3
+SIDE_X = -3.6
+
+
+class A04HowADictFindsAKey(Explainer):
+    storyboard = HOW_A_DICT_FINDS_A_KEY
+
+    def construct(self) -> None:
+        source = box(SOURCE, tone="input", width=6.6, height=0.8, mono=True)
+        source.move_to([SIDE_X, 2.3, 0])
+
+        self.table = DictTable(ENTRIES, index=SLOT_LABELS, tone="durable")
+        self.table.scale(0.84).move_to([TABLE_X, TABLE_Y, 0])
+
+        self.play(FadeIn(source), run_time=0.5)
+        self.show(FadeIn(self.table))
+
+        self.spot = highlight(self.table.entries, tone="durable")
+        self.show(FadeIn(self.spot))
+
+        self.show(Transform(self.spot, highlight(self.table.index, tone="quiet")))
+
+        # The hash is the number itself for a small int, and the mask is seven, so this is
+        # one bitwise and rather than anything the reader has to take on trust.
+        hashed = box("d[9]", tone="focus", width=2.4, height=0.8, mono=True, note="hash(9) & 7 = 1")
+        hashed.move_to([SIDE_X, 0.7, 0])
+        self.show(
+            FadeIn(hashed, shift=DOWN * UNIT),
+            Transform(self.spot, highlight(self.slot(1), tone="focus")),
+        )
+
+        # No words on the arrow. It leaves at an angle, so a label beside it lands on top
+        # of the index array, and the caption underneath is already saying which key it is.
+        wrong = arrow(self.slot(1), self.entry(0), tone="warning", direction=RIGHT)
+        self.show(FadeIn(wrong))
+
+        self.show(
+            FadeOut(wrong),
+            Transform(self.spot, highlight(self.slot(6), tone="focus")),
+        )
+
+        right = arrow(self.slot(6), self.entry(2), tone="focus", direction=RIGHT)
+        found = box("'nine'", tone="focus", width=2.4, height=0.8, mono=True, note="the answer")
+        found.move_to([SIDE_X, -1.2, 0])
+        self.show(FadeIn(right), FadeIn(found, shift=LEFT * UNIT))
+
+        both = VGroup(
+            highlight(self.slot(1), tone="warning"),
+            highlight(self.slot(6), tone="focus"),
+        )
+        self.show(FadeOut(self.spot), FadeIn(both))
+
+    def slot(self, index: int) -> VGroup:
+        """One cell of the index array, which is the thing a lookup actually reads first."""
+        return self.table.index.submobjects[index]
+
+    def entry(self, index: int) -> VGroup:
+        """One entry, in insertion order, which is where the key and the value really live."""
+        return self.table.entries.submobjects[index]
+
+    def show(self, *animations: object) -> None:
+        """Play one beat's worth of change, then hold while the reader takes it in."""
+        seconds = self.beat()
+        self.play(*animations, run_time=min(seconds * 0.4, 1.0))
+        self.wait(max(seconds - 1.0, 0.5))