Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ Setup (a speech recogniser, a model) is in [docs/setup.md](docs/setup.md); commo

The app works on your real screen: be careful with `--visible-to-capture` and with anything that takes screenshots while a private window is open.

The app's windows are hidden from screen capture only for the moment of its own screenshot (`src/main/capture-shield.ts`), so they show in a person's recordings and screen shares the rest of the time. `--visible-to-capture` keeps them in our own screenshots too, which is only for looking at a run. `node_modules/.bin/electron scripts/capture-check.cjs` (needs a visible window, so run it yourself, not in CI) checks that our drawing is absent from our capture and present otherwise, and finds how long the change needs to take effect on your machine.

## Testing

```
Expand Down
4 changes: 4 additions & 0 deletions docs/scene-language.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,8 @@ The capability is a closed solid, not a hollow duct or a general multi-section l

### The mechanism kit

**How the kit is kept.** Each part below is one row of a table (`src/scene/kit-presets.ts`): how its box is fitted, what its `n` counts (default and limits), the values that follow from its size (the crank throw, where a rod's ends sit), its anchors, and the list of shapes it is made of, with numbers as small formulas of its size (`src/scene/kit-expr.ts`). The renderer draws a row with general generators (lathe, cylinder, extruded outline, box, sphere, torus, coil, bolts; `src/overlay/mechanism-parts.ts`), so there is no code per part, and the guide lists the names only. `test/kit-parts-golden.test.ts` holds what every row builds, piece by piece.

Mechanism parts are ready-made `part`s for engines, gearboxes and pumps. Each is built from the same plain shapes a model could write, with good proportions, its own metal and finish (the object's `color` tints the main body), and named **anchors**: a `pin` with `anchor` lands on one by name (an unknown name falls back to the middle of the part). `size` is the box the part fits, centred on `pos`. Cylindrical parts have their axis along y (the crank, cam, gear and bearing along z), so a vertical engine's parts line up with no rotation.

| part | `size` and `n` | anchors |
Expand Down Expand Up @@ -275,6 +277,8 @@ Parts that keep moving once started, in the x,y plane with the shafts along z (t
| `cam` | `id` `by` `lift`? `open`? `phase`? | `id` is a valve (or any follower), `by` a camshaft that is spinning: the valve is pushed down by `lift` (default a fifth of its length) while the cam's total angle is within `open` degrees (default 110) of `phase` (default 180, the nose pointing down), with a smooth rise and fall. Set the camshaft's `rot` to fix its timing |
| `flash` | `id` `by` `angle` `span` `cycle`? | `id` is an object (a small sphere for a spark, a larger one for the burn) that is invisible except for `span` degrees from `angle` in each `cycle` (default 720) of the turning of `by`: it comes on quickly and dies away. For a four-stroke engine 0 degrees is the piston at the top starting the intake stroke, 360 the top of the compression, so the spark is at about 350 and the power stroke runs 360 to 540 |

**How the movement kinds are kept.** Each kind of loop (slide, swing, gate, pulse, float), the crank, the cam and the flash is a row of a catalogue (`src/scene/movement-rows.ts`): the general motion it is (turn, slide, swell, bob, or follow a path: a crank pin's circle with a rod of fixed length, a cam's bump), its default axis, where its cycle starts, the amount it may take, and its numbers as small formulas (`src/scene/kit-expr.ts`). `src/overlay/motion.ts` holds only the few general passes that read the rows; the validator and the timeline read the range, axis and start from the same row. The guide lists names only, and the names a scene may use are read from the tables (the loop kinds from the rows' keys, the parts from the kit table's keys after the parts drawn by their own code), so a new row is usable from a scene with nothing else written (`test/rows-from-scenes.test.ts`). `test/movement-rows-golden.test.ts` holds what every kind poses (9,611 cases recorded from the code that had each as its own pass), with `test/movement-golden.test.ts` for the engine.

A turning part's own `rot` stays as the scene set it, and the angle is added to it; this is exact when the other two angles are 0. A four-stroke engine: the crankshaft `spin` at 30 rpm, each camshaft `spin` `by` the crankshaft with `ratio` 0.5, an intake camshaft with `rot` `[0,0,135]` and an exhaust one with `rot` `[0,0,-135]` (each valve then opens as its cam's nose comes round to point down at it), a `crank` for each piston, and `flash` for the spark and the burn.

A step waits for its spin speed ramps to finish, including ramps down to 0. It then carries the final angle and requested speed into the next step or idle playback. For example, a spin at 2 with `dur: 2` cannot finish its step before 4 seconds, even when the speech is shorter. The continuous turning itself does not hold the step open.
Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "talkthrough",
"version": "0.7.1",
"version": "0.8.0",
"description": "An AI that talks you through anything on your screen, and draws as it talks.",
"keywords": [
"screen assistant",
Expand Down
43 changes: 43 additions & 0 deletions scripts/capture-check-pixels.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
// Pixel reading for scripts/capture-check.cjs, kept apart so it can be tested without a screen (test/capture-check-pixels.test.ts).

/**
* The share (0 to 1) of the pixels in the middle of `rect` (the central 80 %, so antialiased edges and a shadow do not count) that are within `tol`
* of the colour `rgb` on every channel. `bitmap` is { data, width, height } with data in BGRA order, as Electron's nativeImage.toBitmap() gives.
* `rect` is in the bitmap's own pixels.
*/
function share(bitmap, rect, rgb, tol = 24) {
const { data, width, height } = bitmap;
const x0 = Math.max(0, Math.round(rect.x + rect.w * 0.1)), x1 = Math.min(width, Math.round(rect.x + rect.w * 0.9));
const y0 = Math.max(0, Math.round(rect.y + rect.h * 0.1)), y1 = Math.min(height, Math.round(rect.y + rect.h * 0.9));
let hit = 0, total = 0;
for (let y = y0; y < y1; y++) {
for (let x = x0; x < x1; x++) {
const i = (y * width + x) * 4;
total++;
if (Math.abs(data[i + 2] - rgb[0]) <= tol && Math.abs(data[i + 1] - rgb[1]) <= tol && Math.abs(data[i] - rgb[2]) <= tol) hit++;
}
}
return total === 0 ? 0 : hit / total;
}

/** A window's rectangle on its display (device-independent pixels, display-relative) as pixels of a picture of that display, whatever size the picture is. */
function toPicture(rect, display, picture) {
const sx = picture.width / display.bounds.width, sy = picture.height / display.bounds.height;
return { x: (rect.x - display.bounds.x) * sx, y: (rect.y - display.bounds.y) * sy, w: rect.w * sx, h: rect.h * sy };
}

/**
* The smallest wait after which this and every longer wait lost no trial, from { settleMs: failures } counts; null when even the longest failed.
* Waits are taken in ascending order of settleMs.
*/
function leastSafeWait(failuresBySettle) {
const waits = Object.keys(failuresBySettle).map(Number).sort((a, b) => a - b);
let best = null;
for (let i = waits.length - 1; i >= 0; i--) {
if (failuresBySettle[waits[i]] === 0) best = waits[i];
else break;
}
return best;
}

module.exports = { share, toPicture, leastSafeWait };
Loading
Loading