Can-AI-Write-a-Book

Can AI Write a Book? — Longform LLM Benchmark Corpus & Analysis Tools

This repository contains the raw generated manuscripts, custom diagnostic scripts, experiment matrix, and public control prompts for the "Can AI Write a Book?" benchmarking series by Jessica K. Murray.

📖 Read the full pillar write-up: [Can AI Write a Book? Measuring Local Models on 3 Old Computers (https://jessicakmurray.com/can-ai-write-a-book/)

📌 About the Experiment

Can local, open-weights LLMs write a coherent 90,000-word novel on consumer hardware? What is the actual minimum cost floor, and how much "AI slop" (repetition and phrase recycling) comes out of unedited runs?

This ongoing experiment tests multiple model architectures (Dense vs. MoE) across three distinct hardware nodes using BookyAI as a fixed longform wrapper and Ollama as the backend:

Node Name	GPU (VRAM)	System RAM	CPU	Memory Architecture
boo-4070	RTX 4070 (12GB)	128GB	i9-14900KF	Dual-channel
trex-3070	RTX 3070 (8GB)	64GB	i9-10980XE	Quad-channel
backoffice-2060	RTX 2060 (6GB)	64GB	i7-1165G7	Dual-channel

⚠️ Status: Active Benchmark. Raw output folders and run matrix entries are committed as runs complete across all three test nodes.

OS versions differ across nodes and are not controlled. boo-4070 and backoffice-2060 run Windows 11 Pro 25H2 (build 26200.9445). trex-3070 runs Windows 10 Home 22H2 (build 19045.6466) and is not upgradeable to Windows 11 due to CPU compatibility requirements. Comparisons involving trex-3070 should be read as hardware + OS, not hardware alone.

📁 Repository Structure

prompt.md: The verbatim public control prompt, genre parameters, and chapter pipeline used for every run.
run matrix: The complete benchmark matrix tracking wall-clock time, tok/sec, target adherence %, repetition rates, memory footprints, and power consumption.
bookdiff.py: Custom Python script to analyze overlapping 10-word text windows and measure internal repetition rates.
chaptertimes.ps1: PowerShell script to compute generation times per chapter from file output timestamps.
/tests/: Raw, unedited Markdown outputs generated by BookyAI, organized by run_id.

🛠️ How to Run the Analysis Scripts

🛠️ Detailed Script Manuals & Instructions

Can AI Write a Book?

Longform local-LLM benchmark data and analysis tools for the "Can AI Write a Book?" experiment by Jessica K. Murray.

This project explores a deceptively simple question:

Can a local, open-weights language model actually generate a coherent novel-length manuscript on ordinary consumer hardware?

The benchmark looks beyond whether a model can produce 90,000 words. It measures what happens over the full length of a book: generation speed, target-length adherence, repetition, phrase recycling, truncation, drift, and the amount of AI-generated prose that survives later human editing.

The experiments use BookyAI as the longform generation wrapper and Ollama as the local inference backend.

Repository Tools

The repository includes three small analysis utilities:

Script	Purpose	Platform
`runqc.py`	Start here. Runs every check on a finished book in one command and writes the timing	Windows (needs
	CSV, body file, repetition report, QC checklist, matrix row and per-chapter detail	PowerShell for timing)
`bookdiff.py`	Measures internal repetition, chapter drift, and AI-draft carryover after human editing	Windows, macOS, Linux
`chaptertimes.ps1`	Reconstructs approximate chapter generation times from BookyAI output-file timestamps	Windows PowerShell

runqc.py calls the other two. If you are logging a benchmark run, use it. If you are analysing a manuscript that did not come from this pipeline, use bookdiff.py directly. None of these scripts send manuscript text anywhere. Analysis is performed locally. See CHANGELOG.md for what changed and when, including the cases where a fix altered a previously published figure.

Getting Started

Clone the repository:

git clone https://github.com/HelloJessicaM/Can-AI-Write-a-Book.git
cd Can-AI-Write-a-Book

You can also download the repository as a ZIP from GitHub and extract it normally.

Requirements

For `bookdiff.py`

You need:

Python 3.8 or newer
A plain-text or Markdown manuscript
No third-party Python libraries

Check your Python installation with:

python --version

or:

python3 --version

On Windows, you may also have the Python launcher:

py --version

No pip install step is required.

For `chaptertimes.ps1`

You need:

Windows
PowerShell
The original individual chapter files produced during the BookyAI generation run
Their original LastWriteTime timestamps intact

No PowerShell modules or other dependencies are required.

The timing script should ideally be run immediately after a generation completes, before editing the chapter files or otherwise doing anything that might modify their timestamps.

`runqc.py`

runqc.py is the front door. It runs the whole post-generation checklist in order, in one command, and writes everything needed to log a benchmark run. It exists because doing these steps by hand produced eight mislabelled runs in the first six weeks of this project. Every output it writes is stamped with the run ID you supply, so a file can't end up attached to the wrong run. Requirements Python 3.8 or newer, no third-party libraries bookdiff.py in the same folder as runqc.py chaptertimes.ps1 in the same folder, and Windows PowerShell — unless you pass --timings with an existing CSV makebody.py in the same folder, only if you use --export Basic use From anywhere, pointed at a finished BookyAI run folder:

python runqc.py A3-8k "C:\path\to\20261003152816_a3_8k_backoffice_2060_qwen3_8b"

What it does, in order

Timing. Runs chaptertimes.ps1 against the book folder and writes <run_id>_timings.csv. Skipped if you pass --timings. Fallback audit. Compares each chapter's tokens-per-second against the run's median and flags large outliers, which can indicate the wrapper silently generated a chapter with a cloud model. Read the warning in the Known Issues section before trusting this — it fails in both directions when a run has many retries. Body assembly. Builds <run_id>_body.md from the individual chapter_NN.md files, normalising mixed heading shapes as it goes. This is the file every judge reads and every repetition number is computed from. Assembling from the chapter files rather than the wrapper's own export is deliberate: the export includes front and back matter, and an earlier version of this pipeline once measured an outline.md by mistake. Repetition. Runs bookdiff.py repetition --chapters-only on the body and writes <run_id>_repetition.txt. Checklist. Writes <run_id>_checklist.md, a dated to-do list of the steps a script can't do — the wrapper's own quality review, reading ollama ps for the memory footprint and CPU/GPU split, confirming the engine label on every chapter, and the external quality instruments. Matrix row. Writes <run_id>_matrixrow.csv, appends the same row to an append-only matrix_rows.csv ledger (deduplicated by run ID), and prints the row tab-separated for pasting straight into a spreadsheet. Per-chapter detail. Writes <run_id>_chapter_detail.csv with one row per chapter: words, overlap with chapter one, and self-repetition. This is what makes per-chapter metrics computable in a spreadsheet instead of by hand.

Options

Option Meaning --timings PATH Use an existing chaptertimes CSV instead of running it. Required on macOS and Linux. --export PATH Measure this exported manuscript instead of assembling from the chapter files. Needs makebody.py. --out DIR Where to write outputs (default: current directory) --num-ctx N The context window this run used --thinking VALUE Value for the thinking column (default OFF) --machine NAME Machine label, e.g. boo-4070 --model TAG Model tag, e.g. gemma4:26b

How `num_ctx` is determined

Context window is the single setting most likely to be recorded wrong, and a wrong value silently invalidates any comparison built on it. So: If you pass --num-ctx, that value is used. Otherwise it is read from the run ID: A3-8k → 8192, L1-16kr2-4070 → 16384, L3-128k-backoffice → 131072. If the run ID says nothing, the cell is left blank and the script says so. It never falls back to a default. A blank cell is honest; a guessed 32768 is a lie that looks like data. Note that 8k means 8192, not 8000. If you set 8000 in one tool and 8192 in another, the lower value wins silently and your comparison has a confound. Confirm what actually loaded with ollama ps — the CONTEXT column shows the real value.

Naming runs

Put the context in the run ID with a leading separator and a k:

K1-8k        G1-16k-r2        L3-128k-backoffice-gemma412b

Avoid K1-128-4070 (no k, so the context can't be inferred) and avoid putting a bare number that could be read as a context where it isn't one.

Example output

A3-8k
============================================================
  -> chaptertimes
  30 chapters, 60,062 words, 1128.3 min (18.8 h), 1.14 tok/s
  !! 8 chapter(s) far above the median rate of 1.12 tok/s:
       chapter_03.md  5.21 tok/s  (4.7x median)
       ...
  -> assembled .\A3-8k_body.md from 30 chapter files
  -> bookdiff repetition --chapters-only
  repetition: 19.0%
  -> checklist: .\A3-8k_checklist.md
  num_ctx 8192 (read from the run id)
  -> appended to matrix_rows.csv
  -> A3-8k_chapter_detail.csv (30 chapters) for the chapter_detail tab

------------------------------------------------------------
Paste into the experiment matrix (fill size, split, version):

A3-8k   OFF   8192   ...
------------------------------------------------------------

A note on reading the output Three columns need interpreting rather than trusting: `slowest_chapter_sec` includes chapter one's interval, which can be a meaningless large number. If it exceeds the run's wall clock, that's what happened. `chapters_truncated` is a heuristic with four known false-positive classes. A flag on a normal-length chapter is usually a formatting artifact; a flag on a runaway chapter is usually real. The fallback audit is unreliable in runs with many retries. See Known Issues.

`bookdiff.py`

bookdiff.py performs two different kinds of manuscript analysis:

repetition analyzes a single manuscript for repeated text and chapter drift.
compare compares an AI-generated draft against a later edited manuscript and estimates how much original AI wording remains.

General syntax:

python bookdiff.py MODE [files] [options]

You can see built-in help at any time:

python bookdiff.py --help

For help with an individual mode:

python bookdiff.py repetition --help

python bookdiff.py compare --help

If your system uses python3 or py instead of python, substitute that command in all examples below.

1. Measuring Repetition in a Manuscript

Basic command:

python bookdiff.py repetition manuscript.md

For example:

python bookdiff.py repetition "The Affirmation Glitch.md"

Always put paths in quotes if they contain spaces.

The script reads the entire manuscript and analyzes overlapping 10-word windows, also called 10-word n-grams.

For a sequence like:

the old house stood at the end of the empty road

the script records that 10-word sequence, moves forward one word, records the next 10-word sequence, and continues through the manuscript.

It then asks how many of those windows occur more than once.

Example output

A report begins approximately like this:

========================================================================
REPETITION REPORT
  manuscript.md   (89,758 words, 40 chapters)
========================================================================

  Internal repetition: 45.4% of 10-word windows appear more than once

The exact numbers depend on the manuscript.

What "Internal repetition" means

If the report says:

Internal repetition: 12.5%

that means 12.5 percent of the manuscript's overlapping 10-word windows belong to sequences that appear more than once.

This is designed to detect the kind of phrase recycling that can emerge during very long LLM generations.

It is not a plagiarism score, AI detector, writing-quality score, or universal threshold for whether a book is "too repetitive."

Normal novels repeat language too.

Dialogue tags, character names, deliberate refrains, recurring descriptions, quotations, ritual language, poetry, and stylistic motifs can all increase the number.

For that reason, the most useful approach is comparative.

Run the tool on:

multiple outputs from different models;
different settings for the same model;
edited versus unedited manuscripts;
traditionally written novels or other texts you consider reasonable baselines.

The benchmark becomes much more meaningful when the same measurement is used consistently across texts.

Repeated-Passage Report

When sufficiently repeated material exists, bookdiff.py also prints up to ten distinct highly repeated passages.

Example:

Most-repeated passages (distinct):

  windows    find   passage
  --------------------------------------------------------------------
       37       4   she looked at him and did not know what to

There are two columns because repetition can be counted in different ways.

`windows`

This measures overlapping 10-word windows.

A long repeated paragraph can produce many overlapping repeated windows even if the paragraph itself only occurs a few times.

`find`

This approximates the number of times the displayed passage itself occurs as a searchable string.

This is closer to what you would see by copying the displayed phrase and searching for it in an editor.

The distinction is useful because a long passage repeated four times may generate dozens of overlapping repeated windows.

The tool also tries to avoid filling the report with slightly shifted versions of the same repeated passage. Candidate windows sharing at least five consecutive tokens are treated as belonging to the same repetition run.

Only passages occurring more than twice are candidates for this top repeated-passages list.

Chapter Drift

If the script successfully detects at least three chapters, it also produces a drift table:

Drift check:

    ch    words   vs ch1  self-rep
  ---------------------------------
     1    2,315     0.0%      1.2%
     2    2,184     0.1%      1.5%
     3    2,407     0.2%      2.0%

`vs ch1`

This measures how many distinct 10-word windows in that chapter also occur in Chapter 1.

A small amount of overlap is normal.

What is more interesting in a long-generation benchmark is a pattern in which later chapters increasingly overlap Chapter 1.

For example:

Chapter 5     0.2%
Chapter 15    0.5%
Chapter 25    2.8%
Chapter 35    8.7%

That pattern can indicate that the model is increasingly recycling earlier material as the generation continues.

`self-rep`

This measures repeated 10-word windows within that individual chapter.

This can help distinguish whole-book repetition from a particular chapter that became unusually repetitive.

Recurring Short Phrases

The headline number uses a 10-word window, so it is blind to the two-, three- and four-word habits readers actually notice. The short-phrase section catches those and reports them as a rate per 10,000 words, so books of different lengths can be compared:

  Recurring short phrases (below the 10-word window):
    count  per 10k words   phrase
  --------------------------------------------------------------------
      183           25.1   The algorithm of grace
       71            9.7   was no longer just
       62            8.5   She looked at the

A model can score 0.5% on the headline number and still repeat a four-word habit every four hundred words. Compare the rate column against your own baseline, not against zero. Motif Saturation The tier below short phrases: a single image that recurs all book long.

 # Motif saturation (recurring images, measured over 30 chapters):
    count  per 10k  chapters   phrase
  --------------------------------------------------------------------
      324     44.5     30/30   the algorithm
      233     32.0     29/30   the sanctuary
      199     27.3     30/30   code
  ....................................................................
      351     48.2     30/30   Maya   (name/place)
  ....................................................................
       37      5.1      4/30   the "Signal   <- CONCENTRATED

"golden thread" 110 times in 72,000 words shares no 10-word window with itself and is not a four-word habit, so neither number above sees it — but a reader meets it every other page. Read the chapters column first. A phrase in nearly every chapter is a style tic: the model's voice, and it will be in every book that model writes. The same count inside two or three chapters is a runaway passage, which is a different problem with a different fix. Rows concentrated in a small number of chapters are listed separately as CONCENTRATED whatever their count, because ranking on count alone loses them behind ordinary vocabulary.

Things to know about this section specifically:

Words sitting inside a repeated 10-word window are excluded, so this measures the repetition the headline number cannot see rather than restating it. The report prints how much it skipped. Singular and plural are merged here and nowhere else in the report. Names and places are separated out below the dotted line. Frequent everyday words (hand, light, room) are filtered out of single-word rows, because ranking on count alone puts chair above mirror. --motif-all includes them. --motif-rate PER10K changes the floor. Default 1.5, about once every 6,700 words. Lower it to see more.

python bookdiff.py repetition manuscript.md --chapters-only --motif-rate 0.8

Frame Breaks

A check for prose where the narrator stops telling the story and starts describing the document it is inside:

  Frame breaks: 6
    line  kind              text
  --------------------------------------------------------------------
    1018  chapter ref       ...the "pulse" she had identified in Chapter 7.
    1224  meta narration    ...And she was the author of the next chapter.

Five categories are detected: chapter ref, meta narration, production note, model voice and addresses reader. A novel should not know it has chapter numbers; a chapter should not contain Word count: 2,450; and a chapter that opens "Since there was no preceding text provided…" is the assistant answering a prompt rather than the narrator telling a story. Chapter headings and everything from the first back-matter heading onward are excluded, so a hit is prose. Line numbers are real file lines, so you can jump straight there. Measured precision on 23 hits across five real manuscripts is 95%. The remaining false-positive class is a character legitimately reading a chapter of a book, which this check cannot distinguish from a model slip. Read the hits. Verifying a Frequency Claim --count takes an exact phrase and reports its rate. Repeatable:

python bookdiff.py repetition manuscript.md --count "soft glow" --count "the pull to stay"

  Phrase counts you asked for:
    count  per 10k words   phrase
  --------------------------------------------------------------------
       41            5.7   soft glow
        2            0.3   the pull to stay

Built for checking a claim before believing it. A beta reader, human or otherwise, can be confidently wrong about frequency — in this project an AI judge reported a motif "roughly 110 times" that the script found 324 times, and reported another at a tenth of its real rate.


# Analyzing Narrative Chapters Only

By default, repetition analysis includes the text supplied to the script.

To focus on recognized narrative chapters and remove front/back matter, use:

```bash
python bookdiff.py repetition manuscript.md --chapters-only

For example:

python bookdiff.py repetition "The Affirmation Glitch.md" --chapters-only

This is useful when comparing novel-generation runs because acknowledgments, appendices, author notes, discussion questions, and other non-narrative material can distort repetition statistics.

The script recognizes several back-matter headings, including forms of:

Epilogue
Afterword
Endnotes
Glossary
Appendix
Appendices
Further Reading
Discussion Guide
Acknowledgments
About the Author
A Note to Readers
Reader Thanks
Bibliography
Index

Important: Epilogue is treated as back matter by the current script.

If your epilogue is part of the narrative and you want it included in the repetition measurement, do not use --chapters-only without modifying the script.

Saving the Repetition Report

Instead of printing the report only in the terminal, save it to a text file:

python bookdiff.py repetition manuscript.md --out repetition-report.txt

You can combine options:

python bookdiff.py repetition manuscript.md --chapters-only --out repetition-report.txt

The script will confirm:

Report written to repetition-report.txt

This is useful for keeping benchmark results with individual model runs.

Chapter Detection

bookdiff.py automatically looks for common chapter headings.

Supported default patterns include headings such as:

# Chapter 1
## Chapter 1
### Chapter 1

or:

Chapter 1

or:

CHAPTER 1
CHAPTER IV
CHAPTER XII

It also recognizes Markdown headings consisting only of a chapter number, such as:

## 1

The script requires at least three matching chapter headings before treating the manuscript as chapter-structured.

If fewer than three headings are detected, the manuscript is analyzed as one whole file.

In that case, you will not receive the normal chapter-by-chapter drift report.

Custom Chapter Headings

If your book uses a different heading format, provide a regular expression with --chapter-regex.

For example:

python bookdiff.py repetition manuscript.md --chapter-regex "^CHAPTER"

Or:

python bookdiff.py repetition manuscript.md --chapter-regex "^Part [0-9]+"

Or:

python bookdiff.py repetition manuscript.md --chapter-regex "^## Scene [0-9]+"

Important: a custom chapter regex replaces the built-in chapter patterns for that run. It is not added to them.

The custom expression must also identify at least three headings for chapter-based analysis to activate.

Text Normalization

The script performs limited normalization so that typography does not create artificial differences.

Among other things, it normalizes:

curly and straight quotation marks;
curly apostrophes;
em dashes and en dashes;
ellipses;
non-breaking spaces;
Markdown emphasis characters such as *, _, and backticks;
repeated whitespace.

Comparison is also case-insensitive at the token level.

As a result, changing:

She couldn't believe it.

to a typographically different version using curly quotation marks does not make the passage appear newly written.

The goal is to measure meaningful textual changes rather than formatting changes.

2. Comparing an AI Draft With an Edited Manuscript

The second mode measures how much text from an original AI draft survives in a later edited version.

Syntax:

python bookdiff.py compare AI_DRAFT EDITED_MANUSCRIPT

Example:

python bookdiff.py compare "original-ai-draft.md" "edited-manuscript.md"

The order matters.

The first file must be the original AI-generated manuscript:

AI draft -> first argument
Edited version -> second argument

Carryover Measurements

The comparison produces two main measurements.

Identical-sentence carryover

Example:

38.7% of sentences are word-for-word the AI's

The script normalizes case, punctuation, common typography, and Markdown before determining whether a sentence matches the AI draft.

Therefore, "word-for-word" here means essentially the same normalized sequence of words, rather than byte-for-byte identical text.

A sentence whose punctuation was changed but whose wording remains the same will generally still count as a match.

A substantially rewritten sentence will not.

10-word-window carryover

Example:

52.4% of 10-word windows appear in the AI draft

This measurement is more sensitive to partial editing.

Imagine an AI-generated sentence:

Emma walked slowly across the kitchen and looked through the window.

If the edited version changes only a few words, the complete sentence may no longer count as identical.

However, substantial runs of unchanged wording can still produce matching 10-word windows.

For this reason, the window percentage will often be higher than the identical-sentence percentage.

Together, the two numbers provide a more useful picture than either metric alone.

Chapter-by-Chapter Carryover

When chapters are detected in both files, the comparison produces a table similar to:

  ch    words  sentences   windows   status
------------------------------------------------------------------------
   1    2,814      82.1%     88.5%   mostly AI
   2    2,631      64.7%     73.3%   heavily edited
   3    2,907      41.8%     55.2%   mostly yours

The sentences column is the percentage of sentences in the edited chapter whose normalized wording also occurs in the corresponding AI chapter.

The windows column measures 10-word overlap.

The script also assigns descriptive labels based on sentence carryover:

Identical sentence rate	Label
75% or higher	`mostly AI`
60% to under 75%	`heavily edited`
45% to under 60%	`part rewritten`
Under 45%	`mostly yours`

These labels are convenient benchmark shorthand built into the script. They should not be interpreted as legal, publishing-industry, copyright, or philosophical definitions of authorship.

The underlying percentages are the meaningful measurements.

Comparing Manuscripts With Different Chapter Counts

The comparison does not attempt semantic chapter alignment.

If the AI draft contains 40 chapters and the edited book contains 42, the script reports the mismatch and compares the first 40 chapters in order.

Conceptually:

AI Chapter 1 -> Edited Chapter 1
AI Chapter 2 -> Edited Chapter 2
AI Chapter 3 -> Edited Chapter 3
...

It does not attempt to determine that an edited Chapter 12 was split from an original Chapter 11, for example.

If editing substantially changes chapter structure, whole-book window carryover remains useful, but chapter-level sentence percentages should be interpreted cautiously.

Saving a Carryover Report

Use --out:

python bookdiff.py compare "ai-draft.md" "edited.md" --out carryover-report.txt

Custom chapter detection works in comparison mode too:

python bookdiff.py compare "ai-draft.md" "edited.md" --chapter-regex "^CHAPTER" --out carryover-report.txt

The same custom chapter regex is applied to both manuscripts.

What `bookdiff.py` Is Not

The script is intentionally simple and transparent.

It is not:

a machine-learning AI detector;
a plagiarism checker;
a semantic-similarity model;
an authorship classifier;
a copyright determination;
a quality score;
an attempt to determine whether prose "sounds AI."

It measures observable text overlap.

That makes it useful for controlled experiments where the original AI draft is available.

For example, if an author starts with a generated manuscript and spends several editing passes rewriting it, compare can quantitatively show how much exact or near-exact generated language remains after each pass.

Recommended Carryover Experiment

If you are studying human editing of AI prose, save multiple manuscript versions:

draft-00-ai-original.md
draft-01-light-edit.md
draft-02-developmental-edit.md
draft-03-line-edit.md
draft-04-final.md

Then compare every version against the unchanged original AI output:

python bookdiff.py compare draft-00-ai-original.md draft-01-light-edit.md
python bookdiff.py compare draft-00-ai-original.md draft-02-developmental-edit.md
python bookdiff.py compare draft-00-ai-original.md draft-03-line-edit.md
python bookdiff.py compare draft-00-ai-original.md draft-04-final.md

Do not compare each revision only against the immediately preceding revision if your research question is "How much of the original AI prose remains?"

The AI draft should remain the fixed baseline.

`chaptertimes.ps1`

chaptertimes.ps1 analyzes the timestamps of individual BookyAI chapter files to estimate how long each chapter took to generate.

Unlike bookdiff.py, this utility should be considered part of the data-capture stage of a benchmark.

Run it while the original output-file timestamps are still trustworthy.

Why File Timestamps Matter

BookyAI writes individual chapter files during generation.

If chapters are generated sequentially, the difference between the completion timestamp of one chapter and the completion timestamp of the next can be used as an approximation of the later chapter's generation time.

For example:

Chapter 1 finished: 10:00:00
Chapter 2 finished: 10:02:30
Chapter 3 finished: 10:05:10

The script interprets this approximately as:

Chapter 2: 150 seconds
Chapter 3: 160 seconds

There is no timing measurement for Chapter 1 because there is no preceding chapter timestamp from which to calculate its generation interval.

This is why the script reports the wall-clock span from the completion of Chapter 1 through the completion of the final chapter rather than claiming to know the exact beginning of the entire run.

Important: Run Timing Analysis Before Editing Files

chaptertimes.ps1 relies on each file's filesystem LastWriteTime.

Editing and resaving a chapter will change that timestamp.

Some copying, synchronization, archive extraction, cloud-storage, restoration, or file-management workflows may also alter timestamps.

For benchmark-quality data, the safest procedure is:

Allow BookyAI to finish the generation.
Do not edit the chapter files.
Open PowerShell.
Run chaptertimes.ps1 against the original output directory.
Save the generated timing CSV with the benchmark run.
Only then move, rename, edit, or reorganize the manuscript files.

If you no longer trust the timestamps, do not treat reconstructed chapter timing as benchmark data.

Running `chaptertimes.ps1`

First change into the BookyAI output directory containing the individual chapter files:

cd "C:\path\to\bookyai\output"

If the script itself is in that directory:

.\chaptertimes.ps1 -RunId A2

If Windows blocks script execution, you can allow it for only the current PowerShell process:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\chaptertimes.ps1 -RunId A2

This does not require permanently changing the execution policy.

Alternatively, keep chaptertimes.ps1 in the cloned GitHub repository and invoke it from the BookyAI output directory:

cd "C:\path\to\bookyai\output"

& "C:\path\to\Can-AI-Write-a-Book\chaptertimes.ps1" -RunId A2

This is often cleaner because the analysis script stays in the repository while the current working directory remains the run directory.

`-RunId`

RunId labels the benchmark run.

Example:

.\chaptertimes.ps1 -RunId A3b

The value is written into every CSV row and determines the output filename:

A3b_timings.csv

If no RunId is supplied, the default is:

run

which creates:

run_timings.csv

For reproducible benchmark work, always provide an explicit run ID.

Selecting Chapter Files With `-Pattern`

By default, the script examines:

*.md

in the current directory.

You can restrict the files using -Pattern.

For example:

.\chaptertimes.ps1 -RunId A3b -Pattern "chapter*.md"

or:

.\chaptertimes.ps1 -RunId A3b -Pattern "*Chapter*.md"

This can be useful if the output folder contains other Markdown files that are not generated chapters.

The script automatically excludes filenames containing:

outline
summary
character
appendix
reference

However, other unrelated .md files could still be included.

For clean benchmark data, inspect the file list or use a restrictive -Pattern if the directory contains additional Markdown files.

At least two matching chapter files are required.

Otherwise the script exits with:

Found fewer than 2 chapter files. Check the folder and -Pattern.

How Chapter Order Is Determined

The timing script does not sort chapters by filename.

It sorts them by:

LastWriteTime

This is intentional.

The purpose is to reconstruct generation chronology rather than alphabetical file order.

This also means that manually editing a chapter after generation can move it to the end of the timestamp sequence and invalidate the analysis.

Timing CSV Columns

The script creates:

<RunId>_timings.csv

with these columns:

Column	Meaning
`RunId`	Benchmark run identifier supplied with `-RunId`
`File`	Chapter filename
`Finished`	File's `LastWriteTime`, formatted as date and local time
`Seconds`	Seconds since the preceding chapter file's completion
`Words`	Whitespace-based word count
`Tokens`	Estimated token count
`TokPerSec`	Estimated tokens divided by the timestamp interval
`Truncated`	Heuristic indication that the chapter may end abruptly

The first chapter has no value for Seconds or TokPerSec because there is no previous chapter timestamp.

Token Estimates

The script estimates:

tokens = words x 1.33

This is deliberately a rough rule of thumb.

It does not run the manuscript through the model's tokenizer and does not retrieve actual Ollama token telemetry.

Different models and tokenizers can produce substantially different token-to-word ratios, especially with:

code;
unusual punctuation;
non-English text;
fragmented output;
special tokens;
reasoning traces;
Markdown-heavy content.

For this reason, TokPerSec should be interpreted as an internally consistent estimated throughput metric, not an exact tokenizer-level benchmark.

If exact inference throughput is required, use actual runtime/model telemetry instead.

Understanding `Seconds`

For every chapter after the first:

Seconds = current chapter LastWriteTime - previous chapter LastWriteTime

If generation is continuous and sequential, this is a useful approximation of chapter generation time.

However, the number can also include anything that occurred between file completions.

For example:

model-loading delays;
wrapper overhead;
pauses;
retries;
human intervention;
system stalls;
other processing performed between chapters.

This benchmark therefore measures observed wall-clock output cadence, not pure GPU inference time.

That distinction is important when comparing these results with inference engines that report token generation speed directly.

Truncation Detection

The timing script performs a simple truncation check.

A chapter is considered likely complete if its final non-whitespace character is one of several common terminal punctuation characters, such as:

.
!
?
"
'
)
]

Otherwise:

Truncated = YES

This is a heuristic.

A chapter ending abruptly in the middle of a sentence is likely to be correctly flagged.

But false positives and false negatives are possible.

For example, deliberately ending a chapter with:

To be continued

would be flagged even if that ending was intentional.

A generation could also end incorrectly immediately after a period and escape detection.

Treat Truncated = YES as a prompt for manual inspection rather than definitive proof.

Timing Summary

After writing the CSV, the script prints a run summary similar to:

==================== A3b ====================
  chapters              : 40
  wall clock (ch1->end) : 26.9 min
  total words           : 89,758
  overall tok/sec       : 74.00
  median chapter        : 39.2s
  slowest chapter       : 61.7s
  likely truncated      : 2
  target adherence      : 100% of 90,000
================================================

`chapters`

Number of matching chapter files.

`wall clock (ch1->end)`

Elapsed time between the completion timestamp of the first chapter and the completion timestamp of the final chapter.

This is not the complete start-to-finish generation time because the script does not know when Chapter 1 began.

`total words`

Total whitespace-separated words across all included files.

`overall tok/sec`

Estimated total tokens divided by the measured Chapter-1-completion-to-final-completion interval.

Because Chapter 1's token estimate is included while its generation interval is unavailable, this should be regarded as an approximate comparative statistic rather than exact end-to-end throughput.

`median chapter`

Approximate middle chapter-generation interval among chapters with measurable timing.

`slowest chapter`

Longest interval between consecutive chapter completion timestamps.

`likely truncated`

Number of files flagged by the terminal-punctuation heuristic.

`target adherence`

Total word count as a percentage of:

90,000 words

The 90,000-word benchmark target is currently hard-coded in the PowerShell script.

For example:

45,000 words = 50%
90,000 words = 100%
135,000 words = 150%
180,000 words = 200%

If you use chaptertimes.ps1 for a project with a different target length, either interpret this value accordingly or modify the 90000 constant in the summary calculation.

Recommended Benchmark Workflow

For a complete BookyAI/Ollama benchmark run, a reproducible workflow is:

1. Record the configuration

Before starting, record information such as:

run ID
model
quantization
thinking/reasoning setting
hardware
system RAM
GPU VRAM
prompt version
target word count
BookyAI settings
Ollama version

2. Run the generation

Allow the BookyAI run to finish without modifying completed chapter files.

3. Run the QC pipeline immediately

From the original chapter-output directory, before editing anything:

python "C:\path\to\Can-AI-Write-a-Book\runqc.py" A3b "C:\path\to\bookyai\output\run-folder"

This captures the timing CSV while the file timestamps are still trustworthy, assembles the narrative body, measures repetition, writes a QC checklist, and emits a matrix row and per-chapter detail rows.

4. Record what the scripts can't see

Open the generated <run_id>_checklist.md and work through it. The items that need a human are: the wrapper's own quality review, if it has one ollama ps while the model is loaded — the reported size, the CPU/GPU split, and the CONTEXT value confirming the engine label on every chapter, which is the only reliable check against a cloud model having silently written one the external quality instruments

5. Read the repetition report, not just the percentage

Review the repeated-passages list, the short-phrase rates, the motif section's chapters column, and the frame-break hits. The headline number and the motif section measure different failures, and a book can be clean on one and badly saturated on the other.

6. Preserve the raw output

Never overwrite the original generated manuscript if you intend to study later editing. Keep it as an immutable baseline.

8. Edit a copy

For example:

A3b_raw.md
A3b_edited.md

9. Measure human-edit carryover

python bookdiff.py compare "A3b_raw.md" "A3b_edited.md" --out A3b_carryover.txt

This produces a reproducible quantitative record of how much original generated language survived the editing process.

Recommended Run Directory

One possible organization is:

runs/
└── A3b_model-name/
    ├── raw/
    │   ├── chapter-01.md
    │   ├── chapter-02.md
    │   └── ...
    ├── manuscript.md
    ├── A3b_timings.csv
    ├── repetition.txt
    ├── metadata.md
    └── edited/
        ├── manuscript-edited.md
        └── carryover.txt

The exact directory structure is not required by either script.

The important thing is to preserve raw outputs and keep benchmark results associated with the correct run.

Comparing Models Fairly

These tools are most useful when the surrounding experiment is controlled.

If comparing models, try to keep constant:

generation prompt;
BookyAI version and settings;
chapter pipeline;
requested book length;
reasoning/thinking configuration where appropriate;
repetition-analysis settings;
inclusion or exclusion of back matter;
chapter-detection rules.

Hardware differences should be explicitly recorded rather than hidden.

For repetition comparisons, run every manuscript using the same command.

For example:

python bookdiff.py repetition model-a.md --chapters-only
python bookdiff.py repetition model-b.md --chapters-only
python bookdiff.py repetition model-c.md --chapters-only

Do not analyze one book with back matter included and another with --chapters-only and treat the percentages as directly equivalent.

Troubleshooting

`python` is not recognized

Try:

python3 bookdiff.py --help

or on Windows:

py bookdiff.py --help

If none works, install Python 3 before using bookdiff.py.

My manuscript is treated as one chapter

Your chapter headings probably do not match the default patterns.

Use:

python bookdiff.py repetition manuscript.md --chapter-regex "YOUR_REGEX"

Also remember that at least three headings must match.

I don't see a drift report

The script only produces chapter drift when more than two chapters are successfully detected.

Check the line near the beginning of the report:

(90,000 words, 1 chapters)

If it says 1 chapters, chapter detection failed and the whole file was analyzed as one unit.

`--chapters-only` removed my epilogue

That is expected in the current implementation.

Epilogue is one of the headings treated as back matter.

Run without --chapters-only if you want the epilogue included, or modify the BACK_MATTER regular expression in bookdiff.py.

My carryover results look unexpectedly high

Check whether the edited manuscript actually contains long sections copied unchanged from the AI draft.

The 10-word-window metric intentionally detects partially edited sentences and unchanged stretches of prose.

Also make sure you supplied the files in the correct order:

python bookdiff.py compare AI_DRAFT EDITED_VERSION

My carryover chapter table is misaligned

If chapters were inserted, deleted, merged, or split during editing, positional comparison may no longer represent equivalent chapters.

The script pairs chapters by order.

For heavily restructured books, give greater weight to the whole-book carryover numbers or prepare aligned versions specifically for chapter-level comparison.

PowerShell says script execution is disabled

From the current PowerShell session:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

Then run:

.\chaptertimes.ps1 -RunId A3b

The Process scope limits the change to that PowerShell session.

`chaptertimes.ps1` finds too many Markdown files

Use a narrower pattern:

.\chaptertimes.ps1 -RunId A3b -Pattern "chapter*.md"

The default *.md pattern intentionally allows different BookyAI filename conventions, but that means unrelated Markdown files can be included if they are in the same directory and do not match the built-in exclusion terms.

The chapters are in the wrong order

The script sorts files by LastWriteTime, not filename.

If the timestamps no longer represent the original generation order, the timing data has probably been altered.

Do not rename or manually sort the CSV and then treat it as reconstructed generation telemetry.

The chronological timestamps are the measurement.

One chapter shows an enormous generation time

Inspect what happened between the two corresponding file completion timestamps.

The interval may contain:

a paused generation;
system sleep;
a retry;
model loading;
a stalled process;
human interaction;
another interruption.

The script cannot distinguish those events from inference time.

Preserve the result but annotate the benchmark run if you know why the outlier occurred.

Token speed does not match Ollama's reported speed

That is expected.

chaptertimes.ps1 estimates tokens as:

words x 1.33

and divides that estimate by filesystem timestamp intervals.

Ollama can report actual tokenizer/runtime statistics.

These are different measurements and should not be presented as interchangeable.

Data Interpretation

The project intentionally separates several questions that are often collapsed into the vague question, "Can AI write a book?"

A model can successfully generate 90,000 words while still performing poorly in other dimensions.

Useful benchmark dimensions include:

Length adherence
Generation time
Observed throughput
Truncation
Internal repetition
Cross-chapter drift
Structural coherence
Raw output quality
Human editing required
AI-language carryover after editing

For example, a manuscript that hits exactly 90,000 words but has a 45 percent repetition rate represents a very different result from a manuscript of the same length with little phrase recycling.

Likewise, an AI draft that appears polished but ultimately requires nearly every sentence to be rewritten represents a different form of success or failure than one that survives editing largely intact.

The purpose of these scripts is to make some of those differences measurable.

Privacy and Offline Use

Both scripts operate locally.

bookdiff.py uses only Python's standard library and makes no network requests.

chaptertimes.ps1 reads local files and writes a local CSV.

Your manuscript does not need to be uploaded to an API or third-party analysis service.

This makes the tools suitable for unpublished manuscripts and local-model experiments where maintaining an entirely local workflow is important.

Reproducibility Notes

When publishing benchmark results, include enough information for another researcher or hobbyist to understand how the number was generated.

For repetition results, report at minimum:

script/version or commit
manuscript version
whether --chapters-only was used
custom chapter regex, if any
reported word count
repetition percentage

For timing results, report at minimum:

run ID
hardware
model and quantization
BookyAI/Ollama configuration
number of chapters
word count
timestamp methodology
whether the run was interrupted

When possible, preserve the raw output used to calculate the metric.

A percentage without the underlying text, settings, and methodology is much less useful than a reproducible benchmark.

License

The analysis scripts and repository materials are released under the MIT License unless otherwise noted.

See LICENSE for details.

About This Project

"Can AI Write a Book?" is an ongoing experiment examining longform generation with local open-weights language models on consumer hardware.

Rather than evaluating only short benchmark prompts, the project studies what happens when a model is asked to sustain generation across the length of an entire novel.

The project is particularly interested in failure modes that become visible only at long context and long output lengths: repetition, structural drift, runaway length, truncated chapters, reasoning leakage, and increasingly recycled prose.

chaptertimes.ps1 captures what happened while the book was being generated.

bookdiff.py repetition measures what happened inside the generated book.

bookdiff.py compare measures what happened to that generated prose after a human began rewriting it.

Together, they provide a lightweight and reproducible way to study a part of AI-assisted writing that is difficult to see in short-form model benchmarks.

Analyze novel body only (stops at back-matter headings)

python bookdiff.py repetition "runs/A3b_boo-4070_qwen3-8b_thinking-OFF/4070 - The Affirmation Glitch - qwen3:8b - NO thinking.md" --chapters-only

Name		Name	Last commit message	Last commit date
tests		tests
BookyAI Prompts and Settings		BookyAI Prompts and Settings
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
bookdiff.py		bookdiff.py
chaptertimes.ps1		chaptertimes.ps1
run_matrix.csv		run_matrix.csv
runqc.py		runqc.py

Latest commit

History

Folders and files

Repository files navigation

Can-AI-Write-a-Book

Can AI Write a Book? — Longform LLM Benchmark Corpus & Analysis Tools

📌 About the Experiment

📁 Repository Structure

🛠️ How to Run the Analysis Scripts

Can AI Write a Book?

Repository Tools

Getting Started

Requirements

For bookdiff.py

For chaptertimes.ps1

runqc.py

What it does, in order

Options

How num_ctx is determined

Naming runs

Example output

bookdiff.py

1. Measuring Repetition in a Manuscript

Example output

What "Internal repetition" means

Repeated-Passage Report

windows

find

Chapter Drift

vs ch1

self-rep

Recurring Short Phrases

Things to know about this section specifically:

Frame Breaks

Saving the Repetition Report

Chapter Detection

Custom Chapter Headings

Text Normalization

2. Comparing an AI Draft With an Edited Manuscript

Carryover Measurements

Identical-sentence carryover

10-word-window carryover

Chapter-by-Chapter Carryover

Comparing Manuscripts With Different Chapter Counts

Saving a Carryover Report

What bookdiff.py Is Not

Recommended Carryover Experiment

chaptertimes.ps1

Why File Timestamps Matter

Important: Run Timing Analysis Before Editing Files

Running chaptertimes.ps1

-RunId

Selecting Chapter Files With -Pattern

How Chapter Order Is Determined

Timing CSV Columns

Token Estimates

Understanding Seconds

Truncation Detection

Timing Summary

chapters

wall clock (ch1->end)

total words

overall tok/sec

median chapter

slowest chapter

likely truncated

target adherence

Recommended Benchmark Workflow

1. Record the configuration

2. Run the generation

3. Run the QC pipeline immediately

4. Record what the scripts can't see

5. Read the repetition report, not just the percentage

6. Preserve the raw output

8. Edit a copy

9. Measure human-edit carryover

Recommended Run Directory

Comparing Models Fairly

Troubleshooting

python is not recognized

For `bookdiff.py`

For `chaptertimes.ps1`

`runqc.py`

How `num_ctx` is determined

`bookdiff.py`

`windows`

`find`

`vs ch1`

`self-rep`

What `bookdiff.py` Is Not

`chaptertimes.ps1`

Running `chaptertimes.ps1`

`-RunId`

Selecting Chapter Files With `-Pattern`

Understanding `Seconds`

`chapters`

`wall clock (ch1->end)`

`total words`

`overall tok/sec`

`median chapter`

`slowest chapter`

`likely truncated`

`target adherence`

`python` is not recognized

`--chapters-only` removed my epilogue

`chaptertimes.ps1` finds too many Markdown files