docs: add docs/ARCHITECTURE.md and explain the scar tissue in client.ts

README.md

@@ -399,6 +399,9 @@ import {
 
 Contributions are welcome! Please open an issue or submit a pull request.
 
+[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) explains how the client works —
+there is no public Overleaf API, so it authenticates as a browser session.
+
 ## License
 
 MIT © [Alexander Loth](https://alexloth.com)

docs/ARCHITECTURE.md

@@ -0,0 +1,164 @@
+# Architecture
+
+Orientation for anyone about to open `src/client.ts` and wonder why it looks
+like that. For setup and pull request mechanics, see
+[CONTRIBUTING.md](../CONTRIBUTING.md).
+
+## The premise: there is no Overleaf API
+
+Overleaf publishes no REST API for the free tier. olcli is not an API client —
+it **authenticates as a logged-in browser session and calls the same endpoints
+the web editor's own JavaScript calls.**
+
+Almost every design decision follows from that one fact:
+
+- Authentication is a session cookie, because that is what a browser holds.
+- Reading project data means parsing Overleaf's HTML, because that is where a
+  server-rendered page puts it.
+- Write endpoints are whatever the web editor posts to.
+- Nothing is versioned or documented, so **anything here can break when
+  Overleaf ships a redesign.** The layered fallbacks scattered through
+  `client.ts` are not defensive habit; each one is a redesign that already
+  happened.
+
+Requests carry a `User-Agent` of `olcli/<version>`. olcli does not pretend to
+be Chrome — the session is a real one belonging to the user running it.
+
+## Authentication
+
+Two ways in, both ending at the same place:
+
+| Entry point | What it does |
+|---|---|
+| `OverleafClient.fromSessionCookie()` | Takes a cookie the user copied from their browser |
+| `OverleafClient.fromPasswordLogin()` | Submits the login form (self-hosted instances without reCAPTCHA) |
+
+Both then need a second credential. Overleaf requires a **CSRF token** on every
+state-changing request, which is what stops another site using your cookie
+against it. The token is not secret — the web editor needs it in the page to
+make its own requests — so olcli fetches a page with the cookie and reads the
+token out of the HTML (`extractCsrfToken`). From then on every request carries
+both:
+
+```
+Cookie: overleaf_session2=...
+X-Csrf-Token: ...
+```
+
+`applySetCookieHeaders()` folds any `Set-Cookie` from each response back into
+the in-memory jar, so a session that rotates mid-run keeps working — the same
+bookkeeping a browser does.
+
+Credential *storage* lives in `src/config.ts`, deliberately apart from the
+client: env var, then `.olauth` in the current directory, then the global
+config file. The client itself never reads any of them.
+
+## Reading: HTML scraping, then Socket.IO
+
+Project data is server-rendered into `<meta name="ol-*">` tags, so
+`listProjects()` and `getProjectInfo()` parse the page with `cheerio`. Each has
+several fallbacks tried in order, because the tag names and shapes have changed
+more than once.
+
+The file tree is the awkward one. It used to live in `ol-project`; it no longer
+does. `getProjectFromSocket()` recovers it by **speaking Socket.IO 0.9 by
+hand** — handshake for a session id, `xhr-polling` for packets, decode the
+frames, answer the `2::` heartbeats, and pull the tree out of the
+`joinProjectResponse` event.
+
+This is the most fragile surface in the repository, and the least
+self-evident. It is also unavoidable: that payload is where the tree is now.
+Results are cached per project in `folderTreeCache` so a multi-file upload
+does not repeat the whole dance for every file.
+
+## Writing: upload replaces, it does not edit
+
+| Operation | Request |
+|---|---|
+| Read all files | `GET /project/<id>/download/zip` |
+| Write a file | `POST /project/<id>/upload?folder_id=<id>` (multipart, field `qqfile`) |
+| Delete | `DELETE /project/<id>/{doc,file,folder}/<entityId>` |
+| Rename an entity | `POST /project/<id>/<type>/<entityId>/rename` |
+| Rename the project | `POST /project/<id>/rename` |
+| Compile | `POST /project/<id>/compile` |
+
+**The most important thing to understand about writes:** typing in the Overleaf
+editor sends character-level operations over the collaboration socket — an
+operational transform stream that merges concurrent edits. `uploadFile()` does
+not do that. It posts a whole file to the upload endpoint, exactly as if you
+had dragged a same-named file into the web UI.
+
+So a `push` **overwrites**. It does not merge, and it cannot: there is no
+three-way merge to perform, only a file replacing a file. That is why
+`olcli diff` exists — previewing what a push will overwrite is the only
+protection against a collaborator's edit being replaced — and why `diff`
+fetches the remote fresh rather than comparing against the last pull.
+
+Reading the whole project is one request, not one per file: `downloadProject()`
+returns the entire project as a zip. `pull`, `sync` and `diff` all use it.
+
+## The transport
+
+Everything goes through one private method, `httpRequest()`, built on
+`node:http`/`node:https` rather than `fetch`. That is not preference: `fetch`
+validates response headers as Latin-1 and throws on a `Content-Disposition`
+carrying a non-ASCII project name, which made downloads fail for anyone with an
+accented title ([#2](https://github.com/aloth/olcli/issues/2)). It also handles
+redirects, timeouts, and serialising `FormData` into a multipart body.
+
+`--verbose` makes it log every request and response to stderr, which is the
+first thing to reach for when Overleaf changes something.
+
+## Module map
+
+Which files need an Overleaf account to exercise, and which do not. This is the
+main thing to know before adding a feature, because it decides where the logic
+should go.
+
+**Pure — data in, data out. No network, no filesystem, unit-tested:**
+
+| Module | Responsibility |
+|---|---|
+| `diff.ts` | Compare two file trees; render unified diffs |
+| `ignore.ts` | The three ignore layers and the `.pdf`-next-to-`.tex` rule |
+| `paths.ts` | Remote path normalisation; zip-slip containment |
+| `rename-plan.ts` | Plan bulk project renames before applying any |
+| `prompt.ts` | Keystroke handling for the password prompt |
+| `scan.ts` | Walk a local directory, applying ignore rules |
+
+**Talks to Overleaf:**
+
+| Module | Responsibility |
+|---|---|
+| `client.ts` | Every request. The browser-session model lives here |
+| `config.ts` | Credential resolution and storage |
+
+**Entry points, all thin over the two above:**
+
+| Module | Binary |
+|---|---|
+| `cli.ts` | `olcli` — argument parsing and terminal output |
+| `mcp.ts` | `olcli-mcp` — the same operations as MCP tools |
+| `remote-helper.ts` | `git-remote-overleaf` — `gitremote-helpers(7)` protocol |
+| `index.ts` | The programmatic API re-exported from the package root |
+
+New logic belongs in the pure column wherever it can go. That is why `scan.ts`
+exists at all: `push` and `sync` each carried their own copy of the same walk
+loop and had already drifted apart, and `diff` would have made a third. The
+same reasoning produced `rename-plan.ts` and `diff.ts`.
+
+`client.ts` request *construction* can also be tested without an account, by
+pointing the client at a local HTTP server that captures the outgoing request —
+see `test/client.test.ts`.
+
+## When Overleaf breaks it
+
+The usual failure is a redesign moving data somewhere else. Reliable order:
+
+1. `olcli --verbose <command>` — see the actual request and response.
+2. If a page parse returns nothing, fetch the page in a browser with devtools
+   and look for the `ol-*` meta tag. Add a fallback; keep the existing ones,
+   since self-hosted instances run older versions.
+3. If the file tree is what broke, suspect `getProjectFromSocket()` first.
+4. `olcli check` reports which credential source is in play, without printing
+   any secret.

src/client.ts

@@ -1,8 +1,18 @@
 /**
- * Overleaf API Client
+ * Overleaf client
  *
- * Provides programmatic access to Overleaf's REST APIs for project
- * management, file operations, and LaTeX compilation.
+ * Project management, file operations and LaTeX compilation against an
+ * Overleaf instance.
+ *
+ * These are not Overleaf's public APIs - there are none for the free tier.
+ * This client authenticates as a logged-in browser session and calls the same
+ * endpoints the web editor's own JavaScript calls: a session cookie plus a
+ * CSRF token scraped from the page, project data parsed out of `ol-*` meta
+ * tags, and the file tree recovered over the collaboration socket. Nothing
+ * here is versioned or documented by Overleaf, so the layered fallbacks below
+ * are not defensive habit - each one is a redesign that already happened.
+ *
+ * Read docs/ARCHITECTURE.md before changing anything in this file.
  */
 
 import * as cheerio from 'cheerio';
@@ -391,6 +401,22 @@ export class OverleafClient {
     return new OverleafClient({ cookies: bootstrapClient.cookies, csrf: projectCsrf, baseUrl });
   }
 
+  /**
+   * Pull the CSRF token out of a rendered Overleaf page.
+   *
+   * Overleaf requires this on every state-changing request, which is what
+   * stops another site from using your session cookie against it. It is not a
+   * secret - the web editor needs it in the page to make its own requests - so
+   * reading it back out of the HTML is the intended way for a session to
+   * obtain one. See docs/ARCHITECTURE.md.
+   *
+   * The three lookups are not belt-and-braces. Each is where the token lived
+   * at some point: the `ol-csrfToken` meta tag is current, the hidden form
+   * input is what older releases shipped, and the inline-script scrape catches
+   * self-hosted instances older still. Removing the later ones breaks
+   * self-hosted users without breaking anything on overleaf.com, so the
+   * failure would not show up here.
+   */
   private static extractCsrfToken($: cheerio.CheerioAPI): string | undefined {
     let csrf = $('meta[name="ol-csrfToken"]').attr('content');
     if (!csrf) {
@@ -571,7 +597,11 @@ export class OverleafClient {
     const html = response.body as string;
     const $ = cheerio.load(html);
 
-    // Try new Overleaf structure first (PR #82)
+    // There is no projects API; the list is server-rendered into a meta tag,
+    // so this parses Overleaf's own HTML. The three methods below are three
+    // successive shapes that tag has had - newest first, oldest last. A
+    // self-hosted instance can be running any of them, which is why the older
+    // ones stay. See docs/ARCHITECTURE.md.
     let projectsData: any[] = [];
 
     // Method 1: ol-prefetchedProjectsBlob (newest Overleaf)
@@ -757,6 +787,17 @@ export class OverleafClient {
    * Fetch the full project object via the collaboration socket.
    * Returns the `project` field of the joinProjectResponse, which contains
    * the rootFolder tree and other metadata that used to live in ol-project.
+   *
+   * This is a hand-written Socket.IO 0.9 client: handshake for a session id,
+   * `xhr-polling` for packets, decode the frames, answer the `2::` heartbeats,
+   * disconnect with `0::`. No library - the protocol is old enough that
+   * depending on one to speak it would cost more than the forty lines below.
+   *
+   * It is the most fragile surface in the repository and the least obvious,
+   * because it reimplements an undocumented internal protocol rather than
+   * calling an endpoint. It exists because the file tree left the meta tags
+   * and this payload is where it went; there is no HTTP route that returns it.
+   * When the tree is what broke, suspect this method first.
    */
   private async getProjectFromSocket(projectId: string): Promise<any | null> {
     let sid: string | null = null;
@@ -1631,6 +1672,22 @@ export class OverleafClient {
    * If folderTree is provided and fileName contains a path (e.g. 'figures/img.png'),
    * the file will be uploaded into the correct subfolder, creating it if needed.
    */
+  /**
+   * Upload a file, replacing any file of the same name.
+   *
+   * This **overwrites**; it does not edit. Typing in the Overleaf editor sends
+   * character-level operations over the collaboration socket, and those merge
+   * with concurrent edits. This posts a whole file to the upload endpoint -
+   * the same thing as dragging a same-named file into the web UI - so whatever
+   * was there is gone.
+   *
+   * That is why `push` has no merge semantics and cannot grow any: there is no
+   * three-way merge available, only a file replacing a file. It is also why
+   * `olcli diff` exists, and why it fetches the remote fresh rather than
+   * comparing against the last pull - previewing what a push will overwrite is
+   * the only thing standing between a collaborator's edit and its replacement.
+   * See docs/ARCHITECTURE.md.
+   */
   async uploadFile(
     projectId: string,
     folderId: string | null,