diff --git a/README.md b/README.md index ee1d180..211519a 100644 --- a/README.md +++ b/README.md @@ -399,6 +399,9 @@ import { Contributions are welcome! Please open an issue or submit a pull request. +[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) explains how the client works — +there is no public Overleaf API, so it authenticates as a browser session. + ## License MIT © [Alexander Loth](https://alexloth.com) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..2e604e2 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,164 @@ +# Architecture + +Orientation for anyone about to open `src/client.ts` and wonder why it looks +like that. For setup and pull request mechanics, see +[CONTRIBUTING.md](../CONTRIBUTING.md). + +## The premise: there is no Overleaf API + +Overleaf publishes no REST API for the free tier. olcli is not an API client — +it **authenticates as a logged-in browser session and calls the same endpoints +the web editor's own JavaScript calls.** + +Almost every design decision follows from that one fact: + +- Authentication is a session cookie, because that is what a browser holds. +- Reading project data means parsing Overleaf's HTML, because that is where a + server-rendered page puts it. +- Write endpoints are whatever the web editor posts to. +- Nothing is versioned or documented, so **anything here can break when + Overleaf ships a redesign.** The layered fallbacks scattered through + `client.ts` are not defensive habit; each one is a redesign that already + happened. + +Requests carry a `User-Agent` of `olcli/`. olcli does not pretend to +be Chrome — the session is a real one belonging to the user running it. + +## Authentication + +Two ways in, both ending at the same place: + +| Entry point | What it does | +|---|---| +| `OverleafClient.fromSessionCookie()` | Takes a cookie the user copied from their browser | +| `OverleafClient.fromPasswordLogin()` | Submits the login form (self-hosted instances without reCAPTCHA) | + +Both then need a second credential. Overleaf requires a **CSRF token** on every +state-changing request, which is what stops another site using your cookie +against it. The token is not secret — the web editor needs it in the page to +make its own requests — so olcli fetches a page with the cookie and reads the +token out of the HTML (`extractCsrfToken`). From then on every request carries +both: + +``` +Cookie: overleaf_session2=... +X-Csrf-Token: ... +``` + +`applySetCookieHeaders()` folds any `Set-Cookie` from each response back into +the in-memory jar, so a session that rotates mid-run keeps working — the same +bookkeeping a browser does. + +Credential *storage* lives in `src/config.ts`, deliberately apart from the +client: env var, then `.olauth` in the current directory, then the global +config file. The client itself never reads any of them. + +## Reading: HTML scraping, then Socket.IO + +Project data is server-rendered into `` tags, so +`listProjects()` and `getProjectInfo()` parse the page with `cheerio`. Each has +several fallbacks tried in order, because the tag names and shapes have changed +more than once. + +The file tree is the awkward one. It used to live in `ol-project`; it no longer +does. `getProjectFromSocket()` recovers it by **speaking Socket.IO 0.9 by +hand** — handshake for a session id, `xhr-polling` for packets, decode the +frames, answer the `2::` heartbeats, and pull the tree out of the +`joinProjectResponse` event. + +This is the most fragile surface in the repository, and the least +self-evident. It is also unavoidable: that payload is where the tree is now. +Results are cached per project in `folderTreeCache` so a multi-file upload +does not repeat the whole dance for every file. + +## Writing: upload replaces, it does not edit + +| Operation | Request | +|---|---| +| Read all files | `GET /project//download/zip` | +| Write a file | `POST /project//upload?folder_id=` (multipart, field `qqfile`) | +| Delete | `DELETE /project//{doc,file,folder}/` | +| Rename an entity | `POST /project////rename` | +| Rename the project | `POST /project//rename` | +| Compile | `POST /project//compile` | + +**The most important thing to understand about writes:** typing in the Overleaf +editor sends character-level operations over the collaboration socket — an +operational transform stream that merges concurrent edits. `uploadFile()` does +not do that. It posts a whole file to the upload endpoint, exactly as if you +had dragged a same-named file into the web UI. + +So a `push` **overwrites**. It does not merge, and it cannot: there is no +three-way merge to perform, only a file replacing a file. That is why +`olcli diff` exists — previewing what a push will overwrite is the only +protection against a collaborator's edit being replaced — and why `diff` +fetches the remote fresh rather than comparing against the last pull. + +Reading the whole project is one request, not one per file: `downloadProject()` +returns the entire project as a zip. `pull`, `sync` and `diff` all use it. + +## The transport + +Everything goes through one private method, `httpRequest()`, built on +`node:http`/`node:https` rather than `fetch`. That is not preference: `fetch` +validates response headers as Latin-1 and throws on a `Content-Disposition` +carrying a non-ASCII project name, which made downloads fail for anyone with an +accented title ([#2](https://github.com/aloth/olcli/issues/2)). It also handles +redirects, timeouts, and serialising `FormData` into a multipart body. + +`--verbose` makes it log every request and response to stderr, which is the +first thing to reach for when Overleaf changes something. + +## Module map + +Which files need an Overleaf account to exercise, and which do not. This is the +main thing to know before adding a feature, because it decides where the logic +should go. + +**Pure — data in, data out. No network, no filesystem, unit-tested:** + +| Module | Responsibility | +|---|---| +| `diff.ts` | Compare two file trees; render unified diffs | +| `ignore.ts` | The three ignore layers and the `.pdf`-next-to-`.tex` rule | +| `paths.ts` | Remote path normalisation; zip-slip containment | +| `rename-plan.ts` | Plan bulk project renames before applying any | +| `prompt.ts` | Keystroke handling for the password prompt | +| `scan.ts` | Walk a local directory, applying ignore rules | + +**Talks to Overleaf:** + +| Module | Responsibility | +|---|---| +| `client.ts` | Every request. The browser-session model lives here | +| `config.ts` | Credential resolution and storage | + +**Entry points, all thin over the two above:** + +| Module | Binary | +|---|---| +| `cli.ts` | `olcli` — argument parsing and terminal output | +| `mcp.ts` | `olcli-mcp` — the same operations as MCP tools | +| `remote-helper.ts` | `git-remote-overleaf` — `gitremote-helpers(7)` protocol | +| `index.ts` | The programmatic API re-exported from the package root | + +New logic belongs in the pure column wherever it can go. That is why `scan.ts` +exists at all: `push` and `sync` each carried their own copy of the same walk +loop and had already drifted apart, and `diff` would have made a third. The +same reasoning produced `rename-plan.ts` and `diff.ts`. + +`client.ts` request *construction* can also be tested without an account, by +pointing the client at a local HTTP server that captures the outgoing request — +see `test/client.test.ts`. + +## When Overleaf breaks it + +The usual failure is a redesign moving data somewhere else. Reliable order: + +1. `olcli --verbose ` — see the actual request and response. +2. If a page parse returns nothing, fetch the page in a browser with devtools + and look for the `ol-*` meta tag. Add a fallback; keep the existing ones, + since self-hosted instances run older versions. +3. If the file tree is what broke, suspect `getProjectFromSocket()` first. +4. `olcli check` reports which credential source is in play, without printing + any secret. diff --git a/src/client.ts b/src/client.ts index 1d2c9f5..4dbd3fe 100644 --- a/src/client.ts +++ b/src/client.ts @@ -1,8 +1,18 @@ /** - * Overleaf API Client + * Overleaf client * - * Provides programmatic access to Overleaf's REST APIs for project - * management, file operations, and LaTeX compilation. + * Project management, file operations and LaTeX compilation against an + * Overleaf instance. + * + * These are not Overleaf's public APIs - there are none for the free tier. + * This client authenticates as a logged-in browser session and calls the same + * endpoints the web editor's own JavaScript calls: a session cookie plus a + * CSRF token scraped from the page, project data parsed out of `ol-*` meta + * tags, and the file tree recovered over the collaboration socket. Nothing + * here is versioned or documented by Overleaf, so the layered fallbacks below + * are not defensive habit - each one is a redesign that already happened. + * + * Read docs/ARCHITECTURE.md before changing anything in this file. */ import * as cheerio from 'cheerio'; @@ -391,6 +401,22 @@ export class OverleafClient { return new OverleafClient({ cookies: bootstrapClient.cookies, csrf: projectCsrf, baseUrl }); } + /** + * Pull the CSRF token out of a rendered Overleaf page. + * + * Overleaf requires this on every state-changing request, which is what + * stops another site from using your session cookie against it. It is not a + * secret - the web editor needs it in the page to make its own requests - so + * reading it back out of the HTML is the intended way for a session to + * obtain one. See docs/ARCHITECTURE.md. + * + * The three lookups are not belt-and-braces. Each is where the token lived + * at some point: the `ol-csrfToken` meta tag is current, the hidden form + * input is what older releases shipped, and the inline-script scrape catches + * self-hosted instances older still. Removing the later ones breaks + * self-hosted users without breaking anything on overleaf.com, so the + * failure would not show up here. + */ private static extractCsrfToken($: cheerio.CheerioAPI): string | undefined { let csrf = $('meta[name="ol-csrfToken"]').attr('content'); if (!csrf) { @@ -571,7 +597,11 @@ export class OverleafClient { const html = response.body as string; const $ = cheerio.load(html); - // Try new Overleaf structure first (PR #82) + // There is no projects API; the list is server-rendered into a meta tag, + // so this parses Overleaf's own HTML. The three methods below are three + // successive shapes that tag has had - newest first, oldest last. A + // self-hosted instance can be running any of them, which is why the older + // ones stay. See docs/ARCHITECTURE.md. let projectsData: any[] = []; // Method 1: ol-prefetchedProjectsBlob (newest Overleaf) @@ -757,6 +787,17 @@ export class OverleafClient { * Fetch the full project object via the collaboration socket. * Returns the `project` field of the joinProjectResponse, which contains * the rootFolder tree and other metadata that used to live in ol-project. + * + * This is a hand-written Socket.IO 0.9 client: handshake for a session id, + * `xhr-polling` for packets, decode the frames, answer the `2::` heartbeats, + * disconnect with `0::`. No library - the protocol is old enough that + * depending on one to speak it would cost more than the forty lines below. + * + * It is the most fragile surface in the repository and the least obvious, + * because it reimplements an undocumented internal protocol rather than + * calling an endpoint. It exists because the file tree left the meta tags + * and this payload is where it went; there is no HTTP route that returns it. + * When the tree is what broke, suspect this method first. */ private async getProjectFromSocket(projectId: string): Promise { let sid: string | null = null; @@ -1631,6 +1672,22 @@ export class OverleafClient { * If folderTree is provided and fileName contains a path (e.g. 'figures/img.png'), * the file will be uploaded into the correct subfolder, creating it if needed. */ + /** + * Upload a file, replacing any file of the same name. + * + * This **overwrites**; it does not edit. Typing in the Overleaf editor sends + * character-level operations over the collaboration socket, and those merge + * with concurrent edits. This posts a whole file to the upload endpoint - + * the same thing as dragging a same-named file into the web UI - so whatever + * was there is gone. + * + * That is why `push` has no merge semantics and cannot grow any: there is no + * three-way merge available, only a file replacing a file. It is also why + * `olcli diff` exists, and why it fetches the remote fresh rather than + * comparing against the last pull - previewing what a push will overwrite is + * the only thing standing between a collaborator's edit and its replacement. + * See docs/ARCHITECTURE.md. + */ async uploadFile( projectId: string, folderId: string | null,