From 30f46e6e87f03f36dfcde41a3b2814c0fb2879f2 Mon Sep 17 00:00:00 2001 From: "Bryan A. Jones" Date: Tue, 19 May 2026 14:34:22 +0500 Subject: [PATCH 01/22] wip: cache. --- docs/design.md | 25 +- docs/implementation.md | 120 +----- extensions/standalone/src/filewatcher.rs | 9 +- server/cache.pptx | Bin 0 -> 49549 bytes server/src/processing.rs | 393 ++++++++++++++---- server/src/processing/cache.rs | 505 +++++++++++++++++++++++ server/src/processing/tests.rs | 208 +++++----- server/src/translation.rs | 18 +- server/src/webserver.rs | 10 +- 9 files changed, 961 insertions(+), 327 deletions(-) create mode 100644 server/cache.pptx create mode 100644 server/src/processing/cache.rs diff --git a/docs/design.md b/docs/design.md index 173ec9f4..8d1b345a 100644 --- a/docs/design.md +++ b/docs/design.md @@ -79,20 +79,19 @@ These form a set of high-level requirements to guide the project. * A gathering element: given an anchor, it shows the context of all hyperlinks to this anchor. - * If the hyperlink is a heading, the context extends to the next same-level - heading; - * If the hyperlink is a start of context, the context ends at the end of - context or end of file, whichever comes first. - * Otherwise, the context extends to the following code block. + * With no other parameters, a hyperlink to a gathering element includes the + current doc block and the next block if the next block is a code block. + The link may also include start and end query parameters to define a + multi-block span. + * The gather element can include presentation options: specify the order of + gathered blocks as a list or graph; links may be numbered, or include + prev/next links. * A report view: an extended gathering element that operates more like a query, producing nested, hierarchical results from the codebase. * Headings can be collapsed, which code code and doc blocks until the next same-level heading. - * A sequencing/path element: given a starting hyperlink, it produces prev/next - icons to show a startup/shutdown sequence, etc. - * A graph view: shows the entire document as a directed graph of hyperlinks. * An inlined output mode, like Jupyter: includes graphs and console output @@ -107,11 +106,12 @@ These form a set of high-level requirements to guide the project. * Interactive learning support: multiple choice, fill-in-th-blank, short/long answer, coding problem, etc. from Runestone or similar. - * Autogenerated anchors for all anchors (headings, hyperlinks, etc.) + * Lazily autogenerated ids for all targets (headings, hyperlinks, etc.); these + are generated when the target is referenced. * Hyperlinks to identifiers in code (use [ctags](https://github.com/universal-ctags/ctags)); perhaps auto-generate - headings for these identifiers? + headings for these identifiers? Use a language server? * An API view; show only parts of the code that's exported/publicly-accessible. @@ -120,9 +120,8 @@ These form a set of high-level requirements to guide the project. * Files/anchors can be freely moved without breaking links. This requires all anchors to be globally unique. HTML allows upper/lowercase ASCII plus the - hyphen and underscore for IDs, meaning that a 5-character string provides - - > 250 million unique anchors. + hyphen and underscore for IDs, meaning that a 5-character string provides > + 250 million unique anchors. * Make picking a file/anchor easy: provide a searchable, expanded TOC listing every anchor. * Provide edit and view options. (Rely on an IDE to edit raw source.) diff --git a/docs/implementation.md b/docs/implementation.md index e1510040..1f24797a 100644 --- a/docs/implementation.md +++ b/docs/implementation.md @@ -159,10 +159,9 @@ On load: * Classify the file; inputs are mutable global state (which, if present, indicates this is a project build), if the file is a TOC, the file's binary data, and the file's path. Output of the classification: binary, raw text, a - CodeChat document (a Markdown file), or a CodeChat file. The load processing - pipelines + CodeChat document (a Markdown file), or a CodeChat file. -For CodeChat files: +The load processing pipelines for CodeChat files: * (CodeChat files only) Run pre-parse hooks: they receive source code, file metadata. Examples: code formatters. Skip if cache is up to date. @@ -170,14 +169,6 @@ For CodeChat files: * Run post-parse hooks: they receive an array of code and doc blocks. * Transform Markdown to HTML. * Run HTML hooks: - * Update the cache for the current file only if the current file's cache is - stale. To do this, walk the DOM of each doc block. The hook specifies which - tags it wants, and the tree walker calls the hook when it encounters these. - If this requires adding/changing anything (anchors, for example), mark the - document as dirty. - * Update tags whose contents depend on data from other files. Hooks work the - same as the cache updates, but have a different role. They're always run, - while the cache update is skipped when the cache is current. * Determine next/prev/up hyperlinks based on this file's location in the TOC. * Transform the code and doc blocks into CodeMirror's format. @@ -209,112 +200,9 @@ On save: * Save the file to disk. * If dirty, re-load the file. -### Table of contents - -Ideas: - -* Something that reflects the filesystem. Subdirectories are branches, files are - leaves in the TOC tree. Problems: - * Subdirectories should have content, such as a readme. Assume a readme file - titles and provides content for a subdirectory? Or provide a config file - setting to assign this? - * I'd like the ability to relocate files/directories. The means a config file - that tracks this movement. - * We need ignores. - * To reorder files in the TOC, need a config file per directory to store this - ordering. - * Pro: all files are automatically included, so adding a new file is - automatic. The hierarchy is mostly defined by the filesystem, which is nice. - A GUI with drag and drop would make this really simple to maintain. - * Con: a lot of work/rewrite. - * So: readme.md provides a title and contents for a subdirectory. A config - file in each directory specifies ordering of files, titles for non-CodeChat - files (PDFs, etc.), moves of files/directories from other directories, and - ignores. -* Use mdbook's idea -- a very specific structure for a toc.md file. Simple, but - doesn't auto-update as files are added. -* Current TOC isn't immediately useful. Too much flexibility. - -Another topic: how to reconcile headings in a file with the TOC? - -* Separate them -- headings have orthogonal numbering to the TOC. I think this - is simplest. I just need the right way to display it; mdbook is reasonable in - this regard. I'll use this. -* Combine them -- H1 is current number, H2 is a subhead, etc. But this means the - TOC's numbering requires reading the contents of all files referenced by the - TOC, which could be slow. - ### Cache data format -The cache stores the location (file name and ID), numbering (of headings and -figures/equations/etc.), and contents (title text or code/doc blocks for tags) -of a target. Targets are HTML anchors (such as headings, figure titles, display -equations, etc.) or tags. - -Goals: - -* Given a file name and/or ID, retrieve the associated location, numbering, and - contents. -* Perform a search of the contents of all targets, returning a list of matching - targets. -* Given a file name and/or ID, provide a list of all targets in the containing - file. - -Cache data structure: - -* A hashmap of (Path, target data structure). TBD: think about ownership. I - think a page is the owner of all targets. -* A hashmap of (ID, target data structure).  - -Target data structure: - -* Location: the containing page and an `Option` containing the ID, if - assigned. -* Page numbering: `[Option, ...]` where each i32 in the list represents the - number of a H1..6 element (non-TOC) or the numbering of a list item (TOC); - `None` represents a missing level of the hierarchy (e.g. H1 following by and - H3, with no H2 between). -* Type: page, heading, link, tag, caption, equation; numbered items (caption, - equation) also include the current number. Pages include the page data - structure. -* Contents: either a string of HTML (would prefer Markdown) or a vec of code/doc - blocks. Page contents are an empty string. - -Page data structure: - -* Path: the path to this file. -* File info: timestamp, etc. to compare with the filesystem in order to - determine if this cache entry is up to date or no. An option, in the case that - the file doesn't exist -- it's the target of a broken link. -* TOC location: `[i32, ...]` gives the numbering of this page in the TOC; if - it's not in the TOC, this is an empty list. -* Vector of targets on this page. -* (Maybe) first ID on this page. - -Pseudocode: - -1. Create a hashmap of (file paths to index, list of links depending on this - file). Initialize it with the current file. -2. For each file in the hashset: - 1. If this is the first file, we already have its DOM. Otherwise, load the - file from disk and compute the DOM. - 2. Given a file's DOM, first create its page data structure. Pre-existing - cache data provides the TOC numbering. - 3. For each target in the DOM (non-TOC) / numbered item (TOC), add the - target's data structure to the page's vector of targets, updating the - current numbering if this is a numbered item (heading, caption, etc.) and - inserting the HTML to set its number in the DOM. - 4. If this is the first file: for each link in the DOM, if the link is local - and autotitled, look for it in the cache. If it's not in the cache or if - the cache for that file is outdated, add the referring file to the hashset - of files to update if it's not in the hashmap; append this link to its - list of dependent links. - 5. For each link in the list of links depending on this file, update it with - the loaded content. - -References: - -* Hyperlinks with no link text are auto-titled. Look up +Documented elsewhere. ### IDE/editor integration @@ -549,7 +437,7 @@ with descriptions of each setting. want something that includes type validation and allows comments within the config file. Perhaps JSON with a pre-parse step to discard comments then [JSON Typedef](https://jsontypedef.com/)? Possibly, vlang can do this - somewhat, since it wants to decode JSON into a V struct.) + somewhat, since it wants to decode JSON into a V struct. Organization ------------ diff --git a/extensions/standalone/src/filewatcher.rs b/extensions/standalone/src/filewatcher.rs index e6589d1a..694a8ae8 100644 --- a/extensions/standalone/src/filewatcher.rs +++ b/extensions/standalone/src/filewatcher.rs @@ -784,7 +784,7 @@ mod tests { }; use code_chat_editor::{ processing::{ - CodeChatForWeb, CodeMirror, CodeMirrorDiffable, SourceFileMetadata, TranslationResults, + CodeChatForWeb, CodeMirror, CodeMirrorDiffable, SourceFileMetadata, source_to_codechat_for_web, }, webserver::{ @@ -935,13 +935,12 @@ mod tests { // Check the contents. let translation_results = source_to_codechat_for_web( "", - &"py".to_string(), + Path::new("foo.py"), umc.contents.as_ref().unwrap().version, false, - false, + None, ); - let tr = cast!(translation_results, Ok); - let codechat_for_web = cast!(tr, TranslationResults::CodeChat); + let codechat_for_web = translation_results.unwrap(); assert_eq!(umc.contents, Some(codechat_for_web)); // Report any errors produced when removing the temporary directory. diff --git a/server/cache.pptx b/server/cache.pptx new file mode 100644 index 0000000000000000000000000000000000000000..bca3924e71d361ceaf4fbcc2ade6ba96bf490193 GIT binary patch literal 49549 zcmeFZQBF_T=`ab{ve*Opl1Cyy^G6VGRAy;CZf`?QVoApTn84h~vzr>Tk z=*Nhe*_w?zioHEa#mgm@f)NW25yx^~zFxA}4BLE}rGr@k4cig`PoJ&#{h^fyjS=Hk z_OvTSxz5v~(&P|<7=R^fKtsVk_M`$I$1y?bb-pZ#RuOh5Gf+MjgjS^-;O*+dybHK#*dYTM#*CRKSEyq{@qdL3S z^lX{KR74o!3KaF*M8x{>kN{jQ`z16tB2t zB{Ci1a?Z{UNylV#+rH3$X0E-}#};X#Mf9+Dd~#?g3!G1d1zq4G@HG@Ok6QZr(7uQ5 z*B3B=-2WE)X3tLfz~8aI{~g*;-?6XjU~J_`NBdX%e`EfCaG?MHx?YvAAp!jJJLW-l z2qwK*Hz*0`t!#x(vPWM53>iAGwy?!Y=DWM{clGsY0=gzAdGU6L)omYbkgt;JVRZWH zEBND9`qXx)+-&erTkshQS=m5kAz_pB7qoP$a?ONj1>Ng@Mhe`h_w{056-|2Jhqgh+ zd!0HFupc@|bw!o!)RDDSoMWZ?Fyy7v4=t06X*z-wa0|WSX>u4ajgmWyHHMW-ku@8m z%^949qon2GNwVW1DH8-{p=o5RhJ|K(-k6PGB-iRZV3vPLo4vT^&pN9tc+YZ*<{WpH zgxi*)s}BpcA;+F7vO0Oo5Uex03m)AU>PO{@INUO3!u?~+U>jiC6@SM|9Si^f^7k2b zc20D54#tk(Q>MO?xvkA#6Xu`PEt?H|I14>YvcFaT&>lWHm*rLXo8i>wBl;ivx;OdAl z+An8^37vs&3GA0fh#VvoY!=58Gm~hU((!Yi@fQ-{J8bbuOYUXX^5xT-4;Z!s%2}mF zqd7IMbP;DtcGZ5c8ol)>6TV2r3j`kLdWtv2*s+Y|sTp zO>YVDEdf7#K%MI91hl%|osiBlp0opM&V81C^6SHe{r(wodp<)o^CqL-yn>ZeM zb3kb@zkYm7xKpI$*CP+_p(>MEo2&ZSdY5{!*<8F+?RF>bnVcXUi?>ULI>YDFwS~7u zN#M@Tfy9oU67XLCHo&h<6m}3Y?Kc0w%78*@28yKBlT4{3>)-oI``P0X$j&zbTP2%j zm#cA-PgM<$(-T!gy#F|?HCK$K;(h=CynYXBl>Zpkj#lPI#*TD#Onk==!pFP(1a>$LA^Isl_>2B0isR6cn;InvJae^*!s{ zcvG*uxVU^?ELIlHB#SH9GTHljUEz&`j!`scvYt}YIbGuB(|)%eMNMaMD^FJ@=K6Y2 zlvGS3WSLk*n97tlrFrBKl33Sa+VNvlp=hzF4an7WY3*CatuN}FyoImqMzC+WzogxD zl2ZzxWm#d4J5dSEV_Mc}B`3>uG2Nh>3n+jX{5jPm&Gs`jQa&zG-d9kt@UZeE9g@%z z>v(tZB#=)A)`2x>1ysYX)jVllBiLAvaXo%+zuG^oiHkPxV{Oc22l(9f;${?e#f*1J zz%~(ox%BTU(&DMxI7e|~O~S0zIJoRUdc4$9X)htSS1vGUb2`_hl^=7ND-pwi-?6*!I{B51d5D z@m8K-m_UxS#(J~+(k#$iL!Ph;%`ct>-T7@@c!ND}^B3QhLe5om0+rbXan7c%LKmtz;8AO(F1Dd2BW05LVSsk z!a0LB@DbhwIOT3mD|bi~saSpmjx94CB}^26*N9!r>)as0o}R(bsjnm!5r0dF%))j9 zyd~tV0?3nRPyXQqmz6@N9A(XMcoCAJBWA$i1Vnvj)f*grfsWH!)*$s-f4a%-cA!zS zR){KKp6=omSj04YkVK!8*!R&~KCD;4=OG{H%4AaGkjx5D^P3+DhT+8puW{^4a0phN zmVza+IZLmt(A3IM)7hq$-dP6bCS45lg{1`ZeV#_PT`aEGwKF>~GrjdComw$@x!WewAVoKk4thibgYCUnA?=C z_YsaJ>pU2%&S5~SBg;&9T-SEo6n)zJ%!f?kZ9$>qCRhN`y_KVm9zhi%V%;CG779{4 zE-#K-#jKN$?M#@pR4^~m`F9{5+im4B@JmM5`{3mXy1?+xTP3=!9cqDXAjsPx-pROm zjCZvCNC0p@Cr)Gf^}(Y|v@&Up-B&tj-ZZ&2R!k}I?qk74elxJ;F z=vNfGoa;y4;7osf=dAyp&Boe3s84?*%pU2V5ytTM)U0-DbHD=sshfAzd**61mf&TO z*zfR%u-ApvtRAjc%rFvM7?JolaHRJWEQR?N6GcYRYQgHFpK!#8-BD&BqsP--{*e*b zIz7d7;2u6Kc$h$nl+{e2Y|FVFCcI%eIYlbabRbdM?r|iZuUqL+Na_z)D8|xbe&gy4 z(Rd2VvT_VD5TIigg_Xtq%59P263-%yJM!=YdnHBE0Gl44X`i;WIv`r(xkY>{Ng?1; zRCZPr#gfvJ96@Pwda>avpWb;836;(3`6iJq3xE}*4&BCf`IU89-&JkPeu+UU{D5U} zNqA?ZEru^jmu{pF^S9=!`4{~{%asDFLpxmjXVIsqO>}H(iuJyICJiOLTs2LlN-`DE z55PG*SZ9T%Np4iK)1fxw6krTc9K5UO3w8@8PB@<06>%Po>IS>$F8atj>Pgi#YAv!; zr(~(+31S>y`;O29_x!#>uZJa#kONW73ibTrcq~qGqpkAx7IdX)`tv_S_Al-kBMYaL zy1{D(?cfXY9FVFFOmGW()Y&G9V}$8iKBoYDId9972+p@I0Z$S1XI<2nFbIe!zwg)2 zF3uxV(rc7aeu)O7tB11c zt%V`DfYGN27G@<_$wHMAk10(!RC>-0VW>YAHa4#w%Y@<6(cfuxc8SOd|?h$ze%VA0#a7=nTDNr8MrG+1&?Eq_NBVaTs zde`YF`eku9E2J!j+KYrOsag}_R_C7E^DXBQCwzt#zuz$RVKTv4#gFlqo2cD}DebgX zvHDb)5g}EIymb&-DW-4-YN zWs`l=AV&Ku0m_RzYpvbx!ba8Yfu9DHZLDN39acUwR=-ct9YmpIVa()TsK3g+KJ3R% zGd~k0bUj{lRQZlxtg@&}-P7P2oYS#@Rj=ik^$t6K2{w% zIf9Kft^J%};GY1|BbvkH2u0^(q6>O;3~N`&P6>pUwh~}Z*I_yk$^~z;?!0&A@^v~l zS+Ot5=A{}J@Okilgf$B(Rc%QHx>hn^gNb2ns-#7x$OHl}^-WBCCvpU28LHqiEgk`- z{MJ!fLh+=`)^$8+3C(rQs1<#nI!mM#$UXqIFtG`JddSRaP5!*KIgUD2D^at-_!}Y3 zC5{E;BcKc`%!%z!q`3#ulofr^oFAo0w160GzmsvAv?wGls;? zbpb8xfg69s>t1hcT6Q=lp3sf3ejv*}S}v_3)a8FL0o=kpVyI(nVF_^%v1^EbJcf4_ z>G1sGJOdWe{n1p`euz6D?!iW7!X|9Ld)K;LGGkV`%H*b-BK}h=MP03*dJ@=;dUu;i91bQqFpw~kfH)dH9H zkv$I_5;(HSZ#`^S(*faDvGFwq%}EjUpU=AxepzpG`C(i-aGhuZ6PEm!%oZTv#?BW` zZ=Hom3o-Q~NBI6khU~X*?ZT|=^?xdR|3Jy(SjBOoXCuhAva}ramV3rO?#;{L{}_2( zD71dQWZ^?mJR)ys>$D&)ZT`e5MpXlw5pj(vFzhW$eLW8-?b>ym!@Q2LVIyrs?JR&K z`BHqjW0Ns;PY8M`Bk#X)h!2gtHvo_IVv|S-G(wI2l73Nd;e`{T^qoPD z6A8NpT$Mf0Hd}+BMa#u;Fu;}%slT|NaBx@41;gZcaRZwNUbU_qYcgmsp13_jC8)o# zihZeLe_D^x<7@R8H21k$)Id~ zH4BuM#<^v53LWbz zt~7-7wnf<{|7F^2REM z)_Nq_{;~BS;Km(CH3F=nNbmBA68E{JV<2x&I05#~DlNuq=N$9=r}_rXhc43I8BB_; zs|;9}aajguVk63qHMh;L*}YA&e-v|K(|NUK-$+X*sf!^n9~S7~y*2->cfxv#uvtkK!CfMXj2pW{+}C2Z7@9 zi~}!6SPmk!+kdEHT$Hd#hp5G-4*=x(jm8E|P&c=c_t^M`4de}IL9#)>ECN_-d*1hx zGca`sXHiWeO#7BKn4<{=n;ZL&v?wWew=~WmHQg37%NCyjt5mqohHmI9=#ls(J^{J*}1!8H;1%ZDLe2d-0+|m@v z5O&Za*{CEl8gqFGa$2;8|1eezt+2mDmZ=SN9H1^GhtS}qjk@5fV9o@&iY!QlNfE|G zrH5KzlN_*0m+3E9!K3pmB)obE#mjPG$U>C%PTJuyk!0Y~FEbF5BwBgQ8m%a9pFqba z6W{Q%*XhoUy%kQFjI`_4I^1*1p&nq$Hr?}rFCAdIpo!r6ngs>$x)g~ew76;R$D^#g*KP4L>Z&BBKN~eaiN#sQTAk1+vRDJ9% z_m#Vx>B#p7kJ*|lB<$1%DKlEfHE*uGZ#y`ztLD>P=opaF{<@*U%) zdJ>J7&EKB-p(Sv)KF-__7MV=QJ6}&q%csW}qI>=L%TrO;odyy}d;zyR$tVxx z*?@0__MYPmSoCV84P4%^yK#0Lw?UeGX>mN*(QeUzVd*t8%Dg0%QVnKpmXn&stM3#0 zCxtLKKeBS-6sgU9|I;W5I_u4AWfOw;bPMt|;zDE(&S ziDfGkN~!1lh;0v%AO$7K&to$RscH(34*D|GKdBix1ogq*W&91bERURvBsBVULHvuh zn?-}HGmuNBj@4SM#Wms~j6axT0p7A&_!*^p)A%hxVI-@y0$TIBrIKRDV8QZ)uv+!< zAQfe_2j0TE5z4$Tljxw1CG5*Qx&_$EVyH>64HNP>FCnid4yUxG6`!RN?3Q;oAUINC z;kjPB&&2ggt26N0ZDVSTZyK6!#_PxGQezwFCFcpH&=Td@=t2VP(C-(RW`^%Ndz_+E zCFU1~p-^*I+$di7ni3{`O-SrFJTK@sGY|A@c_kKZTCIIEKF?07EHlNVC;KBZCx<)nOPvJML)(FZk z$6QBbOo)J7-4NSCgzE9YTF&@GU@0ch zr3aEQ3d%=>zHtP8(B(^h3e8bj<91ObzmlQQg4ofbp>J3BnjgSD*mQaPzJRj~|cGaD3uq5B;2e)1ilHlM(KU(&3Ga`3sW{FS5!3#d41Ge-72tK$WA$5+(L}z4UUnF-GJQ>n2=zA1@b%HiD(BvpZMh}4Z9mtcyR=+U( z=_p}`z?$%F1xt9OQI^31nALat0_IL?(G7s4Fb6>d#yES~F_EVyDYR_CrnJ;0G4-GnJ#P)g(bNce85&dD);#}+jQ?8K|1MDfckB9MOAH{{ z_Z|NY0sw#q01o)qUH{kZ{y#0J|7F?&elOJczTkiMr!#5FVh|rW_$jzMu+>(If%ijy z(n>-fHF&-RZkL!fe6g%7o5fZ)ox+>}dR}<+7Psrz2b<2Od8e8Wj6Q`*+#YcV-Je=% z%vk!y^V7*OG~QV%MT#5(*;Vg`rK5YjGeB>ebuGj)vNN&~nRWUUOu%Z6Z^EjGNz7M) zVu9gUQW0q?0!{^3{J>3mau(Q>NVbuRf~8}AW`+~y>qbR+2+B=q$P{W`8=W5(yIUvf zc0g`g3MOV>C!!OJG?f5VQ9!Lv@)?*drN;s@pfwv&0ISYa{9XZ^Fq ziK}#E7~a!s|7>jj*TU@Y8p40KF#E4{{cqLuKeOw9XJ!B0x*k-IAy@USlC8e|0)lT~ z{=ZYIl)k&Iv(tYVpa0tV``-Ui+p@`GL+;i!-vRu!Li-E`ET0GoU8N$CX-4J^0GR`= z`MK;yny3>0s?iamC%#5HI)O>J1hwr)QO=scFxkto?B|WJYUa z-+C@t+0)0Yhi8IMPcv|GgQ2z~1#dmot4}J*VYttsY65cBz`pRU%%Od_(#If2CL^QMH=88ttv>DrKT#S6sCUYkPby zY}c7K`D>u!EObqyC$>rFnm_$#C9kWv&LgFU`&@SC%S7`e*DraZSssOaw8-|x^-g(U zQ?%@%b`?s-qq*5!h-v5At#{g2VPDG02-QqQedXe2`s&8)(>|uD&1S3xdmC!r+>Vz% z7YfX+ip09+&}N`N3c5Jl!!7fa!+i#<8dWlvX_{5@oF<%ipwncTXJsPCHlcw>cGHyV zj*_{tHwSKU4WgFvIO|sp&_g;I2yx}O_%>2}Sqbn);9wNDj3M-0^OXeLU6`YV-CwB0>-9)hktGqy#!(2gVImaMMYrjxs%)Q5uz8elLY2f@F4&s32 z`n!1hFxp~D&qof4^#>T_@rwurWP_|{;Eo!Gu)b2Quit3Q2qzft?m3h}B*jTXf zqpF9Bij$vV+B%fs?3(qT$v9sOH!fh_UppMtcLBBS?MhhrWD>wxLKydmknl!4d{OXx zf|CH^4U@8lhp#*zIZWu=6km59zSXr0b-;v)OQ`IWDpYw(P*pZ7vO_56?TO-R7E#mG59`pX$OcTilu} z20uY>K0q66XbvrGBaPr9mu{&*@k+FBnJ@OqDzr}wQ-^1PMI8Twp43i?kuV?-e_Fhu(OH&ZqXJ9maSSY26FPjp;`bV}$?`kbzSfpb`IoR6O5u zV;1pa_u|QUCC_(1eBxMci&6}yoF(c_DiRy>Llf8`tpEopKm(zQQmVf!PQ=A>v$kdQ z&`Sj8S7GKTA6xdxF254(vqVdsZ8|3t?BlV$R{@CV_!IG;>V}e;!+t`ao@* zOg2}YJMlax*={5K){F|IcM%k(PZEIlqy@oH7S2T*iXzoVb~Atecxg^DAl}?L*D5Q_ zWOtRFdxD?lrd|AmekiDsu<6oZwY33|$)(&yF0M(yvLg!ctEB|=Q;6UjuX{ZyTy_08 zc?0HIEaqt7{Ml2g_FTf1$RGqEi2y!;4@?YU2v1C`T)t$g^GB2tWN5|RWHzpOyu*88 zVg&B%{qbg?CpJ!7it!6MU3Tl9^m5Q%EM6kLB7uk$9#074f$vCPKn=T0fo^H@l%Q=2 zhq)Ec_4P6!vb%Cpm*_<1e(mrK1Q+y2sK4t63pz!;qsl?|>|SLvD&5_uks;fB8e({Q zzRsOPR(ocFVD47}&d0Z|{&)3Y7%83t2Kvo7y#E|^f3yCN`uVp@fd0X&cZKiPqkbnt zX{K<(0WFrPCxgcqf2fvelW0VyE}vK1#4VF+L`*UyvtJ}Lj|`Z?i%Jt8`tCOE`mFBr zc3Ci5t-Ri-u6!vwi^cY9o}nh8ifbd+iwnX(OU)WBSv*PAxSh96YGbS;@Y^TQylQfH zK-X58m{veH?m)YcMb%clk}kFG6pJkLR$MYK&zY($3FCNZV$UR%cB-k1^ok%jTp%1D zAEWPHGxlQ6HMKuey{Q+Qr~xyd=Q>+_-4Ir3A=2oA(%34h4)|GEOIO}erQEpX?(AUi zpo5eB`SX|5u}QH_;h28}Z_e*fFb^gr`LKXCfzAi3u)WvJ}fMSU6JmJCzb0k=LWt7VjYXQir#cxCsnOI7`0c! zk%ctg2f}s5Pfl5S?LW(7Ds(q5?5@$L#g~7V7q^U59n@V6qtIkp`QB1M|C$Jh3y^C9 z2&M(G;j@M}AOxlsfJ6FJj}`Hvvr!nBupKL-8??38FtS%{TI%#fQXRsNkvEKx5d!0Q zpyZwFwU57I@S_On9a00L^aj&2f&zPGIz;xtW#OxK@5~@C5$qJ)B|@t}mdURX5ZmCheAzqE@GM(dS;aKQ zaqhk7Vs4l|nxsjN%x&A_&%p!CrY&o=?9fv*nq1@*HsX$48c8X;Nc|jKAt_5JAnw;2 z1w8T~xcd?Zn9CN_>W>9}#?#ZzROrF4fh3E7v5%OqBVd_K5e-2gOvex&9(`Sd*-*fs zqcpxR>p3cKQ_?jak7`gTM1mw%nez5)5-*0&_)lnR*{>nb@|-k{mFlb|E2QocKYup7hLl9j2w9F;*R- zWJAbEzeN>bO09l!phJUxZ19pAaMh|GO46UnRTi~q=H7AUs^QJ-dYZqkXkyi4N0Kd& zGJLp*&Z38{B9q6di8&ADte@609KA@yNso#7qikFX)MYY$7@Xm-`$CXZ1LJn$$V{gA z_(uVfI8gvzENgae#(X^EPB>`>O!D&LHa;Z^!$@wE?YUN+VK0GU>COE3ldjnV-&zc|M9zZS;G(La#m4H7Ev9A(yapH5W@gli3ocm_#GTfM9%nugR^{u94 zVlmTKOX~*>&8}sYuiEv99aYU=Xcluda*z*#vKWl^{Jnx`dMA8> z{%WnGUw{wF^AAf)!?MJSJptW};Zs<*IE>agWQL9YWZFSWd2xW_?11x*%8f#Hs0RG# z_q?%mA4o&`sVL@Y*)*B=W@{LbwiTlmk<$|kWu4;(mwqyB>t>TH9tu?SRjFh4i$bqO z?fHWvWzsPgP?ZvIeI?t2l1rH|-kEX`b%Q`Z;5~Xt;ei5{2+13*FTea;ukLErRHh4U zJ1M=xj=JfV9>pH0XA%})>!@XH;c0cr_hM$aA=82NPD1_K^Wo6>k>OA50F|@x)9>Dc zTHFgBt*DN7+|cgGb0N4U%hB}mw3I+DHfZB`VEkl*Mzr?Gp&V+xN|usRESx*mCsydb z-ftf!dUpD>Cyn1@9X6d8FdGlrSw_Yg{3FXCgaF76p@X92&f4a0#PXUHi99qp&yH-? z%Pa31%PS|}<0E(RbCruEfk?;*Dfka$FLa2`mn`;8IeeU_GN0tV8yN^Gq@oYgHJeaP zcAU3_M|eIRy|2HgiGP?yX}p#Jk&gymmev$o9!8@!M1_bg(TixH0aV7f?q zuSp%p(#OFKKsEPEPF#s%y-D-ONmKjCskn=nxIN1lP#+=SEn&hy`rf1$o~YN|<1EW? zPI&&&*8HGpvw2<3q4JhC-Bhe92&A-OTw|r#0B18(tQGxEjnCJGbPsyQSr+b#4CZbj z7OIoEP4OSJ6y_w&<@vp+r>ej%7&MyXI^-1C?4GEuSsW+j35f#%E5HO((%+b%ue*16>`SaDp>QI+L-eKc)HCYWlYn$t+6VdCIeer=tBpp%)yK0he$<%-(Z zf?T~tfp#;{wv>^OQ{h@34ro9=CM9jP<^0%l1g$oS%=ULHF`K%IlwnT}F;Zx6o=vms z0z~WuC3&;ZaO)`MAR>A$pv#js~${p`dH=ap`oK?hS%sCQhR!a={=)#b>YK5_Vq|&`bN> z*z9@+BemvaZ%tO$06`mX)faScL%94(#O!}tWC&3yfhpXwMvA90c#$(3;k|(%oGo)* zB0;&Y4ZVNowIX8NfSg82ytFivor&_4G%=-x)x-)#4PzeLNVY6W9kCo4B(k z8WL-z&4ON+-gwu$o)hZlsBuz8!z z!@t09FKtfM<@NczIUm`Xnbp3h@0mKPIA0sgCJy%k(;$_gp9}Bh%)bT{m?0sS?q_p$ z9LmN`4BZ6WTWDRltChO%T%J+2lD%JBFfj-9N86uu{(vZsCoI)NxwK5?iHdfUeT(cm z607vREJuBaNG&`Q7cS=ET~Yn|M=1Qc)j{$9hQd9>e+~uKe?eixW|IZpn=Z~5KFL%A zQCNPS@SJa@zz+#)>k!y1{2Zk~lj72ZmEZ4O<&FI%0hVG;#v1K95av*rSd?*q-7!Y2 z%iZ&j#(D41BJ{`l z@maOWj5ro-x^kmCCJtzk68KoC!5~Yt_L69pzlNAUIr&NIc7UmggR; z$+*f$^~oYQOr0qz6BX`_%OvQuY$m0WqdnZ79x$0{ zf~G1><+F{MB`x(5vG)8Bv9!JCT-8Obari$zvf+K1!Oh*ag0F%{kb!MkXwy0r0nOJJ4ih!PB^(e`#!_kqS!Zz`St89>R zrzuGZgb<3nU(NSG7A%D#d(^Y3xacw^HwI-q(ki?FrV;z;{hG1qs?0GR!ir5nbZ1cb z`WGrFpj>W@z{BdYm*8qY2xCv>au-Wh=q-pen-Vui6(=4HlwxVb6bm23;)TnP30Pf$ zg~%V|lZqj3KjJ`1U_W|68cWD`*a?3&AkrY_*GTCWWqA7i(41_5x~c0)n&&LyNcxDu^wg<@B=}HbZDG^3 z8{v0n|PRHnRuSTo9 zH=>9BpU{xs>O`(fl>4lZyiwI|Bf}Gcs1R15#HQ;^l={;(uC$OHFPfZR-w%QPyEFvN zfPky|wimd+kN#&>Ft)$rSMB1j1!VtA8b%`X7UhZSi!amTO9!%h0|*xwR=;KtcT7hB zc~@W9zY=#84G!9Gu8L>U_*PN9TwfpHW=ozuWID%dQmuVUL)_SFDo>*^hVuEUP7d6z zp0GI1`gK~`1-4oVcdqu7i<7Cb8Q!j%*mW-M=-NtSQ$W!*+@X;|;gu!P1N%&BBZQXP z3XPQOolc90*1(M;i!7EK(OufxpTDemX!onIVJFa*E>JH~XId zQidVbJVDhqMo>)c4_gGzE6kJ4?e7N$-1SUV>V3dm9RBHK4O~s*t$twq$}YWn-jnUY zD1%J~^J&f9W<^T!ZLdw`rVs9gnK{+dSX7cIOME{u;e4U#y)|o`=(jC=jQgL>xNRYw zgHe7hj}VD%Hof9hnmTN$i+|C#d1p{>U-Xhjp2FU*exH1zG*yAO`Ed%*m>pR)QCy5F z5qr>hF&UO3?ZWknEIeqs5LN(AC}f>W=qtgEj4+aohyXIUm#b_!8Ji$-w)vid8LjU_ zS1X*tq^vpBp(?Hn4fK;EFv)28$CE|@PW1RX*|KnS8aen!A#gCBm)<=$Nl z0KK8@Og$>}1r~`l-0XaT@?}=T7$jW5{o>9g@FcOk=D}cAxpdjA_rdpU*CpAtWYw8# zubsm+2?G9n%ZM{aFpTJx*uZ83*nlya+tGWm`h&7drJ8OJk|JutNWKrtZO!q>n)STK zvHUbP_x$ZdCV)$lM@`#!jOo0p z#eQRKc-|5?2begbPL^L*=bSz5*ZCfvN0g?8FRNqHFj>XY(hCZ z`QkVUZ)*EC1~32ureG<5Z`*+_ZU5SDhO-2Kzx7rF_LKWFfC&(E$x{W$cXBcC=%;<< zRju=+?4xLadj%-Lv<|e6gIl*1xqLRzHEE=fT%J6UJPD^q{Dwh?s_w{^-;hgRkplCF z0M@t&jNar93>@#c=)R=0a?xtN>`uU%2_B(z@&94iUax%&dk>BvH#3~Qa(KJB$%8QO5DwUV>Ow&SR;zKD{QR&TRdMnzO;(2;#-)m_G&Geo z?f9wo?uEUVK{4#qOncueNfAz}QcvAC2PF6Y|@OgoUFerQX}6X&`1mGVGxkHBq(!~1{Xd6k&oAun5*PCr@gq_+oykq5>D@#bCZ{@*Yp>QOQLIw$|bzOhMsXVlq$!p zfLPLH!mq2$%Yuw6SmBe4_j?(vwg<0U5|%}?L)^2m7R3{|9#Ha%thrsXpWi_Lchn1} z*3G$nAA$HUpq~9-sMq|~Z^$2bdXM;K-RhBW@=CKsbw>8|eCwczPJk7}BmF1Lh!-4+ z$S2JvDGSMyBqT5KqUR+t@pzFXbZ}R{*}00|ZJ{He&;O{J_I%#fv!6!Xb9I-ejsrziaE8lX1S?F98O@VKRz-E7 zIVUP#Z*$@xz#Es#D-aolwVT_rmIZ!)d;#Hs1 zQtZf*-$d+D~B;RKht0PmqEQ*33_fl6+r&6v34n3?!|!pCF-wU~-!d z;E;<)8YMA*JO4U9tcJZdv4^9w!k>J~a%O|HfJ~7$b109hc*QL2+?Of(f^tXlS6(++ z&|XX%ZkThf)tRC|*-Bzq+W|B}htlKdRJaD1vqzz5Pbirjy6RX#2+ni&2sMXuxlP1m zqbdk4qC~?>F8t;PaD(+6D`0yUW8EB{i565~gqo&=O1^9A2lg2?_3a^@8CAo4QVkz6IxnD2;Caz=^2W5D)1@tNajd7kFHR4mV7r-a<_-JVqD#oS4mB z^Ekj*>l^xqds<%j*)@V(-`311%TTm&GSc| z2nN6qs$3zC%sB2}-#>8p*rajABShME1RE&3!Zy_4I(`?#{uFLzFMsgabQXG*!ig8d z2oq};YezoHVu&f}hDJbLO}%8g7Ub~u5MBH<#M(x+|A{vl|2|3~gS~>q@(Mh{hJxM! z|DiZF*E)_cg>aR=7Mn@5&3HwWM#R(1ehG}*hy4T?s~7X;CrUr!-Aj~qz_UAWRJKQ; zcgJ)0fhN;+H|gE*wg*+p>2Xf2^422#DAmOrK}c$bk5Cp`*!!EHAm1tOSPbnl7nvbE zXIoRzc|_fQ<}+~FLMO4u!tW?@ZWMcaMY<_XI3Ju|WO~vb;UJ3;0aQ__d_-HE>x`p8 z_F%&o61$>v(#Qwq3f4AtT+U5XLq)ZsZgk(Gkk5IT6dN_;x;h6x@F4flODVI;vieZF zt8GTbHc9dHMa+#*05=)&h2b_13n7NRh_rC`^npdyny$4c)gDTc)#4KCiesm+?Lgro zT1gkxJwx3#<@cLr(4DEobJ7g@9<_kO5m=ZMcYpxi8~L@s$^$Xn=BX@wT8)_g)aCDY zEY()g7b#?cF@2%9E2!|a8m&n^7!%|Q*iH&Hxw@Nl8AT0#m(cZz)S0i($KQ9pqIPI^ z=wD^ns#}eq7eD-GAk;`9=^2CF%qZL1!WmIP4KU}$EKB?T)VSo`(O)nv@u$|gGdU8${&p-I}Fz{7bJ|wKVC~F)3$+nO!8!Pqkf_-e#Kpa`tCMSEqTQhI0Be$F>NNBg00*^hjge#X?=X z+l2W-=Q@S?+Ga>tnuIHYsuKPl_@stvqUoZy!h9`lz{h^vr&}YLUDB<)T%E|1BtTgy z;A^YhszYAWGoo3g4tDWmt3^BE4YeS6aP;>yQ)AX@e}rKU|H(@MWmL#=T^+UY5VTm_ z4H|eVdvz4ahB(b~ljL^=j(KB8xWkMtc>BY|lOn2KZXJ}Sgt-%a@ZPXt3F&|m zv?IA}30(cB3nNnRMrB9waxVnY724f^MR#|rD2s0ClbB{9;5k(xWyN;ef!*fMEFKC< z)%DmrnHXGaA$CSy<8;Y)rUCLmjDD8Bup+krhLuPK)=7gEsG21kvb3WBbFWWwksS6u zgl6Y-sO_6%ulPF;{IOAHzokCls__X~dH4`E7p^3DPNC0D=lJs%3^VCCiM{m$7oxeq@?jTlJIM-c7>5r@ZgwMgJ^xJvoj=L( zuJmWZ9Mv3cac>c)p(c`cE+TJET-#-H19n|i_&(JPIKQt0bEi!5gH zfRFl=bd3V%daIZOUo6+#;P2I#^t4nN4a~KW^v67k41@m1?1T1mgTp+KTVp@kAnfI{ zH~_^jsQdgs?v3PTGeFTuoC>=y$e&z7r{MI%&W`hf<%30L{0+MsBPch{Ncp>?v zb1!i}H=7C$p1YGlCfFP2BCOS&14N!iFjLPJvTA1Ee)UJ$25IG1A>`)$(()GF8>oPf zRZf!|vq8yyl8T(U+aHBk01&Q`mb2pI*%S-mq8k$+3#O9F?S6(NZ@wfF5~y5KbNK+f zmgl&GoQCAk04uc@!UkIlzaYZ?fg54K5NbBiJF_`^cX8;!<$>&%Pi>i!tcuk~%W@`U zKS@|vX{_~geRJ^_*MJH@xJw{#ZZB}OhoY7O{}-3Z{!HBz-dc)%&1>I^*T77&t$G4U zGb>;wgxZrnsz-igfY~Lirv2J+Xfk0zV0&)mIw) zIfJBt&@@N`fZJe+h3=@Y<&QkkM~E{)bG1H9q~rQ&L}D=YN5{T9tDc zJ~tl+xiAH6Djx1u^?T>{hyFsWHTV3rgkAoCK(Q=Yy>CqC0WY5)+cRVO^}WFG-?K|# z#UtC?Z%i4({}(Xz_iD{awg0FD&icYHnu;Kb%O}J;=UK}4ibvHfxC-xk1fQrxbf~j} z$L(0~lIhM9khmu{8P-B|Tu&qvn1nDoPD7V_|Mf`*eei5zPT10KOHeq&O%1G>H6Y0^ct_UF0{$=h)B}x>q()Qyk)OiW6 zq3ILdvTGMOZEk(1gSgfs3>A@xj#B-uj)u;qB>(pi>NeXEMNyvGa`D^|4D~x;fpYvx zND3>y;V{UKI1S4^T7V<7WlK+l&sr{2^*63)qbih^NvTShyVAaK)$xrhz7L0-BeITU zvqkW=pDs{He4PqCDH_0JQ3hJnkCj8O7c})l99_ce8LV~PYf8e*9IlMmT%nF}$jM9> zvupUKD=Y;XX}R^#qBa@wNqx}roCV|(=6(!P@5r9&Q4nucjW{j};5C46)mh=kby#|= z$^`BCnqz3fD^gYIV_%1jvt4J6?6+4%Vrf>)>B=^Ybi6iIuJ z-M?|AxT^?iA{6pjMa2UxvH@rSxr@+i8IGXL{CFO2rSN-{i9;TcMD$Wxqp&RJPogi? zZFi`-UThlT5y$FB&W|*G-5%B5_U?n{LEa!YrmfunhrPFqZma3i1!ZPtW{R1anJH#w zlo?}6%y!Jo3^B9J%*@OjGgF+{*YDf?P515odd=LK`7^htq|(yTInt_KbyTPJ-p}*Y zh`~a#mfWjg6rk$X+T>P0pD&{drZq{6a~Y-c*Q^p8YPIX4sMq%`Mrj>y{AG=s-ni`E zaXI{&J5d*tpFi>kzhWH@rs-o=kfVv!j zC^RK58h0)&A!%XlKOr|e#}jafvfw$lZ-2plfA;^5b0$aV7`Nfba>gZA9Sf3)t<|o& zWO=T=EphdxX+7B1;PfuhATrD3gBu)aA_@$|ZKs7T`##4bi$}$PNO5&i(H8uP_$^T8 z_6z6oomcI9ZQeuYe(j~diVb}W#ECUg#*qI3&xn#ej_MHX?LMtUu-C+7t+YNt%vyt|Y~VQQm{xH05X=&+5o*F?_B^I8?+N zq&3UG9 z!x5Y?@^@W&u-{X{ghCV_mxvd4CnbhwKPs4@N*anM8KTtKI3fCy_%XQ=ejsQlaQw7R zrY(kZel|?*0z6q=eWAIV{z`NBSYchtV+lnyd1XI2HfK;ocaaZ%Dbwm8lzbxpbpLYo zZ9d?iPH#2^Mr2qYM}|KDHVprM6{cw7>Sp2M`mea`zhU_Q^3lH`(gzk5Z9jz5pmzN7 zJy{xJsrVxjlSWp9#quf=h()ybNo?h%)a3kLBU8i>O{~fCqI(h^j`o3#!oJ9Ux8IiB zr4UBckl{|pekgH+Z%s{qXQZ5xuQGS!fKQw0&G+?P={9BdYdKhRaFdmpRT4S-^E{AN zp-WgKKN-%KwJ?M{7pA;ktEzcbFPh?N$l{X?cF4faO9%E?)c5<2W6!8w(ZCanSkWC% z<&}UU?TkN!*@X><#IzVVgCH}1_GeXjog)h1p$Ceh(^IQgR9Fp7uy_%)5!x8UJM}rX znxq{zzQ$j!` zm7^6dMTMRv)w66>^&MHU_uCGp{qs$sG!`kC8nV*A1cCP?OmDwZpB`%I>T?rA)l0rb z)#Fi9`1N`V5+Y`}iKfclRhQeqxi&;HVjwr$3yo;s@{br|LyU7|_p*)3clzWI-T886 z9@1rhKCzXR--0)3l;Qcg&2ZbMI(KSTv>3%jDkaoeI0se9+{wh|oISOEWptu2-oRg* zU_-tm6RO6BgiunIXj-IvEJLE-LE<*CDI>74z#XN=`e|9X{#RR<=`9k;vKFFwpo{^1 zBGrl(Y#atrW(VW43yatFvp!Pd?q(F@Al5|Tx6rS9ozKLu_?`Q@Yv}WgoCd#dAsySo zb9RV-;2?fs#;UMQ_URy$XFR4phFYwSq%8hkKGLJ5JN`{66-jHYN?T~|J=~1GwlEEs zH44!bdH|MukDxB3bHrtTrfQ#aB) z0Tw1>eWhk`VBK(P?1{ANoO$*`dfe+XBr@WsgVFmg4A&ec=wuK<;bCvS+TKGkF9o8& zg)D&816}M8=_i^@;=fT;rDWhnxk;%reUn?N?Mu<9lgZXqJSFQ!C_guJQrd;T7H9f) zjDz)BWH%UAk9Q(O4!JdXh*W2Q;Gi8~*~Z@gz4lYlJCy~zi`18~^o8Q!5uzF|&T;P0tDD(C}ChG8T;-{_PHveawC2%8s0dHXP%vOH`~jUq;+XB}W|4 zumLjPf-ck!Pf=clx1PIu@~+;NBHDyqYEObPgXUv_{qDeVk@yn=Lqe#oeJNUxja1J7 zupcTiJQwfiP$LcY$4X~M=BA8;Y69*{M zHdF*t^$DP$_vXTR=t24DYmgH?^=D_E0TddL0>`0EE4 z=>73wJfKV5FB!-EsHDR%rfmhuSj$dy-WW1`B&21C$HSfi#7bte6D>{{u~LQRP1r6X zz~pgu>N$4$DU~L9Xo;w>4Z>}eF$yk30Z2D zF{;E>euhD`0TM|{He)UULYi9zQ(l!zKwGu809BGK(v&{3iZ-S|bbxaJPYuzmQnW(*@5qQfC}`5n4ud00&91nh`t z=ktjdqCj_IS#i9X^8v_nK3~+MO}1n?w+(UKQX^f>#zhDETr(&0d&7Hudq^1+B`j^K zRt4Lksi7QG%~>O((}~8~q0IKi(TI>ZsnVsV4tJRwJHLTn<=EpyacY$wl_MR7a4}(K zEB~w?Q!8b)K8eF|>+%!Fkd?wTib+6OHv>l7=l|y9_rJ z-_`(JUXxG*>L9O)&~R8h!}l5YYV!yCj;uy<;aWM?BE*@TE=g$wN5BkQv-!l|7)EC8 zM$W*GkzLxg#uV3+S#rn%eBuzY_(!sd54gx{iv9S--xNlmricz4P+YcwO~T@DkEA#} zo^d{UBdm47v(s_il7AoeM{${Uow`^j2+`l!c>Ct&!=#Qv-<4bF|{h>g94v~HubpLP%%`!qMeJ>=!g<2FrK*f@y=?Pdi;4+ ziL#6U7yZ;)N0&IPnHP!HS<6e-+}5j{4etQys5dY2!$x6x@7`^0HtMT9=L zyZ@ErxI=CH@G5#%HkZZcaFEO^Tn%WK>&Dj@6^x%y*_2Zyk3P%7VbEBHG%(yMtm+xK z&V90wT6}y0>2M8KZQeNeZ18zoqK0N)nzyhrG#FJ>5h*y>3loHm_P&%ePGZ6DW+@x& zob~xCO=R&cE3GM)w6D7Td5dW}`Qki$I_lM9ex;JAL9%h-Hj<`f1glO$FsxcS9L@z4 zau*(Eg?k~lqzy$gH5^fCF$Rl-d%%$xLd)NqJ0IF31tu3C#M&oeV>ka~xNEFx|59vJ z?9ThsGzo_#Vzs9z;?@;6G|Po#>L7ts81X2)fmRGGmErNHn9CGX7`Vv9{-Px--YpS2 zNyN{LLcTvMyc$Rnc!oY()tdL_8FmfC2T7+(DK^?`+s#1+`B&z6u6+VE$0%mt#RbcF zC@g0#^k^

u_*oVA_mONgir%v9zPI$`v+dNf7-={^%CTrAh*>F7UtJ=qX4g=twBLUpH256 zV~4g@hXFjscY^l0&(y)CCImfTdgrxpH&t~4Nb3OP6&DmewqJsu4TrhDLV&vaxq7w- z4>|22c#L6uRs!|30`Ng#zSLotQtW=jiI#=3(->7*`{c5VZA@%D;aw^XM*>MVQQp>M z$LJxDVb;i-khir5WH&a1Qo{LVBaL?6hBSk=9OpULj(i{>=T(!&>)@CvRF%W~`~Y@5 ziD^_v>o6KdAh|HSRk#|RmjdPC)^8X4EcP9h2E~dk$HwC8lD4es1=ak7({lR7 zFnjX68Hw|INHp!GigO+t42m-X_&>x=H>yBq0|M5ov*=Eh&e1e!o*^(o1uNR9Z(w8= zU#(^Q#;`>RS%2+6fWVvU0$=SC8&%p=UEk-a1f*Q2<(pbg-zM}j?i|!}wn($O7n-q| zZ+VX}W!u74NefU%y#=IAhjti6`Zjg$G1_{`gEBTpy|)_#Yo-~<$+a6vX<>ik)_J3n z2TWmGD_KB>@Z=wTm~w0eYJvpgnqG41m}aE&xK=fVwqX6 zGsmeS^S8MTvzzTw%y`nHPndh_k@hHO?K@AsOi9}q4hTU0=c$I>qwE01kGWI%Bh^6m z@15L#GI##XHJ{M>_aQu8bXjNS5x$vdNiZ6&wkenq^yLbvX0?q;2MCClsz=vbDk8N6 z?z&BL^-8era2%wtze0GUS~Z>;Ho%bc$P&Yqt#muRf~ky-AC5SW?RGIG9|_{dsrmY{ z!M?5UQ~rK$cgi(u-daaECta(w2yP3T0tPE~#cQqmZmYVVZ9JD63qk`bq%_L{^E=#K z()|IO8Yjlf7D~z3O@fuF!4~OnwHgIH!GUE$2?XI zWg9Lp9_4f;s~-^0fL70};prxwoJ7aHx9fG8Q=arXC)j}|7ua0Jpk*WJ>4@z5dVek0v(3)*sT2-WgbbwFX3%u zk|ttS;bv@#*TK?hwfvoHW_Jqk5aW(7GR(1pfsLQoR^2A$vMm%joy>}G^G;^L6oH4s zvZzky)P!iUt^+a(ak(u*d+LM(ABzrdi}ApJ$-6iiE*(D({koCmQ@@a)uNnT z^CMAH52KYWZ4uwu{Mjs`LuDzmQC&>eGC;;=rhw>tb3qd~WXisK;yRZ%;t3Ov!IHPQ<2e3W z^yVyOZH`FRMuU6fgk$#7?5Pu3`7J^FV;LGd6W(2O^Tl9fA1mRC{$Nt7{9zXV1P%ZD zA~)unw&5&m&EDsTnuN%XnWBP)elSl1OsR<2>xt-id0J6cGoBcpm{$4RZqN}t6Meup z8k^R+&=ZNX*v3TjHTWsN{VM51Yy2<7E@nK!v}TZsg)Ab$3Te|f!Osr9+*@QWJt|nq zJA)Tek>k-_6atp|kgctJK_8d@ku0w%mI+$rP@S=8mq0 zF}jWJS|C><9pN#j?R5(jAOu5n=8<%Y@*zW@u^+m=y8{-RE4K)9mj%y6XYrf33o|a* z2Y=DQZmr#ND9?oX`pS&y#P%Kc(fTWOB8w>65R<7PVS!g9YXt8%3cttSU;7hbOdlza zjxhP1vt`LAtdop6G8K7IeX$E{6fhvqV}CoLi@?EOohg)(RrR)+jgKEI2fW|D%nQ@} zU@F1e7BE{-blg$G6%!I0mB^Qlrf|C1L#*Bs%IAPD6_r4r%w8&tZG}i2b}j`9!LD+J zc;kGz6v+jaU*S&lZ%yQ)bM-8^9vD|a`v!cE7=9Q2ps#s3s=`^Q}L?{qKz<)eQKxcvw9i>jb-4tKQPUdel~ zn1eOwb;^T+pX-^kfkFFzf8O*P0u&3q)J$37GuQ)fVhZnISt?sdhcVzoB zaUU;SX7#NH^kcl7;w4oD>~LLT_={H%ImpIQM^j~S=7L+3%CyiXiG0+~%GZ9}14a1R z2S;sXX!L?wx+wmbg;l$TBKHbV-@Zr(@6GD)QCP`^3@)fIksf*mfIg3-gOvjp&NmL2tTz*bhyD)TfC<~q zogm#IukBDu$1hkHJ>R#z4A)XvG4(j1E?lvE>MG?cCJ8v}4njPd8_95vPG-8%v(8l8 z|7hHgzp3Gt;;hSmP?QbLT|gfcWphi1^(s6b&yAWR6Mv6R^ysc~d+lWey?Bqp03yNR znqrNp#6f?Mc6Y}xNW1h&`!9-eZwX&H+Q#v29&eUba%F3=y1A1+qKi}`+G3x7t_3Z* zO{i#z6rddqS~U_Pjx4r#WZLv#7G;fKu0hPHfBfpOl37N0zp)H{7HLyXh_1~NeYlcU zl4bxS8~^Y%&MVw<2ErL1eJ3I6E3E>qlMQ+Z*$o-oXc!$B{6~oJUn$CH;sLY&LQz)D zSdMSb9+WBRdOVX=S39<@Cm_k_6&in1n-CZo86mpg?0H%Ng7W1_^DiYROd(%#0#YZO zFWwXkr476bH@AOOkh3~|cQ2jV&UBvr5-0Di?#jNk)FnvBHOa;9P$3S zr*2tzNSU^jZy742dFR1pLp*2LU8AGP-0G%2Vd>5{PG6gvzkKh9!i#`VDMa+)(`9u z=pu^;L9xMLsI)Ie7@sJs5msl#{v_~KQRCp0glhoH;M<7tYbFsMme>qo_9Rz`NKgz6 zl65qekEAe$_I0(_J2Z#yzc859+aB`t&lukG@839d$FN_yz1pXdtx*gPNb|u+_b#pr zVe3ab2cBK*LZhB#5kcY9O9xl5DmmQ{Gu``IQ+i+;0$;SfI7i@^d3(lD=XLl6JMa&J zQ!j7G2PP19?yqg*Fbv-THL;zAvFn97DqPzx0>x1chN2@pwuWBDghGhF@g)4GvO?9tE^{GRX?4qW0i7qG_t&Y2b1mpjXTeOsfzOd;XBX&`u+F66iE4R)eHZnR?2^? zUih2;`Og$%{#*6Jf2qgz->Mh>rVahKzZU*R)crZv?W)Yq z{!*P$;H$Q_>Cx2AxcAdM3y(YJX2ipTTP#_H*I85yd;ltf047)j9Bv>)xC9-GBmfd* z#fC>iEO&zasgA(L%*X%g<@hNl*C=blYoDBY2az?#X3n$2t{OsxtrSeDugmq_P4|rN zD?%D%+zjMHHs{_H0k=`W1#|}!IVJ_^A%;KEW428A{g=+EV;9_8I}XVRUr-)HsF;V2 zHVUpl>K8M$Cxd2#STZHTxE}5D&Svc$Uhq=6VprEfU$KL32`5e#nsbRVsnFmdXnEz% z8%`25=qb_@ewUoDIk^Q0y8!UkIFw&874KQUCJ7$I!o%TL6E;4-AG)-P)ZRA{B3Omw z@!zy7*38s^Nu2Lf8$5ivw*h2LAIh;aSyV3Dwi^f!r-pvTH%cv6k-SZIE8TxHGuX1&(pPMEqMws;)9}_K$;c50lsd;^B z7Rp?7l##w;*`Yn#8F{Sypq*ZiIjaw8_=j{Y8WSsq~&t1_HE zKWPZuq07pHpfa@K{KNND5Kl*pu`C}!#$6}bAj~yyf7ws|=3>oK$d|)ml3anex6yr) zoR0Um#Nq9)5QI{F}ahifOZA`E&rF}w;A1h|r@0~07pLZoY`K8eYwun*BdU!$?A zV7tWaV_)AWBw|+9EQ|87SLcn46TJ(UN$p3ak^wwKF`?CmcLvp&EQqmc{xVB-5$52`z!3-;8$8~UaaG9u zg)uxDRlHYhqcPNwop5-1INM9q#Z>>Vvc>Hr#^x=rt3Iz?3gV9dRb|Zy5EUxt(|!g6 zX+0fD?TSntL}=A4tA2=-H3_N24lg1q_|ta%u8r-Aulz3$%7!sU^M*;F^-%dKP58L3 zEEYHRu&G1vIUKwVRBr!+(w6f z_o_-41lM_oO_goBFQ!=nh2Xy&Kjo%s-s$@{sVA#=*kyg%aIj!6VCbpwH(?L0IDj2( zI4G}fEw_7%byr;U?<%*fE>EwpJu<4W1VKf0L9gjQS~EHyPELbkxR5VF5;+_L3C%Oq871>-Du0)pD8x!LxtI~c|> za&_}LyZDN$7=XF%+^i8buBK>Scro=z-l$UJliy{(iKSv6VL zVBh*ShH!9g&iC*5ccr5!27vbvY7~CDN)`5IJv=jjhTS_=BwQGf~0^m zwwxWZ`y@wcg>y zbS39*X$--QhD1!aSwiU|PttNyiv zt+}I_ii@L@>&Knc9cb!cVq?!_>ttc|zVf~UK$Vk{l>&f)fdPy^9)S0CfH(jG9Q?26 zM}ho!Lcu~oK|(^o!@xkpBElmgBETacAR(ipA|az8BOst+qoQG8Vqsw+qTt|SW8$J? zVqyOEBVZ68-+_dJgMxy?L_$Eq{9i8bJpeRVKpYqV0*o90js^yS2KGMimk=Kq)JJRo zYVf}nFz}ByLc_qq!6ST}(1Z#A2ZMkBhlKd6)gNd3eH;fsqCugPv5G=tsG7i#J7cna zi_eFp5UcCOQk%V|WH)vB(4@e|!NtR;qNbsx`^3S?#m&RZCoUl=B`qT>r~X+(Q%hS% z*Ua3)(#qP#*4543!_&+AOHgn~XjpheWI|$6a!P7idPYHEQE^G>kFxUmhQ=mPb4zPm zU;n`1(D2CU*xdZW;?nZU>e}w!{=wnV@z0afo7=nlhsUSqmtTL`^_QLhDF0&Ef3OSf z!!B@0NC-%nzw817_xz|3Xpm53tkCG9sxT(b802i zcByXuGVO26{;wGp_w4GI6|;pl|*Z3t+;HY?!te&n`;9yqPr9_O-qV3-fj63gp0L^rhJS zqc$wsovXHhzGrMv7r1-)3LM;G_U(4T(B#D2d5hX52;}o+)ng7tI@@s|yv-nk?vB5ZMtSeoHTX z%cbA+p9#PakjD!7I96bts)g~ary8b$&k@QCbSDzM*b=6z{@sta63icNkkoR!&rsPZ z9Q_Wkc?Y098};CaliZLUfd_tb9C0(;Fx(u`4fNbe>EeluR1EO64_?Giz5+se#t>u0l( zaGmHN+Sh|NT4?3sR>u3B@!UEoROj!r#bs(2nax58c#2=$ z%DIQ`w!>mg$I2tg(-At|LiIf2m#0Ec zXfo*2;9tyT5S6ENsWt!)isHS7#EVU5L@uwQ^URzTXrOtumWCMlOy_B|5vfDMoM3m3 z446Ox-d*|jvnmXcxqF|?$8ArpTnj=Jr&Ss@Hd-c^2^ZI=*eZbUJ76bvje97yE$3UQ zSIn|xbN9j(N8I)LT3f-oLDiB)pj&GOL3sL(DB;*n7U)?edCKn{aKX7RKPfz@8=5K{ z!dT@iTx8d_s@JYl{A#a>-*izR71a!nDxg`Hl8Ow`C59>%>u(fn2xUO8Hn-gCPmJ$S zbVl`%HkWtnkgey^Sk{0n?t%ajG-;)LP1BQVNYT8+B*0p{!C6(mzOEy0hM=d1CXo12 zH>3_nmY_v7EO1<{2=L~-x**!yq~5#J$yk=|n#sSRdi>gkR8?6~mP_5I*XT-@o*Hmj zJNshfmV(F{8RHmhgbHMG^TXADR(l6%EbYwH0GsH6Li^{S_)`o;49fUFMUBP2RX=xU zco5;M+5AfT17JJob{)}e4au5jFQ{&y9u}UIEA;h-mTivW3^`J*O-M*Q1t+b4z5_mJ zySqQlzsp?@MVClYjcv8*oZLuM@%>qq(ZF+FZHv{#Nh1-^@ilxdD>T})?%P+qQL6ay z+^sTKMG2Z+VzohPYE8?FF%YAQt6lqy_qn_>>?sRaSmo<}GaUV!9_ijQgE^$?+?^Rt z+|SC2UD?&0@#3&h&bU2vfBaS^ZU~?Q`ON~V*6WwN*0bF%QNXM8t9oA>b@88~8+o6{ zPCh*Ug_}Tf8&A3|6?9CH6WkD;!yCb~+&e(6cEO>8v0&pA-&eKQ>f`c27zg5W9ms4{ ztA4%(kV1^)LkKAEr4CGEd%^5A4+zT3ExNt9`N8!4!YZG;E5Ji{xe*5W%L?rF32t>%}IJPONkoK%Ut*=J&35s!M)BWgR)$pprMIVJ&s)%<7-^=QYM)-=7?0o>-Z{M5LNPp^H>n@T; z&~dl&ttERL{ zQ+4NKAiH;(QRc)-W<&P6lt6WQDi8>oSHMO23I&0fKO#>Rns87gE9WY#w zS_1NEu4zK6J0yPn-XfkWSvJJ5Un@#c<(!w0n#W73z9>THI>s$W{Y`bv35swnl-!v{ z87$H-XZZES#tFe&jMW?oSFsW+J$z)Rj-m?Sh4eHyO?)E6bi>u^SpDp(UNNeKfm#||7;$kT-*QAi1jtjN?q(sq6)o!OA~pXK3p<}nZah< zm5`0{d$nPJ`K-xKH;Rq!pUZZOypxnKo2M6T3m516@|oX;(Pf~INIpH##{$aXOL&jfqfYJzH1%b$w}mVpF9 zg)RsT{i+aNfL-;AT2qFkKfOcTDlqM;hIrM!tj{5P2K)o6+z~si4U0LhkO*aygsRfy z8{s47A5v=a;4fweiDMfc37+;|g}cMyOukm$6@xYXML(Ny)a1ES-6=6JeSCP5xqm?U z`&bHa^ACRkJhagd_zGV4?=dA4#U~DGE_*FX<MGN|I_cvGgZMdo&l#MO?f$bS{p zUiPg03A)p@eq_;SS`W{QrZj349KxlA^sa`GoxZ7sYKw!*LugdT|12$GrQtcXNRu+G z|7HFspdjOx!LD_M`~b18=lGj^ivt7&O&l$lbxAs^OkE1t3N}KH4DRUgo-nJJOg*@2 zka02(Z!eJ~fG_f!<3$c-vI9A5JERse%fuIePonw^n4~n)fEHMkT2a2z(n`Gj|yaskc0F0XA$8U>> z14KszVT<#?BWeHV+M)i28M63R=7}i86uGrV;S`ktEFbiO8k}wMNN^e)$krkaF@?Yy z!F_Dox$;Gzn(erKc&JbCso(xN3&f-LfL(Qw{qyVqzZ=IELOBCNDTL@1iU8_f5+d7D zbfgRP`Pnb)h`j5Hcvs|=$~v&6UtTWzsXloWaO~2*d(;3E{+qz{9RQa!-@iMK3v@>i zo0|09gEO;AWh3b_k5Llk4L1jgD#A5d@A%2BJZB)ZcZvv3U@xncXP42&$3iCjAj0t2 zPL_jMtM)2XZt4gdsN1q#%CP5aiqh40>F8nOe+dr`lxCR!T8=`A=;ACw2%UXx`6`_& z8msgiBv@6QlJexbQdvDaQEs})gxLLwB_s(fWFUqY4>~EJ;VOa?2?<#3PdZsPG$+ZS zE=OYmCl28di?NepogU%0!WbXAF}Y!Dh}_t=^ss)m>9NM!A#Xw)Z)e31M;0}dfP+?M zafiaRf=gjT(`jS-#iG$glB0_pR2%7SBBO#DK>?K1hBkuc5_mDo>PDI-CAQhy4t%43 zn|ftR-1Etnv$_2Jx^7cl1b>6wMO$nrhRyyMu9A!gzQLmT3DeOlb2lv9eN#(&nIo1s zlR!`&U3Xy19&|&mEAIfGoG!)AHpbd(u5VW5sVf!PCe$o=|0~I*Vy*ZG6zF%KO;%8X zm%;95^Ee0>yD7Jtz+ZmFVG#oS&U-#UIlWF{#XxI^3&>2$?}w2#K{E1J)=sEc0J>qy zItbIF{A+%Vh0E${nj3{}YYc-8osGhc2$KX=?xf+T!vOcAp}bE&$g!sURpNcco;7N0 z*Opq#YX<1t5RgPDQN_Dt=HTF(Wv+c^xE-VcW{?w35JZdOZ1$DBe-QLJZEXe7|nqS$jzK^zqYOfMygTj{t zBT+xqq$$DWKS6$h2^S!7t;{TWoAtz4%XB1H>m{c&^R>_hnkA-?W!75YO=LKRF;#J{ z^S`X&tr7TCSf*z+mhe*RQ0T{}!2ZBV95watK+e2|AENx+wUpN>o3S{z$_TC(zMBC@ zslCs}g{=y6#mk?egL}PhCm3L*jlQlNxjy8nS%!(H1MHOrdlhVC6i}jkAzA}JKlZrd z{Wc|QjH0p#v{>W}Nf=iRMFKVR`LHR;53>h5^};~KQ9H@-IP|zGD_qaaTw1huFIhCR zZ@ttFG}}|SXoyI$q3i!{IUD&2BRVvF{b}y5pDV*_vGG+eD8SXv(bc3hyfA9gf~r*; zW0H!6#9y4G{W*G5xX+`_Am)2Pw#USj|Ee>ro?rg#CAwH`X_|&v)JQm5bG1O1kS<_h zUWnktA179<;rbz%PcyY-l%lE;gYQ#{;@1#7q;G$&N?oOYQ>6w8Sq&XRqbAn=QG(kd*Lf( zH@_;UR5g$!*(Ma5sgs?GvAWviN6++rK$=E9<0?C_ZR%NW1|*;$}ucvp}wdo!AY zO`6{pWqw;Y7U^#)%(fGvM+?kzz-kWC53#8pnMUDF(H0bQhy+%A{r^ zsx4Uh?eWRbAX6=d4lH+&JyE^@a^3;`^BuZ*txH99cMkYB2nOxL2R{IXW%M8x85%QY z6}TZ$;BZ)~dN)SXL%YGafa^fY6A~JB89|Y4R2a`DMVIu`*umFN-QRyxeN1s-zbQc} zYpr#O8uMmzIt!@R^^S>j^G@b1$0VD?&py_CekCAA(?y_3M@wz!8txj4l^ZBt{?f1C zC6^Z~6(Wpq2C`E7NbhUH2Z+*&BsT^SlkvnKe z5IU{3RC&Hz*9kuL>*`95s=-e5*tp=Oig6-D?03yqsKD}?dgvq{~xCutPRIkGKu!Ia-jL9hffMe%zN` z9?i{84%mE>0e~%=S}2dOPtC!u-&yGSPpbFYeR#@!It;EM?e74E<_6Yj`VC2Ef^qtn zQr~bU?V)}GfBVbLKH}k)1#Ir6wd^7-O;0p~bvbTn4rC(2FQJ6^caG0$QEs`2E7{ie zvqu?H;ERzwAA|J{@F)>KWDB2=bLo+$jKqO~Xu4Jo*}0YuY7|D%!?NaRr?p?4W5x9g z$MKl$nPQLq0$Be?D5Zuz^U-^jrp+ef?WaSA4o@=(2_)Nb1ogY>u-|W09!E912;0)( zOm)uPt_IV3`LXS6)do9WpXxG>(agQe6Yh{V>G#cxk5R~O^WBVA{BuyBFxqblp*qi< z*J#k3ytntGHv|C)gx0hBeMKK%NUP@}k`C-Wz!>e;Xa+$G=7};l zNCo1Qsgs1rQd%^?NDR=#ot>pqM0kdgywYJFusY zG*u?H^etSQh#>YR4~s7+U92w-!X=Sv?Ks=|2|5}^NLL6g$Eo}c0u{iY&2)|6f175? zE>va?I8pl!fNuA`nhG}`zbw?5U2DZb3>7l(m&|}f}9(P z?*QvlSY3Afp%RvGfmKRpNii>vN~99UcP`CWmXuETc= z=Q3M*;?&ZOlaZUWpbl558_=H0!C%{b|Kv*;ET4Ymc>Z$!_V(*m!;F3`M<2|f$S1{E z*z__h_ewZ*fL zDSJ_0S7h=zq{R}K#QXEFL#F$HD0E5MhGnwmZ=i7u14$s`h6!cA#IBikdQps&g-Ij5 zD1Ks?M->XJ^+#;8^0$diz#jedg*=bjRfB-DJfcdvIW~V0S~<9ZF8vUHS`=IG&>p6) zJ-INlFXAhrKUJ9Og742JMO#Ko?r>MscRLNY3@nK}6&dtKgf08@6OKoOX zhs#dYC^fGf36u|IL#$H-;?xm)q|p#vK39aS)weY_sfrLnUG+|~y$*AA(fO=5b_G7v z1Y1+UYOk;NXk8xL=$Z3OtI71PNf~sX-{px1IJ!MyC{i^vtfw?LqYw~;(;ym{Az$Ke zNGvTi7l?Y1Mjk^BdQYYra(fLk5!Hb;)-02 zz^jL3MU#eY&@@c&Txk%-j(^(Tu-;JY5vP`)&KY>QD5@(r*G!$i|Ml<38y=9)$1cc0TCl&Unlo`O zA!3uI_GDZA*o%f>z|Pran~d1HnG)-OmLOX=D0QN6MRVtnrJCrE~OH zuMtX;1p&VBz!k2cZ5z0u7vJ98v}*2H8dXz&XZ?!`*9&ezIn#+bp1|7~?qQUZ7si0@ zMQ&6;1I=|Xw(8c_Cc>oLv~0#KB?9NsX!^(}uTbQwRBq8!fNu|$*8XE7^pE;yY#HkL z;&kfhCJq-(%2sa{G`t{^2H(hC@rXiRz|HKbFwQ%mp+jdc8?Vtbfe1w*SC!j_Rmdcg zhmtj|4_iNCXo(=m-q*YgJU2+dZXZd$Z*6`1=UqUbC{}|0Ns1P}@6s8)U*MWLVFM}! z^J7E~dvvaIZK0VN^!T1u^f~?uqZ^UgfCtO^s7v!&Xd+eC)c65iBo9bRJBYqTjZKqF z!_!zUh$iSUw1Z3b$Huc_-rW9{!zK1RV1lKb^Ei_n7;7INUTN;s8+ygY0yCLm(LD4H zz!=?wCC%&u&1g43Wso$EIOL%RSi+` z(_$0gqKp%IpvUV!rBrp}|90!Xo~o-p!S2FpZ*Q$@O0Z@YH3RZjb+3U3;$#igXPH|R zuAxl8dJY)a4Uk_{nqRwjM}GXXcDvkiCQCu=m0C5F&Pc&8(Fgswqda#r7o+`$HG|tk zl6z7K51v1kNoc2#JV3Z^k8JLf5;9Be8o= z;oC#qtM(klqTZNjCzf+YAcKLms>7m$p*IKcL8uOO*4q`o%g<~V?#Dxa0_2xCv$yaS z)#aLEjz?b!+Vt^E5yeT!*AW`pu?TFP;e=q7GiTi5H}01}(s7tG%4cZRGq#1_AnjAE$5T9&Is6OKjV$LZg-D)_u1;?JX!EEuZNh!(#{9a&nRLi}g;hJ>wfP3j;cO*0Vmz;zb+4 z!B)x#Lrg}w=y)pc2}l2`j^Asa{PaKCJL{;Zws?;ZAxKKM2oi#X#8A>IEifP=I3QgP z3|)exbhm(jv~+h!NQojLAl)EcN+a*6_g-+W+~@!AZPzSjmTP~`Z}0QlarW8!`M2Fwc?a7X!x8`#_3_vP!PZ6D1%=o-J$Yl~;_}*sOrF$%P+ItkL@s>edHWM^bsQUm# z-!pqYXRmp>z9en0RS{>Ag1GK|Z(UZfQ%(lPbr~=Q<1IQeH$o~wBL+&$5vV!u1Yxb| zW`LPoL+=tPhgcWfJIsQXUEkqT>uUjeMip9?THe*AKpdMyy1jsDfm+(+rEioyMoCHp zC6B9Bb|TC};0Zj9o@=w5k>X@JtqUj-TcOyu4<#n#jm&s5Jqp<8Y9zMu2o?y%SyFrn zW*r@#$+3o|@Tw8BE*Rg!&G6ZrIW#&n)SKF0HO>9vR~D~s5E~c+p;5i1iys~E?*Nfr+qeVm?fu#t|8o zlk*8?)Z;GgsclczS0?WH_JNLBX`6|jnuXq_+l4l!nRGY$8@d1d^kts>@VSTLl(J@P zbH1!<3?IufF0B(y-*0))<7%8l2&)JVbx=@Zy$6*NiM^curUKYJ;1@gzY7#OIRKI6p zN2dEyRDBa=+Q*M|vQz|jW3HZV1~*g)q97y zi?DXgh!U1ZF!~;7A7a!!h@P>@zAZ_?O!rBs;vQo!+3{@H1G`qb9v1J72f;k@=E_xN z{WdiL4BJ9EcHiFD1lS2RU5@S3ejwc(5YmZGHR^WiX zkL@$xO}7jz(MRfbcG@uOZ}b>Hn4i|+9@)1Ty4&{d9`Y*xBP^b=>cFpDNFS**yT7q6 z-dy)J|Du*>%WrO$fSk5NNE=4MB12xNLOitd={AdY0ev7!5T&bkB4L@gTJpwxERj`~ zluDTxX3~w|q%W;hPaI_e&B*MMrW7$p9`WQw$7Hlv8(!5T;z%i_!*?_SS)PEX3unGn zX6{OWtUPB+eI{g%~E|)xMJDnw8G>>13 zhDR^)nhEcUgmbJJqtL1B%}pH&vUITN*45XRNtm#Ey+mn(DY*- zZ*B#361v?aR_1hvbZ)dYSwX=l%{zFKN~tFxsb_cYt4Wg~SF(h$CUEY)2N}X#DS!>d z4j${bP1e}ILIne0PVq1`IkLMSkaIZsk)(#Tb*+2$IJeZLzj@z-Ol(X1iwrA=oCa`?r?N68Z zx>|89pRTu7c&Y#uER$(+;NlM-Gn;`rMBm^k4}ZlOIQ78&6BCLX6N<}^*O2Zv@@DJe z-ZM&8dl9J9;%r|P(9V_8mK6~l6N%!J1wFn^WhI#VxPTF>5Jb|@(6O$VJu$56sqn4) zQ00pDR_zI>%+d4h)@F^%$_!lK!(HVet&zu&((;lvOM(u1h8gXKn*vf=*~P_SpFEh5 zTZm(B_=A0KWRyfGmCHSU#Ge+}&J{d|O#8)j(N zA)mIehPCmfOcRD1XmV|i8-p>Y|kmU&Lj%569QyG8P8$TIv(G{;FoH_UCb6oMRZwnpjL_Zs{R zO~|7z-kk&&<5#|bgCAc(E*&#h8~BPNb+5**KmX1k$-w5D!gk+eT`6_qjv;S>3%5xH z$QsZgpDja=u5K`9v*{|5n(Xop6Q&={Bf&G@+C9sSXn*aPMw0$Ws%ztYLqm(x3CM#Q zJ>X9NbWBV4MYAuJbG0xPuk_M#|6cw#gy~DD_}3#8F&K`(*lQyE0Qx`nsLQVeXa>Ic5s%B-{m>^d2%*0 zHnmxr{n{Eb$5!r|Ks=;Es;}klDSIJOo1z{7Rp&JGP=5Q6rTv(%?nAmXLU1E<(m8K4c9oZu$-kZ;{Q)h?SRB z>^G8$vZa~zWr=*+dB}v)YBMnuFWX1m*ji+7%aET+oxxBOq#r69UYo>Xi?Sff{1#cW zeBG))9eC<>d*~X-*sGtU4-~TAH7^`=EquuWlJ0DEz9+= zHQjaCIwBN3y+@3i&gqcTBm+xr)s~=>xFwaiszP#W+Gj;pm84Zy?{X;lqnbho6hh!t z9q%ykbbeV?^Re8jibT<)wNGG7iEL=*n&*Y#`=ky6G}jVIpKHE!9}D1Kv$yzlok&Xj*yhv*Mz5D@n+}U~@w%lVebRU1 zTv1V~q`sHu3@RL$DU>OLMUzw;1k|-7xT@Xs!)k-_R}+?2!(QO85JA48w1KY-#oy`yufjW5Bjj;& zl21T7vO*=zI*GU2wMm4bJ6apx3XE9gZFYJUt#82f3nu5t6E%0VEc5~HpDSCQ*Z@Zn-NwC3 zHS1Uq++VlRIW?Bld0K#%(aD6}KG9khCXlE5z5eUV7KuP*v`%@72tNg+5 zOD97RITTbP5E=;3ItBz?J{<>z5{aOi zXDkgJEyT%FPi(+)%+jP(yATu0v3w$3$WqbKI)~`Na(9MtyaigSwe>AuGpXcC;pgg~ zSHBu6jha9EjxV#^pMcP$ovFu89b0yKwP63Wpbga zF_haUnay*wn2liY{sxt|C4@W6IHY2C(pE{DeH?85I9WqVSE7Z=-=_?0q0PNpNX`N> z5jwb~>^-kCqW07&z$~y%0UCDW&eX-D>1@fqj^_je{Ks)8;7MhQ-pN|q^q}<@xHF$( zJYWxB>9Z@)YNE2R#cXy`{zkt;tj#d!6rJr0gzl(4azEJSyK;*{p)KHPliNGOnAO3xpQuXOz zLBL(&I&+sStPHIGxl6gdO6DUpc%aE$nZrvY>!=*O!Fl{qm#Iv8sWY;Vk=k1^;y5o# zk7O;mUI8>E$G}-}J2o^nZT1%N@z;bIF7-MZ%u|FCqxrpX;(oOc@jR>%*6@6T&Us0f zt0cGa(qiNLo$bt^yxfRKUERw?{^3=Oneye>~D$7fYoMpj*mm*Wb7iur& zNf@$;ESyZF=UocnuAHq2nT@wpshiCZUA)LoYH9V5{@bfu*E~Z}<<;sC`!q9CWf89e z5e?Ojjw}ix`wJts1_NT=i@1jLayn2uHvJdf$a)^}r4IX1^2LuK^`!8ugV{cFTmXQ=2{&iKL8`@6m)|$17x7Sm@}|xyoVXJ#aGiHJY*v7MBG~21_NwAnfM#sY9djq3MK}BME3Hs-V2>|&&nO>+UDUNX9xg?dIXwNx@MkDD) z!Jhd;t|`5`<7PytMF|Tw+aJBPyKHYc+#PI8uq;BW9lH( zPWGvn*(_#erKnP%x}8kYBnsTs#6JQujG1006}i-P+m|9#2{T|O*S{xZi>edzv0oce zIgxrLq*K~ZZ603K&7e_Zb>A~+zeC$1u^Eb*BX)V8U1z|MDsSe5@qWJPL35_=40F|4k`RQOI*NrxbPfHq`4 zWG)lNYl%9QmNQ*WI}uQKfv>l7iY$_`JA+hbO}!N(HI zL3jV6BdrM`zA3&Dv>2D)bg*%6sr$n)rLmm|+x;5`58uJb)0vRM!go*d9>^^e^nPn@ zuunaNsS!SKGT|fo+?`kcdZUMQ?V;qSyusuMQ57z?6v6&Ht%b>Zan>9-uF5hssAV(G z*x-f5thzy62N$82TGe+1wX7f8`KTNL4eZ|?t2$p(;(g$Mo&yaO|7_#bGq*DM|JHfh z5e0^eD|&JewXHL{p!W(!Hqj4@#oR6NwDj{M3#!PdH4B1BL)-RtNO(Vy*l!GMoz!WK zJ251z=k$)a^A@X;UlsQif~2sg4eNgv);id2=Z?ZC&BI@SC<*#u^_(O>^Dv+;BGd8W+wD9-_=9( z!431~SM@5#`KUC}Bpwe<Z49z=xM?(4V#ps0SpHGf5P z0#HQ64@odYgml--0%O36FyNZMB0d7=TM-dI^t2EWPaV<4F922y19kT+!kp^gh#z89 zhzPl_!Yg$^C4N`dq&ZWWAW$Z7EcJ{PXVhCxm5lzJmIA#!Ju4=M zxet&`~m$V8)~0TkdC{EC}n`ET5B9S$;JyZyc8 zdyxD&r@8A#DC*{#sWp;A%^KY<;lqY zFTCG&tcY>3PI*kgGs{24{X2})nJxt}N*1D}h#{d)$v%+ZkP!n!A>t6@LY(4y)Xw03 z<2Ao6H=5I!X*>HUU$rfc}^5O%(y zgeJei{manv``7cyIOY{(yZx znG^meiQ%7TU+In@Z!n2N-KzFATD2@LR_-` zGUM09Y(xO!67eYjn*9^-%YrdtcEq*I)9gsVs`~fA=a-dC#O%K=pPgm~f$H+k)WKhi nX^6RheZD`N`(e?U+fPY~)Akcit_h0`5PIPON literal 0 HcmV?d00001 diff --git a/server/src/processing.rs b/server/src/processing.rs index baf13c14..9e1c6bff 100644 --- a/server/src/processing.rs +++ b/server/src/processing.rs @@ -1,3 +1,6 @@ +// TODO: Remove this after implementing the cache. +#![allow(unused)] + // Copyright (C) 2025 Bryan A. Jones. // // This file is part of the CodeChat Editor. The CodeChat Editor is free @@ -16,6 +19,10 @@ /// `processing.rs` -- Transform source code to its web-editable equivalent and /// back /// =========================================================================== +// Modules +// ------- +pub mod cache; + // Imports // ------- // @@ -24,16 +31,19 @@ use std::{ borrow::Cow, cell::RefCell, cmp::{max, min}, + collections::HashMap, collections::HashSet, ffi::OsStr, io, iter::Map, + mem, ops::Range, path::{Path, PathBuf}, rc::Rc, slice::Iter, string::FromUtf8Error, sync::LazyLock, + sync::{Arc, Mutex, Weak}, }; // ### Third-party @@ -66,10 +76,14 @@ use serde::{Deserialize, Serialize}; use ts_rs::TS; // ### Local -use crate::lexer::{ - CodeDocBlock, DocBlock, LEXERS, LanguageLexerCompiled, source_lexer, - supported_languages::MARKDOWN_MODE, +use crate::{ + lexer::{ + CodeDocBlock, DocBlock, LEXERS, LanguageLexerCompiled, source_lexer, + supported_languages::MARKDOWN_MODE, + }, + processing::cache::Target, }; +use cache::Cache; // Data structures // --------------- @@ -214,18 +228,6 @@ pub struct StringDiff { pub insert: String, } -/// This enum contains the results of translating a source file to the CodeChat -/// Editor format. -#[derive(Debug, PartialEq)] -pub enum TranslationResults { - /// This file is unknown to and therefore not supported by the CodeChat - /// Editor. - Unknown, - /// A CodeChat Editor file; the struct contains the file's contents - /// translated to CodeMirror. - CodeChat(CodeChatForWeb), -} - /// This enum contains the results of translating a source file to a string /// rendering of the CodeChat Editor format. #[derive(Debug, PartialEq)] @@ -261,10 +263,16 @@ static DOC_BLOCK_SEPARATOR_BROKEN_FENCE: LazyLock = LazyLock::new(|| { // Non-greedy wildcard -- match the first separator, so we don't munch // multiple `DOC_BLOCK_SEPARATOR_STRING`s in one replacement. ".*?", - "\n" + r"(\d+)\n" )) .unwrap() }); +/// After converting Markdown to HTML, this can be used to split doc blocks +/// apart. Since this is post hydration, the element names are normalized to +/// lower case. +static DOC_BLOCK_SEPARATOR_SPLIT_REGEX: LazyLock = LazyLock::new(|| { + Regex::new(r"\d+").unwrap() +}); // Use this as a way to end unterminated fenced code blocks and specific types // of HTML blocks. (The remaining types of HTML blocks are terminated by a blank @@ -294,16 +302,11 @@ const DOC_BLOCK_SEPARATOR_STRING: &str = concat!( r#" ~~~~~~~~~~~~~~~~~~~~~~~ - +{} "# ); -// After converting Markdown to HTML, this can be used to split doc blocks -// apart. Since this is post hydration, the element names are normalized to -// lower case. -const DOC_BLOCK_SEPARATOR_SPLIT_STRING: &str = - ""; // Correctly terminated fenced code blocks produce this, which can be removed // from the HTML produced by Markdown conversion. const DOC_BLOCK_SEPARATOR_REMOVE_FENCE: &str = r" @@ -314,10 +317,13 @@ const DOC_BLOCK_SEPARATOR_REMOVE_FENCE: &str = r" ~~~~~~~~~~~~~~~~~~~~~~~ "; -// The replacement string for the `DOC_BLOCK_SEPARATOR_BROKEN_FENCE` regex. +// The replacement string for the `DOC_BLOCK_SEPARATOR_BROKEN_FENCE` regex. It +// relies on the first capture group in that regex (`$1`) containing the index, +// which it replaces here. const DOC_BLOCK_SEPARATOR_MENDED_FENCE: &str = - "\n\n"; -// + "\n$1\n"; +// + // The column at which to word wrap doc blocks. const WORD_WRAP_COLUMN: usize = 80; // The minimum width for doc block word wrap, since large indents may leave @@ -446,7 +452,7 @@ pub fn codechat_for_web_to_source( } // Translate the HTML document to Markdown. let converter = HtmlToMarkdownWrapped::new(); - let tree = html_to_tree(&code_mirror.doc, None)?; + let tree = html_to_dom(&code_mirror.doc, None)?; dehydrating_walk_node(&tree); return converter .convert(&tree) @@ -633,7 +639,7 @@ pub fn doc_block_html_to_markdown( for (index, code_doc_block) in &mut code_doc_block_vec.iter_mut().enumerate() { if let CodeDocBlock::DocBlock(doc_block) = code_doc_block { last_doc_block_index = Some(index); - let tree = html_to_tree(&doc_block.contents, dom_location)?; + let tree = html_to_dom(&doc_block.contents, dom_location)?; dehydrating_walk_node(&tree); // Calculate the total delimiter width: the delimiter width plus the @@ -838,6 +844,8 @@ pub enum SourceToCodeChatForWebError { // convert the IO error to a string. #[error("unable to parse HTML {0}")] ParseFailed(String), + #[error("no lexer for this file")] + NoLexer, #[error("encoding error {0}")] EncodeFailed(#[from] FromUtf8Error), } @@ -851,14 +859,22 @@ pub fn source_to_codechat_for_web( // The file's contents. file_contents: &str, // The file's extension. - file_ext: &String, + file_path: &Path, // The version of this file. version: f64, // True if this file is a TOC. _is_toc: bool, - // True if this file is part of a project. - _is_project: bool, -) -> Result { + // If provided, the cache for this project; otherwise, this file is not in a + // project. + cache: Option>>, +) -> Result { + // Determine the file's extension, in order to look up a lexer. + let file_ext = &file_path + .extension() + .unwrap_or_else(|| OsStr::new("")) + .to_string_lossy() + .to_string(); + // Determine the lexer to use for this file. let lexer_name; // First, search for a lexer directive in the file contents. @@ -875,13 +891,18 @@ pub fn source_to_codechat_for_web( match LEXERS.map_ext_to_lexer_vec.get(file_ext) { Some(llc) => llc.first().unwrap(), _ => { - // The file type is unknown; treat it as plain text. - return Ok(TranslationResults::Unknown); + // The file type is unknown; we can't lex it. + return Err(SourceToCodeChatForWebError::NoLexer); } } }; // Transform the provided file into the `CodeChatForWeb` structure. + let cache = if let Some(project_cache) = cache { + project_cache + } else { + Arc::new(Mutex::new(Cache::new())) + }; let code_doc_block_arr; let codechat_for_web = CodeChatForWeb { metadata: SourceFileMetadata { @@ -889,11 +910,12 @@ pub fn source_to_codechat_for_web( }, version, source: if lexer.language_lexer.lexer_name.as_str() == MARKDOWN_MODE { - // Document-only files are easy: just encode the contents. + // Document-only files are easy: just encode the contents. Tags are + // only supported in source files. let dry_html = markdown_to_html(file_contents); - let html = hydrate_html(&dry_html) + let html = hydrate_html(&dry_html, file_path, cache) .map_err(|e| SourceToCodeChatForWebError::ParseFailed(e.to_string()))?; - let html = minify(&html)?; + let html = minify(&html.0)?; CodeMirrorDiffable::Plain(CodeMirror { doc: html, doc_blocks: vec![], @@ -923,18 +945,24 @@ pub fn source_to_codechat_for_web( // Walk through the code/doc blocks, ... let doc_contents = code_doc_block_arr .iter() + .enumerate() // ...selecting only the doc block contents... - .filter_map(|cdb| { + .filter_map(|(index, cdb)| { if let CodeDocBlock::DocBlock(db) = cdb { - Some(db.contents.as_str()) + Some((index, db.contents.as_str())) } else { None } }) - // ...then collect them, separated by the doc block separator - // string. - .collect::>() - .join(DOC_BLOCK_SEPARATOR_STRING); + // Add the doc block separator string between each doc block; + // the separator contains the index of this doc block. + .fold(String::new(), |mut acc: String, x: (usize, &str)| { + if !acc.is_empty() { + acc.push_str(&DOC_BLOCK_SEPARATOR_STRING.replace("{}", &x.0.to_string())); + } + acc.push_str(x.1); + acc + }); // Convert the Markdown to HTML. let html = markdown_to_html(&doc_contents); @@ -947,11 +975,14 @@ pub fn source_to_codechat_for_web( // 2. Remove good fences. let html = html.replace(DOC_BLOCK_SEPARATOR_REMOVE_FENCE, ""); // 3. Hydrate the cleaned HTML. - let html = hydrate_html(&html) + let (html, _tags) = hydrate_html(&html, file_path, cache) .map_err(|e| SourceToCodeChatForWebError::ParseFailed(e.to_string()))?; // 4. Split on the separator. - let mut doc_block_contents_iter = html.split(DOC_BLOCK_SEPARATOR_SPLIT_STRING); - // + let mut doc_block_contents_iter: regex::Split<'_, '_> = + DOC_BLOCK_SEPARATOR_SPLIT_REGEX.split(&html); + // 5. TODO Cache updates: process `tags`. + // + // Translate each `CodeDocBlock` to its `CodeMirror` equivalent. let mut len = len_utf16(&code_mirror.doc); for code_or_doc_block in code_doc_block_arr { @@ -986,7 +1017,7 @@ pub fn source_to_codechat_for_web( }, }; - Ok(TranslationResults::CodeChat(codechat_for_web)) + Ok(codechat_for_web) } // Options for a spec-compliant minifier. @@ -1064,6 +1095,8 @@ pub fn source_to_codechat_for_web_string( version: f64, // True if this file is a TOC. is_toc: bool, + // The map of Caches. + cache: Arc>>>>, ) -> Result< ( // The resulting translation. @@ -1073,42 +1106,43 @@ pub fn source_to_codechat_for_web_string( ), SourceToCodeChatForWebError, > { - // Determine the file's extension, in order to look up a lexer. - let ext = &file_path - .extension() - .unwrap_or_else(|| OsStr::new("")) - .to_string_lossy(); - // To determine if this source code is part of a project, look for a project // file by searching the current directory, then all its parents, for a file // named `toc.md`. let path_to_toc = find_path_to_toc(file_path); - let is_project = path_to_toc.is_some(); + let cache: Option>> = path_to_toc.as_ref().map(|path_to_toc| { + cache + .lock() + .unwrap() + .entry(path_to_toc.to_path_buf()) + .or_insert(Arc::new(Mutex::new(Cache::new()))) + .clone() + }); Ok(( - { - let translation_results = source_to_codechat_for_web( - file_contents, - &ext.to_string(), - version, - is_toc, - is_project, - )?; - match translation_results { - TranslationResults::CodeChat(codechat_for_web) => { - if is_toc { - // For the table of contents sidebar, which is pure - // markdown, just return the resulting HTML, rather than - // the editable CodeChat for web format. - let CodeMirrorDiffable::Plain(plain) = codechat_for_web.source else { - panic!("No diff!"); - }; - TranslationResultsString::Toc(plain.doc) - } else { - TranslationResultsString::CodeChat(codechat_for_web) - } + match source_to_codechat_for_web(file_contents, file_path, version, is_toc, cache) { + Err(err) => { + // The no lexer error means we should treat this file type as + // unknown. + if err == SourceToCodeChatForWebError::NoLexer { + TranslationResultsString::Unknown + } else { + // Otherwise, this is an unhandleable error. + return Err(err); + } + } + Ok(codechat_for_web) => { + if is_toc { + // For the table of contents sidebar, which is pure + // markdown, just return the resulting HTML, rather than the + // editable CodeChat for web format. + let CodeMirrorDiffable::Plain(plain) = codechat_for_web.source else { + panic!("No diff!"); + }; + TranslationResultsString::Toc(plain.doc) + } else { + TranslationResultsString::CodeChat(codechat_for_web) } - TranslationResults::Unknown => TranslationResultsString::Unknown, } }, path_to_toc, @@ -1134,7 +1168,7 @@ fn markdown_to_html(markdown: &str) -> String { pub const UNICODE_CURSOR_MARKER: char = '\u{E83B}'; /// Use html5ever to parse a string containing HTML to a DOM tree. -fn html_to_tree( +fn html_to_dom( html: &str, // See the same parameter from `doc_block_html_to_markdown`. dom_location: Option<&(Vec, usize)>, @@ -1190,9 +1224,13 @@ fn html_to_tree( // A framework to transform HTML by parsing it to a DOM tree, walking the tree, // then serializing the tree back to an HTML string. pub fn transform_html)>(html: &str, transform: T) -> io::Result { - let tree = html_to_tree(html, None)?; + let tree = html_to_dom(html, None)?; transform(&tree); + dom_to_html(tree) +} +// Transform a DOM tree back to an HTML string. +pub fn dom_to_html(dom: Rc) -> io::Result { // Serialize the transformed DOM back to a string. let so = SerializeOpts { // Don't include the body node in the output. @@ -1202,7 +1240,7 @@ pub fn transform_html)>(html: &str, transform: T) -> io::Res let mut bytes = vec![]; serialize( &mut bytes, - &SerializableHandle::from(get_dom_body(&tree)), + &SerializableHandle::from(get_dom_body(&dom)), so, )?; let html_out = String::from_utf8(bytes).map_err(io::Error::other)?; @@ -1229,13 +1267,130 @@ fn get_dom_body(document: &Rc) -> Rc { // * (Eventually) record document structure information. // * (Eventually) assign a unique ID to all links that don't have one. // * (Eventually) fill in autocomplete fields. -fn hydrate_html(html: &str) -> io::Result { - transform_html(html, hydrating_walk_node) +fn hydrate_html( + html: &str, + file: &Path, + cache: Arc>, +) -> io::Result<(String, Vec>)> { + let dom = html_to_dom(html, None)?; + let file_entry = cache.lock().unwrap().get_or_create_file(file); + // Move the `target` vec into a `HashMap`; the vec will be re-created by + // moving individual entries back as they're found in the HTML. TODO: + // invalidate all current weak links to all `Targets` in this vec. Must also + // update all weak links to the underlying `File` when moving a `Target`. + let old_target_vec = mem::take(&mut file_entry.lock().unwrap().target); + let old_targets: HashMap>> = old_target_vec + .into_iter() + .map(|target| (target.clone().lock().unwrap().id.clone(), target)) + .collect(); + // This is storage for the state needed for walking the DOM. + let mut walk_context = WalkContext { + cache, + file_entry, + old_targets, + doc_block_index: 0, + links: Vec::new(), + backlinks: Vec::new(), + tags: Vec::new(), + code_doc_block_dependencies: vec![], + }; + walk_context = hydrating_walk_node(dom.clone(), walk_context); + // TODO: + // + // 1. Remove all `old_targets` that weren't moved back to `targets`. + // + // 2. Process `links`: + // + // 1. If the target of this link is in the cache, first check to see if + // it's been processed. If not, mark the current file to be added to + // the end of `pending_files` list. Remove any previous + // `references`/`dependencies`, then re-add them based on the link's + // `Target.backlink_type`: + // + // 1. `None`: if this is an auto-titled link, get its title from the + // `Target` it refers to (this includes the case where the target + // doesn't exist) and add this file to the target's list of + // `references` if it's not already there; also add it to the + // `code_doc_block_dependencies` list; problem: how would we know + // the doc block index? Otherwise, add it to the target's list of + // `dependencies` (again, if it's not already there). Remove it + // from the other list if it's there. + // 2. `Plain`/`wrapped`: give this link an ID if it doesn't have one + // (meaning add it as a new `Target`). As above, add/verify this + // link is in the `references`/`dependencies` and remove the + // opposite link. If this is added, mark the link target's `File` + // as dirty. + // 3. `Gather`: same as above. Also, record the start/end code/doc + // blocks and put this in the code/doc block `tags` list. + // 2. Otherwise, add the file containing this target to set of files to + // process (`Cache::pending_files`) and mark the current file to be + // added to this set after all dirty dependencies are added. + // 3. Process `backlinks`: generate the appropriate HTML. These are + // processed after `links`, to ensure these are updated / assigned an ID + // in case a backlink references a link in this file. Corner case: if a + // gather element refers to tags in the current file, and the code+doc + // block contents of these tags is outdated, then this file need to be + // reprocessed, since the code+doc block contents won't be updated until + // after this function returns. + + Ok((dom_to_html(dom)?, walk_context.tags)) +} + +/// Get the value of an attribute on an element node. +fn get_attr_value(node: &Rc, attr_name: &str) -> Option { + if let NodeData::Element { attrs, .. } = &node.data { + attrs + .borrow() + .iter() + .find(|attr| &*attr.name.local == attr_name) + .map(|attr| attr.value.to_string()) + } else { + None + } +} + +/// Gather text from all text children of this node. +fn get_text_content(node: &Rc) -> String { + let mut text = String::new(); + for child in node.children.borrow().iter() { + if let NodeData::Text { contents } = &child.data { + text.push_str(&contents.borrow()) + } + } + text +} + +/// This provides the needed context when walking the HTML DOM of all doc +/// blocks. +struct WalkContext { + /// The cache for this project. + cache: Arc>, + /// The `cache::File` currently being processed. + file_entry: Arc>, + /// The previous `file_entry.targets` that haven't yet been claimed while + /// processing this file. The key is the `Target`'s ID. + old_targets: HashMap>>, + /// All hyperlinks found in this file. + links: Vec>, + /// Backlinks and gather elements. + backlinks: Vec>, + /// Tags (links to gather elements). + tags: Vec>, + /// For each doc block in the vec of `CodeDocBlock`s, this contains the + /// `Target` of autotitled  links. These are used to determine a tag's + /// indirect dependencies if this doc block is included in a tag. + code_doc_block_dependencies: Vec>>>, + /// The current doc block index, based on parsing the HTML for + /// `codechateditor-separator` elements, which contain this value. + doc_block_index: usize, } -fn hydrating_walk_node(node: &Rc) { +/// Hydrate the HTML of newly-translated doc blocks. +fn hydrating_walk_node(node: Rc, mut walk_context: WalkContext) -> WalkContext { for child in node.children.borrow_mut().iter_mut() { let possible_replacement_child = + // Perform replacements of GraphViz and Mermaid graphs: + // // Look for a `

` tag
         if get_node_tag_name(child) == Some("pre")
             // with no attributes
@@ -1248,7 +1403,7 @@ fn hydrating_walk_node(node: &Rc) {
             && code_children.len() == 1
             && let code_child = code_children.iter().next().unwrap()
             && get_node_tag_name(code_child) == Some("code")
-            // with only a `class=language-mermaid` attribute
+            // with only a `class=language-mermaid/graphviz` attribute
             && let NodeData::Element {
                 attrs: ref_code_child_attrs, ..
             } = &code_child.data
@@ -1279,14 +1434,75 @@ fn hydrating_walk_node(node: &Rc) {
             replace_math_node(child, true)
         };
 
-        // Replace the child if we found a replacement; otherwise, walk it.
+        // Replace the child if we found a replacement.
         if let Some(replacement_child) = possible_replacement_child {
-            replacement_child.parent.set(Some(Rc::downgrade(node)));
+            replacement_child.parent.set(Some(Rc::downgrade(&node)));
             *child = replacement_child;
-        } else {
-            hydrating_walk_node(child);
         }
+
+        // Analyze this node for cacheable data.
+        if let Some(tag_name) = get_node_tag_name(child) {
+            // See if the element has an id/anchor.
+            let _id = get_attr_value(child, "id").unwrap_or_default();
+
+            // Track doc block index from separator elements.
+            if tag_name == "codechateditor-separator"
+                && let Ok(index) = get_text_content(child).trim().parse::()
+            {
+                walk_context.doc_block_index = index;
+            }
+
+            // TODO: write code here.
+            //
+            // When walking, look for:
+            //
+            // 1. A link, or any element that's a back link or gather element.
+            //    Note that links directly to a file are ignored, since files
+            //    are not targets. Add these to the `links`/`backlinks` list.
+            // 2. A target (any item with an id which refers to a file in the
+            //    project tree).
+            //    1. Process the ID:
+            //       1. ID exists in `old_targets` - transfer ownership by
+            //          appending this to `file.targets`, removing it from the
+            //          `old_targets`; update the stale `Weak` pointer to the
+            //          parent `File`. Assert that this ID is unique.
+            //       2. ID doesn't exist in `old_targets` and is unique: add
+            //          this `Target` to `Cache::id`.
+            //       3. ID doesn't exist in `old_targets` and is a duplicate:
+            //          resolve duplicate IDs:
+            //          1. Is the duplicate ID in the same file? In this case,
+            //             we have no way to determine which was the original
+            //             ID. Rename the ID being processed; stop here.
+            //          2. Have all new files been processed? If not, add this
+            //             file to the list of dirty files which will be
+            //             re-processed after new file processing is done. Stop
+            //             here.
+            //          3. Look at the timestamp of this file and of the other
+            //             file which contains the duplicate ID. If this file is
+            //             newer, rename this ID. Otherwise, update the ID and
+            //             mark the other file as dirty.
+            //    2. Check the `contents`: if the `contents` changed, add all
+            //       this target's `dependencies` to the dirty list and update
+            //       the search text for this target.
+            //    3. Check the `backlink_type`. If this changed:
+            //       1. To `Gather` from any other type: add all the target's
+            //          `references` and `dependencies` to the dirty list, since
+            //          some `references` may need IDs and everything needs
+            //          code+doc blocks.
+            //       2. From `Gather` to any other type: Do nothing. The
+            //          code+doc blocks will be removed lazily, since they have
+            //          no impact on the resulting output.
+            //       3. From `None` to `Wrapped` or `Plain`: add `references`
+            //          with no ID to the dirty list.
+            //       4. From `Wrapped` to `None`: nothing to do.
+            //    4. Walk the list of `references`/`dependencies`, removing any
+            //       dropped references.
+        }
+
+        walk_context = hydrating_walk_node(child.clone(), walk_context);
     }
+
+    walk_context
 }
 
 fn replace_math_node(child: &Rc, is_hydrate: bool) -> Option> {
@@ -1746,8 +1962,7 @@ pub fn diff_str(before: &str, after: &str) -> Vec {
 
 // #### Diff support for `CodeMirrorDocBlockVec`
 /// We can't simply implement traits for `CodeMirrorDocBlockVec`, since it's not
-/// a struct. So, wrap that it in a struct, then implement traits on that
-/// struct.
+/// a struct. So, wrap it in a struct, then implement traits on that struct.
 struct CodeMirrorDocBlocksStruct<'a>(&'a CodeMirrorDocBlockVec);
 
 /// Only compare the `contents` of two doc blocks; later, we'll compare the
@@ -1937,7 +2152,7 @@ pub fn diff_code_mirror_doc_blocks(
     // while deletions must be performed beginning to end.
     //
     // Therefore, look for sequences of insertions (adds) or updates where
-    // `from_new` > `from` and swap these sequences.
+    // `from_new` > `from` and swap these sequences.
     let mut immediate_sequence_start_index: Option = None;
     for index in 0..change_specs.len() {
         let is_add = matches!(&change_specs[index], CodeMirrorDocBlockTransaction::Add(_));
diff --git a/server/src/processing/cache.rs b/server/src/processing/cache.rs
new file mode 100644
index 00000000..a8e70aeb
--- /dev/null
+++ b/server/src/processing/cache.rs
@@ -0,0 +1,505 @@
+// TODO: Remove these after implementing the cache.
+#![allow(unused_variables)]
+#![allow(unused)]
+
+// Copyright (C) 2025 Bryan A. Jones.
+//
+// This file is part of the CodeChat Editor. The CodeChat Editor is free
+// software: you can redistribute it and/or modify it under the terms of the GNU
+// General Public License as published by the Free Software Foundation, either
+// version 3 of the License, or (at your option) any later version.
+//
+// The CodeChat Editor is distributed in the hope that it will be useful, but
+// WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
+// details.
+//
+// You should have received a copy of the GNU General Public License along with
+// the CodeChat Editor. If not, see
+// [http://www.gnu.org/licenses](http://www.gnu.org/licenses).
+/// `cache.rs` - Keep a cache used to store all targets in a project
+/// ================================================================
+///
+/// The cache stores the location (file name and ID), numbering (of headings in
+/// the TOC and figures/equations/etc. on a page), and contents (inner HTML or
+/// code/doc blocks for tags) of a target. Targets are HTML elements with an ID,
+/// making them anchors (such as headings, figure titles, display equations,
+/// tags, hyperlinks, etc.), or files.
+///
+/// The goal of the cache is to support auto-titled links, backlinks, and gather
+/// elements, and to ensure that all IDs are unique within a project. This means
+/// that links persist across moving or renaming files, since the IDs will be
+/// found in the cache.
+///
+/// Auto-titled links
+/// -----------------
+///
+/// A hyperlink with empty
+/// [link text](https://spec.commonmark.org/0.31.2/#link-text) is auto-titled --
+/// the contents of the target it references provide the link text. For example,
+/// after processing, the link in the following Markdown
+///
+/// ```Markdown
+/// 

Bar

+/// [](#foo) +/// ``` +/// +/// becomes `[Bar](#foo)`. This works even when the target is located in a +/// different file. Auto-titled links don't support indirection: link A whose +/// link text comes from link B whose link text comes from target C doesn't +/// work; link A will end up with an empty title. +/// +/// Tags +/// ---- +/// +/// A gather element such as `

Bazzy +/// things

` becomes a list of the contents of tags which reference it after +/// processing by the cache. A tag is simply a link to a gather element, such as +/// `[](#baz)`, which becomes `Bazzy things` after +/// auto-titling and auto-assignment of an ID. The tag's content by default +/// includes the contents of the current doc block and the contents of the next +/// code/doc block. Tags can also include an end query parameter to enclose a +/// wider range of code/doc blocks; for example, `[](#baz?end=3)` includes the +/// next 3 code/doc blocks. +/// +/// Tag contents may not include a gather element. They do support indirection: +/// gather element A includes contents from tag B, which contains an auto-titled +/// link to target C. Changes to target C makes B and A dirty. +/// +/// Example output of the gather tag `

Bazzy +/// things

`: +/// +/// ```html +///

Bazzy things

+/// +/// (first item content) +/// ... +/// +/// (last item content) +/// ``` +/// +/// Backlinks +/// --------- +/// +/// Given an ID, a backlink produces a list of links which reference it. This +/// provides a way to create an index, or show what references +/// headings/footnotes/endnotes, etc. Backlinks are like gather elements, but +/// instead of capturing tag contents, they capture target contents. In +/// addition, backlinks don't support indirect dependencies: backlink A, which +/// link B references, doesn't depend on link B's auto-titled text from target +/// C. +/// +/// The default backlink style produces a disclosure widget using a link icon +/// which reveals an unordered list of links when clicked; the plain style +/// simply presents a list of links. Support for ordering backlinks may be added +/// later; these will not support nesting (just as tags don't support nesting). +/// +/// Syntax: `element +/// text`, where `el` is an HTML element (such as `h1-6` or `a`). After +/// processing, this becomes: +/// +/// ```html +/// element text +///
+/// 🔗 +/// +///
+/// +/// ``` +/// +/// Search +/// ------ +/// +/// The cache supports searching the contents of all targets. +/// +/// Table of contents +/// ----------------- +/// +/// I'd like to build a TOC. There are several approaches: +/// +/// * Sphinx, for example, allows a nested hierarchy of TOCs. The disadvantage is +/// that document structure is spread throughout the hierarchy. +/// * PreTeXt bases TOC on a global assignment of chapters, sections, etc. The +/// disadvantage is that moving sections may mean a lot of rewrite to move +/// everything around. +/// * mdbook keeps TOC and a local TOC separate. This means the global TOC can't +/// include the local TOC. +/// +/// My thought is to build something as close to PreTeXt as possible, since +/// that's my primary export target. Also, I want to create a global TOC, and not +/// be constrained by filesystem layout, since that is sometime dictated by the +/// toolchain. So, a list of files in which headings are part of the global TOC +/// makes the most sense to me. This does means that a page may have no h1 +/// headings, though. I need to look at PreTeXt to see how they handle this. +/// Here's a first pass mapping: +/// +/// * book/article = h1 +/// * part = XML only (since it doesn't have any actual contents other than +/// sections) +/// * chapter = h2 +/// * section = h3 +/// * subsection = h4 +/// * subsubsection = h5 +/// * paragraphs = h6 (for paragraphs earlier in the hierarchy, use the XML tag). +/// +/// KISS is very important here. How can I create something I can accomplish? I +/// really want to re-use as much of PreTeXt as I possibly can. I mainly want +/// Markdown because it remove a lot of the mess of paragraphs, em/strong, etc. +/// So, this tool ignores parts when building its simplified TOC. To build a TOC, +/// it simply takes a sequential list of files and scans them for these headings. +/// Later features could include the ability to specify files using wildcards, +/// incorporate ignores, etc. +/// +/// Other PreTeXt conversion notes: hyperlinks map to xrefs, mostly. An +/// auto-title link maps directly to an xref; to be more specific, use the xref +/// tag. A standard internal link maps to an xref with @text=custom to get a +/// fairly similar result; the link title gets munched by PreTeXt (oh, well). +/// Another approach: translate all hyperlinks to PreTeXt url, and use xref +/// directly only for xrefs. That seems like a good idea. Perhaps define a +/// translation table between Markdown and PreText. +/// +/// What if I get rid of backlinks and instead rely on PreTeXt's index for this +/// sort of functionality? That might be simpler. +/// +/// Gather elements are of course unique. I don't know how to translate these. I +/// don't think they map nicely to LP support in PreTeXt, since that's a bit +/// backward. Instead, generate PreTeXt from tags. +/// +/// Next crazy thought: focus only on LP for now, making anything else (e-books) +/// a secondary focus. Keep the current Markdown TOC for simplicity. Support only +/// xrefs (forward pointer to gather elements) and gather elements, which is +/// really the points. In this case, the cache with targets is fine. Existing +/// hyperlink support (to a file, but not to anchors) is fine. xrefs move to IDs. +/// xrefs to gather elements must have IDs, since we need a bidirectional link. +/// Knuth gives doc blocks a name. I give them an ID, which is a bit less +/// intrusive. Perhaps tags are different that xrefs? A tag is an ID, a gather +/// reference, and possibly a length. This sounds close enough to an xref to +/// reuse; probably add a new field (length, span, etc.) +/// +/// The overall goal: record design, specification, and implementation stuff that +/// can't be derived from the code. Document as much/as little of the source as +/// needed. I like the idea of eventual migration toward PreTeXt, but as a +/// secondary goal. I like their TOC approach. But for now, just xrefs + gather +/// tags is all I do. Do all xrefs get an auto-assigned ID? Don't really need it +/// for xrefs to non-gather tags. I prefer minimal. +/// +/// I wish my program had fewer bugs. Writing here is disappointing. But the +/// overall approach is good and makes it easy to add in ideas. +/// +/// Should I take the time to fix exiting bugs first? I'm out of time to get +/// everything done. Sigh. I'll focus first on this, which is core, and record +/// bogs for later. +/// +/// Goals +/// ----- +/// +/// * Given a path to a file, retrieve the associated location, numbering, and +/// contents (a list of all targets in the containing file). +/// * Perform a search of all Target contents, returning a list of matching +/// targets. +/// * Given an id, retrieve the associated `Target`, all `Target`s which +/// reference this id but don't depend on it, and all `Target`s which +/// reference this anchor and also depend on it. +/// +/// Thinking space: +/// +/// * Any file can be submitted for a cache update. After the update finishes, +/// the Server checks to see if this update was to the file currently being +/// edited in the Client. +/// * Non-project files support a subset of this functionality: basically, treat +/// the project as a single file. Backlinks to other files work; tags and +/// backlinks within the current file work. +/// +/// Code changes elsewhere: +/// +/// 1. (Longer-term) modify the pulldown-cmark HTML writer to preserve line +/// numbers. +/// 2. Revise the TOC loader to use mdbook's code to process and update the TOC. +// Imports +// ------- +// +// ### Standard library +use std::{ + collections::{HashMap, HashSet}, + path::{Path, PathBuf}, + rc::Rc, + sync::{Arc, Mutex, Weak}, +}; + +// ### Third-party +use markup5ever_rcdom::Node; + +// ### Local +// +// None. + +/// Data structures +/// --------------- +/// +/// This defines the cache used to store all targets in a project. +pub struct Cache { + /// Provide rapid access to a file by its absolute path; it must be within + /// the project's root directory. This is the sole owner of these `File`s. + pub(super) path: HashMap>>, + /// Provide rapid access to a `Target` by its unique id. + pub(super) id: HashMap>>, + /// All files that need to be processed. Only `File::status::Clean` files + /// that just became dirty should be added, since non-clean files by + /// definition are already in the vec. + pub(super) pending_files: Vec, + /// The root directory of this project. + pub(super) root: PathBuf, + // TODO: search engine data storage. Search fields: target ID, contents, + // file name. Perhaps [Tantivy](https://docs.rs/tantivy/latest/tantivy/)? +} + +/// This stores metadata for given file. For non-page files (non-existent files, +/// images, PDFs, etc.) many of the fields are empty or `None`. +pub(super) struct File { + /// The full path to this file; it must be within the project's root + /// directory. This file may not exist -- it could be created by a broken + /// link. + pub(super) path: PathBuf, + /// The status of this file. + pub(super) status: FileStatus, + /// The TOC's numbering for this file; empty if it's either not in the TOC, + /// or is a prefix/suffix chapter. Taken from + /// [mdbook::book::SectionNumber](https://docs.rs/codam-mdbook/latest/mdbook/book/struct.SectionNumber.html). + pub(super) toc: Vec, + /// All targets on this page, in order of appearance on the page. This is + /// the only owner of `Target` data. + pub(super) target: Vec>>, + /// The first (and hopefully only) `h1` target on this page. + pub(super) h1: Weak>, +} + +/// The status of a file from the cache's perspective. +pub(super) enum FileStatus { + /// The file hasn't been processed yet. Typically, this is a file referenced + /// by a link but not available in the cache. + Pending, + /// The file need to be re-processed. + Dirty, + /// The file has been processed. (It may not exist.) + Clean, +} + +/// Contains all information about a target. A target is any HTML element with +/// an id. This means that links directly to a file are not considered a target +/// or tracked by the cache. +pub(super) struct Target { + /// The file which contains this target. + pub(super) file: Weak>, + /// The id of this target. It must be globally unique within the project. + pub(super) id: String, + /// The line number of this target in `File`. + pub(super) line: usize, + /// The index of the doc block which contains this Target in the vec + /// of `CodeDocBlock`s for this file. + pub(super) code_doc_block_index: usize, + /// The type of backlink for this target. + pub(super) backlink_type: BacklinkType, + /// The HTML contents (or HTML context, if this target has no content, such + /// as ``) of this element. Tags, which contain multiple code + /// and doc blocks, must be rendered to static HTML. + pub(super) contents: String, + /// All references to this target which don't depend on it. The key is the + /// file path for a file, or the ID for a Target. Assume that IDs and file + /// names don't overlap. + pub(super) references: HashMap, + /// Targets may depend on data from another file within this project. + /// Typically, these are auto-titled hyperlinks or backlinks. If this Target + /// is a gather element, this contains both direct dependencies (backlinks + /// for the gather element's anchor) and indirect dependencies (dependencies + /// of each of these backlinks). + /// + /// All references to this target that also depend on it; if this Target + /// changes, all these must be updated. The key is the file path for a file, + /// or the ID for a Target. + pub(super) dependencies: HashMap, +} + +/// Links can have no ID, and therefore are identifiable only by the file they +/// reside in, or they have an ID and are therefore a target. +pub(super) enum LinkType { + File(Weak>), + Target(Weak>), +} + +/// Describe the type of this target's backlink. +pub(super) enum BacklinkType { + /// This target is not a backlink. + None, + /// This target has a gather tag backlink. + Gather, + /// This target has a wrapped backlink. + Wrapped, + /// This target has a plain backlink. + Plain, +} + +/// Query parameters parsed into known link options. +pub(super) enum LinkOptions { + Plain, + AutoTitle, + AutoNumber, + AutoTitleAndNumber, +} + +// Code +// ---- +impl Cache { + pub fn new() -> Self { + Cache { + path: HashMap::new(), + id: HashMap::new(), + pending_files: vec![], + root: PathBuf::new(), + } + } + + /// Look up or create a `File` entry in the cache for the given path. + pub(super) fn get_or_create_file(&mut self, path: &Path) -> Arc> { + self.path + .entry(path.to_path_buf()) + .or_insert_with(|| { + Arc::new(Mutex::new(File { + path: path.to_path_buf(), + status: FileStatus::Pending, + toc: vec![], + h1: Weak::new(), + target: vec![], + })) + }) + .clone() + } +} + +impl Default for Cache { + fn default() -> Self { + Cache::new() + } +} + +#[cfg(test)] +mod tests { + use std::{ + borrow::BorrowMut, + collections::{HashMap, HashSet}, + hash::Hash, + sync::{Arc, Mutex, Weak}, + }; + + use indoc::indoc; + use test_utils::prep_test_dir; + + use crate::processing::cache::{BacklinkType, Cache, File, FileStatus, Target}; + + // Verify basic parsing + #[test] + fn test_1() { + let (temp_dir, test_dir) = prep_test_dir!(); + let bar_cpp = indoc!( + r#" + // # Heading 1 + // + // ## Heading 2 + // + // + // + // [File link](bar.cpp) + // + // [anchor link](bar.cpp#one) + // + // [][baz.cpp) + // + // [](baz.cpp#one) + // + // [](baz.cpp#one?number) + // + // [](baz.cpp#one?title&number) + // + // [][baz.cpp#gathering_tag) + code(); + "# + ); + + let bar_cpp_path = test_dir.join("bar.cpp"); + let file_bar_cpp = Arc::new(Mutex::new(File { + path: bar_cpp_path.clone(), + status: FileStatus::Pending, + toc: vec![1], + // Since we haven't parsed the file, the `h1` hasn't been found. + h1: Weak::new(), + // Same for targets. + target: vec![], + })); + let baz_cpp_path = test_dir.join("baz.cpp"); + + // Create a baz file that's been processed. It contains one heading and + // a gather tag. + let mut file_baz_cpp = Arc::new(Mutex::new(File { + path: baz_cpp_path.clone(), + status: FileStatus::Clean, + toc: vec![2], + // This is filled in below. + h1: Weak::new(), + // This is filled in below. + target: vec![], + })); + file_baz_cpp.borrow_mut().lock().unwrap().target = vec![ + Arc::new(Mutex::new(Target { + file: Arc::downgrade(&file_baz_cpp), + id: "one".to_string(), + line: 1, + code_doc_block_index: 0, + backlink_type: BacklinkType::None, + contents: "Heading one".to_string(), + references: HashMap::new(), + dependencies: HashMap::new(), + })), + Arc::new(Mutex::new(Target { + file: Arc::downgrade(&file_baz_cpp), + id: "gathering_tag".to_string(), + line: 1, + code_doc_block_index: 0, + backlink_type: BacklinkType::Gather, + contents: "Gather tag".to_string(), + references: HashMap::new(), + dependencies: HashMap::new(), + })), + ]; + let h1 = Arc::downgrade(&file_baz_cpp.lock().unwrap().target[0]); + file_baz_cpp.borrow_mut().lock().unwrap().h1 = h1; + + let mut cache_path = HashMap::new(); + cache_path.insert(bar_cpp_path, file_bar_cpp); + cache_path.insert(baz_cpp_path, file_baz_cpp.clone()); + let mut cache_id = HashMap::new(); + cache_id.insert( + "one".to_string(), + file_baz_cpp.lock().unwrap().target[0].clone(), + ); + cache_id.insert( + "gathering_tag".to_string(), + file_baz_cpp.lock().unwrap().target[1].clone(), + ); + let mut cache_anchor = HashMap::new(); + + let mut cache = Cache { + path: cache_path, + id: cache_anchor, + pending_files: vec![], + root: test_dir, + }; + + // Processing a file updates its values in the cache. + //cache.upsert_file_core(&bar_cpp_path, ); + + temp_dir.close().unwrap(); + } +} diff --git a/server/src/processing/tests.rs b/server/src/processing/tests.rs index 8c24b233..82ef0081 100644 --- a/server/src/processing/tests.rs +++ b/server/src/processing/tests.rs @@ -21,7 +21,13 @@ // ------- // // ### Standard library -use std::{io, path::PathBuf, rc::Rc, str::FromStr}; +use std::{ + io, + path::{Path, PathBuf}, + rc::Rc, + str::FromStr, + sync::{Arc, Mutex}, +}; // ### Third-party use indoc::{formatdoc, indoc}; @@ -32,7 +38,7 @@ use pretty_assertions::assert_eq; // ### Local use super::{ CodeChatForWeb, CodeMirror, CodeMirrorDocBlock, SourceFileMetadata, StringDiff, - TranslationResults, find_path_to_toc, + find_path_to_toc, }; use crate::{ lexer::{ @@ -43,9 +49,10 @@ use crate::{ CodeDocBlockVecToSourceError, CodeMirrorDiffable, CodeMirrorDocBlockDelete, CodeMirrorDocBlockTransaction, CodeMirrorDocBlockUpdate, CodechatForWebToSourceError, HtmlToMarkdownWrapped, SourceToCodeChatForWebError, UNICODE_CURSOR_MARKER, byte_index_of, - code_doc_block_vec_to_source, code_mirror_to_code_doc_blocks, codechat_for_web_to_source, - dehydrating_walk_node, diff_code_mirror_doc_blocks, diff_str, doc_block_html_to_markdown, - html_to_tree, hydrate_html, markdown_to_html, source_to_codechat_for_web, + cache::Cache, code_doc_block_vec_to_source, code_mirror_to_code_doc_blocks, + codechat_for_web_to_source, dehydrating_walk_node, diff_code_mirror_doc_blocks, diff_str, + doc_block_html_to_markdown, html_to_dom, hydrate_html, markdown_to_html, + source_to_codechat_for_web, }, }; use test_utils::{cast, prep_test_dir, test_utils::stringit}; @@ -473,8 +480,8 @@ fn test_source_to_codechat_for_web_1() { // A file with an unknown extension and no lexer, which is classified as a // text file. assert_eq!( - source_to_codechat_for_web("", &".xxx".to_string(), 0.0, false, false), - Ok(TranslationResults::Unknown) + source_to_codechat_for_web("", Path::new("foo.xxx"), 0.0, false, None), + Err(SourceToCodeChatForWebError::NoLexer) ); // A file with an invalid lexer specification. Obscure this, so that this @@ -483,10 +490,10 @@ fn test_source_to_codechat_for_web_1() { assert_eq!( source_to_codechat_for_web( &format!("{lexer_spec}unknown"), - &".xxx".to_string(), + Path::new("foo.xxx"), 0.0, false, - false, + None ), Err(SourceToCodeChatForWebError::UnknownLexer( "unknown".to_string() @@ -495,74 +502,62 @@ fn test_source_to_codechat_for_web_1() { // A CodeChat Editor document via filename. assert_eq!( - source_to_codechat_for_web("", &"md".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( - MARKDOWN_MODE, - "", - vec![] - ))) + source_to_codechat_for_web("", Path::new("foo.md"), 0.0, false, None), + Ok(build_codechat_for_web(MARKDOWN_MODE, "", vec![])) ); // A CodeChat Editor document via lexer specification. assert_eq!( source_to_codechat_for_web( &format!("{lexer_spec}markdown"), - &"xxx".to_string(), + Path::new("foo.xxx"), 0.0, false, - false, + None ), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + Ok(build_codechat_for_web( MARKDOWN_MODE, &format!("

{lexer_spec}markdown"), vec![] - ))) + )) ); // An empty source file. assert_eq!( - source_to_codechat_for_web("", &"js".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( - "javascript", - "", - vec![] - ))) + source_to_codechat_for_web("", Path::new("foo.js"), 0.0, false, None), + Ok(build_codechat_for_web("javascript", "", vec![])) ); // A zero doc block source file. assert_eq!( - source_to_codechat_for_web("let a = 1;", &"js".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( - "javascript", - "let a = 1;", - vec![] - ))) + source_to_codechat_for_web("let a = 1;", Path::new("foo.js"), 0.0, false, None), + Ok(build_codechat_for_web("javascript", "let a = 1;", vec![])) ); // One doc block source files. assert_eq!( - source_to_codechat_for_web("// Test", &"js".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + source_to_codechat_for_web("// Test", Path::new("foo.js"), 0.0, false, None), + Ok(build_codechat_for_web( "javascript", "\n", vec![build_codemirror_doc_block(0, 1, "", "//", "

Test")] - ))) + )) ); assert_eq!( - source_to_codechat_for_web("let a = 1;\n// Test", &"js".to_string(), 0.0, false, false,), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + source_to_codechat_for_web("let a = 1;\n// Test", Path::new("foo.js"), 0.0, false, None), + Ok(build_codechat_for_web( "javascript", "let a = 1;\n\n", vec![build_codemirror_doc_block(11, 12, "", "//", "

Test")] - ))) + )) ); assert_eq!( - source_to_codechat_for_web("// Test\nlet a = 1;", &"js".to_string(), 0.0, false, false,), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + source_to_codechat_for_web("// Test\nlet a = 1;", Path::new("foo.js"), 0.0, false, None), + Ok(build_codechat_for_web( "javascript", "\nlet a = 1;", vec![build_codemirror_doc_block(0, 1, "", "//", "

Test")] - ))) + )) ); // A two doc block source file. This also tests references in one block to a @@ -570,19 +565,19 @@ fn test_source_to_codechat_for_web_1() { assert_eq!( source_to_codechat_for_web( "// [Link][1]\nlet a = 1;\n/* [1]: http://b.org */", - &"js".to_string(), + Path::new("foo.js"), 0.0, false, - false, + None ), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + Ok(build_codechat_for_web( "javascript", "\nlet a = 1;\n\n", vec![ build_codemirror_doc_block(0, 1, "", "//", "

Link"), build_codemirror_doc_block(12, 13, "", "/*", "") ] - ))) + )) ); // Trigger special cases: @@ -591,8 +586,8 @@ fn test_source_to_codechat_for_web_1() { // * A doc block in the middle of the file // * A doc block with no trailing newline at the end of the file. assert_eq!( - source_to_codechat_for_web("//\n\n//\n\n//", &"cpp".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + source_to_codechat_for_web("//\n\n//\n\n//", Path::new("foo.cpp"), 0.0, false, None), + Ok(build_codechat_for_web( "cpp", "\n\n\n\n", vec![ @@ -600,11 +595,11 @@ fn test_source_to_codechat_for_web_1() { build_codemirror_doc_block(2, 3, "", "//", ""), build_codemirror_doc_block(4, 5, "", "//", "") ] - ))) + )) ); assert_eq!( - source_to_codechat_for_web("// ~~~\n\n//\n\n//", &"cpp".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + source_to_codechat_for_web("// ~~~\n\n//\n\n//", Path::new("foo.cpp"), 0.0, false, None), + Ok(build_codechat_for_web( "cpp", "\n\n\n\n", vec![ @@ -612,7 +607,7 @@ fn test_source_to_codechat_for_web_1() { build_codemirror_doc_block(2, 3, "", "//", ""), build_codemirror_doc_block(4, 5, "", "//", "") ] - ))) + )) ); // Test Unicode characters and multi-byte Unicode characters in code. @@ -626,32 +621,32 @@ fn test_source_to_codechat_for_web_1() { // These are taken from the // [MDN UTF-16 docs](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String#utf-16_characters_unicode_code_points_and_grapheme_clusters). assert_eq!( - source_to_codechat_for_web("; // σ😄👉🏿👨‍👦🇺🇳\n//", &"cpp".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + source_to_codechat_for_web("; // σ😄👉🏿👨‍👦🇺🇳\n//", Path::new("foo.cpp"), 0.0, false, None), + Ok(build_codechat_for_web( "cpp", "; // σ😄👉🏿👨‍👦🇺🇳\n", vec![build_codemirror_doc_block(22, 23, "", "//", ""),] - ))) + )) ); // Test Unicode characters and multi-byte Unicode characters in strings. assert_eq!( - source_to_codechat_for_web("\"σ😄👉🏿👨‍👦🇺🇳\";\n//", &"cpp".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + source_to_codechat_for_web("\"σ😄👉🏿👨‍👦🇺🇳\";\n//", Path::new("foo.cpp"), 0.0, false, None), + Ok(build_codechat_for_web( "cpp", "\"σ😄👉🏿👨‍👦🇺🇳\";\n", vec![build_codemirror_doc_block(20, 21, "", "//", ""),] - ))) + )) ); // Test Unicode characters and multi-byte Unicode characters in comments. assert_eq!( - source_to_codechat_for_web("// σ😄👉🏿👨‍👦🇺🇳\n;", &"cpp".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + source_to_codechat_for_web("// σ😄👉🏿👨‍👦🇺🇳\n;", Path::new("foo.cpp"), 0.0, false, None), + Ok(build_codechat_for_web( "cpp", "\n;", vec![build_codemirror_doc_block(0, 1, "", "//", "

σ😄👉🏿👨‍👦🇺🇳"),] - ))) + )) ); // Test a fenced code block that's unterminated. See @@ -659,12 +654,12 @@ fn test_source_to_codechat_for_web_1() { assert_eq!( source_to_codechat_for_web( "/* ``` foo\n*/\n// Test", - &"cpp".to_string(), + Path::new("foo.cpp"), 0.0, false, - false + None ), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + Ok(build_codechat_for_web( "cpp", "\n\n\n", vec![ @@ -677,18 +672,18 @@ fn test_source_to_codechat_for_web_1() { ), build_codemirror_doc_block(2, 3, "", "//", "

Test"), ] - ))) + )) ); // Test the other code fence character (the tilde). assert_eq!( source_to_codechat_for_web( "/* ~~~~~~~ foo\n*/\n// Test", - &"cpp".to_string(), + Path::new("foo.cpp"), 0.0, false, - false + None ), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + Ok(build_codechat_for_web( "cpp", "\n\n\n", vec![ @@ -701,38 +696,38 @@ fn test_source_to_codechat_for_web_1() { ), build_codemirror_doc_block(2, 3, "", "//", "

Test"), ] - ))) + )) ); // Test multiple unterminated fenced code blocks. assert_eq!( - source_to_codechat_for_web("// ```\n // ~~~", &"cpp".to_string(), 0.0, false, false), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + source_to_codechat_for_web("// ```\n // ~~~", Path::new("foo.cpp"), 0.0, false, None), + Ok(build_codechat_for_web( "cpp", "\n\n", vec![ build_codemirror_doc_block(0, 1, "", "//", "

\n
"), build_codemirror_doc_block(1, 2, " ", "//", "
"), ] - ))) + )) ); // Test an unterminated HTML block. assert_eq!( source_to_codechat_for_web( "// \n // Test", - &"cpp".to_string(), + Path::new("foo.cpp"), 0.0, false, - false + None ), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + Ok(build_codechat_for_web( "cpp", "\n\n", vec![ build_codemirror_doc_block(0, 1, "", "//", " "), build_codemirror_doc_block(1, 2, " ", "//", "

Test"), ] - ))) + )) ); // Test an unterminated `

` block. Ensure that markdown after this is
@@ -741,19 +736,19 @@ fn test_source_to_codechat_for_web_1() {
     assert_eq!(
         source_to_codechat_for_web(
             "// 
\n // *Test*",
-            &"cpp".to_string(),
+            Path::new("foo.cpp"),
             0.0,
             false,
-            false
+            None
         ),
-        Ok(TranslationResults::CodeChat(build_codechat_for_web(
+        Ok(build_codechat_for_web(
             "cpp",
             "\n\n",
             vec![
                 build_codemirror_doc_block(0, 1, "", "//", "
"),
                 build_codemirror_doc_block(1, 2, " ", "//", "

Test"), ] - ))) + )) ); // Test that minify functions correctly across multiple paragraphs separated @@ -769,19 +764,19 @@ fn test_source_to_codechat_for_web_1() { // Four " ), - &"cpp".to_string(), + Path::new("foo.cpp"), 0.0, false, - false + None ), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + Ok(build_codechat_for_web( "cpp", "\n\n\nthree();\n\n", vec![ build_codemirror_doc_block(0, 3, "", "//", "

One

Two"), build_codemirror_doc_block(12, 13, "", "//", "

Four"), ] - ))) + )) ); // Test that minify functions correctly across multiple paragraphs separated @@ -1319,14 +1314,19 @@ fn test_hydrate_html_1() { // These tests check the translation from Markdown to "wet" HTML (what the // user provides) instead of dry -> wet HTML. assert_eq!( - hydrate_html(&markdown_to_html(indoc!( - "```mermaid + hydrate_html( + &markdown_to_html(indoc!( + "```mermaid flowchart LR start --> stop ``` " - ))) - .unwrap(), + )), + Path::new("foo.md"), + Arc::new(Mutex::new(Cache::new())) + ) + .unwrap() + .0, indoc!( " flowchart LR @@ -1337,15 +1337,20 @@ fn test_hydrate_html_1() { ); assert_eq!( - hydrate_html(&markdown_to_html(indoc!( - "```graphviz + hydrate_html( + &markdown_to_html(indoc!( + "```graphviz digraph { start -> stop } ``` " - ))) - .unwrap(), + )), + Path::new("foo.md"), + Arc::new(Mutex::new(Cache::new())) + ) + .unwrap() + .0, indoc!( " digraph { @@ -1358,8 +1363,9 @@ fn test_hydrate_html_1() { // Ensure math doesn't need escaping. assert_eq!( - hydrate_html(&markdown_to_html(indoc!( - " + hydrate_html( + &markdown_to_html(indoc!( + " ${a}_1, b_{2}$ $a*1, b*2$ $[a](b)$ @@ -1368,8 +1374,12 @@ fn test_hydrate_html_1() { $${a}_1, b_{2}, a*1, b*2, [a](b), 3 b, a \\; b$$ " - ))) - .unwrap(), + )), + Path::new("foo.md"), + Arc::new(Mutex::new(Cache::new())) + ) + .unwrap() + .0, indoc!( r#"

\({a}_1, b_{2}\) @@ -1383,7 +1393,13 @@ fn test_hydrate_html_1() { ); assert_eq!( - hydrate_html(&markdown_to_html("1. foo\u{a0}\n2. bar \n3. baz ")).unwrap(), + hydrate_html( + &markdown_to_html("1. foo\u{a0}\n2. bar \n3. baz "), + Path::new("foo.md"), + Arc::new(Mutex::new(Cache::new())) + ) + .unwrap() + .0, indoc!( "

    @@ -1397,7 +1413,7 @@ fn test_hydrate_html_1() { } fn dehydrate_html(html: &str) -> io::Result> { - let tree = html_to_tree(html, None)?; + let tree = html_to_dom(html, None)?; dehydrating_walk_node(&tree); Ok(tree) } diff --git a/server/src/translation.rs b/server/src/translation.rs index 3775f097..45ce4058 100644 --- a/server/src/translation.rs +++ b/server/src/translation.rs @@ -212,11 +212,11 @@ use std::{ fmt::Debug, path::{Path, PathBuf}, rc::Rc, - sync::LazyLock, + sync::{Arc, LazyLock, Mutex}, }; -use htmd::Node; // ### Third-party +use htmd::Node; use log::{debug, error, warn}; use rand::random; use regex::Regex; @@ -233,9 +233,9 @@ use crate::{ processing::{ CodeChatForWeb, CodeMirror, CodeMirrorDiff, CodeMirrorDiffable, CodeMirrorDocBlock, CodeMirrorDocBlockVec, SourceFileMetadata, TranslationResultsString, UNICODE_CURSOR_MARKER, - byte_index_of, codechat_for_web_to_source, diff_code_mirror_doc_blocks, diff_str, - doc_block_html_to_markdown, minify, remove_tinymce_data, source_to_codechat_for_web_string, - transform_html, + byte_index_of, cache::Cache, codechat_for_web_to_source, diff_code_mirror_doc_blocks, + diff_str, doc_block_html_to_markdown, minify, remove_tinymce_data, + source_to_codechat_for_web_string, transform_html, }, queue_send, queue_send_func, webserver::{ @@ -251,7 +251,7 @@ use crate::{ // ------- // // The max length of a message to show in the console. -const MAX_MESSAGE_LENGTH: usize = 500; +const MAX_MESSAGE_LENGTH: usize = 50000; /// A regex to determine the type of the first EOL. See 'PROCESSINGS\`. pub static EOL_FINDER: LazyLock = LazyLock::new(|| Regex::new("[^\r\n]*(\r?\n)").unwrap()); @@ -402,6 +402,7 @@ struct TranslationTask { to_client_tx: Sender, from_client_rx: Receiver, from_http_rx: Receiver, + cache: Arc>>>>, // These parameters are internal state. /// The file currently loaded in the Client. @@ -487,6 +488,7 @@ pub async fn translation_task( to_client_tx, from_client_rx, from_http_rx, + cache: app_state.cache.clone(), current_file: PathBuf::new(), load_file_requests: HashMap::new(), id: INITIAL_MESSAGE_ID + MESSAGE_ID_INCREMENT, @@ -928,6 +930,7 @@ impl TranslationTask { ( file_to_response( &http_request, + self.cache.clone(), new_version, &self.current_file, Some(&file_contents), @@ -955,6 +958,7 @@ impl TranslationTask { ( file_to_response( &http_request, + self.cache.clone(), self.version, &self.current_file, option_file_contents.as_ref(), @@ -1037,6 +1041,7 @@ impl TranslationTask { &self.current_file, contents.version, false, + self.cache.clone(), ) { Err(err) => { Err(ResultErrTypes::CannotTranslateSource(err.to_string())) @@ -1247,6 +1252,7 @@ impl TranslationTask { &clean_file_path, cfw.version, false, + self.cache.clone(), ) && let TranslationResultsString::CodeChat(ccfw) = ccfws.0 && let CodeMirrorDiffable::Plain(code_mirror_translated) = ccfw.source diff --git a/server/src/webserver.rs b/server/src/webserver.rs index a207e824..24c2f5d5 100644 --- a/server/src/webserver.rs +++ b/server/src/webserver.rs @@ -86,8 +86,8 @@ use url::Url; // ### Local //use crate::capture::EventCapture; use crate::processing::{ - CodeChatForWeb, SourceToCodeChatForWebError, TranslationResultsString, find_path_to_toc, - source_to_codechat_for_web_string, + CodeChatForWeb, SourceToCodeChatForWebError, TranslationResultsString, cache::Cache, + find_path_to_toc, source_to_codechat_for_web_string, }; use crate::capture::{ @@ -415,6 +415,8 @@ pub struct AppState { credentials: Option, // Added to support capture - JDS - 11/2025 pub capture: Option, + /// A hash of project path to Cache. + pub cache: Arc>>>>, } pub type WebAppState = web::Data; @@ -858,6 +860,8 @@ pub async fn try_read_as_text(file: &mut File) -> Option { pub async fn file_to_response( // The HTTP request presented to the processing task. http_request: &ProcessingTaskHttpRequest, + // The map of project caches. + cache: Arc>>>>, // The version of this file. version: f64, // Path to the file currently being edited. This path should be cleaned by @@ -923,6 +927,7 @@ pub async fn file_to_response( file_path, version, is_toc, + cache, ) } else { // If this isn't the current file, then don't parse it. @@ -1608,6 +1613,7 @@ fn make_app_data_with_capture_spool( connection_id: Mutex::new(HashSet::new()), credentials, capture, + cache: Arc::new(Mutex::new(HashMap::new())), }) } From 8af4c855940da20bbda21260654859231413e669 Mon Sep 17 00:00:00 2001 From: "Bryan A. Jones" Date: Sat, 11 Jul 2026 05:41:18 +0500 Subject: [PATCH 02/22] wip: cache redesign. --- server/src/processing.rs | 245 +++++++++------- server/src/processing/cache.rs | 495 ++++++++++----------------------- 2 files changed, 291 insertions(+), 449 deletions(-) diff --git a/server/src/processing.rs b/server/src/processing.rs index 9e1c6bff..f42bf612 100644 --- a/server/src/processing.rs +++ b/server/src/processing.rs @@ -80,8 +80,7 @@ use crate::{ lexer::{ CodeDocBlock, DocBlock, LEXERS, LanguageLexerCompiled, source_lexer, supported_languages::MARKDOWN_MODE, - }, - processing::cache::Target, + }, processing::cache::{Fragment, Target}, }; use cache::Cache; @@ -322,8 +321,6 @@ const DOC_BLOCK_SEPARATOR_REMOVE_FENCE: &str = r" // which it replaces here. const DOC_BLOCK_SEPARATOR_MENDED_FENCE: &str = "
\n$1\n"; -// - // The column at which to word wrap doc blocks. const WORD_WRAP_COLUMN: usize = 80; // The minimum width for doc block word wrap, since large indents may leave @@ -981,7 +978,6 @@ pub fn source_to_codechat_for_web( let mut doc_block_contents_iter: regex::Split<'_, '_> = DOC_BLOCK_SEPARATOR_SPLIT_REGEX.split(&html); // 5. TODO Cache updates: process `tags`. - // // Translate each `CodeDocBlock` to its `CodeMirror` equivalent. let mut len = len_utf16(&code_mirror.doc); @@ -1274,69 +1270,116 @@ fn hydrate_html( ) -> io::Result<(String, Vec>)> { let dom = html_to_dom(html, None)?; let file_entry = cache.lock().unwrap().get_or_create_file(file); - // Move the `target` vec into a `HashMap`; the vec will be re-created by - // moving individual entries back as they're found in the HTML. TODO: - // invalidate all current weak links to all `Targets` in this vec. Must also - // update all weak links to the underlying `File` when moving a `Target`. - let old_target_vec = mem::take(&mut file_entry.lock().unwrap().target); - let old_targets: HashMap>> = old_target_vec - .into_iter() - .map(|target| (target.clone().lock().unwrap().id.clone(), target)) - .collect(); + // ### Prepare `Targets`/`Xrefs` to process this file + // + // Requirement: determine if any files containing cross-references need to + // be rebuilt due to changes in the `Target`s in this file. To accomplish + // this, determine if any `Targets` in this file were added, deleted, or + // modified, then notify all cross-references of these targets that their + // containing file is outdated. Note that "modified" refers only to the + // `Target` state that cross-references depend on. + // + // Because `file_entry.xrefs` have no id to match against, remove all + // `xrefs` on this page from their `Target` dependencies; this will be + // re-generated during the DOM traversal. TODO: remove each `xref.id` from + // `cache.targets[id]` for `xref` in `file_entry.xrefs`. + // + // TODO: implementation. Move the current `file_entry.targets` to a local + // variable `targets`. Match each `Target` encountered in the DOM against + // the `targets`, moving identical entries to `file_entry.targets`. After + // processing the DOM, remaining entries in `targets` are deleted, while new + // entries and modifications were discovered during the DOM traversal. + let old_targets = mem::take(&mut file_entry.lock().unwrap().targets); + // ### Prepare `Fragments`/`GatherElements` to process this file + // + // Requirement: determine if any files containing `Fragment`s need to be + // rebuilt due to changes in gather elements in this file. Similarly, + // determine if any files containing gather elements need to be rebuilt due + // to change in `Fragment`s in this file. Because `Fragment`s have an id, + // use the same approach as `Target`s. Gather elements don't have an id; to + // identify differences, make a copy of all `Fragment.gathers` referenced by + // gather elements in the current file, then remove all gathers in the + // current file from `Fragments.gathers`. Walk the DOM; changes to any + // `Fragment.gathers` which wasn't copied is an addition. Comparison with + // the copied `Fragment.gathers` with current `Fragment.gathers` shows + // deletions. Since Fragments only depend on gather ids and no other state, + // there's no modifications to track (unlike cross-references, where changes + // to Target contents affect its cross-reference state). + // + // Store the previous state of all `Fragment.gathers` referenced by gather + // on this page in a map where key = `Fragment.id`, value = + // `Fragment.gathers`. + let old_fragments_gathers = HashMap::new(); + // For each `id` in `gather.ids` in `file.gathers` on this page: + // + // 1. Insert a copy of `cache.fragments[id].gathers` to a + // `old_fragments_gathers`, unless it's already inserted. + // 2. Remove `gather` from `cache.fragments[id].gathers`. + // + // Finally, empty `file_entry.gathers`. This will be re-created by page + // processing. After processing this page, rebuild any files containing + // fragments whose `gathers` list changed. + // + // Move the `file_entry.fragments` vec into a `HashMap`; follow the same + // logic as `old_targets_vec` above. + let old_fragments = mem::take(&mut file_entry.lock().unwrap().fragments); // This is storage for the state needed for walking the DOM. let mut walk_context = WalkContext { cache, file_entry, old_targets, + old_fragments, doc_block_index: 0, - links: Vec::new(), - backlinks: Vec::new(), - tags: Vec::new(), - code_doc_block_dependencies: vec![], + xrefs: Vec::new(), + fragments: Vec::new(), + gathers: Vec::new(), }; walk_context = hydrating_walk_node(dom.clone(), walk_context); - // TODO: + // The overall processing order after walking the DOM: // - // 1. Remove all `old_targets` that weren't moved back to `targets`. + // 1. All `Target` cross-reference (`Xref`) state is updated. Therefore, + // update the DOM content for all `Xrefs` in this file (whic is stored in + // `walk_context.xrefs`). If the `Xref.id` refers to a `Fragment` instead + // of a `Target`, the DOM will contain an error message. + // 2. All `Fragment`s in this file have updated ids. For + // each `walk_context.gathers`, create a `GatherElement`. Add it to + // `file_entry.gathers` and `file_entry.targets[id].gathers`. + // 3. All `GatherElements` `Fragment` state is now updated. Therefore, + // update the DOM content for all `Fragment`s in this file (stored in + // `walk_context.fragments`). + // 4. On exit from this function, doc block contents will be finalized. + // After that, update each `Fragment.content`. + // 5. `Fragment`s on this page now have updated state needed by + // `GatherElement`s. Update each `GatherElement`. // - // 2. Process `links`: + // TODO: // - // 1. If the target of this link is in the cache, first check to see if - // it's been processed. If not, mark the current file to be added to - // the end of `pending_files` list. Remove any previous - // `references`/`dependencies`, then re-add them based on the link's - // `Target.backlink_type`: + // 1. For each `target` in `old_targets`, mark all `target.xrefs` as + // outdated. + // 2. For each `fragment` in `old_fragments`, mark each `gather.file` in + // `fragment.gathers` as outdated. + // 3. For each `old_fragment_gathers`, compare old and new gathers. If they + // differ, mark the file containing the `Fragment` as outdated, unless + // the file is `file_entry`. + // 4. For each `gather` in `file_entry.gathers`, look up/create the + // corresponding `Fragment`. Add each `gather` to this + // `fragment.gathers`. + // 5. For each `xref` in `walk_context.xrefs`, get/create (add to + // `cache.missingTargetsAndFragments`) the `Target` of this xref. Add + // `file_entry` to `Target.xrefs`. If the `Target` of this xref is not + // clean, mark `file_entry` as dirty. Update the DOM by inserting a link + // based on available `Target` info. Note that `xref` processing was + // deferred until all `Target`s in this file were processed. + // 6. For each `fragment`, generate updated DOM data (a link per gather + // element). // - // 1. `None`: if this is an auto-titled link, get its title from the - // `Target` it refers to (this includes the case where the target - // doesn't exist) and add this file to the target's list of - // `references` if it's not already there; also add it to the - // `code_doc_block_dependencies` list; problem: how would we know - // the doc block index? Otherwise, add it to the target's list of - // `dependencies` (again, if it's not already there). Remove it - // from the other list if it's there. - // 2. `Plain`/`wrapped`: give this link an ID if it doesn't have one - // (meaning add it as a new `Target`). As above, add/verify this - // link is in the `references`/`dependencies` and remove the - // opposite link. If this is added, mark the link target's `File` - // as dirty. - // 3. `Gather`: same as above. Also, record the start/end code/doc - // blocks and put this in the code/doc block `tags` list. - // 2. Otherwise, add the file containing this target to set of files to - // process (`Cache::pending_files`) and mark the current file to be - // added to this set after all dirty dependencies are added. - // 3. Process `backlinks`: generate the appropriate HTML. These are - // processed after `links`, to ensure these are updated / assigned an ID - // in case a backlink references a link in this file. Corner case: if a - // gather element refers to tags in the current file, and the code+doc - // block contents of these tags is outdated, then this file need to be - // reprocessed, since the code+doc block contents won't be updated until - // after this function returns. - - Ok((dom_to_html(dom)?, walk_context.tags)) + // TODO: on return, update fragment contents, then update gathers DOM data + // (a list containing a link to each fragment followed by its contents), + + Ok((dom_to_html(dom)?, walk_context.gathers)) } -/// Get the value of an attribute on an element node. +// Get the value of an attribute on an element node. fn get_attr_value(node: &Rc, attr_name: &str) -> Option { if let NodeData::Element { attrs, .. } = &node.data { attrs @@ -1370,16 +1413,14 @@ struct WalkContext { /// The previous `file_entry.targets` that haven't yet been claimed while /// processing this file. The key is the `Target`'s ID. old_targets: HashMap>>, - /// All hyperlinks found in this file. - links: Vec>, - /// Backlinks and gather elements. - backlinks: Vec>, - /// Tags (links to gather elements). - tags: Vec>, - /// For each doc block in the vec of `CodeDocBlock`s, this contains the - /// `Target` of autotitled  links. These are used to determine a tag's - /// indirect dependencies if this doc block is included in a tag. - code_doc_block_dependencies: Vec>>>, + /// Same as above: currently unclaimed `file_entry.fragments`. + old_fragments: HashMap>>, + /// DOM for all xrefs found in this file. + xrefs: Vec>, + /// DOM for all `Fragment`s. + fragments: Vec>, + /// DOM for all `GatherElement`s + gathers: Vec>, /// The current doc block index, based on parsing the HTML for /// `codechateditor-separator` elements, which contain this value. doc_block_index: usize, @@ -1456,47 +1497,35 @@ fn hydrating_walk_node(node: Rc, mut walk_context: WalkContext) -> WalkCon // // When walking, look for: // - // 1. A link, or any element that's a back link or gather element. - // Note that links directly to a file are ignored, since files - // are not targets. Add these to the `links`/`backlinks` list. - // 2. A target (any item with an id which refers to a file in the - // project tree). - // 1. Process the ID: - // 1. ID exists in `old_targets` - transfer ownership by - // appending this to `file.targets`, removing it from the - // `old_targets`; update the stale `Weak` pointer to the - // parent `File`. Assert that this ID is unique. - // 2. ID doesn't exist in `old_targets` and is unique: add - // this `Target` to `Cache::id`. - // 3. ID doesn't exist in `old_targets` and is a duplicate: - // resolve duplicate IDs: - // 1. Is the duplicate ID in the same file? In this case, + // 1. A cross reference. Add it to `walk_context.xrefs`. + // 2. A `Target` (any item with an id that's not a `Fragment`). + // 1. Process the id: + // 1. id exists in `old_targets` - transfer ownership by + // appending this to `walk_content.file_entry.targets`, + // removing it from the `old_targets`. Assert that this id + // is unique. + // 2. id doesn't exist in `old_targets` and is unique: add + // this `Target` to `walk_content.file_entry.targets`. + // 3. id doesn't exist in `old_targets` and is in the set of + // `cache.missingTargetsOrFragments`: transfer ownership by + // appending this to `walk_content.file_entry.targets`. + // Mark all `target.xrefs` as `Outdated`. + // 4. id is a duplicate (exists + // in `cache.targetsOrFragments`): resolve duplicate ids: + // 1. Is the duplicate id in the same file? In this case, // we have no way to determine which was the original - // ID. Rename the ID being processed; stop here. - // 2. Have all new files been processed? If not, add this - // file to the list of dirty files which will be - // re-processed after new file processing is done. Stop - // here. - // 3. Look at the timestamp of this file and of the other - // file which contains the duplicate ID. If this file is - // newer, rename this ID. Otherwise, update the ID and - // mark the other file as dirty. + // id. Rename the id being processed; stop here. + // 2. Look at the timestamp of this file and of the other + // file which contains the duplicate id. If this file is + // newer, rename this id. Otherwise, update the id and + // mark the other file as `Outdated`. // 2. Check the `contents`: if the `contents` changed, add all - // this target's `dependencies` to the dirty list and update - // the search text for this target. - // 3. Check the `backlink_type`. If this changed: - // 1. To `Gather` from any other type: add all the target's - // `references` and `dependencies` to the dirty list, since - // some `references` may need IDs and everything needs - // code+doc blocks. - // 2. From `Gather` to any other type: Do nothing. The - // code+doc blocks will be removed lazily, since they have - // no impact on the resulting output. - // 3. From `None` to `Wrapped` or `Plain`: add `references` - // with no ID to the dirty list. - // 4. From `Wrapped` to `None`: nothing to do. - // 4. Walk the list of `references`/`dependencies`, removing any - // dropped references. + // this target's `xrefs` to the `Outdated` list and update the + // search text for this target. + // 3. A `GatherElement`: add it to `walk_context.gathers`. + // 4. A `Fragment`: add it to `walk_context.fragments`, then process + // similarly to a `Target`. Note that `content` can't be + // determined yet. } walk_context = hydrating_walk_node(child.clone(), walk_context); @@ -1538,7 +1567,7 @@ fn replace_math_node(child: &Rc, is_hydrate: bool) -> Option> { // When hydrating, there should only be a `class` attribute. if child_attrs_len == 1 { match attr_value_str { - "math math-inline" => Some(("\\(", "\\)", "math math-inline mceNonEditable")), + "math math-inline" => Some(("\(", "\)", "math math-inline mceNonEditable")), "math math-display" => Some(("$$", "$$", "math math-display mceNonEditable")), _ => None, } @@ -1546,9 +1575,9 @@ fn replace_math_node(child: &Rc, is_hydrate: bool) -> Option> { None } } else { - // When dehydrating, there should also be a `contenteditable=false` - // attribute. It may appear in either order relative to `class`, so - // look it up by name. + + + if child_attrs_len == 2 && let Some(contenteditable_attr) = child_attrs .iter() @@ -1556,7 +1585,7 @@ fn replace_math_node(child: &Rc, is_hydrate: bool) -> Option> { && contenteditable_attr.value == *"false" { match attr_value_str { - "math math-inline mceNonEditable" => Some(("\\(", "\\)", "math math-inline")), + "math math-inline mceNonEditable" => Some(("\(", "\)", "math math-inline")), "math math-display mceNonEditable" => Some(("$$", "$$", "math math-display")), _ => None, } diff --git a/server/src/processing/cache.rs b/server/src/processing/cache.rs index a8e70aeb..83bf4d4a 100644 --- a/server/src/processing/cache.rs +++ b/server/src/processing/cache.rs @@ -2,7 +2,7 @@ #![allow(unused_variables)] #![allow(unused)] -// Copyright (C) 2025 Bryan A. Jones. +// Copyright (C) 2026 Bryan A. Jones. // // This file is part of the CodeChat Editor. The CodeChat Editor is free // software: you can redistribute it and/or modify it under the terms of the GNU @@ -20,180 +20,60 @@ /// `cache.rs` - Keep a cache used to store all targets in a project /// ================================================================ /// -/// The cache stores the location (file name and ID), numbering (of headings in -/// the TOC and figures/equations/etc. on a page), and contents (inner HTML or -/// code/doc blocks for tags) of a target. Targets are HTML elements with an ID, -/// making them anchors (such as headings, figure titles, display equations, -/// tags, hyperlinks, etc.), or files. +/// The cache stores the location (file name and ID) and contents (inner HTML or +/// code/doc blocks for fragments) of a target. Targets are HTML elements with +/// an ID. /// -/// The goal of the cache is to support auto-titled links, backlinks, and gather -/// elements, and to ensure that all IDs are unique within a project. This means -/// that links persist across moving or renaming files, since the IDs will be -/// found in the cache. +/// The goal of the cache is to support cross-references and gather elements, +/// and to ensure that all IDs are unique within a project. This means that +/// cross-references and gather elements persist across moving or renaming +/// files, since the IDs will be found in the cache. /// -/// Auto-titled links -/// ----------------- +/// Cross references +/// ---------------- /// -/// A hyperlink with empty -/// [link text](https://spec.commonmark.org/0.31.2/#link-text) is auto-titled -- -/// the contents of the target it references provide the link text. For example, -/// after processing, the link in the following Markdown +/// An `...Generated contents...` is a cross reference. +/// The `id` specifies the destination; the cache then updates the `generated +/// contents` based on the location and contents of the target of the provided +/// ID. This element does not allow an `id` attribute. /// -/// ```Markdown -///

Bar

-/// [](#foo) -/// ``` -/// -/// becomes `[Bar](#foo)`. This works even when the target is located in a -/// different file. Auto-titled links don't support indirection: link A whose -/// link text comes from link B whose link text comes from target C doesn't -/// work; link A will end up with an empty title. -/// -/// Tags -/// ---- -/// -/// A gather element such as `

Bazzy -/// things

` becomes a list of the contents of tags which reference it after -/// processing by the cache. A tag is simply a link to a gather element, such as -/// `[](#baz)`, which becomes `Bazzy things` after -/// auto-titling and auto-assignment of an ID. The tag's content by default -/// includes the contents of the current doc block and the contents of the next -/// code/doc block. Tags can also include an end query parameter to enclose a -/// wider range of code/doc blocks; for example, `[](#baz?end=3)` includes the -/// next 3 code/doc blocks. -/// -/// Tag contents may not include a gather element. They do support indirection: -/// gather element A includes contents from tag B, which contains an auto-titled -/// link to target C. Changes to target C makes B and A dirty. +/// Gather elements +/// --------------- /// -/// Example output of the gather tag `

Bazzy +/// A gather element such as `

Bazzy things

` +/// becomes a list of the contents of fragments it refers to after processing by +/// the cache. A fragment's content by default includes the contents of the +/// current doc block and the contents of the following code/doc block; +/// fragments are not allowed in Markdown documents. Fragments may include the +/// `following` attribute to enclose a specific number of the following code/doc +/// blocks; for example, `` includes the +/// current doc block along with the next 3 code/doc blocks; `following` must be +/// a whole number. The fragment's contents will be replaced with links to any +/// referring doc blocks. TODO: also allow a `` to indicate +/// the last code/doc block of a fragment. +/// +/// Gather elements may include an `id`. Fragment contents may not include a +/// gather element. They do support indirection: gather element A includes +/// contents from fragment B, which contains an cross reference to target C. +/// Changes to target C makes B and A dirty. +/// +/// Example output of the gather tag `

Bazzy /// things

`: /// /// ```html -///

Bazzy things

-/// +///

Bazzy things

+///
/// (first item content) /// ... -/// +/// /// (last item content) /// ``` /// -/// Backlinks -/// --------- -/// -/// Given an ID, a backlink produces a list of links which reference it. This -/// provides a way to create an index, or show what references -/// headings/footnotes/endnotes, etc. Backlinks are like gather elements, but -/// instead of capturing tag contents, they capture target contents. In -/// addition, backlinks don't support indirect dependencies: backlink A, which -/// link B references, doesn't depend on link B's auto-titled text from target -/// C. -/// -/// The default backlink style produces a disclosure widget using a link icon -/// which reveals an unordered list of links when clicked; the plain style -/// simply presents a list of links. Support for ordering backlinks may be added -/// later; these will not support nesting (just as tags don't support nesting). -/// -/// Syntax: `element -/// text`, where `el` is an HTML element (such as `h1-6` or `a`). After -/// processing, this becomes: -/// -/// ```html -/// element text -///
-/// 🔗 -/// -///
-/// -/// ``` -/// /// Search /// ------ /// /// The cache supports searching the contents of all targets. /// -/// Table of contents -/// ----------------- -/// -/// I'd like to build a TOC. There are several approaches: -/// -/// * Sphinx, for example, allows a nested hierarchy of TOCs. The disadvantage is -/// that document structure is spread throughout the hierarchy. -/// * PreTeXt bases TOC on a global assignment of chapters, sections, etc. The -/// disadvantage is that moving sections may mean a lot of rewrite to move -/// everything around. -/// * mdbook keeps TOC and a local TOC separate. This means the global TOC can't -/// include the local TOC. -/// -/// My thought is to build something as close to PreTeXt as possible, since -/// that's my primary export target. Also, I want to create a global TOC, and not -/// be constrained by filesystem layout, since that is sometime dictated by the -/// toolchain. So, a list of files in which headings are part of the global TOC -/// makes the most sense to me. This does means that a page may have no h1 -/// headings, though. I need to look at PreTeXt to see how they handle this. -/// Here's a first pass mapping: -/// -/// * book/article = h1 -/// * part = XML only (since it doesn't have any actual contents other than -/// sections) -/// * chapter = h2 -/// * section = h3 -/// * subsection = h4 -/// * subsubsection = h5 -/// * paragraphs = h6 (for paragraphs earlier in the hierarchy, use the XML tag). -/// -/// KISS is very important here. How can I create something I can accomplish? I -/// really want to re-use as much of PreTeXt as I possibly can. I mainly want -/// Markdown because it remove a lot of the mess of paragraphs, em/strong, etc. -/// So, this tool ignores parts when building its simplified TOC. To build a TOC, -/// it simply takes a sequential list of files and scans them for these headings. -/// Later features could include the ability to specify files using wildcards, -/// incorporate ignores, etc. -/// -/// Other PreTeXt conversion notes: hyperlinks map to xrefs, mostly. An -/// auto-title link maps directly to an xref; to be more specific, use the xref -/// tag. A standard internal link maps to an xref with @text=custom to get a -/// fairly similar result; the link title gets munched by PreTeXt (oh, well). -/// Another approach: translate all hyperlinks to PreTeXt url, and use xref -/// directly only for xrefs. That seems like a good idea. Perhaps define a -/// translation table between Markdown and PreText. -/// -/// What if I get rid of backlinks and instead rely on PreTeXt's index for this -/// sort of functionality? That might be simpler. -/// -/// Gather elements are of course unique. I don't know how to translate these. I -/// don't think they map nicely to LP support in PreTeXt, since that's a bit -/// backward. Instead, generate PreTeXt from tags. -/// -/// Next crazy thought: focus only on LP for now, making anything else (e-books) -/// a secondary focus. Keep the current Markdown TOC for simplicity. Support only -/// xrefs (forward pointer to gather elements) and gather elements, which is -/// really the points. In this case, the cache with targets is fine. Existing -/// hyperlink support (to a file, but not to anchors) is fine. xrefs move to IDs. -/// xrefs to gather elements must have IDs, since we need a bidirectional link. -/// Knuth gives doc blocks a name. I give them an ID, which is a bit less -/// intrusive. Perhaps tags are different that xrefs? A tag is an ID, a gather -/// reference, and possibly a length. This sounds close enough to an xref to -/// reuse; probably add a new field (length, span, etc.) -/// -/// The overall goal: record design, specification, and implementation stuff that -/// can't be derived from the code. Document as much/as little of the source as -/// needed. I like the idea of eventual migration toward PreTeXt, but as a -/// secondary goal. I like their TOC approach. But for now, just xrefs + gather -/// tags is all I do. Do all xrefs get an auto-assigned ID? Don't really need it -/// for xrefs to non-gather tags. I prefer minimal. -/// -/// I wish my program had fewer bugs. Writing here is disappointing. But the -/// overall approach is good and makes it easy to add in ideas. -/// -/// Should I take the time to fix exiting bugs first? I'm out of time to get -/// everything done. Sigh. I'll focus first on this, which is core, and record -/// bogs for later. -/// /// Goals /// ----- /// @@ -219,15 +99,12 @@ /// 1. (Longer-term) modify the pulldown-cmark HTML writer to preserve line /// numbers. /// 2. Revise the TOC loader to use mdbook's code to process and update the TOC. -// Imports -// ------- +/// Imports +// --- // // ### Standard library use std::{ - collections::{HashMap, HashSet}, - path::{Path, PathBuf}, - rc::Rc, - sync::{Arc, Mutex, Weak}, + collections::{HashMap, HashSet}, fs::Metadata, path::{Path, PathBuf}, rc::Rc, sync::{Arc, Mutex, Weak}, }; // ### Third-party @@ -244,12 +121,13 @@ use markup5ever_rcdom::Node; pub struct Cache { /// Provide rapid access to a file by its absolute path; it must be within /// the project's root directory. This is the sole owner of these `File`s. - pub(super) path: HashMap>>, - /// Provide rapid access to a `Target` by its unique id. - pub(super) id: HashMap>>, - /// All files that need to be processed. Only `File::status::Clean` files - /// that just became dirty should be added, since non-clean files by - /// definition are already in the vec. + pub(super) files: HashSet>>, + /// Provide rapid access to a `Target` or `Fragment` by its unique id. + pub(super) targets_and_fragments: HashSet>>, + /// A list of IDs that appeared in `Xref`s or `GatherElement`s but whose + /// `Target` or `Fragment` hasn't been found. + pub(super) missing_targets_and_fragments: HashSet>>, + /// All files with unknown content. pub(super) pending_files: Vec, /// The root directory of this project. pub(super) root: PathBuf, @@ -259,95 +137,124 @@ pub struct Cache { /// This stores metadata for given file. For non-page files (non-existent files, /// images, PDFs, etc.) many of the fields are empty or `None`. +/// +/// TODO: support inclusion in a HashSet using `path` as the key. pub(super) struct File { /// The full path to this file; it must be within the project's root /// directory. This file may not exist -- it could be created by a broken - /// link. - pub(super) path: PathBuf, - /// The status of this file. + /// link. This may not be modified after creating the Target, per HashSet + /// constraints. TODO: create a public getter method to provide read-only + /// access to this field. + path: PathBuf, + /// Metadata used to determine if this data represents the actual state of + /// the file; if the file is newer, then this file is implicitly `Unknown`. + /// `None` if the file doesn't exist or the metadata can't be determined. + pub(super) metadata: Option, + /// The status of this file. Note that this overlaps with `pending_files` + /// and should be kept in sync with it. pub(super) status: FileStatus, - /// The TOC's numbering for this file; empty if it's either not in the TOC, - /// or is a prefix/suffix chapter. Taken from - /// [mdbook::book::SectionNumber](https://docs.rs/codam-mdbook/latest/mdbook/book/struct.SectionNumber.html). - pub(super) toc: Vec, - /// All targets on this page, in order of appearance on the page. This is - /// the only owner of `Target` data. - pub(super) target: Vec>>, - /// The first (and hopefully only) `h1` target on this page. - pub(super) h1: Weak>, + /// All targets on this page. This is the only owner of `Target` data. + pub(super) targets: HashSet>>, + /// All cross references on this page; also the owner. + pub(super) xrefs: Vec>>, + /// All fragments on this page; also the owner. + pub(super) fragments: HashSet>>, + /// All gather elements on this page; also the owner. + pub(super) gathers: Vec>> } /// The status of a file from the cache's perspective. pub(super) enum FileStatus { - /// The file hasn't been processed yet. Typically, this is a file referenced - /// by a link but not available in the cache. - Pending, - /// The file need to be re-processed. - Dirty, - /// The file has been processed. (It may not exist.) - Clean, + /// The file's content is unknown -- either the file hasn't been processed, + /// or it's been modified since it was last processed. + Unknown, + /// The file need to be re-processed to update cross-references or gather + /// elements. + Outdated, + /// The file has been processed. + UpToDate, } /// Contains all information about a target. A target is any HTML element with -/// an id. This means that links directly to a file are not considered a target -/// or tracked by the cache. +/// an id. +/// +/// TODO: support inclusion in a HashSet using `id` as the key. pub(super) struct Target { /// The file which contains this target. pub(super) file: Weak>, /// The id of this target. It must be globally unique within the project. - pub(super) id: String, - /// The line number of this target in `File`. + /// `id` and `innerHtml` define the state of the `Target` that `xrefs` + /// depend on. This may not be modified after creating the Target, per + /// HashSet constraints. TODO: create a public getter method to provide + /// read-only access to this field. + id: String, + /// The inner HTML of this element. + pub(super) innerHtml: String, + /// All files containing cross references to this target. If this `Target`'s + /// state changes, then these need to be rebuilt. + pub(super) xrefs: HashSet>>, + /// The line number of this target in `File`. Is this necessary? pub(super) line: usize, - /// The index of the doc block which contains this Target in the vec - /// of `CodeDocBlock`s for this file. - pub(super) code_doc_block_index: usize, - /// The type of backlink for this target. - pub(super) backlink_type: BacklinkType, - /// The HTML contents (or HTML context, if this target has no content, such - /// as ``) of this element. Tags, which contain multiple code - /// and doc blocks, must be rendered to static HTML. - pub(super) contents: String, - /// All references to this target which don't depend on it. The key is the - /// file path for a file, or the ID for a Target. Assume that IDs and file - /// names don't overlap. - pub(super) references: HashMap, - /// Targets may depend on data from another file within this project. - /// Typically, these are auto-titled hyperlinks or backlinks. If this Target - /// is a gather element, this contains both direct dependencies (backlinks - /// for the gather element's anchor) and indirect dependencies (dependencies - /// of each of these backlinks). - /// - /// All references to this target that also depend on it; if this Target - /// changes, all these must be updated. The key is the file path for a file, - /// or the ID for a Target. - pub(super) dependencies: HashMap, + /// The index of the doc block which contains this Target in the vec of + /// `CodeDocBlock`s for this file. + pub(super) doc_block_index: usize, +} + +/// This defines a cross reference to a `Target`. Currently, this could probably +/// be simplified to just the `id`; keeping the struct to make any future +/// changes easier. +pub(super) struct Xref { + /// The file which contains this target. + pub(super) file: Weak>, + /// The id cross-referenced. + pub(super) id: String, } -/// Links can have no ID, and therefore are identifiable only by the file they -/// reside in, or they have an ID and are therefore a target. -pub(super) enum LinkType { - File(Weak>), - Target(Weak>), +/// This is a unique id that encompasses a series of code/doc blocks, always +/// starting with a doc block, which `GatherElement`s operate on. +/// +/// TODO: support inclusion in a HashSet using `id` as the key. +pub(super) struct Fragment { + /// The file which contains this fragment. + pub(super) file: Weak>, + /// The id of this fragment. It must be globally unique within the project. + /// `id` and `contents` define the state of the `Fragment` that + /// `GatherElements` depend on. + id: String, + /// The code/doc block content of this element rendered as HTML. + pub(super) content: String, + /// All gather elements referencing this `Fragment`. If this `Fragment`'s + /// state changes, then the files containing these need to be rebuilt. + pub(super) gathers: HashSet>>, + /// The line number of this `Fragment` in `File`. Is this necessary? + pub(super) line: usize, + /// The index of the first doc block of this `Fragment` in the vec of + /// `CodeDocBlock`s for this file. + pub(super) doc_block_start_index: usize, + /// The index of the last code/doc block of this `Fragment` in the vec of + /// `CodeDocBlock`s for this file. + pub(super) code_doc_block_end_index: usize, } -/// Describe the type of this target's backlink. -pub(super) enum BacklinkType { - /// This target is not a backlink. - None, - /// This target has a gather tag backlink. - Gather, - /// This target has a wrapped backlink. - Wrapped, - /// This target has a plain backlink. - Plain, +/// This defines a list of `Fragment`s to combine. +pub(super) struct GatherElement { + /// The file which contains this gather element. + pub(super) file: Weak>, + /// The ids gathered. If this changes, all files containing inserted/deleted + /// `Fragments` referenced by these ids need to be rebuilt. But how to track + /// changes to this? It isn't anchored by an ID. + pub(super) ids: Vec, + /// The inner HTML of this gather element. + pub(super) innerHtml: String, + /// The index of the doc block which contains this Target in the vec of + /// `CodeDocBlock`s for this file. + pub(super) doc_block_index: usize, } -/// Query parameters parsed into known link options. -pub(super) enum LinkOptions { - Plain, - AutoTitle, - AutoNumber, - AutoTitleAndNumber, +/// TODO: support inclusion in a HashSet using `id` as the key. +pub(super) enum TargetOrFragment { + Target(Target), + Fragment(Fragment) } // Code @@ -355,8 +262,9 @@ pub(super) enum LinkOptions { impl Cache { pub fn new() -> Self { Cache { - path: HashMap::new(), - id: HashMap::new(), + files: HashSet::new(), + targets_and_fragments: HashSet::new(), + missing_targets_and_fragments: HashSet::new(), pending_files: vec![], root: PathBuf::new(), } @@ -364,15 +272,17 @@ impl Cache { /// Look up or create a `File` entry in the cache for the given path. pub(super) fn get_or_create_file(&mut self, path: &Path) -> Arc> { - self.path + self.files .entry(path.to_path_buf()) .or_insert_with(|| { Arc::new(Mutex::new(File { path: path.to_path_buf(), - status: FileStatus::Pending, - toc: vec![], - h1: Weak::new(), - target: vec![], + metadata: path.metadata().ok(), + status: FileStatus::Unknown, + targets: HashSet::new(), + fragments: HashSet::new(), + gathers: Vec::new(), + xrefs: Vec::new(), })) }) .clone() @@ -397,109 +307,12 @@ mod tests { use indoc::indoc; use test_utils::prep_test_dir; - use crate::processing::cache::{BacklinkType, Cache, File, FileStatus, Target}; + use crate::processing::cache::{Cache, File, FileStatus, Target}; // Verify basic parsing #[test] fn test_1() { let (temp_dir, test_dir) = prep_test_dir!(); - let bar_cpp = indoc!( - r#" - // # Heading 1 - // - // ## Heading 2 - // - // - // - // [File link](bar.cpp) - // - // [anchor link](bar.cpp#one) - // - // [][baz.cpp) - // - // [](baz.cpp#one) - // - // [](baz.cpp#one?number) - // - // [](baz.cpp#one?title&number) - // - // [][baz.cpp#gathering_tag) - code(); - "# - ); - - let bar_cpp_path = test_dir.join("bar.cpp"); - let file_bar_cpp = Arc::new(Mutex::new(File { - path: bar_cpp_path.clone(), - status: FileStatus::Pending, - toc: vec![1], - // Since we haven't parsed the file, the `h1` hasn't been found. - h1: Weak::new(), - // Same for targets. - target: vec![], - })); - let baz_cpp_path = test_dir.join("baz.cpp"); - - // Create a baz file that's been processed. It contains one heading and - // a gather tag. - let mut file_baz_cpp = Arc::new(Mutex::new(File { - path: baz_cpp_path.clone(), - status: FileStatus::Clean, - toc: vec![2], - // This is filled in below. - h1: Weak::new(), - // This is filled in below. - target: vec![], - })); - file_baz_cpp.borrow_mut().lock().unwrap().target = vec![ - Arc::new(Mutex::new(Target { - file: Arc::downgrade(&file_baz_cpp), - id: "one".to_string(), - line: 1, - code_doc_block_index: 0, - backlink_type: BacklinkType::None, - contents: "Heading one".to_string(), - references: HashMap::new(), - dependencies: HashMap::new(), - })), - Arc::new(Mutex::new(Target { - file: Arc::downgrade(&file_baz_cpp), - id: "gathering_tag".to_string(), - line: 1, - code_doc_block_index: 0, - backlink_type: BacklinkType::Gather, - contents: "Gather tag".to_string(), - references: HashMap::new(), - dependencies: HashMap::new(), - })), - ]; - let h1 = Arc::downgrade(&file_baz_cpp.lock().unwrap().target[0]); - file_baz_cpp.borrow_mut().lock().unwrap().h1 = h1; - - let mut cache_path = HashMap::new(); - cache_path.insert(bar_cpp_path, file_bar_cpp); - cache_path.insert(baz_cpp_path, file_baz_cpp.clone()); - let mut cache_id = HashMap::new(); - cache_id.insert( - "one".to_string(), - file_baz_cpp.lock().unwrap().target[0].clone(), - ); - cache_id.insert( - "gathering_tag".to_string(), - file_baz_cpp.lock().unwrap().target[1].clone(), - ); - let mut cache_anchor = HashMap::new(); - - let mut cache = Cache { - path: cache_path, - id: cache_anchor, - pending_files: vec![], - root: test_dir, - }; - - // Processing a file updates its values in the cache. - //cache.upsert_file_core(&bar_cpp_path, ); - temp_dir.close().unwrap(); } } From dd0fa1bb0960acc8b8a7d6fb6fd211f59beefce2 Mon Sep 17 00:00:00 2001 From: "Bryan A. Jones" Date: Sun, 26 Jul 2026 05:52:30 -0500 Subject: [PATCH 03/22] wip: Claude implemenetation. --- CLAUDE.md | 29 +- server/src/processing.rs | 273 +++++----- server/src/processing/cache.rs | 935 +++++++++++++++++++++++++++++---- server/src/processing/tests.rs | 8 +- 4 files changed, 977 insertions(+), 268 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 354a7513..2c897d9d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -34,7 +34,10 @@ void foo(); Architecture ------------ -A Visual Studio Code extension in `extensions/VSCode` exchanges messages with the CodeChat Editor Server, located in `server/` (also terms the Server), which also exchanges message with the CodeChat Editor Client (also termed the Client) located in `client/`. +A Visual Studio Code extension in `extensions/VSCode` exchanges messages with +the CodeChat Editor Server, located in `server/` (also terms the Server), which +also exchanges message with the CodeChat Editor Client (also termed the Client) +located in `client/`. Project build ------------- @@ -44,3 +47,27 @@ All build commands must be executed from the `server/` directory. * To build the entire project, execute `./bt build`. * To build (bundle) only the Client, execute `./bt client-build`. * To run tests, execute `cargo test`. + +Comments +-------- + +This program uses a literate programming approach to improve the overall +comprehensibility of the code. Guidelines for comments: + +* Functions should be preceded by a comment that summarizes their overall + purpose. Each parameter should be preceded by a comment briefly explaining its + purpose; the return value when preset should be preceded by a comment + explaining the data it carries. +* Data structures should be preceded by a command explaining their purpose; each + value in the data structure should be preceded by a command explaining its + role. +* Comments in the code should be limited to those that: +* + 1. Document a connection which cannot easily be determined by inspection -- + for example, explaining the relationship between a web client Ajax call and + the backend server which handles it. + 2. Explain behavior which can only be determined by run-time inspection or + debugging; behavior which can be directly derived from the code should + produce a comment. + 3. Capture requirements or higher-level behavior which specifies the overall + purpose of the code at a higher level than the implementation. diff --git a/server/src/processing.rs b/server/src/processing.rs index f42bf612..60821b2b 100644 --- a/server/src/processing.rs +++ b/server/src/processing.rs @@ -43,7 +43,7 @@ use std::{ slice::Iter, string::FromUtf8Error, sync::LazyLock, - sync::{Arc, Mutex, Weak}, + sync::{Arc, Mutex}, }; // ### Third-party @@ -80,7 +80,8 @@ use crate::{ lexer::{ CodeDocBlock, DocBlock, LEXERS, LanguageLexerCompiled, source_lexer, supported_languages::MARKDOWN_MODE, - }, processing::cache::{Fragment, Target}, + }, + processing::cache::{FileFacts, FragmentFact, GatherFact, TargetFact}, }; use cache::Cache; @@ -1244,6 +1245,20 @@ pub fn dom_to_html(dom: Rc) -> io::Result { Ok(html_out) } +/// Serialize a node's children back to an HTML string -- the node's inner +/// HTML. This defines the contents of a target or gather element stored in the +/// cache. +fn node_inner_html(node: &Rc) -> io::Result { + let so = SerializeOpts { + // Serialize only the node's children, not the node itself. + traversal_scope: TraversalScope::ChildrenOnly(None), + ..Default::default() + }; + let mut bytes = vec![]; + serialize(&mut bytes, &SerializableHandle::from(node.clone()), so)?; + String::from_utf8(bytes).map_err(io::Error::other) +} + /// Get the body element from a top-level DOM. fn get_dom_body(document: &Rc) -> Rc { // HTML is: @@ -1269,112 +1284,54 @@ fn hydrate_html( cache: Arc>, ) -> io::Result<(String, Vec>)> { let dom = html_to_dom(html, None)?; - let file_entry = cache.lock().unwrap().get_or_create_file(file); - // ### Prepare `Targets`/`Xrefs` to process this file + // Read the file's metadata before taking the cache lock, so that no I/O + // happens while the lock is held. This captures the state of the file + // whose content is being processed. + let metadata = file.metadata().ok(); + // ### Collect facts // - // Requirement: determine if any files containing cross-references need to - // be rebuilt due to changes in the `Target`s in this file. To accomplish - // this, determine if any `Targets` in this file were added, deleted, or - // modified, then notify all cross-references of these targets that their - // containing file is outdated. Note that "modified" refers only to the - // `Target` state that cross-references depend on. - // - // Because `file_entry.xrefs` have no id to match against, remove all - // `xrefs` on this page from their `Target` dependencies; this will be - // re-generated during the DOM traversal. TODO: remove each `xref.id` from - // `cache.targets[id]` for `xref` in `file_entry.xrefs`. - // - // TODO: implementation. Move the current `file_entry.targets` to a local - // variable `targets`. Match each `Target` encountered in the DOM against - // the `targets`, moving identical entries to `file_entry.targets`. After - // processing the DOM, remaining entries in `targets` are deleted, while new - // entries and modifications were discovered during the DOM traversal. - let old_targets = mem::take(&mut file_entry.lock().unwrap().targets); - // ### Prepare `Fragments`/`GatherElements` to process this file - // - // Requirement: determine if any files containing `Fragment`s need to be - // rebuilt due to changes in gather elements in this file. Similarly, - // determine if any files containing gather elements need to be rebuilt due - // to change in `Fragment`s in this file. Because `Fragment`s have an id, - // use the same approach as `Target`s. Gather elements don't have an id; to - // identify differences, make a copy of all `Fragment.gathers` referenced by - // gather elements in the current file, then remove all gathers in the - // current file from `Fragments.gathers`. Walk the DOM; changes to any - // `Fragment.gathers` which wasn't copied is an addition. Comparison with - // the copied `Fragment.gathers` with current `Fragment.gathers` shows - // deletions. Since Fragments only depend on gather ids and no other state, - // there's no modifications to track (unlike cross-references, where changes - // to Target contents affect its cross-reference state). - // - // Store the previous state of all `Fragment.gathers` referenced by gather - // on this page in a map where key = `Fragment.id`, value = - // `Fragment.gathers`. - let old_fragments_gathers = HashMap::new(); - // For each `id` in `gather.ids` in `file.gathers` on this page: - // - // 1. Insert a copy of `cache.fragments[id].gathers` to a - // `old_fragments_gathers`, unless it's already inserted. - // 2. Remove `gather` from `cache.fragments[id].gathers`. - // - // Finally, empty `file_entry.gathers`. This will be re-created by page - // processing. After processing this page, rebuild any files containing - // fragments whose `gathers` list changed. - // - // Move the `file_entry.fragments` vec into a `HashMap`; follow the same - // logic as `old_targets_vec` above. - let old_fragments = mem::take(&mut file_entry.lock().unwrap().fragments); - // This is storage for the state needed for walking the DOM. + // Walk the DOM, collecting all cacheable facts -- targets, + // cross-references, fragments, and gather elements -- without touching the + // cache. This keeps the non-`Send` DOM types out of the cache and off its + // critical section; see the design discussion in `cache.rs`. let mut walk_context = WalkContext { - cache, - file_entry, - old_targets, - old_fragments, + facts: FileFacts::default(), doc_block_index: 0, xrefs: Vec::new(), fragments: Vec::new(), gathers: Vec::new(), }; - walk_context = hydrating_walk_node(dom.clone(), walk_context); - // The overall processing order after walking the DOM: - // - // 1. All `Target` cross-reference (`Xref`) state is updated. Therefore, - // update the DOM content for all `Xrefs` in this file (whic is stored in - // `walk_context.xrefs`). If the `Xref.id` refers to a `Fragment` instead - // of a `Target`, the DOM will contain an error message. - // 2. All `Fragment`s in this file have updated ids. For - // each `walk_context.gathers`, create a `GatherElement`. Add it to - // `file_entry.gathers` and `file_entry.targets[id].gathers`. - // 3. All `GatherElements` `Fragment` state is now updated. Therefore, - // update the DOM content for all `Fragment`s in this file (stored in - // `walk_context.fragments`). - // 4. On exit from this function, doc block contents will be finalized. - // After that, update each `Fragment.content`. - // 5. `Fragment`s on this page now have updated state needed by - // `GatherElement`s. Update each `GatherElement`. + walk_context = hydrating_walk_node(dom.clone(), walk_context)?; + // ### Commit facts // - // TODO: + // Apply the collected facts to the cache in a single transaction. This + // diffs them against the file's previous state: added, deleted, and + // modified targets and fragments mark the files which depend on them as + // outdated, this file is linked to every id it references (via + // `Cache::unresolved` for ids with no definition yet), and duplicate ids + // are reported. + let commit = + cache + .lock() + .unwrap() + .commit_file(file, metadata, mem::take(&mut walk_context.facts)); + // TODO: patch the DOM using the committed cache state (re-lock the cache + // and use `Cache::resolve_id`): // - // 1. For each `target` in `old_targets`, mark all `target.xrefs` as - // outdated. - // 2. For each `fragment` in `old_fragments`, mark each `gather.file` in - // `fragment.gathers` as outdated. - // 3. For each `old_fragment_gathers`, compare old and new gathers. If they - // differ, mark the file containing the `Fragment` as outdated, unless - // the file is `file_entry`. - // 4. For each `gather` in `file_entry.gathers`, look up/create the - // corresponding `Fragment`. Add each `gather` to this - // `fragment.gathers`. - // 5. For each `xref` in `walk_context.xrefs`, get/create (add to - // `cache.missingTargetsAndFragments`) the `Target` of this xref. Add - // `file_entry` to `Target.xrefs`. If the `Target` of this xref is not - // clean, mark `file_entry` as dirty. Update the DOM by inserting a link - // based on available `Target` info. Note that `xref` processing was - // deferred until all `Target`s in this file were processed. - // 6. For each `fragment`, generate updated DOM data (a link per gather - // element). + // 1. For each node in `walk_context.xrefs`, replace its generated contents + // with a link to its target; if the id resolves to a `Fragment` or is + // missing, insert an error message instead. + // 2. For each node in `walk_context.fragments`, insert links to the gather + // elements which reference it. + // 3. Report each id in `commit.duplicates` as a warning in the DOM; the + // first definition of an id wins, and later definitions are ignored. + // 4. Schedule reprocessing for each file in `commit.outdated`. // - // TODO: on return, update fragment contents, then update gathers DOM data - // (a list containing a link to each fragment followed by its contents), + // TODO: on return, once doc block contents are finalized, store each + // fragment's content with `Cache::update_fragment_content` (which marks + // the files containing affected gather elements as outdated), then update + // each gather element's DOM data (a list containing a link to each + // fragment followed by its contents). Ok((dom_to_html(dom)?, walk_context.gathers)) } @@ -1406,20 +1363,15 @@ fn get_text_content(node: &Rc) -> String { /// This provides the needed context when walking the HTML DOM of all doc /// blocks. struct WalkContext { - /// The cache for this project. - cache: Arc>, - /// The `cache::File` currently being processed. - file_entry: Arc>, - /// The previous `file_entry.targets` that haven't yet been claimed while - /// processing this file. The key is the `Target`'s ID. - old_targets: HashMap>>, - /// Same as above: currently unclaimed `file_entry.fragments`. - old_fragments: HashMap>>, - /// DOM for all xrefs found in this file. + /// The cacheable facts collected so far; applied to the cache by + /// `Cache::commit_file` after the walk completes. + facts: FileFacts, + /// DOM for all xrefs found in this file, kept so their generated contents + /// can be patched after the cache commit. xrefs: Vec>, /// DOM for all `Fragment`s. fragments: Vec>, - /// DOM for all `GatherElement`s + /// DOM for all `GatherElement`s. gathers: Vec>, /// The current doc block index, based on parsing the HTML for /// `codechateditor-separator` elements, which contain this value. @@ -1427,7 +1379,7 @@ struct WalkContext { } /// Hydrate the HTML of newly-translated doc blocks. -fn hydrating_walk_node(node: Rc, mut walk_context: WalkContext) -> WalkContext { +fn hydrating_walk_node(node: Rc, mut walk_context: WalkContext) -> io::Result { for child in node.children.borrow_mut().iter_mut() { let possible_replacement_child = // Perform replacements of GraphViz and Mermaid graphs: @@ -1484,54 +1436,74 @@ fn hydrating_walk_node(node: Rc, mut walk_context: WalkContext) -> WalkCon // Analyze this node for cacheable data. if let Some(tag_name) = get_node_tag_name(child) { // See if the element has an id/anchor. - let _id = get_attr_value(child, "id").unwrap_or_default(); + let id = get_attr_value(child, "id"); // Track doc block index from separator elements. if tag_name == "codechateditor-separator" && let Ok(index) = get_text_content(child).trim().parse::() { walk_context.doc_block_index = index; + } else if tag_name == "xref" { + // A cross reference: record the referenced id as a fact, and + // keep the node so its generated contents can be filled in + // after the cache commit. Note that this element doesn't allow + // an `id` attribute, so it's never a target. + if let Some(ref_id) = get_attr_value(child, "ref") { + walk_context.facts.xrefs.push(ref_id); + walk_context.xrefs.push(child.clone()); + } + } else if tag_name == "fragment" { + // A fragment; without an id it's meaningless, so it's ignored. + // Its content can't be determined yet -- doc blocks aren't + // finalized during the walk -- so the caller stores it later + // via `Cache::update_fragment_content`. + if let Some(id) = id { + // The `following` attribute selects how many code/doc + // blocks after the current doc block the fragment + // encloses; the default is 1. + let following = get_attr_value(child, "following") + .and_then(|following| following.trim().parse::().ok()) + .unwrap_or(1); + walk_context.facts.fragments.push(FragmentFact { + id, + line: 0, + doc_block_start_index: walk_context.doc_block_index, + code_doc_block_end_index: walk_context.doc_block_index + following, + }); + walk_context.fragments.push(child.clone()); + } + } else { + // A gather element; it may also carry an id, which makes it a + // target as well. + if let Some(gather_ids) = get_attr_value(child, "data-gather") { + walk_context.facts.gathers.push(GatherFact { + ids: gather_ids.split_whitespace().map(str::to_string).collect(), + inner_html: node_inner_html(child)?, + doc_block_index: walk_context.doc_block_index, + }); + walk_context.gathers.push(child.clone()); + } + // Any other element with an id is a target. + if let Some(id) = id + && !id.is_empty() + { + walk_context.facts.targets.push(TargetFact { + id, + inner_html: node_inner_html(child)?, + // Line numbers aren't available until the + // pulldown-cmark HTML writer preserves them; see the + // TODO in `cache.rs`. + line: 0, + doc_block_index: walk_context.doc_block_index, + }); + } } - - // TODO: write code here. - // - // When walking, look for: - // - // 1. A cross reference. Add it to `walk_context.xrefs`. - // 2. A `Target` (any item with an id that's not a `Fragment`). - // 1. Process the id: - // 1. id exists in `old_targets` - transfer ownership by - // appending this to `walk_content.file_entry.targets`, - // removing it from the `old_targets`. Assert that this id - // is unique. - // 2. id doesn't exist in `old_targets` and is unique: add - // this `Target` to `walk_content.file_entry.targets`. - // 3. id doesn't exist in `old_targets` and is in the set of - // `cache.missingTargetsOrFragments`: transfer ownership by - // appending this to `walk_content.file_entry.targets`. - // Mark all `target.xrefs` as `Outdated`. - // 4. id is a duplicate (exists - // in `cache.targetsOrFragments`): resolve duplicate ids: - // 1. Is the duplicate id in the same file? In this case, - // we have no way to determine which was the original - // id. Rename the id being processed; stop here. - // 2. Look at the timestamp of this file and of the other - // file which contains the duplicate id. If this file is - // newer, rename this id. Otherwise, update the id and - // mark the other file as `Outdated`. - // 2. Check the `contents`: if the `contents` changed, add all - // this target's `xrefs` to the `Outdated` list and update the - // search text for this target. - // 3. A `GatherElement`: add it to `walk_context.gathers`. - // 4. A `Fragment`: add it to `walk_context.fragments`, then process - // similarly to a `Target`. Note that `content` can't be - // determined yet. } - walk_context = hydrating_walk_node(child.clone(), walk_context); + walk_context = hydrating_walk_node(child.clone(), walk_context)?; } - walk_context + Ok(walk_context) } fn replace_math_node(child: &Rc, is_hydrate: bool) -> Option> { @@ -1567,7 +1539,7 @@ fn replace_math_node(child: &Rc, is_hydrate: bool) -> Option> { // When hydrating, there should only be a `class` attribute. if child_attrs_len == 1 { match attr_value_str { - "math math-inline" => Some(("\(", "\)", "math math-inline mceNonEditable")), + "math math-inline" => Some(("\\(", "\\)", "math math-inline mceNonEditable")), "math math-display" => Some(("$$", "$$", "math math-display mceNonEditable")), _ => None, } @@ -1575,9 +1547,6 @@ fn replace_math_node(child: &Rc, is_hydrate: bool) -> Option> { None } } else { - - - if child_attrs_len == 2 && let Some(contenteditable_attr) = child_attrs .iter() @@ -1585,7 +1554,7 @@ fn replace_math_node(child: &Rc, is_hydrate: bool) -> Option> { && contenteditable_attr.value == *"false" { match attr_value_str { - "math math-inline mceNonEditable" => Some(("\(", "\)", "math math-inline")), + "math math-inline mceNonEditable" => Some(("\\(", "\\)", "math math-inline")), "math math-display mceNonEditable" => Some(("$$", "$$", "math math-display")), _ => None, } diff --git a/server/src/processing/cache.rs b/server/src/processing/cache.rs index 83bf4d4a..f956e86a 100644 --- a/server/src/processing/cache.rs +++ b/server/src/processing/cache.rs @@ -85,6 +85,46 @@ /// reference this id but don't depend on it, and all `Target`s which /// reference this anchor and also depend on it. /// +/// Design +/// ------ +/// +/// The cache is a single plain-data structure, shared as an +/// `Arc>`; all consistency comes from that one lock, so no +/// per-item locking (and therefore no lock ordering) is needed. Items refer to +/// each other by key -- files by path, targets and fragments by id -- rather +/// than by `Arc`/`Weak` pointers. This keeps the structure acyclic, `Send`, +/// and (in the future) serializable, and avoids garbage-collecting stale weak +/// references. +/// +/// Updating the cache is a two-phase process: +/// +/// 1. Collect: while walking a file's DOM, record all cacheable facts +/// (`FileFacts`) -- targets, cross-references, fragments, and gather +/// elements -- without touching the cache. This keeps the non-`Send` DOM +/// types out of the cache and off its critical section. +/// 2. Commit: `Cache::commit_file` applies the facts in one transaction, +/// diffing them against the file's previous state to compute the set of +/// other files made outdated by this update. +/// +/// Dependencies are tracked at file granularity: each target or fragment +/// stores the set of files (its `dependents`) whose rendered output depends on +/// it. This suffices because the only action ever taken on a dependent is +/// marking its containing file outdated, and it makes indirection (gather A +/// includes fragment B, whose content cross-references target C) work without +/// extra machinery: a change to C outdates B's file; reprocessing B's file +/// changes B's content, which outdates A's file. +/// +/// An id referenced before (or without) being defined is recorded in +/// `Cache::unresolved`, which maps the id to the set of files waiting on it. +/// When the id later appears, those files are marked outdated and become the +/// initial dependents; when a defined id disappears, its dependents move back +/// to `unresolved`. +/// +/// Duplicate ids are never renamed (file timestamps can't reliably identify +/// the original, and renaming would silently modify user content). Instead, +/// the first definition wins and later definitions are reported in +/// `CommitOutcome::duplicates` for the caller to surface as warnings. +/// /// Thinking space: /// /// * Any file can be submitted for a cache update. After the update finishes, @@ -99,34 +139,43 @@ /// 1. (Longer-term) modify the pulldown-cmark HTML writer to preserve line /// numbers. /// 2. Revise the TOC loader to use mdbook's code to process and update the TOC. -/// Imports -// --- +// Imports +// ------- // // ### Standard library use std::{ - collections::{HashMap, HashSet}, fs::Metadata, path::{Path, PathBuf}, rc::Rc, sync::{Arc, Mutex, Weak}, + collections::{HashMap, HashSet}, + fs::Metadata, + mem, + path::{Path, PathBuf}, }; // ### Third-party -use markup5ever_rcdom::Node; - +// +// None. +// // ### Local // // None. -/// Data structures -/// --------------- -/// +// Data structures +// --------------- +// /// This defines the cache used to store all targets in a project. pub struct Cache { /// Provide rapid access to a file by its absolute path; it must be within - /// the project's root directory. This is the sole owner of these `File`s. - pub(super) files: HashSet>>, - /// Provide rapid access to a `Target` or `Fragment` by its unique id. - pub(super) targets_and_fragments: HashSet>>, - /// A list of IDs that appeared in `Xref`s or `GatherElement`s but whose - /// `Target` or `Fragment` hasn't been found. - pub(super) missing_targets_and_fragments: HashSet>>, + /// the project's root directory. This owns all per-file data. + pub(super) files: HashMap, + /// Provide rapid access to a `Target` or `Fragment` by its unique id: the + /// value is the path of the file whose `FileEntry` defines the id. Whether + /// the id names a target or a fragment is determined by looking it up in + /// that entry; see `resolve_id`. + pub(super) ids: HashMap, + /// Ids that appeared in cross-references or gather elements but aren't + /// (currently) defined by any file, mapped to the set of files which + /// reference them. When such an id appears, these files are marked + /// outdated and become the id's initial dependents. + pub(super) unresolved: HashMap>, /// All files with unknown content. pub(super) pending_files: Vec, /// The root directory of this project. @@ -135,40 +184,36 @@ pub struct Cache { // file name. Perhaps [Tantivy](https://docs.rs/tantivy/latest/tantivy/)? } -/// This stores metadata for given file. For non-page files (non-existent files, -/// images, PDFs, etc.) many of the fields are empty or `None`. -/// -/// TODO: support inclusion in a HashSet using `path` as the key. -pub(super) struct File { - /// The full path to this file; it must be within the project's root - /// directory. This file may not exist -- it could be created by a broken - /// link. This may not be modified after creating the Target, per HashSet - /// constraints. TODO: create a public getter method to provide read-only - /// access to this field. - path: PathBuf, +/// This stores the cached data for a given file. For non-page files +/// (non-existent files, images, PDFs, etc.) many of the fields are empty or +/// `None`. +#[derive(Default)] +pub(super) struct FileEntry { /// Metadata used to determine if this data represents the actual state of /// the file; if the file is newer, then this file is implicitly `Unknown`. /// `None` if the file doesn't exist or the metadata can't be determined. pub(super) metadata: Option, - /// The status of this file. Note that this overlaps with `pending_files` - /// and should be kept in sync with it. + /// The status of this file. Note that this overlaps with + /// `Cache::pending_files` and should be kept in sync with it. pub(super) status: FileStatus, - /// All targets on this page. This is the only owner of `Target` data. - pub(super) targets: HashSet>>, - /// All cross references on this page; also the owner. - pub(super) xrefs: Vec>>, - /// All fragments on this page; also the owner. - pub(super) fragments: HashSet>>, - /// All gather elements on this page; also the owner. - pub(super) gathers: Vec>> + /// All targets on this page, keyed by id. + pub(super) targets: HashMap, + /// All cross references on this page. + pub(super) xrefs: Vec, + /// All fragments on this page, keyed by id. + pub(super) fragments: HashMap, + /// All gather elements on this page. + pub(super) gathers: Vec, } /// The status of a file from the cache's perspective. +#[derive(Debug, Default, PartialEq, Eq)] pub(super) enum FileStatus { /// The file's content is unknown -- either the file hasn't been processed, /// or it's been modified since it was last processed. + #[default] Unknown, - /// The file need to be re-processed to update cross-references or gather + /// The file needs to be re-processed to update cross-references or gather /// elements. Outdated, /// The file has been processed. @@ -176,26 +221,21 @@ pub(super) enum FileStatus { } /// Contains all information about a target. A target is any HTML element with -/// an id. -/// -/// TODO: support inclusion in a HashSet using `id` as the key. +/// an id; the id (globally unique within the project) is the key of +/// `FileEntry::targets`, and the containing file is the entry holding this +/// value, so neither is duplicated here. pub(super) struct Target { - /// The file which contains this target. - pub(super) file: Weak>, - /// The id of this target. It must be globally unique within the project. - /// `id` and `innerHtml` define the state of the `Target` that `xrefs` - /// depend on. This may not be modified after creating the Target, per - /// HashSet constraints. TODO: create a public getter method to provide - /// read-only access to this field. - id: String, - /// The inner HTML of this element. - pub(super) innerHtml: String, - /// All files containing cross references to this target. If this `Target`'s - /// state changes, then these need to be rebuilt. - pub(super) xrefs: HashSet>>, - /// The line number of this target in `File`. Is this necessary? + /// The inner HTML of this element. Together with its id, this defines the + /// state of the `Target` that cross-references depend on. + pub(super) inner_html: String, + /// All files containing cross references to this target. If this + /// `Target`'s state changes, then these need to be rebuilt. + pub(super) dependents: HashSet, + /// The line number of this target in its file. Always 0 until the + /// pulldown-cmark HTML writer preserves line numbers; see the TODO in the + /// module docs. pub(super) line: usize, - /// The index of the doc block which contains this Target in the vec of + /// The index of the doc block which contains this `Target` in the vec of /// `CodeDocBlock`s for this file. pub(super) doc_block_index: usize, } @@ -204,29 +244,25 @@ pub(super) struct Target { /// be simplified to just the `id`; keeping the struct to make any future /// changes easier. pub(super) struct Xref { - /// The file which contains this target. - pub(super) file: Weak>, /// The id cross-referenced. pub(super) id: String, } /// This is a unique id that encompasses a series of code/doc blocks, always -/// starting with a doc block, which `GatherElement`s operate on. -/// -/// TODO: support inclusion in a HashSet using `id` as the key. +/// starting with a doc block, which `GatherElement`s operate on. As with +/// `Target`, the id is the key of `FileEntry::fragments` and the containing +/// file is the entry holding this value. pub(super) struct Fragment { - /// The file which contains this fragment. - pub(super) file: Weak>, - /// The id of this fragment. It must be globally unique within the project. - /// `id` and `contents` define the state of the `Fragment` that - /// `GatherElements` depend on. - id: String, - /// The code/doc block content of this element rendered as HTML. + /// The code/doc block content of this element rendered as HTML. Together + /// with its id, this defines the state of the `Fragment` that gather + /// elements depend on. This is empty until the fragment's doc blocks are + /// finalized and the caller stores the result via + /// `Cache::update_fragment_content`. pub(super) content: String, - /// All gather elements referencing this `Fragment`. If this `Fragment`'s - /// state changes, then the files containing these need to be rebuilt. - pub(super) gathers: HashSet>>, - /// The line number of this `Fragment` in `File`. Is this necessary? + /// All files containing gather elements referencing this `Fragment`. If + /// this `Fragment`'s state changes, then these need to be rebuilt. + pub(super) dependents: HashSet, + /// The line number of this `Fragment` in its file; see `Target::line`. pub(super) line: usize, /// The index of the first doc block of this `Fragment` in the vec of /// `CodeDocBlock`s for this file. @@ -238,23 +274,102 @@ pub(super) struct Fragment { /// This defines a list of `Fragment`s to combine. pub(super) struct GatherElement { - /// The file which contains this gather element. - pub(super) file: Weak>, - /// The ids gathered. If this changes, all files containing inserted/deleted - /// `Fragments` referenced by these ids need to be rebuilt. But how to track - /// changes to this? It isn't anchored by an ID. + /// The ids gathered. pub(super) ids: Vec, /// The inner HTML of this gather element. - pub(super) innerHtml: String, - /// The index of the doc block which contains this Target in the vec of + pub(super) inner_html: String, + /// The index of the doc block which contains this element in the vec of /// `CodeDocBlock`s for this file. pub(super) doc_block_index: usize, } -/// TODO: support inclusion in a HashSet using `id` as the key. -pub(super) enum TargetOrFragment { - Target(Target), - Fragment(Fragment) +// ### Facts +// +// Plain data collected while walking a file's DOM, then applied to the cache +// in a single transaction by `Cache::commit_file`. Keeping these free of DOM +// types lets the walk run without holding the cache lock. +/// All cacheable facts found in one file. +#[derive(Default)] +pub(super) struct FileFacts { + /// Every element with an id (excluding fragments), in document order. + pub(super) targets: Vec, + /// The destination id of every cross-reference, in document order. + pub(super) xrefs: Vec, + /// Every fragment, in document order. + pub(super) fragments: Vec, + /// Every gather element, in document order. + pub(super) gathers: Vec, +} + +/// A target found in the DOM; see `Target` for field documentation. +pub(super) struct TargetFact { + /// The id of this target. + pub(super) id: String, + pub(super) inner_html: String, + pub(super) line: usize, + pub(super) doc_block_index: usize, +} + +/// A fragment found in the DOM; see `Fragment` for field documentation. The +/// fragment's content isn't known during the walk (doc blocks aren't finalized +/// yet), so it's stored later via `Cache::update_fragment_content`. +pub(super) struct FragmentFact { + /// The id of this fragment. + pub(super) id: String, + pub(super) line: usize, + pub(super) doc_block_start_index: usize, + pub(super) code_doc_block_end_index: usize, +} + +/// A gather element found in the DOM; see `GatherElement` for field +/// documentation. +pub(super) struct GatherFact { + pub(super) ids: Vec, + pub(super) inner_html: String, + pub(super) doc_block_index: usize, +} + +// ### Commit results +// +/// The result of committing one file's facts to the cache. +pub(super) struct CommitOutcome { + /// Files (other than the committed file) whose rendered output is + /// invalidated by this commit; their status has already been set to + /// `Outdated`. The caller should schedule them for reprocessing. + pub(super) outdated: HashSet, + /// Ids in the committed file that duplicate an already-defined id. These + /// definitions were ignored (the first definition wins); the caller should + /// surface them as warnings. + pub(super) duplicates: Vec, +} + +/// Describes one duplicate id found during a commit. +#[derive(Debug, PartialEq, Eq)] +pub(super) struct DuplicateId { + /// The duplicated id. + pub(super) id: String, + /// The file containing the winning definition. If this is the committed + /// file itself, the id was defined twice within that file. + pub(super) defined_in: PathBuf, +} + +/// The result of looking up an id; borrows from the cache, so it must be used +/// while the cache lock is still held. +pub(super) enum IdResolution<'a> { + /// The id names a target. + Target { + /// The file defining the target. + path: &'a Path, + target: &'a Target, + }, + /// The id names a fragment. + Fragment { + /// The file defining the fragment. + path: &'a Path, + fragment: &'a Fragment, + }, + /// No file defines this id. + Missing, } // Code @@ -262,30 +377,369 @@ pub(super) enum TargetOrFragment { impl Cache { pub fn new() -> Self { Cache { - files: HashSet::new(), - targets_and_fragments: HashSet::new(), - missing_targets_and_fragments: HashSet::new(), + files: HashMap::new(), + ids: HashMap::new(), + unresolved: HashMap::new(), pending_files: vec![], root: PathBuf::new(), } } - /// Look up or create a `File` entry in the cache for the given path. - pub(super) fn get_or_create_file(&mut self, path: &Path) -> Arc> { - self.files - .entry(path.to_path_buf()) - .or_insert_with(|| { - Arc::new(Mutex::new(File { - path: path.to_path_buf(), - metadata: path.metadata().ok(), - status: FileStatus::Unknown, - targets: HashSet::new(), - fragments: HashSet::new(), - gathers: Vec::new(), - xrefs: Vec::new(), - })) + /// Apply the facts collected from one file's DOM walk to the cache in a + /// single transaction. This satisfies two requirements: + /// + /// * Determine if any files containing cross-references need to be rebuilt + /// due to changes in the `Target`s in this file: any target which was + /// added, deleted, or modified marks its dependent files outdated. Note + /// that "modified" refers only to the `Target` state that + /// cross-references depend on (its id and inner HTML). + /// * Determine if any files containing gather elements need to be rebuilt + /// due to changes in the `Fragment`s in this file. Fragment additions + /// and deletions are handled here; content changes are detected by + /// `update_fragment_content`, since a fragment's rendered content is + /// only known after doc block processing completes. + /// + /// Because cross-references and gather elements carry no id to match them + /// against their previous versions, the diff instead unlinks all of the + /// old version's outgoing references, then links all of the new version's. + /// Dependency sets are only ever mutated by this round trip, never used to + /// detect change, so it causes no spurious rebuilds. + pub(super) fn commit_file( + &mut self, + // The file whose facts these are. + path: &Path, + // The file's metadata, captured when its content was read; `None` if + // unavailable. Read this *before* locking the cache, so no I/O happens + // while the lock is held. + metadata: Option, + // The facts collected from the file's DOM. + facts: FileFacts, + // The outcome: outdated files and duplicate ids; see `CommitOutcome`. + ) -> CommitOutcome { + let mut outdated: HashSet = HashSet::new(); + let mut duplicates: Vec = Vec::new(); + + // ### Unlink the old version's outgoing references + // + // Remove this file from the dependents of every id its previous + // version referenced; the new version's references are linked below. + let old_refs: Vec = if let Some(entry) = self.files.get(path) { + entry + .xrefs + .iter() + .map(|xref| xref.id.clone()) + .chain( + entry + .gathers + .iter() + .flat_map(|gather| gather.ids.iter().cloned()), + ) + .collect() + } else { + Vec::new() + }; + for id in &old_refs { + self.unlink_reference(id, path); + } + + // ### Take the old definitions + // + // Move the previous targets and fragments out of the entry (creating + // the entry if this is the first commit for this file). Definitions + // which survive into the new version are matched by id and their + // dependents carried over; the leftovers are deletions. + let entry = self.files.entry(path.to_path_buf()).or_default(); + let mut old_targets = mem::take(&mut entry.targets); + let mut old_fragments = mem::take(&mut entry.fragments); + + // The set of ids the new version references, saved before `facts` is + // consumed; used to link references below. + let new_refs: HashSet = facts + .xrefs + .iter() + .cloned() + .chain( + facts + .gathers + .iter() + .flat_map(|gather| gather.ids.iter().cloned()), + ) + .collect(); + + // ### Install the new targets + let mut new_targets: HashMap = HashMap::with_capacity(facts.targets.len()); + for fact in facts.targets { + // An id already defined by another file is a duplicate: the first + // definition wins, so skip this one. + if let Some(owner) = self.ids.get(&fact.id) + && owner != path + { + duplicates.push(DuplicateId { + id: fact.id, + defined_in: owner.clone(), + }); + continue; + } + // An id defined twice within this file is likewise a duplicate. + if new_targets.contains_key(&fact.id) { + duplicates.push(DuplicateId { + id: fact.id, + defined_in: path.to_path_buf(), + }); + continue; + } + self.ids.insert(fact.id.clone(), path.to_path_buf()); + let dependents = if let Some(old_target) = old_targets.remove(&fact.id) { + // The target survives. If the state cross-references depend on + // changed, its dependents must be rebuilt; either way, they + // remain dependents. + if old_target.inner_html != fact.inner_html { + outdated.extend(old_target.dependents.iter().cloned()); + } + old_target.dependents + } else if let Some(waiters) = self.unresolved.remove(&fact.id) { + // The id was referenced before it existed: the files waiting + // on it must be rebuilt, and they become its dependents. + outdated.extend(waiters.iter().cloned()); + waiters + } else { + HashSet::new() + }; + new_targets.insert( + fact.id, + Target { + inner_html: fact.inner_html, + dependents, + line: fact.line, + doc_block_index: fact.doc_block_index, + }, + ); + } + + // ### Install the new fragments + // + // Same logic as targets; targets and fragments share the id namespace. + let mut new_fragments: HashMap = + HashMap::with_capacity(facts.fragments.len()); + for fact in facts.fragments { + if let Some(owner) = self.ids.get(&fact.id) + && owner != path + { + duplicates.push(DuplicateId { + id: fact.id, + defined_in: owner.clone(), + }); + continue; + } + if new_targets.contains_key(&fact.id) || new_fragments.contains_key(&fact.id) { + duplicates.push(DuplicateId { + id: fact.id, + defined_in: path.to_path_buf(), + }); + continue; + } + self.ids.insert(fact.id.clone(), path.to_path_buf()); + let (content, dependents) = if let Some(old_fragment) = old_fragments.remove(&fact.id) { + // The fragment survives: keep its old content until the + // caller supplies the new content via + // `update_fragment_content`, which also detects content + // changes. + (old_fragment.content, old_fragment.dependents) + } else if let Some(waiters) = self.unresolved.remove(&fact.id) { + outdated.extend(waiters.iter().cloned()); + (String::new(), waiters) + } else { + (String::new(), HashSet::new()) + }; + new_fragments.insert( + fact.id, + Fragment { + content, + dependents, + line: fact.line, + doc_block_start_index: fact.doc_block_start_index, + code_doc_block_end_index: fact.code_doc_block_end_index, + }, + ); + } + + // ### Process deleted definitions + // + // Anything left in the old maps wasn't matched by a same-kind + // definition in the new version. Its dependents must be rebuilt. If + // the id changed kind (target to fragment or vice versa) the + // dependents transfer to the new definition; otherwise the id is gone + // and its dependents wait in `unresolved` for it to reappear. + for (id, old_target) in old_targets { + outdated.extend(old_target.dependents.iter().cloned()); + if let Some(new_fragment) = new_fragments.get_mut(&id) { + new_fragment.dependents.extend(old_target.dependents); + } else { + if self.ids.get(&id).is_some_and(|owner| owner == path) { + self.ids.remove(&id); + } + if !old_target.dependents.is_empty() { + self.unresolved + .entry(id) + .or_default() + .extend(old_target.dependents); + } + } + } + for (id, old_fragment) in old_fragments { + outdated.extend(old_fragment.dependents.iter().cloned()); + if let Some(new_target) = new_targets.get_mut(&id) { + new_target.dependents.extend(old_fragment.dependents); + } else { + if self.ids.get(&id).is_some_and(|owner| owner == path) { + self.ids.remove(&id); + } + if !old_fragment.dependents.is_empty() { + self.unresolved + .entry(id) + .or_default() + .extend(old_fragment.dependents); + } + } + } + + // ### Store the new state + let entry = self.files.get_mut(path).expect("entry was created above"); + entry.metadata = metadata; + entry.status = FileStatus::UpToDate; + entry.targets = new_targets; + entry.fragments = new_fragments; + entry.xrefs = facts.xrefs.into_iter().map(|id| Xref { id }).collect(); + entry.gathers = facts + .gathers + .into_iter() + .map(|fact| GatherElement { + ids: fact.ids, + inner_html: fact.inner_html, + doc_block_index: fact.doc_block_index, }) - .clone() + .collect(); + + // ### Link the new version's outgoing references + // + // Add this file to the dependents of every id it references; ids with + // no definition go to `unresolved`. + for id in &new_refs { + if let Some(owner) = self.ids.get(id) { + let owner = owner.clone(); + if let Some(owner_entry) = self.files.get_mut(&owner) { + if let Some(target) = owner_entry.targets.get_mut(id) { + target.dependents.insert(path.to_path_buf()); + continue; + } + if let Some(fragment) = owner_entry.fragments.get_mut(id) { + fragment.dependents.insert(path.to_path_buf()); + continue; + } + } + } + self.unresolved + .entry(id.clone()) + .or_default() + .insert(path.to_path_buf()); + } + + // ### Mark outdated files + // + // A file never outdates itself: its rendered output was just produced + // from the state committed here. + outdated.remove(path); + for outdated_path in &outdated { + if let Some(outdated_entry) = self.files.get_mut(outdated_path) { + outdated_entry.status = FileStatus::Outdated; + } + } + + CommitOutcome { + outdated, + duplicates, + } + } + + /// Store a fragment's rendered content, once the caller has finalized its + /// doc blocks. If the content changed, all files containing gather + /// elements which reference the fragment are marked outdated. + pub(super) fn update_fragment_content( + &mut self, + // The file containing the fragment. + path: &Path, + // The fragment's id. + id: &str, + // The fragment's code/doc block content, rendered as HTML. + content: String, + // The files (other than `path`) marked outdated by this change; their + // status has already been set to `Outdated`. + ) -> HashSet { + let mut outdated = HashSet::new(); + if let Some(entry) = self.files.get_mut(path) + && let Some(fragment) = entry.fragments.get_mut(id) + && fragment.content != content + { + fragment.content = content; + outdated = fragment.dependents.clone(); + // Gather elements in the fragment's own file are updated by the + // caller in the same processing pass. + outdated.remove(path); + for outdated_path in &outdated { + if let Some(outdated_entry) = self.files.get_mut(outdated_path) { + outdated_entry.status = FileStatus::Outdated; + } + } + } + outdated + } + + /// Look up an id, returning the target or fragment it names along with the + /// defining file, or `Missing` if no file defines it. + pub(super) fn resolve_id(&self, id: &str) -> IdResolution<'_> { + if let Some(owner) = self.ids.get(id) + && let Some(entry) = self.files.get(owner) + { + if let Some(target) = entry.targets.get(id) { + return IdResolution::Target { + path: owner, + target, + }; + } + if let Some(fragment) = entry.fragments.get(id) { + return IdResolution::Fragment { + path: owner, + fragment, + }; + } + } + IdResolution::Missing + } + + /// Remove this file from the dependents of the given id (or from the + /// unresolved waiters, if the id has no definition). + fn unlink_reference( + &mut self, + // The referenced id. + id: &str, + // The file which contained the reference. + referrer: &Path, + ) { + if let Some(owner) = self.ids.get(id) { + let owner = owner.clone(); + if let Some(entry) = self.files.get_mut(&owner) { + if let Some(target) = entry.targets.get_mut(id) { + target.dependents.remove(referrer); + } else if let Some(fragment) = entry.fragments.get_mut(id) { + fragment.dependents.remove(referrer); + } + } + } else if let Some(waiters) = self.unresolved.get_mut(id) { + waiters.remove(referrer); + if waiters.is_empty() { + self.unresolved.remove(id); + } + } } } @@ -295,24 +749,283 @@ impl Default for Cache { } } +// Tests +// ----- #[cfg(test)] mod tests { use std::{ - borrow::BorrowMut, - collections::{HashMap, HashSet}, - hash::Hash, - sync::{Arc, Mutex, Weak}, + collections::HashSet, + path::{Path, PathBuf}, + }; + + use super::{ + Cache, DuplicateId, FileFacts, FileStatus, FragmentFact, GatherFact, IdResolution, + TargetFact, }; - use indoc::indoc; - use test_utils::prep_test_dir; + // ### Test helpers + // + // Build a target fact with unimportant location info. + fn target_fact(id: &str, inner_html: &str) -> TargetFact { + TargetFact { + id: id.to_string(), + inner_html: inner_html.to_string(), + line: 0, + doc_block_index: 0, + } + } + + // Build a fragment fact with unimportant location info. + fn fragment_fact(id: &str) -> FragmentFact { + FragmentFact { + id: id.to_string(), + line: 0, + doc_block_start_index: 0, + code_doc_block_end_index: 1, + } + } + + // Build facts for a file containing a single target. + fn facts_target(id: &str, inner_html: &str) -> FileFacts { + FileFacts { + targets: vec![target_fact(id, inner_html)], + ..Default::default() + } + } + + // Build facts for a file containing a single cross-reference. + fn facts_xref(id: &str) -> FileFacts { + FileFacts { + xrefs: vec![id.to_string()], + ..Default::default() + } + } - use crate::processing::cache::{Cache, File, FileStatus, Target}; + // Shorthand for the status of a cached file. + fn status<'a>(cache: &'a Cache, path: &Path) -> &'a FileStatus { + &cache.files[path].status + } - // Verify basic parsing + // Verify that a cross-reference to an existing target records the + // dependency and resolves. #[test] - fn test_1() { - let (temp_dir, test_dir) = prep_test_dir!(); - temp_dir.close().unwrap(); + fn test_xref_to_existing_target() { + let mut cache = Cache::new(); + let a = PathBuf::from("a.md"); + let b = PathBuf::from("b.md"); + + let outcome = cache.commit_file(&a, None, facts_target("foo", "Foo!")); + assert!(outcome.outdated.is_empty()); + assert!(outcome.duplicates.is_empty()); + + let outcome = cache.commit_file(&b, None, facts_xref("foo")); + assert!(outcome.outdated.is_empty()); + + // The target must know its dependent and resolve to its defining + // file. + let IdResolution::Target { path, target } = cache.resolve_id("foo") else { + panic!("expected a target"); + }; + assert_eq!(path, a); + assert_eq!(target.inner_html, "Foo!"); + assert_eq!(target.dependents, HashSet::from([b.clone()])); + } + + // Verify that referencing an id before its definition marks the referring + // file outdated when the definition appears. + #[test] + fn test_forward_reference() { + let mut cache = Cache::new(); + let a = PathBuf::from("a.md"); + let b = PathBuf::from("b.md"); + + cache.commit_file(&b, None, facts_xref("foo")); + assert!(matches!(cache.resolve_id("foo"), IdResolution::Missing)); + assert_eq!(cache.unresolved["foo"], HashSet::from([b.clone()])); + + // Defining the id resolves the reference: `b` must be rebuilt and + // becomes a dependent. + let outcome = cache.commit_file(&a, None, facts_target("foo", "Foo!")); + assert_eq!(outcome.outdated, HashSet::from([b.clone()])); + assert_eq!(*status(&cache, &b), FileStatus::Outdated); + assert!(cache.unresolved.is_empty()); + let IdResolution::Target { target, .. } = cache.resolve_id("foo") else { + panic!("expected a target"); + }; + assert_eq!(target.dependents, HashSet::from([b.clone()])); + } + + // Verify that only a change to a target's content outdates its + // dependents. + #[test] + fn test_target_change_outdates_dependents() { + let mut cache = Cache::new(); + let a = PathBuf::from("a.md"); + let b = PathBuf::from("b.md"); + cache.commit_file(&a, None, facts_target("foo", "Foo!")); + cache.commit_file(&b, None, facts_xref("foo")); + + // Recommitting identical content causes no rebuilds... + let outcome = cache.commit_file(&a, None, facts_target("foo", "Foo!")); + assert!(outcome.outdated.is_empty()); + assert_eq!(*status(&cache, &b), FileStatus::UpToDate); + + // ...while changed content outdates the dependent. + let outcome = cache.commit_file(&a, None, facts_target("foo", "Bar!")); + assert_eq!(outcome.outdated, HashSet::from([b.clone()])); + assert_eq!(*status(&cache, &b), FileStatus::Outdated); + } + + // Verify that deleting a target moves its dependents to `unresolved`, and + // that a later re-definition (in another file) finds them again. + #[test] + fn test_target_deletion() { + let mut cache = Cache::new(); + let a = PathBuf::from("a.md"); + let b = PathBuf::from("b.md"); + let c = PathBuf::from("c.md"); + cache.commit_file(&a, None, facts_target("foo", "Foo!")); + cache.commit_file(&b, None, facts_xref("foo")); + + // Delete the target: the dependent is rebuilt and now waits on the + // id. + let outcome = cache.commit_file(&a, None, FileFacts::default()); + assert_eq!(outcome.outdated, HashSet::from([b.clone()])); + assert!(matches!(cache.resolve_id("foo"), IdResolution::Missing)); + assert_eq!(cache.unresolved["foo"], HashSet::from([b.clone()])); + + // The id reappears in a different file: the waiter is rebuilt again. + let outcome = cache.commit_file(&c, None, facts_target("foo", "Foo!")); + assert_eq!(outcome.outdated, HashSet::from([b.clone()])); + } + + // Verify that duplicate ids are reported, with the first definition + // winning. + #[test] + fn test_duplicate_ids() { + let mut cache = Cache::new(); + let a = PathBuf::from("a.md"); + let b = PathBuf::from("b.md"); + cache.commit_file(&a, None, facts_target("foo", "Foo!")); + + // A duplicate in another file loses to the existing definition. + let outcome = cache.commit_file(&b, None, facts_target("foo", "Imposter!")); + assert_eq!( + outcome.duplicates, + vec![DuplicateId { + id: "foo".to_string(), + defined_in: a.clone() + }] + ); + let IdResolution::Target { path, target } = cache.resolve_id("foo") else { + panic!("expected a target"); + }; + assert_eq!(path, a); + assert_eq!(target.inner_html, "Foo!"); + assert!(cache.files[&b].targets.is_empty()); + + // A duplicate within a single file is reported against that file. + let outcome = cache.commit_file( + &b, + None, + FileFacts { + targets: vec![target_fact("bar", "1"), target_fact("bar", "2")], + ..Default::default() + }, + ); + assert_eq!( + outcome.duplicates, + vec![DuplicateId { + id: "bar".to_string(), + defined_in: b.clone() + }] + ); + } + + // Verify that removing a cross-reference unlinks the dependency. + #[test] + fn test_unlink_on_recommit() { + let mut cache = Cache::new(); + let a = PathBuf::from("a.md"); + let b = PathBuf::from("b.md"); + cache.commit_file(&a, None, facts_target("foo", "Foo!")); + cache.commit_file(&b, None, facts_xref("foo")); + + // Recommit `b` without the cross-reference; changing the target must + // no longer outdate `b`. + cache.commit_file(&b, None, FileFacts::default()); + let outcome = cache.commit_file(&a, None, facts_target("foo", "Bar!")); + assert!(outcome.outdated.is_empty()); + } + + // Verify the fragment/gather flow: a gather element depends on a + // fragment, and only a change to the fragment's content outdates the + // gathering file. + #[test] + fn test_fragment_gather_flow() { + let mut cache = Cache::new(); + let a = PathBuf::from("a.py"); + let b = PathBuf::from("b.py"); + + cache.commit_file( + &a, + None, + FileFacts { + fragments: vec![fragment_fact("frag")], + ..Default::default() + }, + ); + cache.commit_file( + &b, + None, + FileFacts { + gathers: vec![GatherFact { + ids: vec!["frag".to_string()], + inner_html: "Gathered".to_string(), + doc_block_index: 0, + }], + ..Default::default() + }, + ); + let IdResolution::Fragment { fragment, .. } = cache.resolve_id("frag") else { + panic!("expected a fragment"); + }; + assert_eq!(fragment.dependents, HashSet::from([b.clone()])); + + // Storing the fragment's first content outdates the gathering file; + // storing identical content afterwards does not. + let outdated = cache.update_fragment_content(&a, "frag", "content".to_string()); + assert_eq!(outdated, HashSet::from([b.clone()])); + assert_eq!(*status(&cache, &b), FileStatus::Outdated); + let outdated = cache.update_fragment_content(&a, "frag", "content".to_string()); + assert!(outdated.is_empty()); + } + + // Verify that an id changing kind (target to fragment) transfers its + // dependents and rebuilds them. + #[test] + fn test_id_changes_kind() { + let mut cache = Cache::new(); + let a = PathBuf::from("a.py"); + let b = PathBuf::from("b.py"); + cache.commit_file(&a, None, facts_target("foo", "Foo!")); + cache.commit_file(&b, None, facts_xref("foo")); + + // The id becomes a fragment: the cross-referencing file must be + // rebuilt (its cross-reference is now an error), and the dependency + // edge transfers. + let outcome = cache.commit_file( + &a, + None, + FileFacts { + fragments: vec![fragment_fact("foo")], + ..Default::default() + }, + ); + assert_eq!(outcome.outdated, HashSet::from([b.clone()])); + let IdResolution::Fragment { fragment, .. } = cache.resolve_id("foo") else { + panic!("expected a fragment"); + }; + assert_eq!(fragment.dependents, HashSet::from([b.clone()])); } } diff --git a/server/src/processing/tests.rs b/server/src/processing/tests.rs index 82ef0081..2ee3431b 100644 --- a/server/src/processing/tests.rs +++ b/server/src/processing/tests.rs @@ -788,12 +788,12 @@ fn test_source_to_codechat_for_web_1() { // 1 "# ), - &"cpp".to_string(), + Path::new("foo.cpp"), 0.0, false, - false + None ), - Ok(TranslationResults::CodeChat(build_codechat_for_web( + Ok(build_codechat_for_web( "cpp", "\n", vec![build_codemirror_doc_block( @@ -803,7 +803,7 @@ fn test_source_to_codechat_for_web_1() { "//", r"

1" ),] - ))) + )) ); } From 29f8e58bc384d0e2ee1614bc6ca761163cfcdce3 Mon Sep 17 00:00:00 2001 From: "Bryan A. Jones" Date: Fri, 7 Aug 2026 15:10:34 -0500 Subject: [PATCH 04/22] Fixme: bug for later. --- client/src/CodeMirror-integration.mts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/client/src/CodeMirror-integration.mts b/client/src/CodeMirror-integration.mts index 8b479561..a016d011 100644 --- a/client/src/CodeMirror-integration.mts +++ b/client/src/CodeMirror-integration.mts @@ -1357,7 +1357,7 @@ export const DocBlockPlugin = ViewPlugin.fromClass( resolve(), ); // Untypeset math in the old doc block and the current - // doc block before moving its contents around. + // doc block before moving its contents around. TODO: `tinymceDiv === null` in production at least once. const tinymceDiv = document.getElementById(TINYMCE_INST)!; mathJaxUnTypeset(tinymceDiv); From 7a5c716ef41230c5d34adbe31bd62add57a6c914 Mon Sep 17 00:00:00 2001 From: "Bryan A. Jones" Date: Fri, 7 Aug 2026 15:11:31 -0500 Subject: [PATCH 05/22] Clean: remove stray character. --- CLAUDE.md | 1 - 1 file changed, 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 2c897d9d..5876c484 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -62,7 +62,6 @@ comprehensibility of the code. Guidelines for comments: value in the data structure should be preceded by a command explaining its role. * Comments in the code should be limited to those that: -* 1. Document a connection which cannot easily be determined by inspection -- for example, explaining the relationship between a web client Ajax call and the backend server which handles it. From 118c107900af667eba83d43358c1dc3f5d4a5f95 Mon Sep 17 00:00:00 2001 From: "Bryan A. Jones" Date: Fri, 7 Aug 2026 15:12:11 -0500 Subject: [PATCH 06/22] wip: cache redesign. --- client/src/tinymce-config.mts | 6 +- server/src/processing.rs | 105 ++++---- server/src/processing/cache.rs | 451 +++++++++++++++++++-------------- 3 files changed, 317 insertions(+), 245 deletions(-) diff --git a/client/src/tinymce-config.mts b/client/src/tinymce-config.mts index bb891605..f3b6dd49 100644 --- a/client/src/tinymce-config.mts +++ b/client/src/tinymce-config.mts @@ -143,8 +143,10 @@ export const init = async ( "bold italic underline codeformat | quicklink h2 h3", // Needed to allow custom elements. - extended_valid_elements: "graphviz-graph[scale],wc-mermaid", - custom_elements: "graphviz-graph,wc-mermaid", + extended_valid_elements: + "graphviz-graph[scale],wc-mermaid,xref[contenteditable|ref],fragment[contenteditable|id]", + // Per the [docs](https://www.tiny.cloud/docs/tinymce/latest/content-filtering/#custom_elements), `~` marks tags as an inline element, not a block element. + custom_elements: "graphviz-graph,wc-mermaid,~xref,~fragment", }, ); diff --git a/server/src/processing.rs b/server/src/processing.rs index 60821b2b..b71f28fa 100644 --- a/server/src/processing.rs +++ b/server/src/processing.rs @@ -251,8 +251,8 @@ pub enum TranslationResultsString { /// Match the lexer directive in a source file. static LEXER_DIRECTIVE: LazyLock = LazyLock::new(|| Regex::new(r"CodeChat Editor lexer: (\w+)").unwrap()); -/// If this matches, it means an unterminated fenced code block. This should -/// be replaced with the `

` terminator. +/// If this matches, it means an unterminated fenced code block. This should be +/// replaced with the `
` terminator. static DOC_BLOCK_SEPARATOR_BROKEN_FENCE: LazyLock = LazyLock::new(|| { Regex::new(concat!( // Allow the `.` wildcard to match newlines. @@ -953,8 +953,9 @@ pub fn source_to_codechat_for_web( } }) // Add the doc block separator string between each doc block; - // the separator contains the index of this doc block. + // the separator contains the index of this doc block in the vec of code/doc blocks. .fold(String::new(), |mut acc: String, x: (usize, &str)| { + // TODO: why are we skipping empty doc blocks here? This seems incorrect. Remove this and run tests. if !acc.is_empty() { acc.push_str(&DOC_BLOCK_SEPARATOR_STRING.replace("{}", &x.0.to_string())); } @@ -1033,38 +1034,46 @@ static MINIFY_OPTIONS: LazyLock = LazyLock::new(|| { cfg }); -// A static config for Ammonia. +// A static config for Ammonia. TODO: additional updates based on cache spec. static AMMONIA_OPTIONS: LazyLock = LazyLock::new(|| { let mut b = Builder::default(); // Add custom tags produced during hydration, plus `input` (task list // checkboxes produced by pulldown-cmark) and `iframe` (embedded media // inserted via TinyMCE), neither of which Ammonia allows by default. - b.add_tags(&["wc-mermaid", "graphviz-graph", "input", "iframe"]) - // Allow any element to be assigned an ID. - .add_generic_attributes(&["id"]) - // This allows math produced by pulldown-cmark and updated by the - // hydration code. - .add_allowed_classes( - "span", - &["math", "math-inline", "math-display", "mceNonEditable"], - ) - // `code` tags can have `class=language-*`. Since Ammonia doesn't - // support a regex like this, just allow anything. - .add_tag_attributes("code", &["class"]) - // Task list checkboxes are rendered as ``. - .add_tag_attributes("input", &["type", "checked", "disabled"]) - // Allow the attributes TinyMCE/the IDE place on embedded `