Skip to content

2026-10-29 Engineering Team #154

Open

Description

@janzill

Agenda

Reminder: Phase 12 moonshot workshop is happening next Tuesday.

This meeting: Phase 11 close out, towards release.

ActivitySim

Discussion requested: estimation notebooks in CI

  • Stefan and Sijia asked to discuss #1136 Park-and-ride lot choice estimation (draft). It fails CI because its new notebook has no matching estimation data. The telework notebooks for #1126 hit the same problem. Sijia's questions:
    1. Does every new component need an example estimation notebook, even one that only calls standard Larch methods?
    2. If so, must the notebook run in CI?
    3. If so, add estimation data to the MTC example used by CI, or build a SANDAG set?

Issues

Open PRs

  • #1127 Python versions. PR review done, changes requested and applied, will be merged once approved.
  • #1130 Skip failed sampling choices. PR review done, changes requested and applied, will be merged once approved.
  • #1125 Summarize memory reduction. Jeff to add abench results.
  • #1107 Calibration. Jan reviewing. PSRC and MWCOG have tested it. Agency acceptance contact: Joe C.
  • #1126 Explicit telecommute. Joe F. reviewed on 23 Oct with minor comments for Sijia. Agency reviewer and acceptance criteria still to confirm (Joe C. with Sijia). Estimation notebooks: see the CI discussion above.
  • #1110 Peak-memory reduction. Jeff addressed review comments. MTC full-scale runs on Windows done, question regarding memory release on Windows but otherwise good to go.
  • #1086 Application and Analysis Guide. Still a draft, Joe F. waiting for consolidated feedback?
  • #1103 Faster RNG. On hold. David posted SANDAG results on 22 Oct: with full EET and no Poisson sampling, disaggregate accessibility runtime roughly halves but stays slower than legacy Monte Carlo (and therefore Poisson sampling). He argues for keeping the code for future full-EET work, such as system-wide shadow pricing. Jeff to try to reproduce with EET benchmarks. Defer to Phase 12?
  • Python 3.14 PR on outerloop fork, blocked by Sharrow #89 and Larch #21. PRs open, need reviewer.

PopulationSim

Activity

  1. janzill commented on Nov 1, 2026

    @janzill
    Author

    Meeting Notes

    Meeting Overview

    Participants

    Duration

    50 minutes

    Executive Summary

    The last engineering meeting before the Phase 12 moonshot workshop, held as Phase 11 closes out towards a release. People invited to join the bench were added to the meeting series this week, and Tim Veitch and Billy Charlton attended for the first time.

    Most of the meeting concerned estimation notebooks for new components. Draft #1136 (park-and-ride lot choice estimation) fails CI because the fake estimation survey data, derived from the MTC example, does not cover the component; the telework notebooks for #1126 were moved outside CI for the same reason. Sijia questioned the value of near-identical notebooks for simple MNL components. Jan and Amir favoured a notebook for every component, Amir stressing structured tests in notebooks over bare estimation calls. David noted that every component added for SANDAG lacks estimation coverage. Coverage, and whether to build SANDAG-based test data, was tabled for Phase 12; no decision was made on how #1136 should pass CI in the meantime.

    Elsewhere: the faster-RNG pull request was deferred to Phase 12 and will not be merged now; PopulationSim intermediate geographies was agreed for merge, with the release-tooling update to land before the end-of-phase PopulationSim release; the Sharrow 2.16 skim-loading slowdown may be addressed by the new h5py-based OpenMatrix release rather than by duplicating code in Sharrow; and Amir's memo on alpha and beta releases and structured testing is to go on the next agenda. RSG believes the PopulationSim weight-cap issue may be a misunderstanding of the setting and will post its findings.

    Meeting Notes

    Meeting Format and New Participants

    • Joe C has added the people invited to join the bench to this meeting series, ahead of contracting being in place. Tim Veitch (Veitch Lister Consulting, Australia) and Billy Charlton (Berlin) attended for the first time.
    • Jan outlined how the meetings run. Jeff and Jan post the agenda shortly beforehand as an issue in ActivitySim/meeting-notes, and an AI-generated, human-checked summary is posted to the same issue afterwards. Meeting content is public. Joe C manages the recordings, and access is arranged through him.

    Phase 12 Moonshot Workshop

    • The Phase 12 moonshot workshop is next Tuesday (Wednesday morning in Australia, 7 or 8 a.m. depending on location). Jeff has shared a Word document through Google Drive, which has also been sent to TfNSW.
    • Joe C will send out a survey to help ground the conversation, with responses requested by the end of Monday, followed by a plan for getting the most out of the workshop.

    Estimation Notebooks in CI

    • #1136 (draft) adds estimation support to park-and-ride lot choice, a component funded by TransLink and implemented by RSG, mostly by David. PSRC has implemented the component in its model, added park-and-ride lot choices from its survey to its estimation inputs, and made minor code changes so that estimation data bundles are created for it. Stefan considers David the right reviewer.
    • The PR fails CI because it adds an example notebook, which expects data bundles that CI cannot build: the MTC survey files used in CI do not support park-and-ride lot choice. Stefan saw two options — skip the test for now and add estimation testing later, or fold it in now, which requires adding the component to the CI survey data.
    • Sijia hit the same problem with the two telework notebooks (arrangement and duration) for #1126, which she has committed to a separate folder outside CI. She posed three questions: whether every new component needs an example notebook; if so, whether it must run in CI; and if so, whether to add data to the MTC example or build a SANDAG set. In her view the park-and-ride notebook, with substantial bespoke code, is worth keeping, but the telework notebooks are near-identical to existing simple MNL notebooks that only call Larch APIs and add little. Many existing components also lack notebooks.
    • David described current CI: output from the MTC example is converted into fake estimation survey data, with no real survey data included; CI builds estimation data bundles from it and runs the existing notebooks on them, and that is the full extent of estimation testing. The data is very clean and covers only MTC components. Everything added for the SANDAG model but absent from MTC lacks coverage, including school escorting, the joint tour frequency-composition model, and parking location choice (distinct from park-and-ride lot choice). Full coverage was an early goal, which is why all MTC components are included, but components were later added without estimation support as priorities moved elsewhere. He sees whether to build a SANDAG set as a Phase 12 prioritisation question for the consortium, and suggested any such work also consider how typical survey-data gaps are handled — for example, a worker who does not go to work and reports no workplace location.
    • Jan finds from experience that an example notebook for every component helps new users, even where it is near-identical to another, because the system then appears complete. Fake data also allows a quick schema comparison against real bundle output. If the effort of creating fake survey data is relatively low, he would support adding tests. MTC extended, which exercises functionality rather than serving as a real example model, could carry fake data for each new component if the aim is to extend it with every addition. He noted that the workplace-location gap also arises in several Australian jurisdictions, where work and school locations are not reported unless visited.
    • Amir favours separate notebooks for each component in the consortium repository even where the procedures are similar, and considers structured tests within notebooks more valuable than a single cell that runs estimation and returns parameters. TfNSW built standard tests for each component during its estimation work with WSP and RSG, but strict TfNSW code-sharing policies currently prevent sharing them; Shaun Tabone is pursuing a way to share code and findings with partners.
    • Estimation coverage was tabled for Phase 12, with input to be sought from users. No decision was made on how #1136 or the #1126 notebooks should handle CI in the meantime.

    Slower OMX Skim Loading in Sharrow 2.16

    • Sharrow #90: SANDAG reports a considerable slowdown in skim loading with Sharrow 2.16. Jeff's open Sharrow #87 reads OMX files directly with h5py; Jan has not yet reviewed it. From experience with AequilibraE, Jan considers the OpenMatrix package slow at reading, and the straightforward fix to be bypassing it and reading with h5py directly. He asked whether SANDAG considers the review a priority.
    • Pedro said the OpenMatrix Python package has recently been rewritten to use h5py in place of PyTables. That may resolve the slowdown, unless the cause lies in OpenMatrix's own mechanics rather than the HDF5 driver, and he suggested trying it first to avoid duplicating code across packages. Jan agreed with the principle and will test the updated package locally.
    • Billy said the rewrite is listed as beta but is ready to release as version 0.4. Anyone needing the old version can pin it, and he does not believe anyone else depends on it. He will remove the beta designation.

    Python Version Support

    • #1127 Python versions moves CI to Python 3.11, 3.12 and 3.13. Python 3.10, ActivitySim's current main version, has reached end of life, with security updates having stopped in October 2026. The PR also includes package changes prompted by the recent certificate-authority problems. Jeff reviewed it and requested one change, a leftover Python 3.10 reference, which Jan has fixed; it is expected to merge soon.
    • Python 3.14 support is on the Outer Loop fork only, because Sharrow and Larch were not 3.14-ready. Jan has opened Sharrow #89 and Larch #21. Larch took more work: it relied on an old SciPy version, and a SciPy solver used for nested logit models has had its back end rewritten and moved from 32-bit to 64-bit floats, which caused problems in its linkage to Larch. That is now implemented. Jan considers Jeff the reviewer for both; Jeff was not present, so this will be picked up at a later meeting.

    Remaining ActivitySim Issues and Pull Requests

    • #999 (resume_after fails on write_data_dictionary with multiprocessing): Sijia has not yet looked at it and will do so when time allows. It is not urgent and can be backlogged.
    • #1130 Skip failed sampling choices, reported by Amir: Jan implemented it and Jeff reviewed it with minor comments. Jan needs to make the PR description more descriptive; it will come in once Jeff approves it.
    • #1125 Summarize memory reduction: Sijia's change reduces memory use in the final summarize step, which runs non-parallel. It has no performance testing yet; Jeff is to add abench results in the coming days.
    • abench is a repository in the ActivitySim organisation, added by Jeff, providing tooling for reproducible ActivitySim runs in a Linux Docker container that record memory and runtime. Jeff has AI tooling that can be pointed at a PR to post before-and-after comparisons automatically, guarding against runtime and memory regressions.
    • #1107 Calibration: Jan has started the review, which is considerable, and expects to finalise review and testing in the next few days. So far it looks good, with minor comments. PSRC and MWCOG have tested it, and Jan expects it to reach main soon, subject to agency approval.
    • #1086 Application and Analysis Guide: Joe F. was not present. Consolidated feedback is still awaited, coordinated by Joe C.

    Peak-Memory Reduction

    • #1110: Jeff has addressed the review comments. Jan's MTC full-scale runs on Windows, testing several options, finished on the morning of the meeting, and he will post the results to the PR in the next days. A small question remains about memory release on Windows, which may need brief discussion, but the PR is otherwise good to go.
    • Chunking trades memory for runtime: because choosers in a discrete choice model are independent, a large chooser table can be cut into n slices and processed sequentially. Some parts of ActivitySim's computation sat outside the chunking loop, so memory could not be pushed below a certain floor whatever the chunk size. #1110 brings those parts into the loop.
    • The work came out of the Boston model, where a model with several thousand zones was run on 64 GB. It is not wired into every model; trip destination choice is the main beneficiary, where memory use falls considerably at the cost of runtime. Large-scale tests showed no effect on unchunked runs and reasonable behaviour on chunked runs.

    Faster RNG Deferred to Phase 12

    • #1103: Jan summarised David's SANDAG results. Using EET as a sampling engine helps runtime considerably but remains nowhere near Poisson sampling. Earlier tests showed EET sampling and Poisson sampling to have similar levels of noise, so in Jan's view there is no good reason to use EET sampling.
    • David's case for keeping the code is future work that needs many error terms, such as shadow pricing. Location choice with around 3,000 alternatives and importance sampling with replacement (30 draws is common in practice) requires a large number of error terms. Current shadow pricing reallocates workers away from over-subscribed zones at random, which has produced outliers where people without good alternatives are pushed to very poor options over long distances. Reallocating instead the people who lose the least utility — an idea Joel Freedman raised during the Phase 10 EET work — would keep the error terms and resembles the optimisation algorithms of constrained choice models.
    • David agreed with that summary. The question may be moot depending on the moonshot discussion and what the random number generator looks like in ActivitySim 2.0. He sees no need to merge into main now and no urgency, and since the changes are isolated deep in the random number code he does not expect many merge conflicts in the meantime.
    • Jan noted that keeping a PR open carries technical-debt risk as main moves on, but that a counter-based RNG of the kind Jeff implemented would also be needed for consistent Poisson sampling with two-zone systems and pre-sampling, as found during last phase's inconsistency investigation.
    • The PR was deferred to Phase 12, to be considered as part of a possible wider rewrite of the computational engine, and will not be merged now.

    PopulationSim Release Work

    • #225 Intermediate geographies was originally Joe F's work. Nick took it over and added CI and cross-platform solver stabilisation, which were then split into separate PRs. Joe F reviewed and approved it. With Joe F absent, Jan relied on that approval and his own quick look, and it was agreed to merge.
    • A PopulationSim release is planned for the end of Phase 11 in the coming weeks, and #226 Docs, CI and release modernisation should land first. The current release is 0.10, but the documentation link leads to an old 0.5 version, and even the newest documentation is out of date on uv and related tooling. The PR updates the documentation, automates its publishing, and publishes releases directly to PyPI so that PopulationSim can be installed with pip or uv, in line with ActivitySim. It should not change functionality.
    • Jeff said he would arrange an AI review of #226. It had not arrived, so the PR is waiting on it.
    • Dependabot PRs #227 and #223: Jan will check whether they agree with the uv lock file in #226, then either close them or update #226.

    Release and Testing Practice

    • Phase 11 will end with a new release. Release practice is currently ad hoc, and the release strategy going forward is a discussion point once the new bench members are on board.
    • Amir drafted a memo on alpha and beta releases and structured testing and sent it to Joe C a couple of weeks ago. It bears on how PRs are reviewed and merged into main and what systematic checks apply. Jan had not seen it.
    • Jan agreed that development practice needs formalising as more people contribute and adoption widens. Releases currently depend on one person running the model on a very large machine and checking results partly by hand, which misses things, and there is no large-scale testing outside releases. With stochastic models, individual choice changes may not matter so long as results remain statistically valid. Open questions include periodic full-scale runs and their scheduling, whether a dedicated machine is needed and at what cost, and automated statistical checks of result differences. The current plan is to take this up once the new bench is assembled, early in Phase 12.
    • Amir asked for the item to go on the next meeting's agenda, with a follow-up with Joe C, who holds the memo. Jan agreed.

    PopulationSim Household Weight Cap

    • PopulationSim #221: Amir asked about progress on the maximum household weight not being enforced. David said RSG has looked into it and thinks it may be a misunderstanding of what the setting means; he will summarise RSG's findings on the issue and ping Amir and Nick. Amir has added numbers to the issue description.

    Action Items

    • Joe C (@joecastiglione) will send out the pre-workshop survey, with responses requested by the end of Monday, and a plan for the Phase 12 moonshot workshop.
    • Billy (@billyc) will remove the beta designation from the h5py-based OpenMatrix Python package and release it as version 0.4.
    • Jan (@janzill) will test the h5py-based OpenMatrix release against the skim-loading slowdown in Sharrow #90.
    • Sijia (@i-am-sijia) will look at #999 when time allows.
    • Jan (@janzill) will update the description of #1130.
    • Jeff (@jpn--) will add abench results to #1125.
    • Jan (@janzill) will finalise the review and testing of calibration #1107.
    • Jan (@janzill) will post the MTC full-scale Windows results to #1110.
    • Jeff (@jpn--) will arrange the AI-assisted review of PopulationSim #226.
    • Jan (@janzill) will check Dependabot PRs #227 and #223 against the uv lock file in #226, and close them or update #226.
    • Jan (@janzill) will follow up with Joe Castiglione (@joecastiglione) on Amir's release and testing memo and put it on the next engineering meeting agenda.
    • David (@dhensle) will summarise RSG's findings on PopulationSim #221 on the issue and ping Amir and Nick.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    meetingMeeting notes.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions