Repository navigation
Conversation
Proposes RFC `610.0000` (`Pachunk`), an in-place content-addressed installer and delta updater for the Flutter SDK.
|
I'm currently working on getting the prototype in a github repo to share. |
| ### 5.2 Open Questions for Reviewers | ||
|
|
||
| 1. How should `pachunk` integrate with `flutter_tools` and the official install/upgrade flow? | ||
| * Option A (Recommended): Make `pachunk` the primary installation method on `flutter.dev` so `flutter` "Just Works" out of the box. Because `flutter_tools` already uses `flutter precache` to download engine and tool artifacts today, have `flutter precache`, `flutter upgrade`, and `flutter downgrade` delegate to `pachunk` under the hood (migrating artifact download responsibility into `pachunk` over time) with an automatic fallback to legacy `.zip` archive downloads during transition. |
There was a problem hiding this comment.
For both of these options it's not clear to me from the descriptions:
- What does the user download from flutter.dev initially by clicking on a link.
- What are they missing at that point, and how do they get the rest.
Asking the user to copy-paste a command is not my favorite when others do it, but would be an option as well if it is technically necessary.
Also worth noting here that we currently document an "Install with VSCode" flow that would also need to be modified, I think? https://docs.flutter.dev/install/with-vs-code
There was a problem hiding this comment.
Good questions and worth arguing here - maybe I should leave off the recommended for now and just have it as an opinion.
- What does the user download from flutter.dev initially by clicking on a link.
I think a script or command much like https://brew.sh/ or https://github.com/jtmcdole/flutter_worktree#unattended-installation.
- What are they missing at that point, and how do they get the rest
If its the script, we could have it automatically install for the host/arch. If its just a binary, they would need to run it to download and install a version.
- We could consult
release_<host>.jsonfor latest stable / beta if they use--stableor--beta - We could offer code block copy-paste like brew for the different components.
VSCode: noted! I... assume devtools just downloads the sdk and didn't invent a new method of downloading Flutter? @kenzieschmoll
There was a problem hiding this comment.
Added a section on what it could look like to the phased rollout plan. I believe the details of this can be left to the infra and tooling team to answer. Still want to hear from @kenzieschmoll for devtools.
There was a problem hiding this comment.
DevTools is shipped as a binary in the Dart SDK, so it doesn't do any downloading of Flutter at all.
There was a problem hiding this comment.
For the install with VS Code flow, the Dart-Code VS Code extension currently runs a git clone to install the Flutter SDK: https://github.com/Dart-Code/Dart-Code/blob/master/src/extension/sdk/utils.ts#L270
There was a problem hiding this comment.
CC @DanTup. Implementing this proposal would require some changes to the Dart-Code extension for how the Flutter SDK is installed.
There was a problem hiding this comment.
Thanks for the heads up. If the end result is a single copy/pastable command (for non-IDE users), it should be pretty trivial for Dart-Code to just do that instead of a git clone.
I did once look at downloading the archive version of Flutter instead of git clone, but I think there were some complications around what unzip tools were pre-installed across all supported Windows and Linux distros 🙃
- Remove "exact" numbers, e.g. 10,183 files, from all over the RFC - use relative numbers bigger: - address prototype changes (format of packfile, techdebt, pointer plane)
justinmc
left a comment
justinmc
left a comment
There was a problem hiding this comment.
This would save a lot of people a lot of time! Sounds great if we can pull it off.
One thing we should keep an eye on as we change this tooling is reliability, like the ability to recover from failed partial downloads, weird states, etc. Even the current tooling has bitten me many times here.
Also FYI the PR description says "610.0001" but everything else says "610.0000".
| 3. Is the Android team open to stripping non-alloc DWARF `.debug_*` sections from `libflutter.so` inside default `flutter.jar` builds? | ||
| * Today, unstripped DWARF `.debug_*` sections make up over 90% of `libflutter.so` inside `flutter.jar` (`android-arm64/artifacts.zip` is `~170 MB` compressed), even though `symbols.zip` already archives debug symbols separately. | ||
| * Recommended: Strip `.debug_*` from default `flutter.jar` artifacts and expose granular per-platform/per-ABI `--include-debug-symbols` flags in `pachunk`. |
There was a problem hiding this comment.
I have a PR that I'm going to put into draft (or should already be draft) that was experimenting with some of the engine flags for compiling as well. I will caveat that I'm not an expert on any one system or its flags.
|
|
||
| ### 5.2 Open Questions for Reviewers | ||
|
|
||
| 1. How should `pachunk` integrate with `flutter_tools` and the official install/upgrade flow? |
There was a problem hiding this comment.
I think there is actually a lot of details here to discuss for a long-term plan, and I'm not sure we are going to know the exact shape of this until we start experimenting with it. I think the first step is to use pachunk within the flutter tool to download artifacts just to actually kick the tires and work out any issues with it. But I do agree that eventually the goal would be for it to make it so that end-users don't need a full git checkout.
From my point of view, I think I eventually see pachunk as just part of the flutter tool, essentially. We will still need a flutter/dart bash script (or flutter.bat/dart.bat for windows) that bootstraps the whole downloading process. I think we should consider making the flutter tool itself a per-platform AOT-compiled binary (which will also include pachunk), and the bootstrapping bash scripts will do a simple HTTP get of this binary if it doesn't exist. This is fine because there is no real benefit to using pachunk on initial download.
But this means that the flutter tool (and pachunk by extension) need to learn to be self-updating. The bootstrapping scripts should invoke the flutter tool if it is there, and the flutter tool needs to be able to determine if it needs updating, and then download its own update via pachunk and then die and relaunch as the new binary.
There's probably a lot more nitty-gritty details to get into, but I think eventually this is the direction I see this sort of thing going. That means that a user starting from scratch basically just starts with the bootstrapping script rather than doing a git checkout or downloading a full SDK archive from flutter.dev as they would today.
There was a problem hiding this comment.
lot of details here to discuss
Indeed, open questions are open.
first step is to use
pachunkwithin the flutter tool to download artifacts
You are (or I am) in luck; pachunk the prototype already supports --cache-only mode which downloads binaries only. That's certainly one way to kick the tires, but not the only one we could start with (e.g. building pachunk as releases on github).
pachunk as just part of the flutter tool
I do not. Its generic enough to serve a purpose outside of the flutter tool.
We will still need a flutter/dart bash script (or flutter.bat/dart.bat for windows) that bootstraps the whole downloading process
We actually do not need that if pachunk is distributed outside of the tool. It will already have downloaded the correct things. Right now I have pachunk building the AOT snapshot, but I could make it do the executable.
bootstrapping bash scripts will do a simple HTTP get of this binary if it doesn't exist
Indeed, it could do the downloading of the release from github or wherever we compile and distribute it. Do we want to do that for every invocation of flutter.bat/dart.bat?
This is fine because there is no real benefit to using pachunk on initial download
I don't follow this line of thinking. You mean from downloading of the SDK from the Flutter Archives vs git clone; flutter --version?
But this means that the flutter tool (and pachunk by extension) need to learn to be self-updating. The bootstrapping scripts should invoke the flutter tool if it is there, and the flutter tool needs to be able to determine if it needs updating, and then download its own update via pachunk and then die and relaunch as the new binary.
Flutter tool could just know what version to download and then ask Pachunk to download it. That relationship could be one way rather than circular.
basically just starts with the bootstrapping script
I thought I called out "brew.sh" as an example in here, but that could be one possibility.
There was a problem hiding this comment.
But this means that the flutter tool (and pachunk by extension) need to learn to be self-updating. The bootstrapping scripts should invoke the flutter tool if it is there, and the flutter tool needs to be able to determine if it needs updating, and then download its own update via pachunk and then die and relaunch as the new binary.
In a way, the tool is already self-updating. Performing a flutter upgrade downloads the updated artifacts, at which point the tool deletes some files to indicate it should be rebuilt and then invokes flutter again to trigger the rebuild. As long as pachunk is updated as part of the artifact download step, this should continue to work as is.
There was a problem hiding this comment.
@jtmcdole wrote:
pachunk as just part of the flutter tool
I do not. Its generic enough to serve a purpose outside of the flutter tool.
To clarify, I do agree that pachunk itself should be standalone and not have any flutter-specific concerns built into it. What I mean is that from the point of view of the end user, the fact that the flutter tool uses pachunk is just an implementation detail. That is to say, I wouldn't want the user story for flutter installation to be "first download pachunk, and then use pachunk to download the flutter tool, and then make sure that pachunk is in your path or in a specific location so that the flutter tool can invoke it" or whatever. The fact that pachunk is part of the process should more or less be transparent to the user.
We will still need a flutter/dart bash script (or flutter.bat/dart.bat for windows) that bootstraps the whole downloading process
We actually do not need that if pachunk is distributed outside of the tool. It will already have downloaded the correct things. Right now I have pachunk building the AOT snapshot, but I could make it do the executable.
What I mean is that whatever happens, we still need a script that actually runs when you type flutter or dart on the command line. The hope is that the bootstrapping scripts could be greatly simplified though. All the logic about deciding what artifacts to download at what hash and so on could be pulled into the flutter tool itself, if all the bootstrapping script does is either invoke an existing prebuilt flutter tool snapshot or download one with a simple HTTP request.
This is fine because there is no real benefit to using pachunk on initial download
I don't follow this line of thinking. You mean from downloading of the SDK from the Flutter Archives vs git clone; flutter --version?
I'm saying that if you have a fresh git checkout or a "bare" end-user flutter install (like you're suggesting in this doc) but you haven't actually downloaded any artifacts at all, then you're not doing any differential updates, and so pachunk isn't going to save anything over a simple HTTP get. So in that scenario, the bootstrapping script can just grab the flutter tool AOT artifact (and any other pachunk bits that the flutter tool uses as a dependency) via a normal HTTP request and not really lose out on any efficiency.
There was a problem hiding this comment.
The fact that pachunk is part of the process should more or less be transparent to the user.
Agreed; but how that happens is an open question:
- Where is it stored.
- Does it update itself while running
- Does flutter just call it or do we use ffi.
we still need a script that actually runs when you type flutter or dart
If we only need "flutter.bat" so flutter autocompletes on the windows commandline, sure. The bootstrapping we currently do is a bear to work with and we (I) get it wrong from time to time. I'm 100% down to compile the flutter tool locally after updates.
I'm saying that if you have a fresh git checkout or a "bare" end-user flutter install (like you're suggesting in this doc) but you haven't actually downloaded any artifacts at all, then you're not doing any differential updates, and so pachunk isn't going to save anything over a simple HTTP get
You're always going to have to start from somewhere, so yes a download is going to be requried. Though I would say "bare user install" (sans git) automatically comes with the binaries if you're using pachunk up front (e.g. brew.sh install that does magic for the user). Pachunk does download less than a 2.2GB sdk archive since its deduplicating.
|
to be clear: I "WANT_LGTM" from all the reviewers eventually - no plans on jumping the queue. |
8 participants
Proposes RFC
610.0001pachunk(from "patch + chunk"), an in-place content-addressed installer and delta updater for the Flutter SDK.Flutter distributes SDK releases and engine artifacts through full-history
gitclones and monolithic archives (.zipand.tar.xz) on GCS. A fresh release archive unpacks at least 2.5 GB on Linux (1.5 GBarchive), 3.3 GB on Windows (1.9 GBarchive), and 4.1 GB on macOS ARM64 (2.2 GBarchive; 6.24 GB onceflutter precachepopulatesbin/cache/)...What if delta-upgrades (3.47.5 -> 3.47.6) were only 50MB and 20 seconds?
What if you could upgrade/downgrade from any version to any version?
What if Flutter had a dedicated installer?
Tracking issue: flutter/flutter#194032
Pre-launch Checklist
design doc.dart run bin/assign_rfc_number.dartlocally when instructed).