Many GAP packages come with large data sets. Shipping them inside the package archive makes the package huge for everybody, including the people who never touch the data. The usual alternatives are worse: telling users to download a tarball by hand and unpack it in exactly the right place, or writing a download function per package — and today essentially none of those verify what they downloaded.
ArtifactManager offers a common mechanism instead. A package declares its
artifacts — named blobs of data living on a web server — in a file
artifacts.json next to its PackageInfo.g. When the package first needs
one, ArtifactManager downloads it, verifies it against the declared SHA256
checksum, unpacks it, and hands back a directory. The data is cached where the
user wants it, and can be listed and removed again.
This addresses gap-system/gap#4285.
Status: early. The on-disk format is settled, but the feature set is not complete — see Limitations below.
gap> LoadPackage("ArtifactManager");
gap> ShowArtifacts();
package artifact status size description
------------------------------------------------
transgrp dat32 installed 312.3 MB Transitive groups of degree 32
transgrp dat48 absent (30.0 GB) Transitive groups of degree 48
total on disk: 312.3 MBUseful things to know:
RemoveArtifact("transgrp", "dat32")reclaims the space; the data is downloaded again next time it is needed.- By default the store is
~/.gap/artifacts. Set the user preferenceArtifactManager/ArtifactStore, or the environment variableARTIFACTMANAGER_STORE, to move it. On a shared machine the environment variable is the better knob, since it needs no per-user configuration. - Anything bigger than 1 GB is not downloaded automatically; you get an error
telling you to call
FetchArtifactexplicitly. Change this with theMaxAutoDownloadSizepreference. AllowDownloads := falseworks offline.OverrideArtifact("transgrp", "dat48", "/scratch/trans48")points GAP at a copy you already have. ArtifactManager will then never download, verify or delete it.ArtifactStoreDiagnostics()says what it found and what it is missing.
Add artifacts.json to your package's root directory:
{
"gapArtifactManifest": 1,
"package": "transgrp",
"artifacts": {
"dat32": {
"description": "Transitive groups of degree 32",
"version": "1.0",
"size": 314572800,
"license": "GPL-2.0-or-later",
"provenance": "https://doi.org/10.5281/zenodo.5935751",
"strip": 1,
"download": [
{ "url": "https://example.org/trans32.tar.gz", "sha256": "aaaa..." },
{ "url": "https://mirror.example.org/trans32.tar.gz", "sha256": "aaaa..." }
]
}
}
}DescribeArtifactURL("https://example.org/trans32.tar.gz") downloads the file
once and prints the stanza above with the checksum and size filled in, so you
do not have to compute them by hand.
Then, in your package code, replace whatever you did before with:
dir := ArtifactDirectory("transgrp", "dat32");
f := Filename(dir, "trans32.grp");ArtifactDirectory downloads on the first call and returns immediately after
that. It raises an error if the data cannot be provided, and the error says
what to do about it. Use IsArtifactAvailable if you would rather decide for
yourself — it never downloads and never raises.
Add ArtifactManager to NeededOtherPackages if your package cannot work
without its data, or to SuggestedOtherPackages and guard the calls with
IsPackageLoaded("artifactmanager") if the data is optional.
The full field reference is in the manual, chapter "Declaring artifacts".
- Checksums are verified before anything is unpacked, so no untrusted
bytes ever reach
tar. - Archives are listed before extraction and rejected if any member has an
absolute path or a
..component. - An artifact's directory name contains its checksum, so a changed declaration installs somewhere new. A stale cache cannot be mistaken for current data.
- Because of that, two GAP sessions installing the same artifact at once can only produce identical bytes, so no locking is needed; the loser of the race simply uses the winner's copy.
- An interrupted download leaves at most a directory under
<store>/tmp, never a half-installed artifact.CleanArtifactTemp()sweeps those up. - Sources are tried in order, so a mirror that has gone bad is skipped.
- Download backends that do not follow HTTP redirects are used only as a last resort, because a redirect silently returning an empty body is exactly how data downloads broke when servers moved from http to https.
- Artifacts up to about 1 GB. The pieces needed for very large data
(streaming downloads, resumable transfers, hashing a file without an
external helper) are missing from GAP and from the packages we build on.
Issues have been filed; see
dev/upstream.md. Until they land, something like transgrp's 30 GB degree-48 data is out of scope, and pretending otherwise would only produce a path that runs out of memory on the wrong backend and restarts from zero on any interruption. - No garbage collection yet.
RemoveArtifactandRemoveAllArtifactscover reclaiming space by hand, andShowArtifactsmarks data that no installed package asks for any more asstale. Collecting it automatically is the next piece of work; the last-use times it will need are already being recorded, andPinArtifactalready exists so that anything you pin now is safe by the time it arrives. - One artifact, one archive. Fetching individual files out of a large remote collection — what AtlasRep needs — is planned but not implemented.
- Windows support is best effort and untested.
GAP 4.13 or newer, and the utils package for downloading.
The IO package is strongly recommended. It is not required, but without it several things GAP itself cannot do — file sizes, atomic renames, reading a file as raw bytes, telling the time — have to be done by running external programs instead.
Please report issues at https://github.com/gap-packages/ArtifactManager/issues.
This package is distributed under the terms of the GNU General Public License v2.0 or later. This is also the license used by GAP itself and by many GAP packages.