Repository navigation
fix(compose): run one fetcher for every fetcher source - #373
Merged
Merged
Conversation
catinspace-au
force-pushed
the
fix/shared-fetcher
branch
from
November 12, 2026 08:07
57dbfbd to
386364a
Compare
The published install runs fixed containers without reading the engine's instance index. Console-added fetcher sources never ran. The engine merges every fetcher source into one file through apps.yaml shared_instance. This stack runs one fetcher on that file. - dfe-fetcher is a committed service on every profile with fetcher_enabled. It reads /etc/dfe/apps/dfe-fetcher/fetcher.yaml and gets no generated instances. instances.py drops its per-instance cursor directory. - DFE_ENGINE_DEPLOYMENT_HOSTED_INSTANCES names the demo's transformed sources. The engine rejects console-added transform sources when saved. The stack file names no hosted instances. - A published install has no make. DFE_ENGINE_DEPLOYMENT_CREDENTIAL_COMMAND gives the first-login password. DFE_ENGINE_DEPLOYMENT_APPLY_COMMAND recreates a service from the file's own OCI reference. - The demo declares no fetcher instance. Its fetcher waits for demo-sources so its config holds the demo sources at startup. make demo checks that fetcher and clears cursor files for its sources on demo-stop. - make sources skips seed sources the fetcher cannot take. It prints the engine's reason. Only one of the three AWS seeds runs until hyperi-io/dfe-fetcher#289 adds a keyed aws family. - single-grpc no longer offers the fetcher or the transforms. The engine runs per-source apps only for sources on Kafka. The three vrl outage scenarios move to single-kafka. - The archiver writes to /var/lib/dfe-archiver/archive, the path the dfe-archiver chart mounts by default and the engine's default destination. Needs the dfe-engine release that adds shared_instance and the three deployment settings.
catinspace-au
force-pushed
the
fix/shared-fetcher
branch
from
November 12, 2026 08:12
386364a to
663ac3b
Compare
The shared fetcher needs dfe-engine v1.22.90. The 2.2.2-rc.5 stack pins that version. The 2.2.2 stack doesn't yet. The dial template pins the candidate so the release builds its compose files against those images.
single-grpc no longer runs the transforms. The operating page says the cursor volume holds one directory for the one fetcher.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The published install runs fixed containers without reading the engine's instance index. Console-added fetcher sources never ran. The engine merges every fetcher source into one file through apps.yaml shared_instance. This stack runs one fetcher on that file.
Needs the dfe-engine release that adds shared_instance and the three deployment settings.