Monitor and handle the files your partners send you, on local folders, S3 or SFTP.
Data teams in finance, logistics and retail still depend on files delivered by other companies every business day. The recurring questions are always the same: Did every partner deliver today? Which job failed overnight? Copy these files from the partner's bucket to ours. Find the return files that mention these accounts. Unzip what arrived. dropkit answers them with one configuration format and one storage abstraction.
$ dropkit watch watch.yaml
## 2 issue(s) found - 18/02/2026
**daily-portfolio-files**: 41 OK, 1 issue(s)
- ⚠️ **FUND_C** LATE: no file dated 13/02, 18/02 (latest: `FUND_C/stock/stock_20260211.csv`, 11/02/2026 19:02)
**failed-jobs**: 0 OK, 1 issue(s)
- 🚨 **load_positions** ALERT: load_positions failed with code 3: timeout reading partner file
Note the date window above: 18/02/2026 is Ash Wednesday, so on the B3 calendar "D-1" is Friday 13/02, not Tuesday (Carnival).
| Command | Problem it solves |
|---|---|
dropkit watch config.yaml |
Delivery SLAs. Each partner folder (auto-discovered) must have a file dated within N business days, with per-partner tolerances. Job alerts: a SQL query whose new rows (since the last run) are sent once. Reports go to Teams, Slack, e-mail, any webhook or the console. |
dropkit transfer SRC DEST |
Copy between any two storages (S3 → S3 across accounts, SFTP → S3…). Idempotent: files already there with the same size are skipped. |
dropkit find LOCATION --contains … |
Search by name fragments (or a CSV of them) and a modification window, e.g. "return files from June mentioning these 300 accounts". |
dropkit unzip SRC DEST |
Extract each new archive exactly once, with the password from an env var. Half-extracted folders never become visible. |
dropkit rename DIR --match RX --to REPL |
Bulk regex rename that plans first and refuses collisions and overwrites. |
dropkit consolidate "GLOB" -o OUT |
Stack many CSVs (decimal comma supported), aggregate, and write one file per group, with a source_file column for traceability. |
flowchart LR
subgraph Storage["Storage interface"]
L[LocalStorage]
S[S3Storage]
F[SFTPStorage]
end
W[watch: FileDeliveryCheck] --> Storage
T[transfer / find] --> Storage
Q[watch: SqlCheck] --> DB[(any SQLAlchemy DB)]
W & Q --> R[runner: one report] --> N[Notifiers: console · webhook/Teams/Slack · e-mail]
C[(YAML + env vars)] --> W & Q & N
- One storage interface (
list,list_dirs,read_bytes,write_bytes) behind every feature. Supporting GCS or Azure means adding one class, without touching the features. - Optional extras keep the core light:
pip install "dropkit[s3,sftp,sql]". - Secrets only from the environment.
${VAR}in the YAML; SFTP keys via env vars; boto3 profiles or instance roles for S3. A missing variable fails loudly. - Strict configuration. An unknown key (
max_lgo: 1) is an error, not a silent default. - A monitor must never go silent. A check that crashes (expired credentials, server
down) appears in the report as ERROR, and one broken notification channel doesn't
stop the others. Exit code
1= needs attention, so it plugs into cron, Task Scheduler, Airflow or GitHub Actions. - At-most-once alerts for SQL. The watermark is persisted atomically and keeps the column's own type (text timestamps compare as text, datetimes as datetimes).
- Safe file operations. Atomic writes, idempotent transfers and extractions, and renames planned before they run.
pip install -e ".[s3,sftp,sql]"
dropkit watch examples/watch.yaml --dry-run # see what would be sent
dropkit transfer s3://their-bucket/exports?profile=partner s3://our-bucket/landing --pattern "*.csv" --dry-run
dropkit find sftp://ops@drop.example.com/returns --contains-file accounts.csv --since 2026-06-01 -o found.csv
dropkit unzip ./inbox ./extracted --password-env PARTNER_ZIP_PASSWORD --exclude "RENEG_*"
dropkit rename ./downloads --match "^(.*)\.ret_\d+$" --to "\1.ret" --dry-run
dropkit consolidate "exports/*.csv" --sep ";" --decimal "," --group-by REF_DATE BUCKET --sum PV PROVISION --split-by REF_DATE -o out/See examples/watch.yaml for a full monitoring config: file SLAs on
S3 and SFTP, a failed-jobs SQL alert, and Teams/Slack notifiers.
pytest covers the business-day calendar (including Carnival), every delivery status
(OK, LATE, MISSING, ignored groups, per-group overrides, dates inside folder names), SQL
alerts that fire once, notifier body templates, config validation, S3 (mocked with
moto), and all file operations, including corrupt archives and rename collisions.
MIT