Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion crates/bin/escrow_manager/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,30 @@ Configuration options are set via a single JSON file. The structure of the file
| `dry_run` | If `true`, skip contract calls (useful for testing) |
| `port_metrics` | Port for Prometheus metrics server (default: 9090) |
| `update_interval_seconds` | Polling interval for the main loop |
| `balance_fill_factor` | Funding margin, in the range (0, 1] (default: 0.8) |

## Funding Margin

Each cycle, every receiver's debt is taken as the maximum of: receipts over the last 28 days, RAVs
on active allocations, and the manual `debts` floor. The escrow balance is then stepped up until debt sits below
`balance_fill_factor` of the balance, so the target settles at roughly `debt / balance_fill_factor`:

| `balance_fill_factor` | Target balance |
|---|---|
| `0.6` | ~1.67x debt |
| `0.8` (default) | ~1.25x debt |
| `0.95` | ~1.05x debt |

Steps double from 2 GRT and then grow linearly in 10,000 GRT increments, so the target only
approximates that ratio for small balances — below ~16,000 GRT the step granularity dominates and
the effective margin is wider.

Lowering the margin frees capital but leaves less headroom to absorb query volume between cycles.
Note that total deposits are capped at 10,000 GRT per cycle (`MAX_ADJUSTMENT`), which bounds how
fast a thin margin can be refilled. **The manager never withdraws**, so raising
`balance_fill_factor` only affects receivers still being topped up — balances already above their
target drain only as receivers collect. Validate changes with `dry_run: true` first, and watch
`escrow_target_grt` against `escrow_balance_grt`.

## Sender and Signers

Expand Down Expand Up @@ -106,12 +130,14 @@ curl http://localhost:9090/metrics
|--------|------|-------------|
| `escrow_total_debt_grt` | Gauge | Total outstanding debt across all receivers |
| `escrow_total_balance_grt` | Gauge | Total escrow balance across all receivers |
| `escrow_total_target_grt` | Gauge | Total target escrow balance across all receivers |
| `escrow_total_adjustment_grt` | Gauge | Total GRT deposited in the last cycle |
| `escrow_receiver_count` | Gauge | Number of receivers being tracked |
| `escrow_loop_duration_seconds` | Histogram | Duration of each polling cycle |
| `escrow_debt_grt{receiver}` | Gauge | Outstanding debt per receiver |
| `escrow_balance_grt{receiver}` | Gauge | Escrow balance per receiver |
| `escrow_adjustment_grt{receiver}` | Gauge | Last adjustment per receiver |
| `escrow_target_grt{receiver}` | Gauge | Target escrow balance per receiver |
| `escrow_adjustment_grt{receiver}` | Gauge | Last adjustment per receiver (0 if at or above target) |
| `escrow_deposit_ok` | Counter | Successful deposit transactions |
| `escrow_deposit_err` | Counter | Failed deposit transactions |
| `escrow_deposit_duration` | Histogram | Deposit transaction duration |
10 changes: 10 additions & 0 deletions crates/bin/escrow_manager/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,22 @@ pub struct Config {
/// Port for metrics server
#[serde(default = "default_port_metrics")]
pub port_metrics: u16,
/// Fraction of a receiver's target escrow balance that its debt is allowed to reach before the
/// balance is raised to the next step. This sets the funding margin: the target balance settles
/// at roughly `debt / balance_fill_factor`, so 0.6 funds ~1.67x debt and 0.8 funds ~1.25x.
/// Must be in the range (0, 1].
#[serde(default = "default_balance_fill_factor")]
pub balance_fill_factor: f64,
}

fn default_port_metrics() -> u16 {
9090
}

fn default_balance_fill_factor() -> f64 {
0.8
}

#[derive(Debug, Deserialize)]
pub struct Kafka {
pub config: BTreeMap<String, String>,
Expand Down
6 changes: 4 additions & 2 deletions crates/bin/escrow_manager/src/contracts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,17 @@ sol!(
);
use ERC20::ERC20Instance;
sol!(
#[allow(missing_docs)]
// `sol!` generates a constructor per event, and some events (e.g. GraphDirectoryInitialized)
// have more parameters than clippy's threshold. Nothing we can restructure.
#[allow(missing_docs, clippy::too_many_arguments)]
#[sol(rpc)]
#[derive(Debug)]
PaymentsEscrow,
"src/abi/PaymentsEscrow.abi.json"
);
use PaymentsEscrow::{PaymentsEscrowErrors, PaymentsEscrowInstance};
sol!(
#[allow(missing_docs)]
#[allow(missing_docs, clippy::too_many_arguments)]
#[sol(rpc)]
#[derive(Debug)]
GraphTallyCollector,
Expand Down
72 changes: 63 additions & 9 deletions crates/bin/escrow_manager/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,13 @@ async fn main() -> anyhow::Result<()> {
.and_then(|s| serde_json::from_str(&s).map_err(anyhow::Error::from))
.context("failed to load config")?;

anyhow::ensure!(
(config.balance_fill_factor > 0.0) && (config.balance_fill_factor <= 1.0),
"balance_fill_factor must be in the range (0, 1], got {}",
config.balance_fill_factor,
);
tracing::info!(balance_fill_factor = config.balance_fill_factor);

if config.dry_run {
tracing::info!("dry run mode enabled, contract calls will be skipped");
}
Expand Down Expand Up @@ -215,6 +222,7 @@ async fn main() -> anyhow::Result<()> {
.total_debt_grt
.set(debts.values().sum::<u128>() as f64 / GRT as f64);

let mut total_target: u128 = 0;
let adjustments: Vec<(Address, u128)> = receivers
.into_iter()
.filter_map(|receiver| {
Expand All @@ -223,25 +231,37 @@ async fn main() -> anyhow::Result<()> {
debts.get(&receiver).copied().unwrap_or(0),
config.debts.get(&receiver).copied().unwrap_or(0) as u128 * GRT,
);
let next_balance = next_balance(debt);
let next_balance = next_balance(debt, config.balance_fill_factor);
total_target += next_balance;
let adjustment = next_balance.saturating_sub(balance);
// Record the target and adjustment for every receiver, including those already at
// or above their target. Skipping them would leave the gauges holding the last
// value they were set to, indefinitely.
let receiver_str = format!("{receiver:?}");
metrics::METRICS
.target_grt
.with_label_values(&[&receiver_str])
.set(next_balance as f64 / GRT as f64);
metrics::METRICS
.adjustment_grt
.with_label_values(&[&receiver_str])
.set(adjustment as f64 / GRT as f64);
if adjustment == 0 {
return None;
}
tracing::info!(
?receiver,
balance_grt = (balance as f64) / (GRT as f64),
debt_grt = (debt as f64) / (GRT as f64),
target_grt = (next_balance as f64) / (GRT as f64),
adjustment_grt = (adjustment as f64) / (GRT as f64),
);
let receiver_str = format!("{receiver:?}");
metrics::METRICS
.adjustment_grt
.with_label_values(&[&receiver_str])
.set(adjustment as f64 / GRT as f64);
Some((receiver, adjustment))
})
.collect();
metrics::METRICS
.total_target_grt
.set(total_target as f64 / GRT as f64);

let total_adjustment: u128 = adjustments.iter().map(|(_, a)| a).sum();
tracing::info!(total_adjustment_grt = ((total_adjustment as f64) * 1e-18).ceil() as u64);
Expand Down Expand Up @@ -298,9 +318,12 @@ async fn main() -> anyhow::Result<()> {
}
}

fn next_balance(debt: u128) -> u128 {
/// Target escrow balance for a receiver with the given debt. The balance steps up while debt
/// reaches `fill_factor` of the current step, so the target settles at roughly `debt / fill_factor`
/// once the steps are fine-grained (above `MAX_ADJUSTMENT`, where they stop doubling).
fn next_balance(debt: u128, fill_factor: f64) -> u128 {
let mut next_round = (MIN_DEPOSIT / GRT) as u32;
while (debt as f64) >= ((next_round as u128 * GRT) as f64 * 0.6) {
while (debt as f64) >= ((next_round as u128 * GRT) as f64 * fill_factor) {
next_round = next_round
.saturating_mul(2)
.min(next_round + (MAX_ADJUSTMENT / GRT) as u32);
Expand Down Expand Up @@ -356,7 +379,38 @@ mod tests {
(100 * GRT, 256 * GRT),
];
for (debt, expected) in tests {
assert_eq!(super::next_balance(debt), expected);
assert_eq!(super::next_balance(debt, 0.6), expected);
}
}

#[test]
fn next_balance_fill_factor() {
// A higher fill factor packs debt closer to the target balance, funding a thinner margin.
let tests = [
(0, MIN_DEPOSIT),
(MIN_DEPOSIT, MIN_DEPOSIT * 2),
(30 * GRT, 64 * GRT),
(70 * GRT, 128 * GRT),
// 100 GRT of debt is funded to 256 GRT at 0.6, but only 128 GRT at 0.8.
(100 * GRT, 128 * GRT),
];
for (debt, expected) in tests {
assert_eq!(super::next_balance(debt, 0.8), expected);
}
}

#[test]
fn next_balance_margin_converges_above_step_cap() {
// Once the steps stop doubling, the target tracks debt / fill_factor closely.
for fill_factor in [0.6, 0.8, 0.95] {
let debt = 580_000 * GRT;
let target = super::next_balance(debt, fill_factor);
let ratio = (target as f64) / (debt as f64);
let expected = 1.0 / fill_factor;
assert!(
(ratio >= expected) && (ratio < (expected + 0.05)),
"fill_factor {fill_factor}: ratio {ratio} not just above {expected}",
);
}
}
}
13 changes: 13 additions & 0 deletions crates/bin/escrow_manager/src/metrics.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,15 @@ lazy_static! {
pub struct Metrics {
pub total_debt_grt: Gauge,
pub total_balance_grt: Gauge,
pub total_target_grt: Gauge,
pub total_adjustment_grt: Gauge,
pub receiver_count: IntGauge,
pub loop_duration: Histogram,
pub deposit: ResponseMetrics,
// Per-receiver metrics
pub debt_grt: GaugeVec,
pub balance_grt: GaugeVec,
pub target_grt: GaugeVec,
pub adjustment_grt: GaugeVec,
}

Expand All @@ -34,6 +36,11 @@ impl Metrics {
"total escrow balance across all receivers in GRT"
)
.unwrap(),
total_target_grt: register_gauge!(
"escrow_total_target_grt",
"total target escrow balance across all receivers in GRT"
)
.unwrap(),
total_adjustment_grt: register_gauge!(
"escrow_total_adjustment_grt",
"total GRT deposited in the last cycle"
Expand Down Expand Up @@ -62,6 +69,12 @@ impl Metrics {
&["receiver"]
)
.unwrap(),
target_grt: register_gauge_vec!(
"escrow_target_grt",
"target escrow balance per receiver in GRT",
&["receiver"]
)
.unwrap(),
adjustment_grt: register_gauge_vec!(
"escrow_adjustment_grt",
"last adjustment per receiver in GRT",
Expand Down
Loading
Loading