gleamunison - v3.24.0

gleamunison/llm_budget

LLM budget and circuit policy — pure data in, decision out.

One owner for the vocabulary: budgets, usage, circuit state, and the refusal names. The stateful holder (gleamunison_llm_budget.erl) applies these functions; it never invents its own decisions. Rates and limits arrive as data (env-derived at the edge), never hardcoded here.

Types

Budget

</> 
pub type Budget {
  Budget(
    max_tokens: Int,
    max_cost_micro_usd: Int,
    max_concurrent: Int,
    window_ms: Int,
    allowed_models: List(String),
    circuit_failure_threshold: Int,
    circuit_reset_ms: Int,
  )
}

Constructors 

  • Budget(
  max_tokens: Int,
  max_cost_micro_usd: Int,
  max_concurrent: Int,
  window_ms: Int,
  allowed_models: List(String),
  circuit_failure_threshold: Int,
  circuit_reset_ms: Int,
)

    Arguments

    max_tokens

    Aggregate token ceiling per window (prompt + completion).

    max_cost_micro_usd

    Aggregate cost ceiling per window, micro-USD (1/1_000_000 USD).

    max_concurrent

    In-flight call ceiling across the runtime.

    window_ms

    Rolling window length; counters reset when it elapses.

    allowed_models

    Non-empty = only these models may run. Empty = unpinned (documented).

    circuit_failure_threshold

    Consecutive failures that open the circuit.

    circuit_reset_ms

    How long an open circuit refuses before a half-open probe.

Circuit

</> 
pub type Circuit {
  Closed
  Open(opened_at: Int)
  HalfOpen
}

Constructors 

  • Closed
  • Open(opened_at: Int)
  • HalfOpen

Refusal

</> 
pub type Refusal {
  TokenBudgetExceeded(limit: Int)
  CostBudgetExceeded(limit_micro_usd: Int)
  ConcurrencyExceeded(limit: Int)
  CircuitOpen(retry_after_ms: Int)
  ModelNotPinned(model: String)
}

Constructors 

  • TokenBudgetExceeded(limit: Int)
  • CostBudgetExceeded(limit_micro_usd: Int)
  • ConcurrencyExceeded(limit: Int)
  • CircuitOpen(retry_after_ms: Int)
  • ModelNotPinned(model: String)

Usage

</> 
pub type Usage {
  Usage(
    tokens: Int,
    cost_micro_usd: Int,
    in_flight: Int,
    window_started_at: Int,
  )
}

Constructors 

  • Usage(
  tokens: Int,
  cost_micro_usd: Int,
  in_flight: Int,
  window_started_at: Int,
)

Values

admit

</> 
pub fn admit(
  budget: Budget,
  usage: Usage,
  circuit: Circuit,
  model: String,
  now: Int,
) -> Result(Nil, Refusal)

The admission decision. Checked in cheapest-to-most-expensive order: pin, circuit, concurrency, then aggregate budgets.

after_failure

</> 
pub fn after_failure(
  circuit: Circuit,
  consecutive_failures: Int,
  threshold: Int,
  now: Int,
) -> Circuit

The holder owns the consecutive-failure counter; this decides what a failure means given the circuit we were in. A failed half-open probe re-opens immediately.

begin

</> 
pub fn begin(usage: Usage) -> Usage

cost_micro_usd

</> 
pub fn cost_micro_usd(
  prompt_tokens: Int,
  completion_tokens: Int,
  prompt_rate: Int,
  completion_rate: Int,
) -> Int

Cost arithmetic stays here and only here: rates are micro-USD per 1M tokens, supplied by the caller as data.

effective_usage

</> 
pub fn effective_usage(
  usage: Usage,
  window_ms: Int,
  now: Int,
) -> Usage

Rolling-window view: when the window has elapsed the counters read as zeroed (the holder persists the roll on the next write).

end

</> 
pub fn end(usage: Usage) -> Usage

model_allowed

</> 
pub fn model_allowed(budget: Budget, model: String) -> Bool

new_usage

</> 
pub fn new_usage(now: Int) -> Usage

parse_allowed_models

</> 
pub fn parse_allowed_models(text: String) -> List(String)

Parse the operator-facing allowlist: comma-separated, trimmed.

record_success

</> 
pub fn record_success(circuit: Circuit) -> Circuit

refusal_name

</> 
pub fn refusal_name(refusal: Refusal) -> String

refusal_retryable

</> 
pub fn refusal_retryable(refusal: Refusal) -> Bool

spend

</> 
pub fn spend(
  usage: Usage,
  tokens: Int,
  cost_micro_usd: Int,
) -> Usage
✨ Search Document