gleamunison/llm_middleware

LLM handler middleware (gap analysis concept A, decomplected).

BAML complects retry/fallback/mocking into a client registry + hosted studio. Here they are plain handler composition: a middleware is just an OpHandler wrapping another OpHandler. Same contract as llm.gleam: continuation receives {ok, CompletionJsonBin} | {error, ReasonBin}.

Types

RetryPolicy

</>

pub type RetryPolicy {
  RetryPolicy(max_attempts: Int, base_delay_ms: Int)
}

Constructors

RetryPolicy(max_attempts: Int, base_delay_ms: Int)

Values

budget_handler

</>

pub fn budget_handler(
  inner: fn(
    List(dynamic.Dynamic),
    fn(dynamic.Dynamic) -> dynamic.Dynamic,
  ) -> dynamic.Dynamic,
  provider: llm_types.Provider,
  model: String,
) -> fn(
  List(dynamic.Dynamic),
  fn(dynamic.Dynamic) -> dynamic.Dynamic,
) -> dynamic.Dynamic

Budget/circuit gate as handler composition. Admit before the call, release the concurrency slot exactly once, then meter actuals and report the outcome. Refusals return the typed refusal name as the handler error; the inner handler never runs.

Settlement errors are reported, not asserted: a bookkeeping failure must not crash a call whose completion already succeeded, and the inner handler’s result is always what the continuation receives.

fallback_handler

</>

pub fn fallback_handler(
  handlers: List(
    fn(
      List(dynamic.Dynamic),
      fn(dynamic.Dynamic) -> dynamic.Dynamic,
    ) -> dynamic.Dynamic,
  ),
) -> fn(
  List(dynamic.Dynamic),
  fn(dynamic.Dynamic) -> dynamic.Dynamic,
) -> dynamic.Dynamic

Try handlers in order; the first {ok, _} wins. All failing returns the last handler’s error.

recording_mock

</>

pub fn recording_mock(
  response: llm_types.Completion,
  record: fn(BitArray) -> Nil,
) -> fn(
  List(dynamic.Dynamic),
  fn(dynamic.Dynamic) -> dynamic.Dynamic,
) -> dynamic.Dynamic

Deterministic mock that also reports the messages JSON it received — the fixture recorder for the eval family (gap analysis concept E).

retry_handler

</>

pub fn retry_handler(
  inner: fn(
    List(dynamic.Dynamic),
    fn(dynamic.Dynamic) -> dynamic.Dynamic,
  ) -> dynamic.Dynamic,
  policy: RetryPolicy,
) -> fn(
  List(dynamic.Dynamic),
  fn(dynamic.Dynamic) -> dynamic.Dynamic,
) -> dynamic.Dynamic

At most three attempts with at most 1000ms delay per retry. Terminal settings/auth/decode/cancellation errors must not repeat a keyed call.