gleamunison/llm_middleware

LLM handler middleware (gap analysis concept A, decomplected).

BAML complects retry/fallback/mocking into a client registry + hosted studio. Here they are plain handler composition: a middleware is just an OpHandler wrapping another OpHandler. Same contract as llm.gleam: continuation receives {ok, CompletionJsonBin} | {error, ReasonBin}.

Types

RetryPolicy

</>

pub type RetryPolicy {
  RetryPolicy(max_attempts: Int, base_delay_ms: Int)
}

Constructors

RetryPolicy(max_attempts: Int, base_delay_ms: Int)

Values

fallback_handler

</>

pub fn fallback_handler(
  handlers: List(
    fn(
      List(dynamic.Dynamic),
      fn(dynamic.Dynamic) -> dynamic.Dynamic,
    ) -> dynamic.Dynamic,
  ),
) -> fn(
  List(dynamic.Dynamic),
  fn(dynamic.Dynamic) -> dynamic.Dynamic,
) -> dynamic.Dynamic

Try handlers in order; the first {ok, _} wins. All failing returns the last handler’s error.

recording_mock

</>

pub fn recording_mock(
  response: llm.Completion,
  record: fn(BitArray) -> Nil,
) -> fn(
  List(dynamic.Dynamic),
  fn(dynamic.Dynamic) -> dynamic.Dynamic,
) -> dynamic.Dynamic

Deterministic mock that also reports the messages JSON it received — the fixture recorder for the eval family (gap analysis concept E).

retry_handler

</>

pub fn retry_handler(
  inner: fn(
    List(dynamic.Dynamic),
    fn(dynamic.Dynamic) -> dynamic.Dynamic,
  ) -> dynamic.Dynamic,
  policy: RetryPolicy,
) -> fn(
  List(dynamic.Dynamic),
  fn(dynamic.Dynamic) -> dynamic.Dynamic,
) -> dynamic.Dynamic

Retry the inner handler with linear backoff (base_delay_ms * attempt). Any {error, _} result is retryable; {ok, _} passes straight through.