gleamunison/rank

BM25 ranking over tokenized documents: pure data in, ranked data out. Used by find and the MCP search tool. Intentionally standalone from the datalog engine’s own in-DB BM25: the CLI path must not transact a database to answer one query.

Types

Doc

</>

pub type Doc(a) {
  Doc(id: a, tokens: List(String))
}

Constructors

```
Doc(id: a, tokens: List(String))
```

DocV

</>

pub type DocV(a) {
  DocV(id: a, text: String)
}

Constructors

```
DocV(id: a, text: String)
```

Vector

</>

Sparse embedding: dim -> count. 1024 dims is enough that collisions are rare at codebase scale, and collisions are deterministic (same trigram, same dim) so they blur rather than corrupt.

pub type Vector {
  Vector(entries: dict.Dict(Int, Int))
}

Constructors

```
Vector(entries: dict.Dict(Int, Int))
```

Values

cosine

</>

pub fn cosine(a: Vector, b: Vector) -> Float

cosine_shared

</>

pub fn cosine_shared(a: Vector, b: Vector) -> #(Float, Int)

Cosine plus the number of shared trigram dims. One shared trigram is coincidence (word-start “^no” is in both “no” and “not”); two is evidence. Callers use the count as a match gate.

embed

</>

pub fn embed(text: String) -> Vector

rank

</>

pub fn rank(
  query: String,
  docs: List(Doc(a)),
) -> List(#(Float, a))

Rank docs against a query string. Returns #(score, id) descending, zero-score docs dropped. Empty query or empty corpus yields [].

rank_hybrid

</>

pub fn rank_hybrid(
  query: String,
  docs: List(DocV(a)),
) -> List(#(Float, a))

Hybrid rank: normalized BM25 (exact token relevance) + normalized cosine over trigram embeddings (subword/morphological similarity). One sum, no knobs. Zero-score docs dropped; descending.

tokenize

</>

pub fn tokenize(text: String) -> List(String)