gleamunison/rank

BM25 ranking over tokenized documents: pure data in, ranked data out. Used by find and the MCP search tool. Intentionally standalone from the datalog engine’s own in-DB BM25: the CLI path must not transact a database to answer one query.

Types

Doc

</>

pub type Doc(a) {
  Doc(id: a, tokens: List(String))
}

Constructors

```
Doc(id: a, tokens: List(String))
```

Values

rank

</>

pub fn rank(
  query: String,
  docs: List(Doc(a)),
) -> List(#(Float, a))

Rank docs against a query string. Returns #(score, id) descending, zero-score docs dropped. Empty query or empty corpus yields [].

tokenize

</>

pub fn tokenize(text: String) -> List(String)