gleamunison/lexer

Types

pub type ParseError {
  ParseError(message: String, line: Int, col: Int)
}

ParseError(message: String, line: Int, col: Int)

</>

pub type Token {
  Symbol(String)
  IntVal(Int)
  FloatVal(Float)
  LParen
  RParen
  Quote
  UnterminatedString(String)
  Comment(String)
}

```
Symbol(String)
```
```
IntVal(Int)
```
```
FloatVal(Float)
```
```
LParen
```
```
RParen
```
```
Quote
```
```
UnterminatedString(String)
```
```
Comment(String)
```
A ; comment, text preserved including the leading semicolon. Emitted only outside string literals (strings scan in their own loop, so a ; inside quotes is string content by construction). The parser treats comments as whitespace; the formatter reattaches them. Nothing is ever silently dropped.

</>

pub type TokenInfo {
  TokenInfo(token: Token, line: Int, col: Int)
}

TokenInfo(token: Token, line: Int, col: Int)

</>

pub fn tokenize(input: String) -> List(TokenInfo)