litertlm-go¶

A Go binding for Google's LiteRT-LM for high-performance local on-device LLM inference.

Inspired by Hybridgroup's Yzma.

📖 Full documentation: vladimirvivien.github.io/litertlm-go

Features¶

💬 Stateful Chat & Conversations — Multi-turn chat orchestration with system prompts, message history, and token tracking.
🖼️ Multimodal Inputs — Process text, images, and audio inputs in any order using a unified Go interface.
🛠️ Automated Tool Calling — Register standard Go functions as tools that the model can call automatically and in parallel.
🎯 Structured JSON Output — Extract model outputs directly into Go structs with type-safe generic helpers.
⚡ High Performance & GPU Acceleration — Run on CPU or GPU (WebGPU/Direct3D 12) with built-in memory safety.
🔍 Low-Level Control — Access the full LiteRT-LM C-API when you need custom inference loops, scoring, or custom memory management.
🤖 Supported Models — Fully compatible with Gemma 4, Gemma 3/3n, Qwen 3/2.5, Phi-4, and other LiteRT-LM families.

Install¶

go get github.com/vladimirvivien/litertlm-go@latest

Quickstart¶

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/vladimirvivien/litertlm-go/pkg/litertlm"
)

func main() {
    ctx := context.Background()
    client, err := litertlm.New(ctx,
        litertlm.WithLib(os.Getenv("LITERTLM_LIB")),
        litertlm.WithModel(os.Getenv("LITERTLM_MODEL")),
    )
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }
    defer client.Close()

    text, err := client.Generate(ctx, "Write a haiku about the sea.")
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }
    fmt.Println(text)
}

LITERTLM_LIB=/abs/path/to/dist/lib \
LITERTLM_MODEL=/abs/path/to/gemma-4-E2B-it.litertlm \
    go run main.go

Full walkthrough: Getting started.

Building the C library¶

LiteRT-LM does not ship prebuilt C API shared libraries. Build them using the platform guide: LITERTLM-BUILD.md (Linux/macOS) or LITERTLM-BUILD-WINDOWS.md.

Minimum supported upstream: LiteRT-LM v0.13.1.

Examples¶

Self-contained programs covering every public API surface live under examples/. See the examples index for the full list.

License¶

Apache-2.0, same as LiteRT-LM itself.