import "github.com/egdaemon/eg/runtime/x/wasi/egllm"
const (
// EnvAnthropicAPIKey is the standard variable holding the anthropic api key.
EnvAnthropicAPIKey = "ANTHROPIC_API_KEY"
// EnvAnthropicBaseURL is the standard variable overriding the anthropic api url.
EnvAnthropicBaseURL = "ANTHROPIC_BASE_URL"
// DefaultAnthropicEndpoint is the hosted anthropic api.
DefaultAnthropicEndpoint = "https://api.anthropic.com"
)
DefaultEndpoint is where the managed llama-server listens, a unix socket. must match the --host of llama-server.service.
const DefaultEndpoint = "unix:///run/llama-server/llama.sock"
func CacheDirectory(dirs ...string) string
func Chat(ctx context.Context, c Client, model string, messages ...Message) (string, error)
Chat sends the messages, in order, to the llama.cpp server's OpenAI-compatible chat completions endpoint and returns the generated content.
func Endpoint() string
Endpoint returns the base url of the llm server: the value of eg.EnvComputeLLMEndpoint, otherwise the managed llama-server's. either an http(s) url or a unix socket, see unixScheme.
func Env() []string
attempt to build the environment that sets up the llama.cpp model cache directory.
func Generate(ctx context.Context, c Client, model string, prompt string) (string, error)
Generate sends prompt as a single user message to the llama.cpp server's OpenAI-compatible chat completions endpoint and returns the generated content.
func Install(runtime shell.Command) eg.OpFn
Install copies the embedded llama-server systemd unit into place, replacing any unit already present, so the image doesn't need to provide one. run it before Serve.
func Prepare(c eg.ContainerRunner) eg.OpFn
build llama.cpp container.
func Pull(runtime shell.Command, model string) eg.OpFn
Pull warms the local model cache for the given model so that starting llama-server (which loads the model at process start, unlike ollama's decoupled daemon+pull model) never blocks on a cold download. It also records which model and cache the systemd unit should use, since the unit does not inherit the runtime's environment.
func Runner() eg.ContainerRunner
container for this package.
func Runtime() shell.Command
Create a shell runtime that properly sets up the llama.cpp caching environment.
func Serve(runtime shell.Command) eg.OpFn
func Shutdown(runtime shell.Command) eg.OpFn
func With(model string, op eg.OpFn) eg.OpFn
run the provided operation with the given model. a llama-server is started for the duration of the operation, unless the endpoint has been overridden (see eg.EnvComputeLLMEndpoint) in which case the operation is run against that server as is.
ClaudeError is an error response from the anthropic api, it carries the reason the api gave, which the status alone doesn't, e.g.) a depleted credit balance is a 400. it unwraps to the httpx.Error of the response, so the status code can still be checked with httpx.IsStatusError.
type ClaudeError struct {
Status int // http status code.
Type string // api error type, e.g.) invalid_request_error. empty when the body isn't an api error.
Message string // api error message, or the raw body when it isn't an api error.
// contains filtered or unexported fields
}
func (t *ClaudeError) Error() string
func (t *ClaudeError) Unwrap() error
type Client interface {
Do(r *http.Request) (*http.Response, error)
Host() string
}
Judge generates a response to a conversation.
type Judge interface {
Do(ctx context.Context, m ...Message) (string, error)
}
func Echo() Judge
Echo is a Judge that responds with the conversation it was given, useful when no model is available.
Message is a single turn of a chat, role is one of "system", "user" or "assistant".
type Message struct {
Role string `json:"role"`
Content string `json:"content"`
}
Server is implemented by judges that need setup and teardown around the operations that use them, e.g.) a managed llama-server.
type Server interface {
// With runs ops in sequence while the judge is available.
With(ops ...eg.OpFn) eg.OpFn
}