llama.cpp includes a program called llama-server that loads a GGUF model file and exposes it over HTTP, including endpoints compatible with the OpenAI chat API, so existing tools and SDKs can talk to a model running on your own machine. This guide walks through starting the server, the flags that matter, calling it from code, and the security and troubleshooting points people most often miss. Flag names change between releases, so confirm them with llama-server --help on your build.

Why use llama-server

Ollama, covered in our local LLM guide, wraps llama.cpp in a friendlier package. Running llama-server directly gives you finer control: you choose the exact model file and quantization, set context size and GPU offload yourself, and track upstream releases quickly. The trade-off is that you manage more details. Teams use it for private prototypes, offline tools and as a stand-in for a paid API during development.

Step 1: install and get a model

Install llama.cpp from the project’s releases page, a package manager or by building from source, following the README for your platform. You then need a model in GGUF format, the single-file container llama.cpp uses for weights and metadata. Pick a size that fits your memory: our guides to GPU memory sizing and quantization explain how file size, quantization level and context length add up. As a rule of thumb, the model file plus working memory for the context must fit in RAM or video memory.

The server documentation also describes a flag that fetches a model directly from a Hugging Face repository, which saves manual downloading. Check the current flag name in --help.

Step 2: start the server

A minimal launch looks like this:

llama-server -m ./models/your-model.gguf --port 8080 -c 8192

The flags you will use most:

  • -m points to the GGUF file.
  • -c sets the context window in tokens. Larger values use more memory because the key-value cache grows with context.
  • -ngl sets how many layers to offload to the GPU. Raise it until you run out of video memory, then back off.
  • --port and --host choose where the server listens. The default host is local only.
  • --api-key requires a key on requests.

When the model finishes loading, the server prints the address it listens on. The project also ships a simple browser chat interface, which is useful for checking that the model works before you write any code.

Step 3: call it like the OpenAI API

Because the server speaks the same request format, you can point an OpenAI-style client at it by changing the base URL. With the official Python SDK, set the base URL to http://localhost:8080/v1, supply any placeholder key if none is required, and send a normal chat completion request. A plain HTTP call works too:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Summarize GGUF in one sentence."}]}'
Source code displayed on a computer screen
Any OpenAI-compatible client or script can call a model served from your own machine. Photo: Slashme / Wikimedia Commons, CC0

The response follows the familiar structure, with the reply under choices. Not every OpenAI feature is supported, and behavior of newer parameters varies by version, so test the specific features you rely on, such as tool calling or structured output, rather than assuming parity.

Security: keep it local unless you mean otherwise

By default the server should accept connections only from your own machine. Binding to all interfaces exposes the model, and anything it can read, to your network. If you must share it, put it behind a reverse proxy with authentication and TLS, set an API key, and restrict access by firewall. Never expose an unauthenticated model endpoint to the internet. Treat prompts and outputs as sensitive: logs may contain them.

Troubleshooting

  • Out-of-memory at startup: lower -c, use a smaller quantization, or reduce -ngl.
  • Very slow generation: the model may be running on CPU; check the startup log for the GPU backend and offloaded layer count.
  • Truncated or odd replies: the prompt may be exceeding the context window, or the model’s chat template may not match. Use a model whose GGUF includes the correct template.
  • Client errors: confirm the base URL ends in /v1 and that the port matches.

Performance tips

Keep the context window no larger than you need, because the key-value cache scales with it; see the glossary entry on the KV cache. Batch size and the number of parallel slots trade latency against throughput, so test with your real workload. Update llama.cpp regularly, since inference speed and model support improve quickly.

What to watch next

  • New quantization formats and speedups landing in llama.cpp releases.
  • Broader support for tool calling and structured output over the compatible API.
  • Improvements in multimodal input through the server.
  • Hardware-specific backends for new NPUs and GPUs.

Key takeaways

  • llama-server turns a GGUF file into a local HTTP API with OpenAI-style endpoints.
  • Context size and GPU offload (-c, -ngl) are the main levers for memory and speed.
  • Point existing OpenAI-compatible clients at http://localhost:PORT/v1.
  • Keep the server bound to localhost or protect it with a key and a proxy.
  • Check --help for current flags and test the features you depend on.

Frequently asked questions

What is llama-server?

It is the HTTP server included with llama.cpp. It loads a GGUF model and serves chat and completion endpoints, including ones compatible with the OpenAI API.

Is llama-server the same as Ollama?

No. Ollama is a separate tool that builds on llama.cpp with model management and a simpler interface. llama-server gives direct control over files and flags.

Can I expose llama-server to the internet?

Only with care. Use an API key, TLS and a reverse proxy, and restrict access, because an open endpoint lets anyone use your hardware and read what the model sees.