Login Register

Running Local LLMs: A Comparison of Ollama, llama.cpp, LM Studio, and vLLM

DaDesktop Team ·

There are multiple avenues for executing Large Language Models (LLMs) locally. Some solutions are engineered for immediate accessibility, while others prioritise granular control or are architected to handle high-volume, multi-user demands. Selecting the appropriate tool hinges on your specific requirements: whether you seek a straightforward local chat interface, a highly configurable inference engine, or a robust production-grade API.

Ollama

Ollama stands out as one of the most accessible entry points for local model execution. The process is streamlined: install the software, pull a model, and launch it via the command line. It also offers a local API endpoint, facilitating integration with other applications and tools.

Advantages:

  • Straforward installation and model handling
  • User-friendly command-line interface
  • OpenAI-compatible API endpoints
  • Hardware acceleration support for NVIDIA, AMD, Apple Silicon, and Vulkan-based GPUs
  • Ability to customise models and parameters via Modelfiles
  • Handling of concurrent requests, provided sufficient memory is available

Limitations:

  • Offers less low-level control compared to llama.cpp
  • Model management is tightly coupled to the Ollama ecosystem
  • Not the optimal choice for scenarios demanding maximum serving throughput or distributed inference

Complexity: Low. An excellent option for quickly deploying a model without navigating complex inference configurations.

llama.cpp

llama.cpp is a lightweight C/C++ inference engine designed for efficient model execution across diverse hardware. It utilises GGUF models and provides extensive control over the loading and execution processes.

Advantages:

  • Granular control over context, GPU offloading, batching, threads, quantization, and other inference parameters
  • Extensive hardware compatibility, including CUDA, HIP, Metal, Vulkan, and SYCL
  • Support for various quantization levels, ranging from low-bit formats to 8-bit
  • Capability to split models across multiple GPUs
  • Hybrid CPU and GPU utilisation for models exceeding available VRAM
  • Inclusion of llama-server for an OpenAI-compatible API

Limitations:

  • Requires more configuration than Ollama or LM Studio
  • GGUF models typically require separate download and management
  • Effective use often demands a solid understanding of inference parameters

Complexity: Medium. Ideal for users seeking precise control over model execution or wishing to experiment with performance tuning and quantization.

LM Studio

LM Studio is a desktop application facilitating the download, configuration, and execution of local LLMs. It features a graphical interface for discovering models and managing settings such as GPU offloading and context size.

Advantages:

  • Intuitive graphical user interface
  • Model search and download integration via Hugging Face
  • Pre-load visibility into model and resource specifications
  • OpenAI-compatible API server capabilities
  • Headless operation support via its llmster server
  • Support for GGUF via llama.cpp and MLX models on Apple Silicon

Limitations:

  • Less low-level control compared to direct use of llama.cpp
  • Less suited to certain server-side deployments due to its desktop nature
  • Not primarily optimised for large-scale, multi-user serving

Complexity: Low. A strong choice for experimenting with local models without extensive command-line interaction.

vLLM

vLLM is architected for serving LLMs to applications and multiple concurrent users. Its primary strength lies in efficient high-concurrency serving, leveraging techniques such as PagedAttention, continuous batching, prefix caching, and distributed inference.

Advantages:

  • High throughput for handling multiple concurrent requests
  • Continuous batching and efficient KV-cache management
  • OpenAI-compatible API server
  • Direct compatibility with many Hugging Face models
  • Support for quantization methods including FP8, INT4, GPTQ, AWQ, GGUF, and others
  • Support for tensor, pipeline, expert, and other parallelism strategies
  • Designed specifically for production-grade inference and serving

Limitations:

  • More complex setup and configuration requirements
  • Primarily targeted at Linux environments
  • Often overkill for single-user, interactive model usage
  • Requires verification of hardware and model compatibility prior to deployment

Complexity: High. Best suited for professionals deploying inference services rather than casual local model usage.

Which one should you choose?

  • Seeking ease of use: Opt for Ollama or LM Studio. Choose Ollama for a command-line focused approach with a simple API, or LM Studio for a graphical interface.
  • Prioritising inference control: Select llama.cpp. It offers direct control over model loading, quantization, context, GPU offloading, and other settings.
  • Requiring a local API: Ollama, llama.cpp, or LM Studio are all viable options, as they all provide OpenAI-compatible APIs.
  • Serving multiple users: Choose vLLM. Its continuous batching and distributed inference capabilities are tailored for this scenario.
  • Experimenting with quantization: Consider llama.cpp or LM Studio.

Run it on DaDesktop

If your local hardware lacks sufficient GPU power, you can execute these tools on a DaDesktop cloud desktop. Select a GPU with adequate VRAM for your chosen model, launch the desktop environment, and install your preferred inference software.

Ollama and LM Studio are well-suited for creating simple local environments. llama.cpp offers greater control over hardware and inference settings. vLLM is the recommended option when you need to expose a model as a high-throughput API.

View available GPUs to compare VRAM and other specifications.