HeyRocky
Run Hermes Locally with Ollama

Run Hermes Locally with Ollama

Ollama can serve an open-weight model on the same machine as Hermes through http://localhost:11434/v1. This removes per-token API charges for the model call, but it does not make hardware, electricity, latency, or maintenance free.

Local inference also does not automatically make the whole workflow offline. Web tools, remote browsers, messaging platforms, telemetry configured outside Hermes, and any cloud fallback can still send data over the network. Define the privacy boundary before calling the setup fully local.

Choose hardware and model deliberately

Model size must fit available RAM or VRAM with room for the operating system, Hermes, and context cache. Smaller chat models may answer questions but fail to emit reliable tool calls. For agentic work, choose an Ollama model that explicitly supports tools and test it with the actual Hermes tool schema.

The official guide uses gemma4:31b as its tool-capable example and recommends a 64,000-token context for agentic work. Model catalogs and hardware support change, so check the current Ollama library and measure the chosen model on the target machine.

Setup checklist

  • Install Ollama from its official distribution and confirm the service is running.
  • Pull a model that fits the machine and advertises tool-calling support.
  • Query Ollama’s model list at http://localhost:11434/api/tags.
  • Send one direct request to the local /v1/chat/completions endpoint.
  • Run hermes setup and choose Custom Endpoint.
  • Set the base URL to http://localhost:11434/v1 and use the exact pulled model name.
  • Configure a context window large enough for the intended agent workflow.
  • Start Hermes and test prose, a read-only file action, and a harmless terminal action.
  • Add a cloud fallback only if its cost and data boundary are acceptable.
ollama --version
ollama pull gemma4:31b
curl http://localhost:11434/api/tags
hermes setup
hermes

Ollama does not require an API key on its default local endpoint. In the Hermes custom-endpoint setup, leave the key empty or use the wizard’s documented no-key option.

Performance and context

CPU-only inference can work but may be slow. Use ollama ps to see whether a model is loaded and whether GPU offload is active. If a local request times out, the official guide documents HERMES_API_TIMEOUT in the Hermes environment file; it is not a config.yaml key.

Ollama’s default context may be too small for Hermes tools and conversation history. Create a model variant with a larger num_ctx only after checking memory impact. Long context can reduce speed and increase memory use even when the model weights fit.

Common pitfalls

  • Picking a conversational model that cannot produce structured tool calls.
  • Calling the route private while web tools or a cloud fallback remain enabled.
  • Exposing Ollama on 0.0.0.0 without authentication or a network-access plan.
  • Testing only /api/tags and never making a completion request.
  • Increasing context beyond available memory and causing swapping or severe latency.
  • Assuming zero API cost means zero operating cost.
  • Enabling a gateway before proving the model can complete the normal agent loop.

Verification steps

  1. Confirm Ollama is listening only on the intended interface.
  2. Query /api/tags and record the exact local model ID.
  3. Send a direct /v1/chat/completions request and confirm a valid JSON response.
  4. Start a fresh Hermes session and verify the configured provider, base URL, and model.
  5. Ask Hermes to perform one harmless read-only tool action and confirm the model returns a valid tool call.
  6. Monitor memory, latency, and ollama ps during a representative task before enabling cron or gateway use.
  7. If a fallback exists, test and document when data leaves the local machine.

Official documentation

https://hermes-agent.nousresearch.com/docs/guides/local-ollama-setup

Related pages