Using Ollama as an LLM endpoint on FRCE

Ollama can be used on FRCE to run an open-weight large language model (LLM) as an HTTP service. A Slurm job allocates the required compute resources, starts Ollama on the assigned compute node, and makes the model available through both an OpenAI-compatible API and the native Ollama API.

This is useful when an application expects to communicate with an LLM through an API rather than running the model itself. For example, Biomni, LangChain, LiteLLM, the OpenAI Python SDK, and other tools can use a configurable model endpoint. Running the endpoint on FRCE allows model inference to occur using FRCE GPU resources rather than requiring an externally hosted model API.

An Ollama endpoint can also be useful for testing or comparing open models, developing workflows around a particular model, or providing several tools with a consistent API for accessing an LLM.

Important: The endpoint exists only while the Slurm job is running. It should be thought of as a temporary service associated with a compute allocation, not as a permanently running FRCE service.

How it works

1. Submit the Ollama server job to Slurm using sbatch.
2. Slurm assigns a GPU compute node and starts the Ollama server there.
3. The job writes an .info file containing the hostname, port, model, and API addresses.
4. Configure the application that needs an LLM to use the endpoint listed in the .info file.
5. Slurm captures output from the job and Ollama server in a .log file.

Starting an endpoint

Save the following example as ollama-endpoint.sbatch and submit it from an FRCE login node:

sbatch ollama-endpoint.sbatch

This example requests one L40S GPU, 8 CPU cores, 96 GB of memory, and an eight-hour allocation and serves qwen3-coder:30b. These are starting values rather than requirements. See Customizing the job below for selecting an available GPU, choosing another model, and adjusting the requested resources.

#!/bin/bash
#SBATCH --job-name=ollama-endpoint
#SBATCH --partition=gpu
#SBATCH --gres=gpu:l40s:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=96G
#SBATCH --time=08:00:00
#SBATCH --output=ollama-endpoint-%j.log
#
# Serves an open-weight model from a Slurm allocation as an HTTP endpoint.
# Any OpenAI-compatible client can point at it: LiteLLM, LangChain, the
# openai SDK, Continue, Aider, or your own code.
#
#   sbatch ollama-endpoint.sbatch
#   cat  ollama-endpoint-<jobid>.info     <- connection details
#   tail -f ollama-endpoint-<jobid>.log   <- server log
#
# Overrides:
#   sbatch --export=ALL,MODEL=llama3.1:70b,PORT=11500 ollama-endpoint.sbatch
#
#   MODEL          model to serve; must already be in the store (ollama list)
#   PORT           listen port                      (default 11434)
#   CONTEXT        context window in tokens         (default 65536)
#   OLLAMA_MODELS  model store path

set -euo pipefail

source /etc/profile.d/modules.sh 2>/dev/null || true
module load ollama

# The site module points OLLAMA_MODELS at $HOME/.ollama/models, a per-user
# copy seeded by user_sync.sh. Use the shared store instead.
# Verify contents with:  OLLAMA_MODELS=<path> ollama list
export OLLAMA_MODELS="${OLLAMA_MODELS:-/mnt/nasapps/production/ollama/models}"

MODEL="${MODEL:-qwen3-coder:30b}"
PORT="${PORT:-11434}"
CONTEXT="${CONTEXT:-65536}"

export OLLAMA_HOST="0.0.0.0:${PORT}"
export OLLAMA_CONTEXT_LENGTH="${CONTEXT}"
export OLLAMA_KEEP_ALIVE=-1     # keep the model resident for the whole job
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_NUM_PARALLEL=1    # one client, full context, no KV cache split

NODE=$(hostname -f)
INFO="${SLURM_SUBMIT_DIR:-$PWD}/ollama-endpoint-${SLURM_JOB_ID}.info"

echo "node=${NODE} port=${PORT} gpus=${CUDA_VISIBLE_DEVICES:-none}"
echo "OLLAMA_MODELS=${OLLAMA_MODELS}"

ollama serve &
SERVER_PID=$!
trap 'kill $SERVER_PID 2>/dev/null || true' EXIT

for _ in $(seq 1 60); do
  curl -sf "http://127.0.0.1:${PORT}/api/tags" >/dev/null && break
  sleep 2
done

# Written before warm-up so it exists even if the model load is slow.
cat > "$INFO" <<EOF
JOB=${SLURM_JOB_ID}
NODE=${NODE}
PORT=${PORT}
MODEL=${MODEL}
CONTEXT=${CONTEXT}

# OpenAI-compatible API - most clients, LiteLLM, LangChain, openai SDK
OPENAI_BASE_URL=http://${NODE}:${PORT}/v1
OPENAI_API_KEY=not-needed

# Native Ollama API - the ollama python package, langchain_ollama
# (langchain_ollama ignores its base_url argument; set OLLAMA_HOST instead)
OLLAMA_HOST=http://${NODE}:${PORT}

# Quick check:
#   curl http://${NODE}:${PORT}/v1/chat/completions \
#     -d '{"model":"${MODEL}","messages":[{"role":"user","content":"hi"}]}'
EOF

# No pull. The model must already be in the store; a pull against a read-only
# shared store would kill the job under set -e.
echo "warming ${MODEL}, first load off NFS can take several minutes..."
curl -s --max-time 900 "http://127.0.0.1:${PORT}/api/generate" \
  -d "{\"model\":\"${MODEL}\",\"prompt\":\"ready\",\"stream\":false}" >/dev/null \
  && echo "warm-up ok" \
  || echo "WARNING: warm-up did not complete; first request will be slow"

echo "=============================================================="
cat "$INFO"
echo "=============================================================="
echo "connection details written to: ${INFO}"

wait $SERVER_PID

Customizing the job

The GPU type, CPU and memory allocation, maximum running time, model, port, and context size can all be changed. The appropriate settings depend on the model and the application using the endpoint.

Choosing a GPU

FRCE has several types of GPUs. Before submitting the endpoint job, use freen from an FRCE login node to see current availability:

freen

Look for the entries in the gpu partition. The GPU type is shown in parentheses and the FreeGPUs column shows current availability. For example, output may contain entries similar to:

Partition      FreeNds      FreeCPUs      FreeGPUs
----------------------------------------------------
gpu (h200)      0 / 3       192 / 288      0 / 8
gpu (p100)     11 / 18      495 / 632     35 / 46
gpu (v100)      0 / 9        96 / 324     22 / 72
gpu (a100)      1 / 2        63 / 64       3 / 4
gpu (l4)        2 / 2        48 / 48       8 / 8
gpu (v100)      5 / 6       200 / 240     40 / 48
gpu (l40s)      6 / 15      624 / 800     30 / 48

Availability changes continuously as jobs start and finish, so freen provides a current snapshot rather than a guarantee that a resource will still be free when the job is submitted.

The example requests a single L40S:

#SBATCH --partition=gpu
#SBATCH --gres=gpu:l40s:1

To request a different GPU type, change l40s to an appropriate available type such as l4, a100, v100, or h200. The final number specifies the number of GPUs requested.

Different GPU types have different amounts of GPU memory and different performance characteristics. Smaller models can generally run on a wider range of GPUs, while larger models and large context windows require more GPU memory. The size displayed by ollama list is the stored model size and should not be treated as the complete amount of GPU memory required at runtime.

Choosing a model

A selection of Ollama models is maintained on FRCE shared storage. Load the Ollama module and use ollama list from a login node to see what is currently available:

module load ollama
ollama list

For example, the shared store may contain models such as:

This list is only an example. Run ollama list to see the models currently installed on FRCE.

For initial testing, a smaller model can be useful because it loads more quickly and requires fewer GPU resources. A larger or more specialized model can then be selected if the application benefits from it.

The default model is set in the example script with:

MODEL="${MODEL:-qwen3-coder:30b}"

You can edit that line or select a model when submitting the job without changing the script:

sbatch --export=ALL,MODEL=qwen3:8b ollama-endpoint.sbatch

Using your own Ollama models

The example endpoint uses the shared FRCE model store. This avoids maintaining duplicate copies of commonly used models and is the preferred choice when the model you need is already available.

Users may also maintain their own Ollama model store. To download a model that is not in the shared store, point OLLAMA_MODELS at a writable directory you own before running ollama pull. For example:

module load ollama
export OLLAMA_MODELS=$HOME/.ollama/models
ollama pull <model-name>
ollama list

The endpoint job must use the same OLLAMA_MODELS location when serving that model. The example script is already written to honor an OLLAMA_MODELS value supplied by the user, so this can be done without editing the script:

sbatch --export=ALL,OLLAMA_MODELS=$HOME/.ollama/models,MODEL=<model-name> ollama-endpoint.sbatch

Alternatively, change the default model-store line in a personal copy of the script:

export OLLAMA_MODELS="${OLLAMA_MODELS:-$HOME/.ollama/models}"

Keep in mind that models can consume substantial storage space. When a suitable model is already available in the shared FRCE store, using the shared copy avoids storing a duplicate in your home or project space.

Other Slurm and Ollama settings

Tip: For a first test, start with a relatively small model and a GPU that is currently available. Once the application is successfully using the endpoint, increase the model size or context window if the workflow benefits from it.

Several Ollama settings can be changed when submitting the job without editing the script. For example:

sbatch --export=ALL,MODEL=llama3.1:70b,CONTEXT=32768,PORT=11500 ollama-endpoint.sbatch

Slurm resource requests such as the GPU type, number of GPUs, memory, and time can be changed in the #SBATCH directives or supplied as options to sbatch.

Finding the endpoint

When the job starts, it creates an information file in the directory from which the job was submitted:

ollama-endpoint-<jobid>.info

For example, if Slurm reports job ID 123456, view the connection information with:

cat ollama-endpoint-123456.info

The file will look similar to:

JOB=123456
NODE=compute-node.example.gov
PORT=11434
MODEL=qwen3-coder:30b
CONTEXT=65536

OPENAI_BASE_URL=http://compute-node.example.gov:11434/v1
OPENAI_API_KEY=not-needed

OLLAMA_HOST=http://compute-node.example.gov:11434

In most cases, an application that supports an OpenAI-compatible API should be given the value of OPENAI_BASE_URL. Applications that use the native Ollama API should instead use the value of OLLAMA_HOST.

Important: FRCE compute nodes are not directly accessible from outside the FRCE environment. The Ollama endpoint therefore cannot be called directly from a desktop, laptop, or other external system. The application using the endpoint must itself be running on a system within FRCE that can communicate with the assigned compute node.

Using the endpoint with another application

The application using the Ollama endpoint must run within the FRCE environment. For example, a Biomni workflow, Python program, Jupyter session, or other LLM-enabled application running on FRCE can connect to the endpoint using the information in the .info file.

The endpoint cannot be accessed directly from a desktop or other system outside FRCE.

Many LLM-enabled applications allow a custom OpenAI-compatible server to be specified. For these applications, use the values reported by the .info file:

OPENAI_BASE_URL=http://<node>:<port>/v1 OPENAI_API_KEY=not-needed

For example, an application such as Biomni can be configured to use a custom model-serving endpoint instead of a hosted model API. Run Biomni within FRCE, use the model name reported by MODEL, and point its custom OpenAI-compatible base URL at OPENAI_BASE_URL.

Other software may use names such as base_url, api_base, or endpoint for the same setting. The important value for an OpenAI-compatible client is the URL ending in /v1.

Using an FRCE endpoint means that the LLM inference itself occurs on FRCE. The application calling the model may have other network connections or services of its own, so using an FRCE-hosted model endpoint does not by itself mean that every component of the application operates locally.

Understanding the .info and .log files

The .info file is intentionally written before model warm-up is attempted. This means that connection information may already be available while a large model is still loading.

Follow the log while the endpoint starts with:

tail -f ollama-endpoint-<jobid>.log

A successful model warm-up will report:

warm-up ok

If warm-up takes too long or does not complete, the job reports a warning and continues running. In that case, the first request to the model may take longer while the model is loaded.

Testing the endpoint

Once the endpoint is running, a quick OpenAI-compatible request can be made using the node, port, and model reported in the .info file:

curl http://<node>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model>","messages":[{"role":"user","content":"hi"}]}'

A successful response confirms that Ollama is running, the requested model can be loaded, and the client can reach the compute node.

Checking or stopping the endpoint

Because the Ollama server is running as a Slurm job, standard Slurm commands can be used to check or stop it.

Check the job:

squeue -j <jobid>

Stop the endpoint before its requested time limit:

scancel <jobid>

When the Slurm job ends, the Ollama server and its HTTP endpoint also end. The .info and .log files remain in the submission directory, but the address recorded in the .info file is no longer an active endpoint. Start a new job to create another endpoint.

Access considerations

This example configures Ollama to listen on the FRCE compute-node network interface. The endpoint is intended for use by applications running within FRCE and is not directly reachable from systems outside the FRCE environment.

OPENAI_API_KEY=not-needed is provided only for client libraries that require an API-key value. It is not an authentication mechanism.