UVF IT: Minimal LLM Gateway with LiteLLM

Welcome to another official UVF IT build guide! In this tutorial we’ll spin up a tiny AI gateway that speaks the OpenAI API format, using the open‑source litellm library. You’ll end up with a local server that can forward requests to any free LLM endpoint (e.g., OpenAI's gpt-3.5-turbo with a free trial key).

We’ll keep the stack ultra‑simple: Python 3.10+, pip, and SQLite (used only for the optional request‑logging demo). No cloud services, no paid APIs – just a few commands and a tiny Flask app.

Before you start…

You need Python 3.10+ and pip. Older versions will fail to install litellm.
The real LiteLLM proxy supports auth, rate‑limiting, and dozens of providers – we only implement a single forward‑to‑OpenAI route.
If you hit OpenAI’s rate limits you’ll see HTTP 429; the clone does not retry automatically.

Minimal LiteLLM Clone Architecture

Phase 1: Scaffold the project

Create a fresh directory for the clone and isolate its dependencies with a virtual environment. This keeps the project tidy and avoids version clashes with other Python work you may have. Once the env is active we install only Flask and LiteLLM – the two libraries the proxy needs to run.

We also generate a basic requirements.txt so you can reinstall the exact versions later or share the project with a teammate. All of these steps are one‑liners you can copy‑paste into a terminal.

When you finish this phase you should have a new folder containing a virtual environment and a src subfolder ready for code. Running python -m pip list inside the env will show flask and litellm among the installed packages.

mkdir minimal_llm_gateway && cd minimal_llm_gateway
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install flask litellm
pip freeze > requirements.txt
mkdir src

Phase 2: Define the data model (SQLite logger)

We add a tiny SQLite helper that records each incoming request: when it arrived, which model was used, and how many tokens were consumed. This log is optional but invaluable when you start debugging latency or cost issues.

The helper creates a requests.db file on first run and exposes two functions – init_db() to ensure the table exists and log_request() to insert a new row. The rest of the proxy can call log_request() after it receives a response from LiteLLM.

Because SQLite lives in a single file, no extra server is required. The code is pure Python and fits in a single module named logger.py inside the src directory.

logger.py

import sqlite3
from pathlib import Path

DB_PATH = Path(__file__).parent / "requests.db"

def init_db():
    """Create the requests table if it doesn't exist yet."""
    with sqlite3.connect(DB_PATH) as conn:
        conn.execute(
            """
            CREATE TABLE IF NOT EXISTS request_log (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                timestamp TEXT NOT NULL,
                model TEXT NOT NULL,
                prompt_tokens INTEGER,
                completion_tokens INTEGER,
                total_tokens INTEGER
            )
            """
        )
        conn.commit()

def log_request(timestamp: str, model: str, prompt_tokens: int, completion_tokens: int, total_tokens: int):
    """Insert a single request record into the DB.
    All arguments are simple scalars; the caller formats the timestamp.
    """
    with sqlite3.connect(DB_PATH) as conn:
        conn.execute(
            "INSERT INTO request_log (timestamp, model, prompt_tokens, completion_tokens, total_tokens) VALUES (?,?,?,?,?)",
            (timestamp, model, prompt_tokens, completion_tokens, total_tokens),
        )
        conn.commit()

# Initialise the DB when the module is first imported
init_db()

Phase 3: Build the proxy API (Flask + litellm)

We will create a tiny Flask app that mimics OpenAI's /v1/chat/completions endpoint. The handler receives the same JSON payload a typical OpenAI client would send, then forwards the relevant fields to LiteLLM's completion function. LiteLLM abstracts many providers, so we only need to pass the model name, messages, and temperature. Finally we format LiteLLM's response back into OpenAI's structure and return it to the caller.

app.py

import os
from flask import Flask, request, jsonify
from litellm import completion

app = Flask(__name__)

# Simple in‑memory API key check (optional but helpful)
EXPECTED_KEY = os.getenv("PROXY_API_KEY", "sk-fake-key")

@app.route("/v1/chat/completions", methods=["POST"])
def chat_completions():
    # Basic auth header validation
    auth = request.headers.get("Authorization", "")
    if not auth.startswith("Bearer ") or auth.split()[1] != EXPECTED_KEY:
        return jsonify({"error": "Invalid API key"}), 401
    data = request.json
    # Extract fields required by LiteLLM
    model = data.get("model")
    messages = data.get("messages", [])
    temperature = data.get("temperature", 0.7)
    # Convert OpenAI message format to LiteLLM's expected list of dicts
    litellm_messages = [{"role": m["role"], "content": m["content"]} for m in messages]
    # Call LiteLLM
    resp = completion(model=model, messages=litellm_messages, temperature=temperature)
    # Build OpenAI‑compatible response
    openai_resp = {
        "id": resp.get("id", "chatcmpl-123"),
        "object": "chat.completion",
        "created": resp.get("created", 0),
        "model": model,
        "choices": [{
            "index": 0,
            "message": {"role": "assistant", "content": resp.get("choices", [{}])[0].get("message", {}).get("content", "")},
            "finish_reason": resp.get("choices", [{}])[0].get("finish_reason", "stop")
        }]
    }
    return jsonify(openai_resp)

if __name__ == "__main__":
    # Default to port 4000 to match the outline
    app.run(host="0.0.0.0", port=4000)

Phase 4: Run locally and test

Now we spin up the Flask proxy and fire a quick curl request that looks exactly like an OpenAI call. Make sure you have set the PROXY_API_KEY environment variable to a dummy key you will also use in the curl header. The response should contain a choices array with the assistant's reply, proving the gateway works end‑to‑end.

# Install dependencies (run once)
python -m pip install flask litellm

# Export a fake API key for the proxy and the test request
export PROXY_API_KEY=sk-testkey123

# Start the Flask server in the background
nohup python app.py > server.log 2>&1 &
# Give it a moment to start
sleep 2

# Send a test request mimicking OpenAI's client
curl -X POST http://127.0.0.1:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $PROXY_API_KEY" \
  -d '{
        "model": "gpt-3.5-turbo",
        "messages": [{"role": "user", "content": "Hello, who are you?"}],
        "temperature": 0.5
      }'

Phase 5: Verify & extend

After the request completes, we can inspect the SQLite log that LiteLLM maintains by default. The log shows each request's model, prompt, response, and timestamps, confirming that the proxy not only forwards but also records activity. From here you might try a different model name (e.g., gpt-4) or add a simple auth check, rate limiting, or even containerize the service for production use.

# LiteLLM stores logs in ~/.litellm/database.db by default
sqlite3 ~/.litellm/database.db "SELECT id, model, created_at FROM request_logs ORDER BY id DESC LIMIT 5;"

# Example of swapping model (no code change needed, just curl again)
curl -X POST http://127.0.0.1:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $PROXY_API_KEY" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Tell me a joke."}], "temperature": 0.7}'

Verifying the Stack

Check	Command / Why it matters
Server is listening on http://127.0.0.1:4000	`netstat -ltnp \| grep 4000` or see Flask startup line in server.log
curl returns a JSON with a `choices` array	The curl command above prints the OpenAI‑style response containing `choices`
SQLite log contains at least one row	`sqlite3 ~/.litellm/database.db "SELECT COUNT(*) FROM request_logs;"` should output a number >= 1

You did it!

Your minimal LiteLLM gateway is now running locally. Feel free to swap the OpenAI key for any other provider supported by litellm, add API‑key checking, or containerize it for deployment. Happy hacking!