Learn when to use cloud models vs local open source models for ops workloads. Run Llama locally with Ollama for sensitive data and high-volume tasks — zero data leaves your network.
### You have more options than just OpenAI Most people building with LLMs default to OpenAI. It is easy, fast, and the models are excellent. But for ops workloads specifically, defaulting to cloud APIs creates real problems: **Log data contains sensitive information** — IP addresses, user IDs, internal service names, database schemas. Sending this to an external API may violate your security policy or compliance requirements. **High query volume gets expensive fast** — If you are analyzing every alert, every log anomaly, and every deployment across a large infrastructure, the token costs add up quickly. **Latency matters during incidents** — An API call to a US data centre from India adds 200-400ms per call. During a fast-moving incident where you are making multiple calls, this matters. **Air-gapped environments** — Some production environments have no outbound internet access by policy. Cloud APIs are simply not available. This does not mean cloud models are wrong. It means you need to choose deliberately. ---
### Cloud models — what they are good at | Model | Provider | Best For in Ops | |-------|----------|----------------| | GPT-4o | OpenAI | Complex RCA, multi-step reasoning, postmortem writing | | GPT-4o-mini | OpenAI | High-volume simpler tasks, alert classification, summaries | | Claude Sonnet | Anthropic | Long context analysis, large log files, architecture review | | Claude Haiku | Anthropic | Fast, cheap, good for classification and structured output | | Gemini 1.5 Flash | Google | Very long context (1M tokens), good for large log analysis | **When cloud is the right choice:** * You need the best possible reasoning quality * Data sensitivity is not a concern * Query volume is moderate * You want zero infrastructure to manage ### Open source models — what they are good at | Model | Size | Best For in Ops | |-------|------|----------------| | Llama 3.1 8B | 8B params | Fast local inference, alert triage, simple classification | | Llama 3.1 70B | 70B params | Better reasoning, closer to GPT-4o-mini quality locally | | Mistral 7B | 7B params | Fast, efficient, good instruction following | | CodeLlama 13B | 13B params | Code generation, script writing, config analysis | | Phi-3 Mini | 3.8B params | Extremely fast, works on CPU, good for simple tasks | **When local/open source is the right choice:** * Log data, credentials, or PII cannot leave your network * Air-gapped environment * High query volume where API costs would be prohibitive * You need sub-100ms latency * You want full control over the model version ---
### What Ollama is Ollama is an open source tool that makes running LLMs locally as simple as running a Docker container. It handles model downloading, memory management, and exposes a simple API that works the same way as OpenAI's API. You install it once, pull a model like `ollama pull llama3.1`, and immediately have a local LLM running. ### Installing Ollama **Linux:** ```bash curl -fsSL https://ollama.com/install.sh | sh ``` **Mac:** ```bash # Download from https://ollama.com/download # Or with Homebrew: brew install ollama ``` **Start the Ollama server:** ```bash ollama serve # Runs on http://localhost:11434 by default ``` ### Pulling and running models ```bash # Pull a model — downloads it to your machine (~4-5GB for 8B models) ollama pull llama3.1 # Pull a smaller, faster model for testing ollama pull phi3 # Run a model interactively in the terminal ollama run llama3.1 # List models you have downloaded ollama list # Check how much memory a model needs ollama show llama3.1 --modelfile ``` ### Choosing the right model size for your hardware | RAM Available | Recommended Model | Why | |--------------|-------------------|-----| | 8GB | Phi-3 Mini (3.8B) | Fits comfortably, reasonable quality | | 16GB | Llama 3.1 8B | Good balance of speed and quality | | 32GB | Llama 3.1 8B or Mistral 7B | Fast inference, good quality | | 64GB+ | Llama 3.1 70B | Near GPT-4o-mini quality locally | If you have a GPU with 8GB+ VRAM, inference is 5-10x faster than CPU. ### Using Ollama from Python Ollama exposes an API that is compatible with OpenAI's format. You can use it with the OpenAI Python library by changing the base URL: ```python from openai import OpenAI # Point the OpenAI client at your local Ollama server # This means you can swap between local and cloud with one line change client = OpenAI( base_url="http://localhost:11434/v1", api_key="ollama" # Ollama does not need a real key — this is a placeholder ) def analyze_alert_locally(alert_text: str) -> str: """ Send an alert to a locally running Llama model for analysis. No data leaves your machine. """ response = client.chat.completions.create( model="llama3.1", # must match the model you pulled with ollama pull messages=[ { "role": "system", "content": "You are a senior SRE. Analyze alerts and give brief, actionable responses." }, { "role": "user", "content": alert_text } ], temperature=0.1 # low temperature = more consistent, deterministic responses ) return response.choices[0].message.content # Test it alert = """ ALERT: HighMemoryUsage Service: redis-cache Current: 89% Threshold: 80% Duration: 12 minutes """ result = analyze_alert_locally(alert) print(result) ``` ### Switching between local and cloud with one config change A good pattern is to make your code work with either local or cloud: ```python import os from openai import OpenAI def get_llm_client(use_local: bool = False): """ Return an LLM client configured for either local Ollama or cloud OpenAI based on the use_local flag. In production: set USE_LOCAL_LLM=true in your environment for sensitive workloads, false for complex reasoning tasks. """ if use_local: return OpenAI( base_url="http://localhost:11434/v1", api_key="ollama" ), "llama3.1" else: return OpenAI( api_key=os.getenv("OPENAI_API_KEY") ), "gpt-4o-mini" # Usage — swap between local and cloud with one environment variable use_local = os.getenv("USE_LOCAL_LLM", "false").lower() == "true" client, model_name = get_llm_client(use_local=use_local) response = client.chat.completions.create( model=model_name, messages=[{"role": "user", "content": "Explain a Redis connection timeout"}] ) print(response.choices[0].message.content) ``` > **Note:** The `temperature` parameter controls how random or creative the model's responses are. For ops tasks where you want consistent, factual answers — set it low (0.0 to 0.2). For creative tasks like generating documentation, higher temperature (0.5 to 0.8) produces more varied output. ---
### Map your ops tasks to the right model Not all ops tasks need the same model. Using a powerful (and expensive) cloud model for every task is like using a sledgehammer to hang a picture frame. | Task | Complexity | Recommended | |------|-----------|-------------| | Alert classification — is this a real alert or noise? | Low | Local Phi-3 or GPT-4o-mini | | Log summarization — what happened in these 100 lines? | Low-Medium | Local Llama 3.1 8B | | RCA — what is the root cause of this incident? | High | GPT-4o or Claude Sonnet | | Runbook generation — write a runbook for this alert | Medium | GPT-4o-mini or local Llama 70B | | Postmortem writing — draft a full postmortem | High | GPT-4o or Claude Sonnet | | Code generation — write a Python monitoring script | Medium | CodeLlama or GPT-4o-mini | | Sensitive log analysis — logs with PII or credentials | Any | Local only, never cloud | | High-volume triage — 500 alerts per hour | High volume | Local model, cost-prohibitive on cloud | ### The four questions to ask when choosing **1. Can this data leave my network?** If no → local model only, full stop. **2. How complex is the reasoning required?** Simple classification or summary → smaller, faster model is fine. Multi-step diagnosis, nuanced analysis → use a frontier cloud model. **3. What is the query volume?** Under 1000 queries/day → cloud API is usually affordable. Over 10,000 queries/day → calculate costs, local may be cheaper. **4. What does latency look like?** Real-time incident response → local models on good hardware win. Async batch processing → cloud latency is not a problem. ### Cost comparison example Suppose your AIOps system analyzes 5000 alerts per day, each requiring approximately 500 input tokens and 200 output tokens. **GPT-4o-mini:** * Input: 5000 × 500 = 2.5M tokens × $0.00015 = **$0.375/day = ~$137/year** **Local Llama 3.1 8B on a ₹40,000 server:** * Electricity + hardware amortized: **~₹500/month** * API cost: **₹0** At high volume, local pays for itself in months. At low volume, cloud is simpler and cheaper than buying hardware. ---
### Running Ollama as a persistent service For a team setup where multiple engineers or systems query the same local model: ```bash # Run Ollama as a systemd service (Linux) sudo systemctl enable ollama sudo systemctl start ollama # Or run with Docker for better isolation docker run -d \ --name ollama \ -p 11434:11434 \ -v ollama_data:/root/.ollama \ ollama/ollama # Inside the container, pull your model docker exec ollama ollama pull llama3.1 ``` ### A minimal local model API wrapper ```python # local_model_api.py # A simple FastAPI wrapper around Ollama for your ops team from fastapi import FastAPI from pydantic import BaseModel from openai import OpenAI app = FastAPI() # Connect to local Ollama ollama_client = OpenAI( base_url="http://localhost:11434/v1", api_key="ollama" ) class AnalysisRequest(BaseModel): text: str # the alert, log, or question to analyze task: str # "rca", "summarize", "classify" @app.post("/analyze") def analyze(request: AnalysisRequest): """ Analyze ops text using the local model. Accepts alert text, log lines, or questions. Returns the model's analysis. """ # Different system prompts for different tasks system_prompts = { "rca": "You are a senior SRE. Identify the root cause and next steps.", "summarize": "Summarize this in 3 bullet points for an on-call engineer.", "classify": "Classify this alert as: CRITICAL, WARNING, or INFO. One word only." } system = system_prompts.get(request.task, system_prompts["summarize"]) response = ollama_client.chat.completions.create( model="llama3.1", messages=[ {"role": "system", "content": system}, {"role": "user", "content": request.text} ], temperature=0.1 ) return { "task": request.task, "result": response.choices[0].message.content, "model": "llama3.1-local" } ``` Run it: ```bash pip install fastapi uvicorn openai uvicorn local_model_api:app --host 0.0.0.0 --port 8080 ``` Any system on your network can now call `http://your-server:8080/analyze` and get AI analysis without any data leaving your infrastructure. ---
### Do not trust benchmarks alone Public benchmarks like MMLU measure general knowledge. They do not measure how well a model handles your specific runbooks, your alert formats, or your team's terminology. Before committing to a model, test it on your actual data: ```python # model_eval.py # Quick evaluation of a model on your real ops data test_cases = [ { "input": "Redis memory at 89%. Logs show: connection timeout after 3000ms", "expected_keywords": ["memory", "redis", "threshold", "flush", "bigkeys"] }, { "input": "Pod CrashLoopBackOff on auth-service. Last exit code: 137", "expected_keywords": ["OOM", "memory limit", "kubectl logs", "previous"] }, { "input": "Classify this: CPU at 45%, all services healthy, no errors", "expected_keywords": ["INFO", "normal", "no action"] } ] def evaluate_model(client, model_name: str, test_cases: list) -> dict: """ Test a model on your real ops scenarios. Returns a simple pass/fail score based on keyword presence. """ passed = 0 for case in test_cases: response = client.chat.completions.create( model=model_name, messages=[ {"role": "system", "content": "You are a senior SRE."}, {"role": "user", "content": case["input"]} ], temperature=0.1 ) answer = response.choices[0].message.content.lower() # Check if response contains expected keywords hits = sum(1 for kw in case["expected_keywords"] if kw.lower() in answer) score = hits / len(case["expected_keywords"]) if score >= 0.5: # at least half the expected keywords present passed += 1 print(f"PASS ({score:.0%}): {case['input'][:50]}...") else: print(f"FAIL ({score:.0%}): {case['input'][:50]}...") print(f" Response: {answer[:100]}...") return {"model": model_name, "score": f"{passed}/{len(test_cases)}"} # Run eval on both local and cloud from openai import OpenAI local_client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") cloud_client = OpenAI() print("=== Local Llama 3.1 ===") local_result = evaluate_model(local_client, "llama3.1", test_cases) print("\n=== GPT-4o-mini ===") cloud_result = evaluate_model(cloud_client, "gpt-4o-mini", test_cases) print(f"\nResults: Local={local_result['score']} Cloud={cloud_result['score']}") ``` Run this evaluation before choosing a model for production. The results will tell you whether the quality difference justifies the cost and data sharing trade-off.
You have more options than just OpenAI Most people building with LLMs default to OpenAI. It is easy, fast, and the model...
Cloud models — what they are good at Model Provider Best For in Ops GPT-4o OpenAI Complex RCA, multi-step reasoning, pos...
What Ollama is Ollama is an open source tool that makes running LLMs locally as simple as running a Docker container. It...
Map your ops tasks to the right model Not all ops tasks need the same model. Using a powerful (and expensive) cloud mode...
Running Ollama as a persistent service For a team setup where multiple engineers or systems query the same local model: ...
Do not trust benchmarks alone Public benchmarks like MMLU measure general knowledge. They do not measure how well a mode...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.