I started running language models locally because it was technically interesting. I kept running them because a few small jobs turned out to be genuinely useful.

That distinction took some time. The first temptation was to route everything through a local model: chat, coding, search, home automation, maybe even a cheerful summary of the weather. Most of those experiments added latency and another service to maintain without improving my day.

The jobs that stuck share a pattern. They are narrow, tolerant of imperfect wording, and benefit from keeping the input on my own hardware. None of them require a model to be an oracle.

Turning noisy text into a first pass

My most reliable use is classification and cleanup. Logs, alerts, copied notes, and transcripts arrive in inconsistent shapes. A local model can turn them into a predictable first pass: a short summary, a category, and a list of possible actions.

I deliberately call it a first pass. If a monitoring alert says a disk is failing, the model does not decide whether to replace it. It extracts the host, device, symptoms, and relevant lines so I can inspect the evidence faster.

This works well because the output format is constrained. I ask for JSON with a small schema, validate it, and reject anything malformed. The surrounding script handles timestamps, file paths, and routing. The model only handles the fuzzy language part.

Language models are good at transforming text. They are a poor substitute for jq, a database query, or an exact threshold check.

Searching my own notes

The second job is local question answering over material I wrote or saved. I use embeddings to retrieve a few relevant chunks, then give only those chunks to the model with instructions to cite the source files.

This is less magical than the demos suggest, which is why it is useful. It does not “know” my notes. It helps me find the paragraph where I documented a reverse proxy exception or the checklist I used when replacing a drive.

The retrieval layer does most of the important work. Chunk size, filenames, headings, and good source text matter more than an elaborate system prompt. If retrieval returns irrelevant notes, a larger model mostly produces a more confident distraction.

I also keep the original excerpt beside every answer. For operational notes, verification should be one glance away.

Drafting commands, not running them

Local models have also earned a place as command-line drafting tools. I can paste an error and ask for likely diagnostic commands, or describe a file transformation and get a starting point for rg, sed, or ffmpeg.

The useful safety boundary is that generated commands print to the terminal or a scratch file. They do not execute automatically. I want to inspect paths, flags, and quoting before anything touches a server.

For recurring operations, the final answer becomes a script or an Ansible task that can be reviewed and tested. I do not keep asking the model to improvise the same procedure. The model helps discover the automation; deterministic tooling owns it afterward.

Private drafts and redaction

The least flashy job may be the one I use most: working with text I would rather not send to a hosted service. A local model can draft a clearer version of rough notes, suggest a title, or identify obvious names and credentials before I share an excerpt elsewhere.

Local processing is not automatic privacy. Prompts can still land in shell history, application logs, backups, or an unsecured web interface. I treat the model server like any other internal service: bind it deliberately, restrict access, review logging, and update it.

I also do not assume redaction is complete just because a model says it is. Pattern matching catches structured secrets; manual review catches context. The model is an extra pass, not a compliance system.

What did not stick

I stopped using local models for facts that need to be current, large coding tasks that exceed the available context, and automation where one wrong answer has consequences. I also abandoned “AI everywhere” integrations that needed a paragraph of prompt engineering to perform a simple rule.

My test now is straightforward:

  • Is language ambiguity the hard part?
  • Can I verify or discard the output cheaply?
  • Does local processing provide a real privacy or availability benefit?
  • Would ordinary code be simpler?

If the last answer is yes, I write the code.

Local LLMs became useful when I stopped treating them as smaller replacements for hosted frontier models. In my homelab, they are text-processing components with clear limits. That role is less exciting than an all-purpose assistant, but it is the one that survived contact with daily use.