LoRA Fine-Tuning on Your Own Hardware: Meta Muse Just Made the Argument for You
A practical homelab guide to LoRA fine-tuning open source models on consumer GPUs, covering QLoRA VRAM needs, Unsloth training code, dataset prep, and when RAG is the better fit.
LoRA Fine-Tuning on Your Own Hardware: Meta Muse Just Made the Argument for You
Ars Technica's piece on Meta Muse exposing users' personal and business secrets when they share conversations landed on HN and sat there at 337 points, quietly. Everyone nodded, nobody was surprised. If you've been paying attention, you already know the pattern: any prompt you send to a hosted model is a data transfer with a company whose interests are not aligned with yours. Your incident postmortems, your customer schemas, your half-drafted legal strategy — all of it goes into someone else's logging pipeline, and their retention policy is a PDF you will never read.
The usual answer is "self-host a model," and I've written about that before — the local helpers post covers the basics of running models on your own box. But there's a step most people skip. A base model, even a good 8B one, is a generalist. It doesn't know your log format. It doesn't know that in your infrastructure "prod" means three clusters, not one. You can stuff all of that into a system prompt and burn 4,000 tokens of context on every call, or you can bake it into the weights with LoRA.
LoRA fine-tuning used to be something you needed a research group for. In 2026, with a single 24GB consumer card and a weekend, it's a practical homelab project. This is how I do it.
What LoRA actually changes
Low-Rank Adaptation freezes the pretrained weights and injects small trainable matrices into specific layers — usually the attention projections and MLP blocks. Instead of updating 8 billion parameters, you update maybe 20-40 million. The adapter file that comes out the other end is often 20-200MB.
That number is the whole point. You're not producing a new foundation model. You're producing a behavioral patch you can attach to a base model at load time, version in Git, and swap in and out without re-downloading a 16GB checkpoint.
There's a wrinkle worth knowing: because the base weights are frozen, you're not teaching the model new facts reliably, and you're definitely not teaching it new reasoning. You are teaching it format, tone, and narrow task behavior. "Always emit a JSON block with these exact keys." "Answer in the voice of our runbook." "Given a CVE description, classify severity using our rubric." That's the sweet spot.
If you need factual recall of your own corpus, a LoRA is the wrong tool — you want retrieval. I wrote up a whole pipeline for that in the RAG with local models and DuckDB 2.0 post, and DuckDB 2.0's LTS release makes the data-shuffling side of this genuinely pleasant now. In practice I use both: RAG for knowledge, LoRA for behavior and format compliance. They solve different problems and arguing about which one "wins" is a waste of a weekend.
The hardware reality
Let's not pretend. You need a GPU with decent VRAM, and you need it exposed to whatever environment you're training in. On a Proxmox host, that means PCIe passthrough into a VM, which is its own afternoon of work — I covered the sharp edges in the Proxmox tips and tricks post. Getting the VFIO binding right and keeping the host from grabbing the card on boot is the difference between a smooth training run and three hours of silence.
Rough VRAM floors for QLoRA (4-bit quantized base + LoRA adapters), which is what you'll actually use on consumer hardware:
| Model size | QLoRA training | LoRA fp16 | Inference (Q4_K_M) |
|---|---|---|---|
| 1-3B | 6GB | 12GB | 2GB |
| 7-8B | 10-12GB | 24GB+ | 5GB |
| 13-14B | 16-20GB | 40GB+ | 8GB |
| 32B | 24GB (tight, short seq) | 80GB+ | 20GB |
A 3090 or 4090 handles an 8B QLoRA fine at 2048-token sequences. A 7B on a 12GB 3060 is doable with a small rank and short context. If you're on CPU-only or an integrated GPU, stop here and go build a better RAG pipeline; I mean that sincerely, not as a brush-off.
The actual workflow
I use Unsloth for this because it's roughly 2x faster than vanilla PEFT on the same card and the memory savings are real, not marketing. Here's the shape of it.
python -m venv .venv && source .venv/bin/activate
pip install "unsloth[cu124-torch260]" trl peft accelerate bitsandbytes datasets
Your dataset should be JSONL with a consistent instruction/input/output shape. I generate mine from real support tickets and runbook entries, then dedupe with a quick DuckDB query and split 95/5 for eval. Fifty to five hundred good examples goes further than a sloppy ten thousand.
from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="meta-llama/Llama-3.1-8B-Instruct",
max_seq_length=2048,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
model,
r=16, # rank; 8-32 is the practical range
lora_alpha=16,
lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
use_gradient_checkpointing="unsloth",
)
ds = load_dataset("json", data_files="train.jsonl", split="train")
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=ds,
args=SFTConfig(
output_dir="./out",
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
num_train_epochs=3,
learning_rate=2e-4,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
logging_steps=5,
save_strategy="epoch",
bf16=True,
),
)
trainer.train()
model.save_pretrained("./adapter")
tokenizer.save_pretrained("./adapter")
Two knobs matter more than the rest. Learning rate: 2e-4 for QLoRA is the safe default; drop to 1e-4 if you see the loss cratering in the first fifty steps, which is how overfitting announces itself. Rank: r=16 is enough for format and tone work. Jumping to r=64 in the hope of teaching the model your entire product manual is the single most common mistake I see. You'll get memorization of your training examples and nothing that generalizes.
Three epochs on a few hundred examples takes 20-40 minutes on a 4090. I've run this while making dinner.
From adapter to something you can actually call
The adapter isn't directly servable by most runtimes. Merge it into the base model and convert to GGUF.
# merge adapter into base weights
python -c "
from unsloth import FastLanguageModel
m, t = FastLanguageModel.from_pretrained('./adapter', load_in_4bit=False)
m.save_pretrained_merged('./merged', t, save_method='merged_16bit')
"
# convert and quantize
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
python convert_hf_to_gguf.py ../merged --outfile ../models/runbook-v1-f16.gguf --outtype f16
./build/bin/llama-quantize ../models/runbook-v1-f16.gguf ../models/runbook-v1-Q4_K_M.gguf Q4_K_M
Then wire it into Ollama with a two-line Modelfile:
FROM ./models/runbook-v1-Q4_K_M.gguf
PARAMETER temperature 0.2
PARAMETER stop "<|eot_id|>"
ollama create runbook-v1 -f Modelfile
ollama run runbook-v1 "classify this alert: disk_pressure on node-7"
That's the full loop. Start it Friday night, have a callable endpoint Saturday morning.
Which tool for which job
I've used all of these across different projects. None is universally right.
| Tool | Best for | VRAM floor (8B QLoRA) | Learning curve | Notes |
|---|---|---|---|---|
| Unsloth | Single-GPU, speed-obsessed | ~10GB | Low | Fastest consumer path; patches upstream HF internals, so breakage happens on major releases |
| Axolotl | YAML-driven, reproducible runs | ~12GB | Medium | Config file checked into Git; good for re-running the same recipe months later |
| LLaMA-Factory | GUI-ish workflow, many model families | ~12GB | Low-Medium | Broad model coverage; abstraction can hide what's actually happening |
| Raw PEFT + TRL | Understanding the mechanics | ~14GB | High | Slowest, but you learn why the others exist |
| mlx-lm | Apple Silicon | N/A | Low | Genuinely works on an M-series Mac; different ecosystem, don't expect portability |
My opinion: start with Unsloth to get a result, then move to Axolotl once you have a recipe you care about reproducing. Raw PEFT is worth doing once, on a tiny model, just so the abstractions stop being magic.
Evaluation is where everyone lies to themselves
Here's the honest caveat: the training loss going down tells you almost nothing. I've trained maybe a dozen adapters over the last two years and only four are still running anywhere. The rest looked fine on the loss curve and were subtly worse than the base model on general prompts while being marginally better on the narrow task.
So before you train, hold out 20-50 examples and write down what a correct answer looks like. After training, run both the base model and your adapter against that set, blind if you can manage it. I keep a eval.jsonl in the same repo as the training data and a small script that dumps both outputs side by side. It takes ten minutes to build and saves you from shipping a regression.
Also watch for the adapter degrading general capability. If your model suddenly can't answer a simple factual question it handled before, your rank is too high, your learning rate is too high, or your dataset is too narrow. Mix in 10-20% general instruction data if you're finetuning for something very specific.
What to do this weekend
If the Meta Muse story made you uneasy, don't just move your prompts to a different vendor. Pick one narrow, repetitive task you currently paste into a hosted chat — alert classification, commit message formatting, turning messy logs into summary lines — and fine-tune an 8B model on 200 real examples of it. Run it locally. Keep the data on your own disk.
Version the adapter directory in Git alongside your training config and eval set. When you inevitably retrain in three months with better data, you'll want to know exactly what changed.
And measure it. If the adapter isn't reliably beating the base model on your held-out set, don't ship it — you've just added 150MB and a merge step for nothing.
Recommended tools & services
Disclosure: some links below are affiliate links. If you buy through them I may earn a commission at no extra cost to you. I only list tools I actually run.
| Tool | Why it is here | |
|---|---|---|
| Amazon - SSDs & NVMe | Boot drives and storage for your homelab | Visit |
| Amazon - Mini PCs | Low-power always-on nodes (ProDesk, EliteDesk, N100 boxes) | Visit |
| Amazon - PC Components | CPUs, RAM and cases for a build | Visit |
| Amazon - GPUs | Consumer GPUs for local inference (16GB VRAM is the sweet spot) | Visit |
| Amazon - RAM & Memory | More RAM is the cheapest upgrade for VMs and containers | Visit |