Fine-tune a small LLM for legal labeling

81.7% shows a 3B SmolLM3 can beat frontier models on one legal-labeling task.

This guide is for developers who want a practical path from a general open model to a specialist that handles one repeated decision well.

By the end, you will have a fine-tuned small language model, a repeatable evaluation loop, and a clear rule for when to route hard requests to a larger model.

Before you start

• An account with access to a GPU host or local machine with an NVIDIA GPU.
• Python 3.10+ and pip 23+.
• Node is not required.
• Hugging Face account and access token.
• PyTorch 2.2+ with CUDA 12.1 or newer.
• Transformers 4.40+, Datasets 2.19+, PEFT 0.11+, and Accelerate 0.30+.
• A labeled legal contract dataset in CSV or JSONL form.
• Enough disk space for the base model, adapter weights, and evaluation set.

Step 1: Pick a narrow legal task

Your first outcome is a task definition that a small model can learn from examples instead of broad reasoning.

Choose one repeated decision, such as contract clause tagging, support-ticket routing, or legal issue classification. Keep the label set small and stable, and write one sentence that defines the input and one sentence that defines the expected output.

For example, use contract text as input and one of a fixed set of tags as output. If humans cannot label the same example consistently, the model will not learn a clean pattern.

You should see a task spec with clear labels, example inputs, and a short success criterion such as “match human tags on held-out contracts.”

Step 2: Prepare the dataset for LoRA fine-tuning

Your second outcome is a training file the model can consume without extra cleanup during training.

Split your data into train, validation, and test sets. Remove duplicates, normalize label names, and convert each record into a prompt-response pair. If you use JSONL, keep one example per line so you can stream the data into training.

{"prompt":"Classify this clause: ...","response":"termination"}

Make sure the validation and test sets contain contracts the model has never seen. You should see counts for each split and no overlap between them.

Step 3: Fine-tune SmolLM3 with LoRA

Your third outcome is an adapter checkpoint that teaches the base model your legal labels without retraining every weight.

Start from a small open base model such as SmolLM3, then attach LoRA adapters and train only the low-rank layers. This keeps the run cheap enough for a single GPU and makes it practical to iterate on the dataset.

Use your framework of choice to launch training, then save only the adapter weights and tokenizer files. Keep the base model frozen so you can compare runs fairly.

You should see training loss fall, validation loss stabilize, and an adapter folder appear at the end of the run.

Step 4: Measure against frontier baselines

Your fourth outcome is an honest scorecard that tells you whether the specialist model is actually useful.

Run the same held-out test set through the fine-tuned model and through one or two frontier baselines, then compare exact-match accuracy or macro F1. In the source benchmark, the fine-tuned 3B model finished at 81.7% after 74 minutes, ahead of GPT-5.5 at 76.7% and Claude Sonnet 4.6 at 77% on that narrow task.

Track both quality and operational cost. A small model can win on one workflow even if it is not better at open-ended reasoning. You should see a table of predictions, scores by model, and a clear winner for your target task.

Step 5: Add routing for hard cases

Your fifth outcome is a production path that keeps cheap requests on the small model and escalates edge cases to a larger one.

Set a confidence threshold or a rule-based fallback. If the small model is uncertain, send the request to a frontier model and log both decisions. This gives you low latency for common cases and a safety net for ambiguous ones.

Keep the router simple at first. You can route by score margin, label entropy, or a classifier trained on past failures. You should see most traffic handled by the small model and a smaller share forwarded upstream.

Step 6: Ship a monitoring loop

Your sixth outcome is a system that stays accurate after the first release.

Create a weekly review set from real production examples, then re-run the same evaluation script after every dataset or prompt change. Watch for label drift, hallucinated tags, and regressions after adapter updates.

If performance drops, retrain on fresh examples before expanding the model’s scope. You should see a stable dashboard with live accuracy, fallback rate, and a list of recent failures.

Common mistakes

• Using a broad task with vague labels. Fix: narrow the scope to one repeated decision and define labels before training.
• Training on noisy or duplicated examples. Fix: deduplicate, normalize labels, and hold out a clean test set.
• Skipping fallback routing. Fix: send low-confidence requests to a frontier model and log every escalation.

What's next

After this, try adding retrieval for fresh policy text, distillation for cheaper inference, or quantization so the same specialist can run on smaller hardware.