Fine-tuning is useful for consistent formats, a particular tone, domain jargon or narrow tasks where prompting falls short. Parameter-efficient methods such as LoRA train only small adapter weights, making it affordable on a single GPU.
For adding knowledge that changes often, retrieval is usually better than fine-tuning. See the LoRA fine-tuning overview.