Every enterprise is experimenting with large language models. Most are using them wrong. They're feeding sensitive corporate data into public APIs, accepting generic outputs that miss domain nuances, and wondering why the ROI isn't materialising. The gap between a demo and a production AI system is a custom model trained on your data, deployed on your infrastructure, governed by your policies.
This guide walks through the complete spectrum of LLM customisation — from zero-effort prompt engineering to training a model from scratch — with honest assessments of when each approach makes sense, what it costs, what it requires, and where enterprises get it wrong.
Why Aren't Off-the-Shelf LLMs Enough for Enterprise Use?
General-purpose LLMs like GPT-4, Claude, and Gemini are trained on internet-scale public data. They're remarkably capable at general tasks, but they share fundamental limitations for enterprise deployment:
- No proprietary knowledge. They don't know your products, internal processes, customer history, regulatory obligations, or institutional knowledge. A healthcare LLM that doesn't know your hospital's clinical protocols is a liability, not an asset.
- Generic domain understanding. They use general terminology when your industry has specific, consequential vocabulary. In legal, financial, and medical contexts, the difference between synonyms can be the difference between compliance and violation.
- No data sovereignty. Using cloud APIs means your queries — and the proprietary data they contain — leave your infrastructure. For regulated industries, this is often a non-starter under PDPA, HIPAA, or equivalent frameworks.
- Inconsistent output quality. Without domain-specific training, outputs require heavy human review. The 80% accuracy that impresses in demos falls apart in production, where every edge case matters.
- No competitive moat. If your AI uses the same model as your competitors, with the same training data, your AI capabilities are identical. Custom models on proprietary data create defensible advantage.
The good news: you don't need to build a model from scratch to close these gaps. The customisation spectrum offers multiple on-ramps at different cost, complexity, and capability levels.
What Is the LLM Customisation Spectrum?
Think of LLM customisation as a spectrum from lightweight to heavyweight. Each level requires more data, more compute, more expertise, and more time — but delivers deeper customisation. The art is choosing the minimum intervention that meets your requirements.
Level 1: Prompt Engineering. Zero training required. You craft system prompts, few-shot examples, and chain-of-thought instructions that guide the base model toward your desired behaviour. This is where every enterprise should start. It costs nothing beyond engineering time, can be iterated in minutes, and often gets you 60–70% of the way to production quality. The limitation: the model's knowledge and reasoning patterns don't change. You're steering, not teaching.
Level 2: Few-Shot Learning. You provide 5–50 examples of desired input-output pairs within the prompt context. The model learns the pattern from examples without any weight updates. Effective for format standardisation, classification tasks, and output style. Limited by context window size — you can't fit thousands of examples in a prompt.
Level 3: Retrieval-Augmented Generation (RAG). Instead of training the model on your data, you build a retrieval pipeline that fetches relevant documents at query time and provides them as context. The model's weights don't change, but it can now answer questions about your proprietary data with citations. RAG is the sweet spot for most enterprise document intelligence use cases — current, auditable, and cost-effective. We've covered RAG architecture in depth in our AI Document Intelligence guide.
Level 4: Parameter-Efficient Fine-Tuning (LoRA/QLoRA). This is where you start modifying model weights — but surgically. LoRA (Low-Rank Adaptation) adds small trainable matrices alongside the frozen base model, typically updating less than 1% of parameters. QLoRA adds quantisation for memory efficiency. This teaches the model domain-specific reasoning patterns, output formats, and terminology without the compute cost of full fine-tuning. You need 1,000–10,000 high-quality training examples. Training a 7B model with LoRA takes hours on a single GPU, not weeks on a cluster.
Level 5: Full Fine-Tuning. All model parameters are unfrozen and updated on your domain-specific data. This delivers deeper customisation than LoRA but requires significantly more compute — multiple high-end GPUs for days to weeks — and 10,000–100,000 training examples. Risk of catastrophic forgetting (the model loses general capabilities while learning domain-specific ones) is real and must be managed through careful data mixing and evaluation.
Level 6: Continued Pre-Training. You take a pre-trained base model and continue its pre-training on a large corpus of domain-specific text (millions of tokens). This doesn't use instruction-response pairs — it's raw text that teaches the model domain language, concepts, and relationships at a fundamental level. Think of it as making the model "natively fluent" in your domain before fine-tuning for specific tasks. This requires substantial compute and data but produces models with genuinely deep domain understanding.
Level 7: Training from Scratch. You pre-train a model from random weights on your curated dataset. This is the nuclear option — billions of tokens of training data, clusters of GPUs running for weeks or months, millions of dollars in compute. Almost no enterprise needs this. It makes sense only when your domain is so specialised that no existing model provides a viable starting point, or when you need complete control over every byte of training data for regulatory reasons.
How Do You Decide Which Customisation Level Is Right?
The decision framework is surprisingly straightforward. Answer these questions in order:
Can prompt engineering close the gap? If the base model understands your domain but needs formatting, tone, or workflow guidance, start here. Test rigorously before moving to heavier approaches. Many enterprises skip this step and overspend on fine-tuning for problems solvable with better prompts.
Does the model need access to proprietary data? If yes, RAG is almost always the right next step. It's cheaper, faster to deploy, keeps data current, and provides auditability. Don't fine-tune when what you actually need is retrieval.
Does the model need to reason differently? If the model has access to the right information (via RAG) but still produces inadequate analysis, reasoning, or outputs, fine-tuning is warranted. LoRA first — full fine-tuning only if LoRA proves insufficient after proper hyperparameter tuning.
Is the domain fundamentally different from the model's training distribution? Highly specialised domains — biomedical, materials science, legal in non-English languages — may require continued pre-training to build baseline domain fluency before fine-tuning is effective.
Do regulatory requirements demand full data provenance? Some regulated environments require demonstrating exactly what data trained the model. Training from scratch — or continued pre-training from a known checkpoint on fully audited data — may be a compliance requirement, not a performance one.
What Data Do You Need for Each Level of Customisation?
Data requirements scale with customisation depth, but quality always trumps quantity:
- Prompt engineering: Zero training data. You need deep domain understanding to write effective prompts, plus a test set of 50–100 representative queries for evaluation.
- Few-shot: 10–50 high-quality input-output examples that represent the diversity of your use case.
- RAG: Your document corpus, cleaned and structured. Quality of chunking and embedding matters more than volume. A well-chunked 10,000-document corpus outperforms a poorly chunked 100,000-document one.
- LoRA fine-tuning: 1,000–10,000 instruction-response pairs. Each must be expert-validated. Bad training examples teach bad behaviour — there's no fixing garbage data with more epochs.
- Full fine-tuning: 10,000–100,000 examples plus a carefully curated mix of general-capability data to prevent catastrophic forgetting. Data preparation typically takes 2–3x longer than the training itself.
- Continued pre-training: 1–100 million tokens of domain-specific text. Internal documents, industry publications, technical standards, regulatory texts. This is the most data-intensive approach short of training from scratch.
- Training from scratch: Billions of tokens. Curated, deduplicated, quality-filtered. This is a dataset engineering project in its own right.
What Infrastructure Is Required for LLM Training and Deployment?
Infrastructure requirements vary dramatically across the customisation spectrum. Here's what each level demands:
Prompt engineering and RAG need inference infrastructure only. A capable GPU (NVIDIA A100, H100, or DGX Spark) running the base model, plus a vector database for RAG. NVIDIA's TensorRT optimises inference throughput by 2–6x, and Triton Inference Server handles production model serving with batching, queuing, and multi-model management.
LoRA fine-tuning is remarkably efficient. A single NVIDIA A100 (80GB) can fine-tune a 7B parameter model in hours. QLoRA pushes this further — fine-tuning a 70B model on a single GPU via 4-bit quantisation. NVIDIA NeMo provides the framework, handling distributed training, checkpointing, and experiment tracking. DGX Spark with its 128GB unified memory makes this accessible on-premise without a data centre.
Full fine-tuning of large models requires multi-GPU setups. A 70B model needs 4–8 A100s for full fine-tuning. Training time ranges from days to weeks depending on dataset size and model architecture. NeMo and DeepSpeed handle distributed training across GPUs with data parallelism, tensor parallelism, and pipeline parallelism.
Continued pre-training and training from scratch require GPU clusters. We're talking 32–256+ GPUs running for weeks. This is DGX SuperPOD territory, or equivalent cloud compute on AWS, Google Cloud, or Azure. PyTorch FSDP, NeMo Megatron, and DeepSpeed ZeRO manage the distributed computation.
For deployment, the full NVIDIA stack is critical: CUDA for GPU computation, cuDNN for neural network primitives, TensorRT for inference optimisation, Triton for production serving, and NIM for pre-packaged, optimised inference microservices. RAPIDS accelerates data preprocessing at GPU speed.
What Are the Common Pitfalls in Enterprise LLM Customisation?
We see the same mistakes repeatedly across enterprise engagements. Here are the ones that cost the most time and money:
Catastrophic forgetting. Fine-tuning on domain data causes the model to lose general capabilities — it can now answer domain questions but can't write a coherent paragraph or follow basic instructions. Mitigation: mix domain-specific training data with general instruction-following data at a 70:30 to 80:20 ratio. Evaluate on both domain benchmarks and general capability benchmarks throughout training.
Overfitting on small datasets. With 1,000 training examples and 7 billion parameters, overfitting is essentially guaranteed without regularisation. The model memorises your examples instead of learning patterns. Mitigation: LoRA with low rank (r=8–16), early stopping based on validation loss, and data augmentation where possible.
Evaluation debt. Enterprises build and deploy custom models without robust evaluation frameworks. They don't know if the model is getting better or worse, can't quantify accuracy for different query types, and can't detect regression. Before training a single example, build your evaluation pipeline: curated test sets, automated scoring, domain expert review protocols, and continuous monitoring in production.
Data quality neglect. The excitement of model training overshadows the unglamorous work of data preparation. Inconsistent formats, contradictory examples, outdated information, and label noise in training data produce unreliable models. Budget 60% of project time for data preparation and quality assurance.
Ignoring the deployment gap. A model that runs in a Jupyter notebook is not a production system. Inference latency, throughput, memory efficiency, model versioning, A/B testing, monitoring, and rollback capabilities are all engineering challenges that must be solved before users see the model. This is where TensorRT, Triton, and NIM earn their place in the stack.
How Do You Handle Data Governance and Compliance?
For regulated industries, data governance isn't a feature — it's a prerequisite. Custom LLM development must address:
Data residency. Training data, model weights, and inference queries must remain within jurisdictional boundaries. Malaysia's PDPA, Singapore's PDPA, and similar frameworks impose strict data localisation requirements. On-premise training on NVIDIA DGX infrastructure guarantees data never leaves client premises.
Training data provenance. Every data point used in training must be traceable — origin, consent status, processing history, and retention policy. This is non-negotiable for healthcare and financial services. NeMo's experiment tracking provides the foundation, but enterprise-grade provenance requires integration with data cataloguing and governance tools.
Model auditability. Regulators increasingly require explainability — not just what the model outputs, but why. For fine-tuned models, this means documenting training data composition, hyperparameters, evaluation results, and known limitations. For RAG-augmented models, citation trails provide auditability by design.
Right to deletion. If a data subject requests deletion under PDPA or GDPR, their data must be removed from training datasets and — potentially — the model retrained without it. This is architecturally simpler with RAG (delete the document, re-index) than with fine-tuned models (retrain entirely). Factor this into your architecture decisions.
Access controls. Different users should access different model capabilities based on role. A clinician querying a medical LLM should access different data than an administrator. Fine-grained access control must be implemented at the application and retrieval layers.
How Does NovaGenAI Build Custom LLMs for Enterprises?
NovaGenAI's approach to custom LLM development is defined by pragmatism, not hype. We follow a systematic methodology:
Start with the minimum effective intervention. We evaluate prompt engineering and RAG before recommending fine-tuning. We've seen engagements where 80% of the value was delivered by a well-architected RAG pipeline, with fine-tuning adding the remaining 20% for edge cases. We never prescribe heavy customisation when lightweight approaches deliver equivalent outcomes.
Build on NVIDIA NeMo. Our training infrastructure uses NVIDIA NeMo for model customisation — LoRA, full fine-tuning, and continued pre-training. NeMo provides distributed training, mixed-precision computation, experiment tracking, and seamless integration with TensorRT for inference optimisation. We leverage NIM for deployment-ready inference microservices that include pre-built health checks, scaling, and monitoring.
Deploy on client infrastructure. Models are trained and deployed on-premise using NVIDIA DGX systems or in the client's private cloud. Proprietary data never leaves client infrastructure. For organisations with existing cloud commitments, we deploy hybrid architectures with sensitive workloads on-premise and non-sensitive workloads in the cloud.
Continuous evaluation and improvement. Every deployed model is continuously monitored against domain-specific benchmarks. Accuracy degradation triggers automated alerts and retraining workflows. Models improve over time as new data becomes available and evaluation criteria sharpen.
The result: enterprise LLMs that understand your domain, run on your infrastructure, comply with your regulations, and get better every month. Not a demo — a production system.
