On this page
Read
The useful mental model
Training sets the model's parameters. Inference holds that trained model version steady, passes a new input through it, and reads the resulting output. Saving an input or correction can support later training, but the inference call does not by itself teach the deployed model.
In plain English
Training is when the model learns its settings. Inference is when you give that trained model a new input and use those settings to get an output.
Where builders use it
- 01Real-time assistant responses
- 02Batch classification jobs
- 03Recommendation and routing scores
Why it matters
Inference is the part users experience repeatedly. Its latency, throughput, model size, input length, and serving hardware shape product responsiveness and operating cost.
Why now
AI products increasingly choose among real-time, streaming, batch, local, and hosted inference. That serving choice affects latency, capacity, privacy, and cost without being the same decision as how the model was trained.
A common misconception
“Every inference request teaches the model and updates its parameters.”
Ordinary inference applies the trained parameters to an input. Updating parameters requires a separate training, fine-tuning, or explicitly adaptive process.
See
See the mental model
The visual is generated from the same structured knowledge as the written explanation.
A new input passes through a fixed deployed model version to produce a prediction. Reviewed corrections move to a separate training step rather than changing the model during that inference call.
Watch
Watch the explainer
A short visual explanation built from the same reviewed knowledge object.
Inference: using a trained model without retraining it
See a deployed classifier route a new request, then follow reviewed corrections into a separate evaluation and training path instead of silently changing the model during inference.
Read transcript
A support team's assistant gets a new request and has to pick a route. A trained model makes that pick, and the act of using it is called inference. The request goes into the model, and a route comes out, with a score. The model has fixed settings, learned earlier during training. Inference only uses them. It does not change them. Now an operator corrects a wrong route. Saving that correction is useful, but the model does not learn from it. The team can gather many corrections, check them, and train a new version. They test it, and only then switch to it. Until then, every request uses the same known version. Training changes the model. Inference uses it.
Builder example
Route workshop requests without silent retraining
A deployed classifier receives each new workshop-support request and predicts one of four routes for the operations team.
Failure
The team assumes that correcting a bad route in the product automatically teaches the deployed model, so the same failure continues while nobody prepares new training data or a replacement model version.
Intervention
Use the current model version for inference, log its input and reviewed outcome, and send approved corrections to a separate evaluation and retraining process.
Observable result
Each request receives an immediate route from the known model version, while model behavior changes only through a deliberate, evaluated training step and deployment of a new version.
Sources
Go deeper
- Google for Developers - Supervised learning: inference
- Google Cloud - Get online inferences from a custom trained model
- Google for Developers - Fine-tuning, distillation, and prompt engineering
Last researched 11 October 2026