# Inference

> Part of the AI Garage Jargons glossary: https://aigarage.in/jargons (all terms: https://aigarage.in/jargons.md)

**Category:** Machine Learning
**Tags:** prediction, latency, serving, production

The process of applying a trained model to new input to produce a prediction, score, classification, or generated output.

**In plain terms:** Training is when the model learns its settings. Inference is when you give that trained model a new input and use those settings to get an output.

**Use cases:**
- Real-time assistant responses
- Batch classification jobs
- Recommendation and routing scores

**Why it matters:** Inference is the part users experience repeatedly. Its latency, throughput, model size, input length, and serving hardware shape product responsiveness and operating cost.

**Why now:** AI products increasingly choose among real-time, streaming, batch, local, and hosted inference. That serving choice affects latency, capacity, privacy, and cost without being the same decision as how the model was trained.

**AI Garage example:** A workshop intake workflow could run inference on each new response to predict the best support route. The prediction can guide the workflow immediately, while reviewed corrections are collected separately for a future training or fine-tuning run.

**Mental model:** Training sets the model's parameters. Inference holds that trained model version steady, passes a new input through it, and reads the resulting output. Saving an input or correction can support later training, but the inference call does not by itself teach the deployed model.

**Builder scenario: Route workshop requests without silent retraining**

- **Context:** A deployed classifier receives each new workshop-support request and predicts one of four routes for the operations team.
- **Failure:** The team assumes that correcting a bad route in the product automatically teaches the deployed model, so the same failure continues while nobody prepares new training data or a replacement model version.
- **Intervention:** Use the current model version for inference, log its input and reviewed outcome, and send approved corrections to a separate evaluation and retraining process.
- **Outcome:** Each request receives an immediate route from the known model version, while model behavior changes only through a deliberate, evaluated training step and deployment of a new version.

**Common misconceptions:**
- **Claim:** Every inference request teaches the model and updates its parameters.
  **Correction:** Ordinary inference applies the trained parameters to an input. Updating parameters requires a separate training, fine-tuning, or explicitly adaptive process.

**Visual model (system):** Inference uses the current model. Learning from reviewed outcomes requires a separate, deliberate update path.

**Visual description:** A new input passes through a fixed deployed model version to produce a prediction. Reviewed corrections move to a separate training step rather than changing the model during that inference call.

**Related concepts:** [Neural Network](https://aigarage.in/jargons/neural-network), [Fine-tuning](https://aigarage.in/jargons/fine-tuning), [API](https://aigarage.in/jargons/api), [Decision Model](https://aigarage.in/jargons/decision-model)

**Video explainer:** [Inference: using a trained model without retraining it](https://nqqbubjesbyuwrdhqxnk.supabase.co/storage/v1/object/public/aigarage-public-media/jargons/inference/v2/inference.mp4) (45 seconds)

See a deployed classifier route a new request, then follow reviewed corrections into a separate evaluation and training path instead of silently changing the model during inference.

**Published:** 2026-10-12T00:45:14+05:30
**Captions:** [English](https://aigarage.in/jargons/inference/captions.vtt)
**Narration:** generated

**Transcript:** A support team's assistant gets a new request and has to pick a route.

A trained model makes that pick, and the act of using it is called inference.

The request goes into the model, and a route comes out, with a score.

The model has fixed settings, learned earlier during training.

Inference only uses them. It does not change them.

Now an operator corrects a wrong route.

Saving that correction is useful, but the model does not learn from it.

The team can gather many corrections, check them, and train a new version.

They test it, and only then switch to it.

Until then, every request uses the same known version.

Training changes the model. Inference uses it.

**Last researched:** 2026-10-11

**Sources:**
- [Google for Developers - Supervised learning: inference](https://developers.google.com/machine-learning/intro-to-ml/supervised)
- [Google Cloud - Get online inferences from a custom trained model](https://docs.cloud.google.com/vertex-ai/docs/predictions/get-online-predictions)
- [Google for Developers - Fine-tuning, distillation, and prompt engineering](https://developers.google.com/machine-learning/crash-course/llm/tuning)

**URL:** https://aigarage.in/jargons/inference