Deploying ML and AI workloads

• Vertex AI — Google Cloud’s managed ML/AI platform (training, model registry, endpoints, Model Garden, generative AI). In April 2026 Google rebranded and extended it as the Gemini Enterprise Agent Platform; the APIs (aiplatform.googleapis.com), console pages, and existing resources keep working, and both names appear in docs.
• Container image — a packaged application plus dependencies, stored in Artifact Registry.
• Artifact Registry — GCP’s registry for container images and packages.
• Cold start — the delay when a service must start a new instance to handle a request.
• Scale to zero — running zero instances (and paying nothing) when there is no traffic.
• GPU / TPU — accelerators for ML. GPUs (L4, A100, H100, and newer) are general; TPUs are Google’s custom matrix chips.
• vLLM — an open-source LLM serving engine (see VRAM and running LLMs locally).
• Node pool — a group of identical VMs inside a GKE cluster.
• Autopilot — GKE mode where Google manages nodes and you pay per pod resources.
• Model Garden — catalog of Google, partner, and open models deployable on Vertex AI.
• Pub/Sub — GCP’s managed messaging service (asynchronous events and queues).

Most GCP deployment questions reduce to “how much do I want to manage?” versus “how special is my workload?”:

                      less management  ───────────────────────────►  more control
 API call to a        Vertex AI managed       Cloud Run              GKE                 Compute Engine
 hosted model         endpoints / Model       (containers,           (Kubernetes,        (raw VMs,
 (Gemini, MaaS)       Garden deployment       serverless, GPU)       full ecosystem)     custom everything)

Need	Best fit
Use Gemini or another hosted model through an API	Vertex AI / Gemini API (no deployment)
Serve a custom or fine-tuned model with managed scaling, versions, monitoring	Vertex AI online prediction endpoints
Deploy an open model (Llama, Qwen, Gemma) with one click	Model Garden deployment to a Vertex endpoint, or self-host on Cloud Run / GKE
HTTP service, agent, RAG API, light model, scale to zero	Cloud Run
Large-scale LLM serving, multi-GPU, custom scheduling, many models	GKE (with vLLM / Triton / JetStream)
Training jobs	Vertex AI custom training, or GKE / Batch / Compute Engine
Nightly batch scoring	Vertex batch prediction, Cloud Batch, or BigQuery ML
Event-driven glue (a file arrives -> run inference)	Cloud Run functions + Pub/Sub / Eventarc
Experiments and notebooks	Vertex AI Workbench or Colab Enterprise
Data/ML pipelines	Vertex AI Pipelines (Kubeflow Pipelines) or Cloud Composer (Airflow)
Full control, unusual drivers, HPC	Compute Engine

What it is. You give it a container (or source code to build) that listens on a port; Cloud Run runs it, scales instances with request load, and scales to zero when idle. Billing is per request-duration (or per instance-time with CPU always allocated), rounded to 100 ms.

gcloud run deploy summarizer \
  --source . \                       # builds with Cloud Build + buildpacks/Dockerfile
  --region europe-west4 \
  --service-account summarizer-sa@my-proj.iam.gserviceaccount.com \
  --set-env-vars GOOGLE_CLOUD_PROJECT=my-proj,GOOGLE_CLOUD_LOCATION=europe-west4 \
  --min-instances 0 --max-instances 20 \
  --concurrency 40 \                 # requests handled in parallel per instance
  --no-allow-unauthenticated         # require IAM-authenticated callers

GPUs. Cloud Run supports NVIDIA GPUs (L4 and larger) on services and jobs, with scale to zero and per-second billing, so serving a small/medium open model (Gemma, a 7-8B quantized model with vLLM or Ollama) is cheap for spiky traffic. Limits: one or few GPUs per instance, model weights must be loaded at start (cold starts of tens of seconds; mitigate with min instances, baking weights into the image, or loading from a Cloud Storage volume mount).

Why it is the default for AI apps. Agents, RAG back ends, and Gemini-calling APIs are mostly I/O-bound (they wait on model calls), so concurrency per instance is high and cost is low. Authentication with IAM (service-to-service), secrets via Secret Manager, VPC connectivity, and traffic splitting for canary releases are built in.

Cloud Run jobs run containers to completion (batch scoring, data prep, fine-tune launches) instead of serving requests; Cloud Run functions (formerly Cloud Functions 2nd gen) are the function-shaped wrapper for event triggers.

Watch out for: request timeout (up to 60 minutes), instance memory ceiling, no persistent local disk (use Cloud Storage/Filestore mounts), and CPU throttling when not handling a request unless “always allocated” is set (matters for background work and streaming).

Online prediction endpoints host a model behind an autoscaled HTTPS endpoint, with model versioning, traffic splitting between versions, request logging, and monitoring.

from google.cloud import aiplatform

aiplatform.init(project="my-proj", location="europe-west4")

model = aiplatform.Model.upload(
    display_name="churn-xgb",
    artifact_uri="gs://my-bucket/models/churn/",
    serving_container_image_uri="europe-docker.pkg.dev/vertex-ai/prediction/xgboost-cpu.2-1:latest",
)
endpoint = model.deploy(
    machine_type="n1-standard-4",
    min_replica_count=1, max_replica_count=5,        # note: min >= 1, endpoints do not scale to zero
)
print(endpoint.predict(instances=[[0.3, 12, 1]]))

Pre-built containers exist for scikit-learn, XGBoost, TensorFlow, and PyTorch; for anything else bring a custom container that exposes a prediction route and health route.

Model Garden is the catalog from which you can deploy open models (Llama, Gemma, Qwen, Mistral, and others) onto a Vertex endpoint with chosen GPU type, using optimised serving stacks (vLLM and similar) without writing the container yourself. It also lists model-as-a-service (MaaS) offerings, where partner or open models are served by Google as a pay-per-token API with nothing to deploy.

Trade-offs versus Cloud Run:

	Vertex endpoint	Cloud Run
Scale to zero	No (min 1 replica billed continuously for dedicated nodes)	Yes
Model management (registry, versions, A/B traffic split, monitoring/skew detection)	Built in	You build it
GPU options	Wide (A100/H100, multi-GPU nodes)	Fewer, smaller
Best for	Steady production traffic, governed models	Spiky traffic, lighter models, custom app logic

Google Kubernetes Engine is the choice for large-scale or complex serving: multi-GPU/multi-node LLM inference, many models sharing a GPU pool, custom schedulers, service meshes, or when the organisation already runs on Kubernetes.

• Autopilot (Google manages nodes, you pay per pod request) vs Standard (you manage node pools and GPU types).
• GPUs and TPUs attach to node pools; use node selectors/taints, and GPU sharing (time-slicing, MIG) to avoid wasting an expensive card on a small model.
• Inference stack: vLLM, TGI, Triton, or JetStream (for TPU) in containers; GKE Inference Gateway and the llm-d-style routing add model-aware load balancing (KV-cache-aware routing) and autoscaling on queue depth or GPU utilisation rather than CPU.
• Cost levers: Spot node pools for batch training, Dynamic Workload Scheduler / flex-start and reservations to obtain scarce GPUs, cluster autoscaler to scale nodes to demand.

# Minimal vLLM Deployment on a GPU node pool
apiVersion: apps/v1
kind: Deployment
metadata: { name: qwen-vllm }
spec:
  replicas: 1
  selector: { matchLabels: { app: qwen-vllm } }
  template:
    metadata: { labels: { app: qwen-vllm } }
    spec:
      nodeSelector: { cloud.google.com/gke-accelerator: nvidia-l4 }
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model=Qwen/Qwen2.5-7B-Instruct", "--max-model-len=16384"]
          resources: { limits: { nvidia.com/gpu: 1 } }
          ports: [{ containerPort: 8000 }]

Rule of thumb. If a single container on Cloud Run or a Vertex endpoint meets the need, do not take on cluster operations. Choose GKE when scale, GPU utilisation, or ecosystem integration justifies the platform work.

• Vertex AI custom training jobs — you submit a container or Python package plus a machine spec (CPU/GPU/TPU, count); Google provisions, runs, logs, and tears it down. Supports distributed training, hyperparameter tuning, and Spot/flex capacity. Outputs go to Cloud Storage and can be registered in the Model Registry.
• Vertex AI supervised tuning for Gemini and some open models — managed fine-tuning from a JSONL dataset, no infrastructure at all.
• Vertex AI Pipelines — orchestrate steps (prepare data -> train -> evaluate -> register -> deploy) as a reproducible DAG; each step is a container.
• Cloud Batch — managed batch job scheduling on VMs (including GPU and Spot) for HPC and ETL-style jobs not tied to Vertex.
• Compute Engine — raw VMs. GPU VM families: G2 (L4), A2/A3 (A100/H100), and newer; Deep Learning VM images come preconfigured with drivers. Maximum control, maximum operational burden; GPUs bill while the VM exists.
• TPUs — Cloud TPU for training and serving with JAX/PyTorch-XLA; strong for large-scale training and cost-efficient inference of supported models.

gcloud ai custom-jobs create \
  --region=europe-west4 --display-name=finetune-run-1 \
  --worker-pool-spec=machine-type=a2-highgpu-1g,accelerator-type=NVIDIA_TESLA_A100,accelerator-count=1,replica-count=1,container-image-uri=europe-docker.pkg.dev/my-proj/ml/trainer:latest

Typical architectures

1. LLM-powered API (no model hosting): client -> Cloud Run (your logic, prompts, tools) -> Gemini on Vertex AI. Cheapest, simplest, scale to zero.
2. RAG service: Cloud Run -> embeddings + Vector Search / AlloyDB / BigQuery vector -> Gemini. Documents ingested by a Cloud Run job or Dataflow into the index.
3. Self-hosted open model: Cloud Run GPU (small/spiky) or GKE + vLLM (large/steady) behind an internal load balancer, authenticated with IAM.
4. Classic ML (tabular, vision): train with Vertex custom training or BigQuery ML, register, deploy to a Vertex endpoint, monitor for drift.
5. Async/batch inference: Pub/Sub or Cloud Tasks -> Cloud Run jobs / Vertex batch prediction -> results to BigQuery or Cloud Storage.

Pitfalls

• Idle accelerators bill continuously. Vertex dedicated endpoints and GPU VMs/nodes cost money at zero traffic; undeploy test endpoints.
• GPU quota and capacity are per project and region and often default to zero: request quota early, and have a second region in mind.
• Cold starts with large weights. Pre-bake weights into the image or mount them from a bucket; keep a minimum instance for latency-critical paths.
• Region mismatch between data, model, and caller adds latency and egress cost.
• Authentication: use service accounts with least privilege (roles/aiplatform.user to call models) and Application Default Credentials; do not ship API keys in images.
• Observability: Cloud Logging, Cloud Trace, and Vertex model monitoring; for LLM apps add prompt/response logging (with PII care) and token metrics.