Install
Inference & Serving
Serving runtimes, Triton, vLLM, ONNX Runtime, and scalable endpoints.
- 3 Tracked terms
- Last 30 days Feed window
What this topic collects on
An article joins this feed when it matches these terms. Each one is also a search of its own.
Related topics
Latest in Inference & Serving
2B Gemma 4 Deployment with Cloud Run, NVIDIA L4, MCP SDK 2.x, and Claude Code
4+ day, 2+ hour ago (1508+ words) This article provides a step by step deployment guide for Gemma 4 E2B to a Cloud Run hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment with Claude Code. This project…...
External Rollout Engines Roadmap??? Vime
6+ day, 22+ hour ago (685+ words) An external rollout engine is a vLLM engine that is not launched by the vime training job. Another system deploys and owns the engine lifecycle; vime connects to those engines during training, registers a router, and syncs updated actor weights…...
Rustls 0.23.44 Released With ML-DSA Certificates Enabled By Default
1+ week, 11+ hour ago (200+ words) Rustls as the modern TLS library implementation written in the Rust programming language is out with a new feature release. This morning's Rustls 0.23.44 release enables post-quantum secure ML-DSA certificates by default... - Categories Computers Display Drivers Graphics Cards Linux Gaming Memory…...
Three Gemma 4 Deployments on One T4G for Under $3: What the Runtime Changes, and What It Doesn't
1+ week, 6+ day ago (1818+ words) This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared…...
Gemma 4 in Pure JAX: What Ports from TPU to GPU, and What Doesn't
2+ week, 2+ day ago (1003+ words) This article is about running a hand-written Gemma 4 port in pure JAX on three different accelerators, and about the two places the abstraction leaks. The code is here: This project aims to serve one Gemma 4 checkpoint from one JAX port…...
Pure JAX on G5g: Serving Gemma 4 on Graviton and a T4G
2+ week, 2+ day ago (1363+ words) This article provides a step by step deployment guide for serving Google's Gemma 4 on an AWS EC2 G5g instance using pure JAX. The code is here: This project aims to serve a modern open model on the cheapest whole CUDA GPU AWS…...
Deploying a QAT Checkpoint Your Serving Stack Can't Load: Gemma 4 E2B in Pure JAX on One TPU
3+ week, 5+ day ago (1484+ words) Cloud TPU v6e-1 (ct6e-standard-1t, one v6e chip, 32 GB HBM), Compute Engine flex-start, europe-west4-a. All timings below measured 2026-08-19 unless stated otherwise. There is a particular kind of dead end where every component is healthy, every version is current, and the thing still does…...