Nvidia Launches NeMo Switchyard to Curb Enterprise AI Inference Costs

Nvidia has introduced NeMo Switchyard, a software platform designed to address soaring infrastructure costs and uncertain ROI currently hindering enterprise AI adoption, as first reported by The Register.

Switchyard acts as a proxy between the inference server's API endpoint and the models. Rather than routing every request to a single expensive model, the software can be configured to direct prompts to different models based on cost, latency, or output quality. Nvidia claims that by routing certain requests to smaller, locally hosted, or cheaper models, enterprises can reduce job completion costs by 74% compared to using Claude Opus 4.8 alone, though this comes with an approximate six-point accuracy tradeoff.

This release coincides with the launch of Nemotron 3.5-30B-A3B-Lightning, a 30 billion-parameter MoE model designed for general-purpose, low-latency use. Nvidia is also promoting task-specific models like Nemotron Parse, a one-billion-parameter model optimized for extracting context from PDFs. Joey Conway, senior director of AI software and models at Nvidia, told The Register that offloading specialized tasks to these smaller models can improve accuracy while reducing costs.

Model routing is becoming a standard efficiency play for large organizations. The Register notes that OpenAI utilized similar routing for GPT-5, and reports that The Wall Street Journal noted AT&T implemented its own "smart router." By shifting from proprietary to open-weight models, AT&T reportedly saved 80% to 90% in specific applications; the company currently powers 25% of its AI workloads with open models and expects that figure to reach 70-80% in the coming years.