CLOUDTECHNOLOGIES- Classical ML, embeddings and batch jobs often run well on CPUs.
- Quantization lets smaller language models fit comfortably in server RAM.
- CPU token generation is memory-bound, so latency grows with model size.
- Training and serving large models at scale still requires GPUs.
Conversations about AI infrastructure tend to focus on GPUs, and for training large models that focus is justified. But much of the practical AI work businesses do is inference on modest models, data preparation and classical machine learning, and a lot of it runs well on CPUs.
Knowing where CPUs are a good fit, and where they are not, can save significant cost and complexity.
Workloads that run well on CPUs
Several common AI tasks are well suited to modern server processors:
- Classical machine learning: gradient-boosted trees, logistic regression and similar models for fraud scoring, forecasting, recommendations and classification.
- Embeddings: generating vectors for semantic search and retrieval-augmented generation (RAG) with compact embedding models.
- Smaller language models: models with a few billion parameters for summarization, extraction, classification and routing.
- Batch jobs: document processing, tagging and enrichment where results are needed in hours, not milliseconds.
- Data preparation: cleaning, tokenization, feature engineering and ETL, which run on CPUs regardless of where the model runs.
How quantization changes the picture
Quantization stores model weights at lower precision, for example as 8-bit or 4-bit integers instead of 16-bit floating point. That shrinks memory requirements substantially, often with only a small loss in output quality for many tasks.
Runtimes such as llama.cpp and ONNX Runtime are optimized for running quantized models on CPUs. A model with a few billion parameters, quantized to 4 bits, needs only a few gigabytes of memory. A server with hundreds of gigabytes of RAM can hold larger quantized models, several models side by side or a model next to a large vector index, although generation speed, not capacity, then becomes the limit.
Latency vs. throughput
Two measures decide whether CPU inference is good enough. Latency is how long a single request takes; throughput is how much work the system completes per second overall.
For language models, generating each token on a CPU is largely limited by memory bandwidth, so response speed drops as models grow. That is acceptable for background jobs, internal tools and moderate traffic, but it can feel slow in interactive chat with larger models. Batching requests, keeping prompts concise and caching frequent results all improve efficiency.
When you need GPUs instead
It is important to be realistic about the limits. GPUs are the right tool, and usually the only practical one, for:
- Training large models or fine-tuning them beyond small adapters.
- Serving large language models to many concurrent users with low latency.
- High-volume image, video or speech generation.
Many teams use both: GPUs where massive parallelism is essential, and CPUs for everything around them, from data pipelines to embeddings and lightweight inference.
A cost-efficient starting point with Infinity Cloud
For teams starting with AI, CPU servers offer a practical way to prototype and run production inference without committing to GPU infrastructure. Infinity Cloud Technologies offers bare-metal servers with AMD EPYC™ 9355 (32 cores / 64 threads) or EPYC 9555 (64 cores / 128 threads) processors and 256 GB or 512 GB of RAM. Many cores and large memory suit CPU inference, embedding generation and data preparation, and EPYC platforms use ECC registered DDR5 memory for integrity on long-running jobs.
If a project grows into training large models, that work belongs on GPUs; CPU servers remain useful for the pipelines around it. The servers run from Infinity Cloud’s data center in Romania, within the European Union, so the data used for inference can stay in the EU. Bare-metal pricing is available on request.