Optimizing CPU-Based SLM Inference on Lenovo ThinkAgile HX V4 and FX V4 Servers with Nutanix

Generative AI adoption is accelerating across enterprises and SMEs as organizations look for practical ways to improve productivity, automate knowledge workflows, summarize business content, and deploy AI assistants close to enterprise data. Small Language Models (SLMs) provide an opportunity to deliver these capabilities on modern CPU-based infrastructure, reducing dependency on specialized GPU platforms while maintaining a simpler and more cost-effective operational model.

This technical brief establishes proven architectural patterns and a deployment foundation for running Small Language Models (SLMs) on CPU infrastructure powered by Lenovo ThinkAgile HX V4 and FX V4 servers with Nutanix software and Intel Xeon 6 processors. The benchmark translates benchmark data into scalable deployment blueprints, providing data-backed sizing guidance for small and medium enterprises (SMEs) to balance latency and throughput for enterprise-wide use cases.

The benchmark architecture evaluated in this paper optimizes Small Language Model (SLM) deployment via quantized mixed-precision formats (INT4/INT8) executed locally within an isolated, NUMA-bound virtual machine environment. For enterprises evaluating distributed, multi-node scaling of uncompressed model weights utilizing native, up-streamed frameworks, see the companion Lenovo Press technical brief, Optimizing Distributed SLM Inference on Lenovo ThinkAgile VX V4 and FX V4 Servers with VMware Cloud Foundation 9.0.

The Lenovo ThinkAgile HX650 V4 and FX650 V4 are 2-socket, 2U systems and the Lenovo ThinkAgile HX630 V4 and FX630 V4 are 2-socket, 1U systems. These hyperconverged systems feature the Intel Xeon 6 two-socket processors (formerly code named "Granite Rapids"). ThinkAgile HX V4 and FX V4 hyperconverged systems are designed for deploying industry-leading hyperconvergence software from Nutanix on Lenovo enterprise platforms.

Figure 1. ThinkAgile HX650 V4 (top) and HX630 V4 (bottom) designed for Nutanix hyperconverged infrastructure

The Nutanix Cloud Platform (NCP) delivers cloud‑scale advantages to enterprise applications and databases by combining ThinkAgile HX V4 and FX V4 and public cloud infrastructure. It provides robust enterprise storage, integrated data protection, built‑in resilience, unified management, advanced analytics, and end‑to‑end security and consistent performance for mission‑critical workloads.

ThinkAgile FX offers a unique, industry first flexibility for software-defined approach to hyperconvergence, leveraging the ability to move between hypervisors of your choice to deliver computing, storage and management in a tightly integrated software stack and future-proof your investment with seamless HCI software transitions.

Figure 2. ThinkAgile FX650 V4 (top) and FX630 V4 (bottom) designed for flexible hyperconverged infrastructure

The hardware configuration represents a typical SME deployment model where generative AI inference can be deployed on Lenovo ThinkAgile HX/FX V4 systems with Nutanix and Intel Xeon 6 Processors.

The following table lists the virtual machine configuration.

The inference stack was based on OpenVINO GenAI, which provides optimized generative AI pipelines for Intel hardware for CPU based inference.

The following table lists the model and runtime configuration.

The inference framework used for this validation was OpenVINO GenAI, which provides optimized generative AI pipelines on top of the OpenVINO Runtime. OpenVINO GenAI exposes the LLMPipeline API for running language models on devices such as CPU and provides native performance metrics through PerfMetrics (openvinotoolkit.github.io). The model selected for testing was Microsoft Phi-3 Mini 128K Instruct, a small language model from the Microsoft Phi-3 family (huggingface.co) suitable for use cases such as knowledge assistance, summarization, document interaction, and workflow automation.

In the results, total throughput is summed across the four workers because each worker generates tokens independently. TTFT and TPOT are not summed up; they are interpreted as per-worker responsiveness metrics under concurrent execution. This distinction allows the analysis to separate full-system capacity from per-request latency behavior.

Each test point was executed by launching four independent OpenVINO GenAI workers, with one worker pinned to each NUMA node and each worker used 24 inference threads, one stream, one warm-up run, and five measured runs. The warm-up run was used to reduce cold-start effects before collecting measured results. Values reported represent the mean across 5 measured execution runs following 1 warmup run.

The benchmark collected the following OpenVINO GenAI PerfMetrics.

• TTFT (Time to First Token) - Measures how quickly the first token is produced; key for interactive responsiveness (response-start latency).

  The metric is extracted via OpenVINO GenAI’s PerfMetrics API for a single batched generate() call. Empirical verification confirmed prefill executes as a parallel batched operation: wall-clock completion of first-token generation for all sequences matches the reported mean TTFT (within 1%) at Batch 32, the largest tested configuration, confirming reported TTFT represents true per-request first-token latency.

• TPOT (Time per Output Token) - Measures sustained generation speed after the first token (streaming generation speed).

  Represents aggregate worker throughput efficiency across the concurrent processing pipeline. Per-user streaming latency for an individual request inside a batch is calculated as Batch Size × System TPOT.

• Throughput - Measures generated tokens per second
• Generate duration - Shows model generation time for the fixed 256-token response

The goal was to measure peak throughput and understand how model precision, batch size, input token length, and NUMA-local execution affect responsiveness and system capacity.

The results are interpreted relevant to architecture patterns for deploying AI models:

• Interactive (Realtime two-way user conversation) - Ultra low TTFT and predictable benchmark elapsed time.
• Streaming (Continuous one-way ingestion or output) - low TPOT and stable generation behavior.
• Shared services (Maximize infrastructure usage and reduce costs) - balance between latency and aggregate throughput.
• Throughput-oriented workloads – Maximum processing volume and system throughput and batch efficiency.

INT4 delivered the strongest overall CPU inference performance for Phi-3, leading in Time to First Token (TTFT), Time per Output Token (TPOT), and total system throughput across the full workload matrix. The tradeoff rule is that larger batch sizes increase aggregate throughput and improve TPOT but also increase TTFT and total response completion time.

• Throughput
• Batch size
• Input token length

Throughput

The following table shows INT4 provides the fastest response start, sustained token generation, and highest total throughput for the most latency-sensitive test case. INT4 provided the best balance of responsiveness, sustained generation speed, and total system throughput. INT8 remained a viable compressed alternative, but it did not outperform INT4 in the benchmark. FP16 provided the lowest throughput and slowest sustained generation in this workload, making it more appropriate as a higher-precision reference configuration than as the default performance choice.

The table shows mathematical averages across all tested batch sizes and token lengths to illustrate macro-level throughput scaling across FP16, INT8, and INT4 precisions. They do not represent a single operational batch profile.

The following two figures show that INT4 achieved the lowest average TTFT and highest throughput across the workload matrix, indicating the fastest response-start behavior.

Figure 3. Inference Latency by Precision (Lower is Better)

Figure 4. Average Total Throughput by Precision

Batch size

Larger batch size impacts total system throughput and latency across all tested precision formats. As batch size increased, total throughput increased sharply and TPOT decreased, indicating better sustained decoding efficiency. At the same time, TTFT increased, which means larger batches improved capacity but reduced response-start responsiveness.

The following table and figure show TTFT increases as batch size grows, indicating that higher-throughput configurations introduce additional response-start latency. INT4 maintains the lowest TTFT across the tested batch-size range, while INT8 and FP16 show higher response-start latency. TPOT decreases as batch size increases, indicating improved sustained token generation efficiency at higher concurrency levels. INT4 maintains the lowest TPOT across the tested batch-size range, followed by INT8 and FP16.

Figure 5. Total system throughput vs. batch size at 128 input tokens

The key interpretation is that batch size should be selected based on deployment intent:

• Batch 1-4 per NUMA worker is best suited to interactive or latency-sensitive use cases.
• Batch 8-16 per NUMA worker provides a balanced operating range for shared SME services.
• Batch 16-32 per NUMA worker is best suited to throughput-oriented or queued workloads.

Larger batches improve system throughput and token generation efficiency, but they also increase response-start latency.

Input token length

Longer prompts increased the amount of work required during prompt processing, which increased TTFT and benchmark elapsed time while reducing total system throughput. This effect was most visible at higher batch sizes, where more prompt tokens were processed concurrently across the four NUMA-local workers.

The following table summarizes the effect of input token length at batch 16 per NUMA worker. This batch size was selected because it represents a balanced shared-service operating point: high enough to show system capacity behavior, but not as throughput-oriented as batch 32.

The following figure shows total system throughput as input token length increases from 128 to 1024 tokens.

Figure 6. System throughput vs. input token length at batch 16 per NUMA worker

The total system throughput decreases as input token length increases. INT4 maintains the highest throughput across all input lengths, followed by INT8 and FP16. The decline is expected because longer prompts require more prompt-processing work before and during generations.

The benchmark supports the following conclusions:

• While INT4 quantization yields up to ~2.5x higher throughput than FP16 at low batch sizes (and ~1.3x on average across the workload matrix), enterprise deployments should conduct task-specific accuracy evaluations prior to production rollout. Public evaluations indicate that modern 4-bit weight-only quantization typically preserves most benchmark accuracy, though degradation is method- and task-dependent.
• INT8 and FP16 also delivered acceptable response-start latency for the tested workloads. Across the full-system benchmark dataset, average TTFT remained below 200 ms for all three precision formats, which supports a responsive user experience for many interactive and shared-service SLM use cases.
• Larger batch sizes improved throughput and reduced TPOT, but increased TTFT. This confirms the expected trade-off between system capacity and response-start latency. For most deployments, batch sizes between 1 and 16 per NUMA worker provide the most practical balance between responsiveness and capacity.
• Longer input prompts reduced throughput and increased benchmark elapsed time, especially at higher batch sizes, confirming that prompt length should be included in sizing assumptions.

The validation of Small Language Model (SLM) inference on Lenovo ThinkAgile HX V4 and FX V4 systems demonstrates that generative AI workloads can be effectively delivered on CPU‑based hyperconverged infrastructure without requiring dedicated GPU platforms for every deployment scenario. Lenovo ThinkAgile HX V4 and FX V4 provide an enterprise‑ready foundation for deploying SLM-based AI inference alongside business data, applications, and operational workflows. This enables broader adoption of generative AI across diverse enterprise use cases, while maintaining simplicity, scalability, and cost efficiency.

The following table lists the Bill of Materials for a single ThinkAgile HX650 V4 node in the configuration validated in this paper. Four identically configured nodes formed the Nutanix cluster described in Table 1.

For more information, see these resources:

• Lenovo ThinkAgile HX650 V4 Hyperconverged System
  https://lenovopress.lenovo.com/lp2133-lenovo-thinkagile-hx650-v4-hyperconverged-system
• Lenovo ThinkAgile HX630 V4 Hyperconverged System
  https://lenovopress.lenovo.com/lp2132-lenovo-thinkagile-hx630-v4-hyperconverged-system
• Lenovo ThinkAgile FX650 V4 Hyperconverged System
  https://lenovopress.lenovo.com/lp2338-lenovo-thinkagile-fx650-v4-hyperconverged-system
• Lenovo ThinkAgile FX630 V4 Hyperconverged System
  https://lenovopress.lenovo.com/lp2337-lenovo-thinkagile-fx630-v4-hyperconverged-system
• OpenVINO GenAI Documentation — Performance Metrics
  https://openvinotoolkit.github.io/openvino.genai/docs/guides/performance-metrics/
• OpenVINO API Documentation — openvino_genai.PerfMetrics
  https://docs.openvino.ai/2025/api/genai_api/_autosummary/openvino_genai.PerfMetrics.html
• OpenVINO API Documentation — openvino_genai.LLMPipeline
  https://docs.openvino.ai/2025/api/genai_api/_autosummary/openvino_genai.LLMPipeline.html
• OpenVINO GenAI Documentation
  https://openvinotoolkit.github.io/openvino.genai/
• Microsoft Phi-3 Model Collection — Hugging Face
  https://huggingface.co/collections/microsoft/phi-3
• Microsoft Phi-3 Mini 128K Instruct — Hugging Face Model Card
  https://huggingface.co/microsoft/Phi-3-mini-128k-instruct
• OpenVINO Phi-3 Mini 128K Instruct INT4 Model — Hugging Face
  https://huggingface.co/OpenVINO/Phi-3-mini-128k-instruct-int4-ov
• Intel AI Development Software
  https://www.intel.com/content/www/us/en/developer/topic-technology/artificial-intelligence/development-software.html

Cristian Ghetau is an Advisory Engineer for Lenovo in Romania and has experience in Cloud Infrastructure technologies. He has had more than 13 years of experience working with virtual environments from VMware, Microsoft, Oracle, Linux.