AI’s Next EC2 Moment Begins Before the GPU

CPU virtualization made physical servers shareable. The next leap in AI economics may come from controlling the reasoning state that reaches every model call.

10-minute read

Every major era of computing has been shaped by a scarce resource.

In the early enterprise era, servers were expensive and difficult to procure. During the rise of cloud computing, organizations struggled with rigid capacity, long deployment cycles, and machines sized for peak demand but left idle most of the time.

Today, the scarce resource is accelerated computing.

Enterprises want to deploy AI across customer service, finance, software development, research, compliance, operations, and decision support. Yet the GPUs used to run large models remain expensive, capacity is difficult to plan, and workloads are increasingly unpredictable.

That has led infrastructure leaders to ask a familiar question:

Can GPUs be virtualized the way CPUs were?

In several ways, they already are.

GPUs can be partitioned, time-sliced, shared across processes, and multiplexed across many inference requests. Modern model-serving systems are becoming remarkably effective at extracting more work from every accelerator.

But GPU virtualization alone will not produce the next EC2 moment.

CPU virtualization solved a machine utilization problem. Large language model inference introduces a second constraint: every request carries context, memory, history, and operational state that compete for scarce accelerator resources.

GPU sharing improves the supply side of AI infrastructure.

The next major efficiency gain may come from reducing the demand created by each model call.

Cloud economics began with abstraction

Cloud computing did not transform enterprise technology simply because hyperscalers bought servers at enormous scale.

The deeper breakthrough was the separation of the logical machine from the physical one.

Before virtualization became widespread, an enterprise application was commonly assigned its own server. Even when that application used only a small fraction of the machine’s capacity, the organization still had to purchase, power, cool, secure, patch, and maintain the entire system.

A software layer called a hypervisor changed that model. It allowed multiple virtual machines to run on one physical server while presenting each workload with what appeared to be its own processor, memory, storage, and network interfaces.

The hypervisor did more than let applications take turns using the CPU. It isolated workloads, controlled memory access, scheduled processor time, and gave each application a stable logical environment independent of the physical machine underneath it.

This abstraction unlocked several reinforcing advantages.

Applications no longer had to be designed for a specific server. Infrastructure could be provisioned through software. Idle capacity from one workload could be reassigned to another. Large fleets could absorb variations in demand because not every customer reached peak usage at the same time.

This produced statistical multiplexing. By pooling many workloads, cloud providers could serve aggregate demand with less total hardware than every customer would have required independently.

Amazon EC2 turned that architectural breakthrough into a commercial model. Customers requested logical servers on demand while Amazon optimized the physical fleet beneath them.

The contract was simple:

The customer saw a stable unit of computing. The provider optimized the infrastructure underneath it.

The search for an EC2 moment in AI is ultimately a search for an equally powerful abstraction.

Why LLM inference is different

At first glance, GPUs appear to present the same opportunity.

A high-end accelerator is expensive. Many workloads do not use one continuously. If several customers can safely share the device, utilization should increase and the cost of each unit of work should fall.

That logic is sound, but large language model inference creates a fundamentally different type of workload.

A model-serving system typically keeps the model’s parameters, commonly called its weights, loaded into GPU memory. Those weights can be shared across many users. The individual requests sent to the model, however, create their own state.

A request may contain user instructions, conversation history, retrieved documents, software tool results, generated artifacts, workflow status, approval information, and the tokens already produced during the current response.

A token is a small unit of information processed by the model. Longer prompts contain more tokens, which means the model must perform more work before it can begin generating an answer.

When a model first receives a request, it processes the input tokens in a phase called prefill. During prefill, the model interprets the prompt and creates the internal state needed to begin producing a response.

It then generates new tokens, generally one at a time, during a phase called decode.

Reprocessing the full input for every generated token would be prohibitively expensive. To avoid that, the model stores information about previously processed tokens in a structure known as the key-value cache, or KV cache.

The KV cache improves performance, but it consumes high-bandwidth memory, commonly called HBM. This is the extremely fast memory located close to the GPU, and it is one of the most valuable and constrained resources in an AI server.

The cache also grows as the input and output sequences become longer. An active request can therefore continue occupying scarce GPU memory even when it is temporarily not consuming compute.

That creates a meaningful difference from CPU virtualization.

When a CPU pauses one workload and schedules another, most of the first application’s memory remains elsewhere in the system. The processor can switch tasks relatively quickly.

When an LLM request stops using GPU compute, its sequence state may remain resident in expensive accelerator memory. A long prompt may also require substantial prefill work before the first output token appears. Large models may span several GPUs, making device placement and the connections between those devices important.

The constrained resource is therefore not simply GPU time. It is a combination of compute capacity, memory capacity, memory bandwidth, GPU interconnects, batch space, KV-cache state, prompt-processing time, and latency requirements.

Traditional time-slicing cannot solve all of those constraints by itself.

The AI infrastructure industry has not waited for a single universal GPU hypervisor.

Instead, it has developed several complementary forms of sharing. Each operates at a different layer of the inference stack.

Together, they create a form of pseudo-sharing. More workloads can use the same accelerator infrastructure, although not through one abstraction as clean and universal as the virtual machine.

NVIDIA’s Multi-Instance GPU, commonly called MIG, divides supported GPUs into smaller hardware-isolated instances. Each partition receives a defined portion of the accelerator’s compute capacity and memory and appears to the application as a smaller GPU.

Virtual GPU platforms provide a related abstraction by exposing configurable GPU profiles to virtual machines. These technologies are useful when several workloads are too small to justify an entire accelerator and require stronger isolation than ordinary process sharing can provide.

The limitation is rigidity.

A workload must fit the available partition shape. A large model may require the entire GPU or several GPUs, while unused capacity inside a smaller partition may not be immediately available to another workload.

Hardware partitioning improves isolation and predictability. It does not create unlimited elasticity.

GPUs can also be shared over time.

With time-slicing, several processes alternate access to the same physical accelerator. This can work well for development environments, intermittent workloads, and applications that do not continuously saturate the device.

NVIDIA’s Multi-Process Service, or MPS, allows several applications using the CUDA programming platform to share GPU execution resources more efficiently. CUDA is the software platform developers use to run general-purpose programs on NVIDIA GPUs.

MPS can improve concurrency when one process alone cannot fully occupy the accelerator. But it remains a mechanism for coordinating GPU work, not a fully isolated virtual-machine boundary.

Time-sharing makes the accelerator accessible to more workloads. It does not make those workloads smaller, and it does not eliminate competition for memory.

Some of the most important advances in LLM inference occur above the hardware layer.

Instead of assigning the model to one user until the entire response is complete, modern serving systems combine work from many users into a shared batch.

GPUs are highly efficient when they can apply the same mathematical operation across many pieces of data at once. Batching takes advantage of that parallelism.

Traditional batching works poorly for generative AI because responses have different lengths. One request may finish after twenty tokens while another continues for several hundred. Under rigid batching, completed capacity can remain unused until every sequence in the group finishes.

Continuous batching solves this by allowing requests to enter and leave the batch between token-generation steps. When one response finishes, another request can take its place without waiting for every other sequence to complete.

In this architecture, the GPU is not divided into virtual machines. Instead, model execution is continuously multiplexed across many active requests.

This is much closer to the practical LLM equivalent of CPU time-sharing.

Modern serving engines also apply virtualization principles to the memory used by active requests.

A technique called PagedAttention applies an idea similar to virtual memory in operating systems to the KV cache.

Without paging, a serving system may reserve large contiguous areas of GPU memory for each request. Because sequence lengths are unpredictable, some of that memory may remain unused. Gaps between allocations can also create fragmentation, meaning free memory exists but is divided into pieces that are difficult to use effectively.

PagedAttention divides the KV cache into smaller blocks. The serving system can allocate, map, reuse, and release those blocks as requests grow and finish.

This does not eliminate the KV cache. It makes the cache easier to manage and allows GPU memory to be packed more efficiently.

That distinction matters:

PagedAttention improves how accepted model state is stored. It does not decide whether that state should have entered the model request in the first place.

Pipeline and model optimization

Inference systems are also reorganizing how model work is executed.

Chunked prefill divides a large prompt into smaller pieces so one long input does not monopolize the accelerator and delay other users.

Prefill-decode disaggregation assigns prompt processing and token generation to separate pools of hardware. This allows each phase to be optimized according to its own compute and latency characteristics.

Prefix caching reuses model state when several requests begin with the same instructions, policies, or document content.

Speculative decoding uses a smaller model to propose likely tokens, which a larger model then verifies.

Quantization represents model weights using fewer bits, reducing memory consumption and sometimes improving execution speed.

These are important advances. They improve how efficiently inference infrastructure executes the work it receives.

But they all begin after the workload has largely been defined.

The unresolved problem sits before the model call

Modern inference infrastructure is becoming remarkably effective at processing the requests it receives.

It can partition accelerators, schedule processes, batch requests continuously, page memory, reuse prefixes, and reorganize model execution.

Yet it generally accepts one critical assumption:

The information submitted to the model is the information the model should process.

For a simple chatbot, that assumption may be manageable.

For an enterprise AI agent, it becomes increasingly expensive.

An AI agent combines a model with memory, enterprise data, and software tools to complete multi-step work. It may retrieve records, call applications, generate documents, request approvals, and continue a workflow over hours, days, or weeks.

As the agent works, it accumulates state. That state may include documents, tool results, identifiers, generated artifacts, previous decisions, approvals, policies, conversation history, failed attempts, retrieved evidence, and the results of earlier model calls.

Over time, this forms an enterprise state graph, which is the connected record of everything the agent has observed, generated, or been instructed to preserve.

The conventional response has been to send more of that state into the model’s context window.

But a larger context window is capacity, not a strategy.

The fact that a model can accept more information does not mean every available record should participate in every decision.

More context creates more prefill work and a larger KV cache. It can reduce concurrency, complicate scheduling, and make processing batches less predictable.

It can also reduce the quality of the result.

Stale records may conflict with current ones. Duplicate tool results can obscure authoritative evidence. Earlier model-generated summaries may be less reliable than their original sources. Irrelevant documents can distract the model from the task at hand.

There is also a governance consequence. Every record placed in context becomes information the model may use. This expands the surface that must be controlled, audited, secured, and explained.

For the past several years, the industry has focused on a single question:

How do we make models process more context?

That remains an important challenge.

It may not be the first one.

A more fundamental question comes earlier:

What state should participate in the next action at all?

The missing layer is reasoning-state virtualization

Operating systems do not load every file stored on a computer into physical memory merely because the file exists.

Enterprise agents should not place every available record into a model call merely because the record is accessible.

The complete state available to an agent and the active state required for its next decision should be treated as two different things.

Operational state is everything the system may need over the life of a workflow. It includes documents, memories, tool outputs, artifacts, approvals, policies, identifiers, previous decisions, and historical runs.

Reasoning state is the smaller, task-specific working set required to produce the next correct action.

A new infrastructure layer can sit between them.

Its role is to understand the current task, identify relevant and authoritative evidence, preserve required constraints and dependencies, and construct decision-ready context before the model call begins.

This is not simply prompt compression.

Prompt compression asks how selected text can be represented using fewer tokens.

Reasoning-state virtualization asks which evidence, records, instructions, artifacts, and decisions should participate in the model call in the first place.

Compression changes the representation of an already selected payload.

Selection determines the payload itself.

The term virtualization is useful here, but it should be understood as an architectural analogy rather than hardware emulation. A state-virtualization layer creates a logical working set from a much larger body of operational state, allowing the model to reason over the information relevant to the immediate task without loading the entire history into context.

The role of this layer is to identify the load-bearing state, meaning the information whose inclusion or omission could change the correctness of the result.

That might include an authoritative database record, the exact identifier needed for a tool action, the latest approved version of a document, an active policy exception, a workflow gate, a generated artifact, a required output schema, or a user instruction that must remain in force.

Everything else remains available in the operational record.

It is simply not forced into every model call.

Why this changes GPU economics

Reasoning-state virtualization does not partition a GPU or schedule a CUDA kernel.

It changes the workload that reaches the inference system.

When unnecessary context is removed before inference, the model has fewer input tokens to process during prefill. A smaller input generally creates a smaller initial KV cache, freeing high-bandwidth memory for other active requests.

If each request consumes less memory, the serving engine may be able to support more concurrent sequences on the same hardware. Better-controlled working sets can also reduce extreme variation in prompt length, making batches easier to assemble and schedule.

Most importantly, state selection can prevent expensive accelerator resources from being spent on information that does not contribute to the outcome.

The relationship is not perfectly linear.

Reducing input tokens by a given percentage does not guarantee an identical reduction in total GPU cost. Output generation may dominate some workloads. Model weights may occupy most of the available memory. The serving engine may need to cross a capacity threshold before freed memory can accommodate another request.

But infrastructure economics are often shaped by exactly those thresholds.

Fitting one additional sequence into memory can matter. Avoiding another model replica can matter. Preventing one oversized prompt from disrupting a latency-sensitive batch can matter.

Across a large fleet, modest reductions in the resources consumed by each successful model call can compound into meaningful capacity gains.

The relationship can be summarized simply:

GPU sharing allocates the supply. State virtualization reduces the demand created by each decision.

Consider a long-running procurement agent

Imagine an AI agent supporting a procurement process that lasts several weeks.

During that process, the agent accumulates multiple versions of a contract, vendor proposals, pricing spreadsheets, legal comments, approval messages, meeting notes, policy documents, tool results, and previous recommendations.

Its next task may be narrow:

Determine whether the latest proposal exceeds the business unit’s approval threshold and identify the required approver.

The model may need the latest proposal, the authoritative approval policy, the correct business-unit identifier, the approved budget, any active exception, and the required output format.

It may not need superseded proposals, unrelated contract clauses, entire meeting transcripts, rejected pricing alternatives, summaries of obsolete documents, or every software tool available to the agent.

Nothing has to be deleted.

The complete workflow history remains available to the system. The next model call simply receives the working state required for the current decision rather than the accumulated weight of the entire process.

This is how state virtualization can make the model less expensive to operate while also focusing it on the evidence that matters.

State selection must be a control plane

The difficult part is not removing text.

The difficult part is preserving meaning.

A production-grade state-selection layer must understand the authority of each record. It must distinguish an official system response from a user assertion, an older model-generated summary, or an unverified retrieval result.

It must understand freshness and determine whether a record is current or has been superseded. It must enforce scope so the selected information belongs to the correct customer, account, tenant, transaction, and workflow.

It must also preserve dependencies. A later action may require an identifier, artifact, approval, or decision produced many steps earlier.

Policy introduces another requirement. Some instructions, approvals, and exceptions must remain on the critical path regardless of their apparent textual relevance.

Risk matters as well. The consequences of omitting information vary considerably between a low-stakes summary and a financial approval, legal review, or record-changing tool action.

The system must also preserve provenance so it can explain why a record was included or excluded.

Most importantly, it must know when not to optimize.

When confidence is insufficient, the safer choice is to preserve more context or pass through the broader baseline. The objective is not the smallest possible prompt.

It is the smallest decision-sufficient state that preserves the required outcome.

What enterprises should measure

Token reduction alone is not enough.

An organization can reduce tokens and still increase total cost if users must retry failed answers, tool actions become less reliable, or state selection introduces excessive latency before every model call.

The more useful measure is:

Correct, policy-compliant business outcomes per GPU dollar, delivered within the required response time.

A serious evaluation should measure outcome quality, infrastructure efficiency, serving capacity, latency, governance, and fallback behavior together.

It should determine whether the system preserved the evidence required to answer correctly and complete the intended action. It should examine input processing, KV-cache occupancy, accelerator time, concurrency, throughput, and the number of GPU replicas needed at a target workload.

It should also measure the overhead introduced by state selection. A system that saves model time but adds several seconds before each call may not belong inside an interactive agent loop.

Governance must be evaluated directly. Authoritative records, policies, identifiers, approvals, evidence, and output requirements must remain intact.

Fallback behavior matters just as much. A trustworthy system must recognize when broader context is necessary and decline to optimize aggressively.

For the CFO, this is the difference between durable unit-cost leverage and superficial token savings.

For the CTO, it offers a path to better infrastructure efficiency without replacing the underlying model or accelerator architecture.

For the CIO and risk organization, it creates a more explicit boundary around the evidence used for each AI-assisted decision.

The future AI stack will contain several virtualizers

Cloud computing converged around the virtual machine as a dominant abstraction.

AI infrastructure is likely to require several cooperating abstractions.

Hardware virtualization will determine which workloads receive which portions of an accelerator. Runtime virtualization will determine which requests execute together and when. Memory virtualization will determine how model and sequence state are allocated, retained, paged, and reused.

Model optimization will reduce the computation and memory required to produce an output.

Reasoning-state virtualization will determine what information is necessary to produce the correct output in the first place.

Each layer addresses a different source of inefficiency. No single layer will create cloud-scale AI economics by itself.

Together, they can turn a fixed accelerator fleet into substantially more useful enterprise capacity.

The next cloud abstraction begins before the model call

The first cloud era separated applications from physical servers.

The next AI infrastructure era must separate an agent’s complete operational memory from the active state required for its next action.

That separation will become more important as agents run longer, call more tools, create more artifacts, inherit more policies, and accumulate more organizational history.

The winning architecture will not simply ask increasingly powerful models to read everything faster.

It will decide what deserves to be read in the first place.

It will preserve the evidence that can change the answer, exclude state that only consumes time, memory, and money, and give the accelerator a smaller and more precise unit of useful work.

The next EC2 moment for AI will not begin inside the GPU.

It will begin with the decision about what deserves to reach it.