Microsoft reveals GitHub Copilot infrastructure gaps in 13.5M sessions
In brief
- Microsoft and UIUC analyzed 13.5M Copilot sessions from 3.2M users in June 2026
- 87% of LLM calls are agent-initiated; median 15 calls per turn, mean exceeding 40
- KV-cache hit rates drop from 90% within-turn to 55% between-turn to 8% on model switches
- Tool-failure retry loops amplify compute 4x; idle-time prediction recovers 86-90% wasted time
Agentic patterns at scale
The study reveals how GitHub Copilot behaves in production—and the findings challenge conventional infrastructure design. In a typical session, the median session involves about 15 LLM calls per turn, with the mean exceeding 40 in some scenarios. More striking: approximately 87% of all LLM calls in Copilot sessions are agent-initiated, not user-initiated. This means the AI itself is driving most of the compute, not human prompts.
The researchers identified five distinct user archetypes, and the gap between the lightest and heaviest users spans a 50-fold range in token consumption. This heterogeneity matters for resource planning—one-size-fits-all infrastructure can't serve both ends of that spectrum efficiently.
Cache efficiency craters between turns
KV-cache, the mechanism that lets language models avoid recomputing previous context, hits at roughly 90% within a single turn. But once a user issues a new turn, the hit rate drops to around 55%. Switch models or compact the context window, and it craters to 8%.
The implication is stark: current scheduling strategies that treat every request as independent are burning compute cycles on redundant calculations. Researchers recommend scheduling should move from per-request to per-turn or per-session granularity, keeping cache-hot workloads together rather than scattering them.
Retry cascades and idle time
Tool-failure-driven retry loops can amplify compute workload demands by up to four times. When a tool call fails, the agent retries—often multiple times—creating a cascade that multiplies the original request's cost. Infrastructure needs to account for the retry amplification problem, building headroom for tool-failure cascades rather than assuming a linear cost model.
Idle time between user actions is another inefficiency. Researchers found that 86-90% of total idle time in sessions could be captured by lightweight predictors. The study proposes that idle-time prediction should become a first-class infrastructure concern, not an afterthought, enabling proactive resource reclamation instead of holding compute warm indefinitely.
