Vinoth Haldorai
← All thoughts

The Building Blocks of the Enterprise Context Layer

The four largest hyperscalers have provided guidance to the market that they would be spending approximately $700 Billion in 2026 on their capital expenditure, out of which a majority of 2/3rd would be tied to building their AI infrastructure backbone directed almost entirely at compute, data centres and power. According to Goldman Sachs this is expected to grow to a cumulative capex of $5 Trillion between 2025 and 2030. This is remarkable and uncharacteristic of the typically capital-light software companies the market has been accustomed to over the last decade. This spend doubled from their 2025 levels, and now comfortably outpaces their respective free cash flows, which have put their traditional DCF based valuations under pressure. The industry is fundamentally changing, and software majors are building up assets in a way a utility major would do. Amazon’s free cash flow is set to turn negative in 2026. Meta’s shares fell over 9% the day it raised its spending guidance. Clearly the firms are betting on AI as if their survival depends on it, and taking uncomfortable measures to do so.

On the other hand, enterprises around the world are yet to realise value from their AI investments. According to a report from MIT, 95% of enterprise generative-AI pilots show no measurable impact on their P&L. S&P Global found that 42% of firms abandoned most AI initiatives in 2025. Gartner reported that at least 30% of AI POCs fail. All these independent reports converge on the fact that AI initiatives are failing to realise value. The reports also converge on why these AI initiatives fail. MIT names it as a “learning gap” of integration and context, not model quality. Each of the other independent reports blames data, integration and adoption, not the model. According to a Gartner study in 2026, “organizations that report successful AI initiatives invest up to four times more (as % of revenue) in foundational areas, such as data quality, governance, AI-ready people and change management, compared to those that experience poor outcomes from AI”.

But what precisely do these “foundational areas” mean? When all these reports state that AI projects fail to scale, this failure is attributed to the architectural problem of providing the correct information, at the right time to the right AI consumer. In other words, it is a “context problem”. The model capability is no longer the binding constraint. An OpenAI published study described how building the right context layer helped employees go from question to insight in minutes, not days.

What is a Context Layer

A Context Layer is the architectural tier that sits between the AI models or agents and an organisation’s data, tools, knowledge and rules. Its job is to assemble, govern and deliver the right information to the model at inference time. It is about translating raw metadata into governed business meaning with provenance trails that make answers auditable.

How it evolved

The evolution of Context Layer

The context layer evolved from an afterthought to a named category in roughly 4 phases. From 2020 to 2022, context simply meant the prompt. RAG was published as a research technique, but production systems mostly relied on prompt engineering. From 2022 to 2024, RAG industrialised. Vector databases (Pinecone, ChromaDB, Qdrant), orchestration frameworks (LangChain, LlamaIndex), function calling from mid-2023, and a long-context arms race that took the range from 4K to over a million tokens all materialised. But so did the realisation that bigger context windows and native retrieval did not solve grounding; they introduced cost, latency and “context rot”.

The third phase, from November 2024, was the protocol era. Anthropic released the Model Context Protocol as an open standard for connecting models to tools and data. OpenAI, Google DeepMind, Microsoft and thousands of teams adopted it. The official MCP registry launched in September 2025 and grew to nearly 2000 entries within months, and in December 2025 Anthropic donated MCP to the Agentic AI Foundation under the Linux Foundation, making it vendor-neutral and community-governed. By late 2025 there were more than 10,000 public MCP Servers deployed. In parallel “context engineering” was coined by Andrej Karpathy and Tobi Lutke as the successor to prompt engineering. Context engineering is the practice of architecting the entire information environment, not just the prompt but memory, tools, retrieval, and state.

The fourth phase arrived in 2026, which elevated the context layer to a named category. By March 2026, a16z had picked up the category in “Your Data Agents Need Context” confirming what practitioners had worked out: agents fail in production because they cannot read enterprise meaning, not because the models are weak. Bessemer’s 2026 infrastructure roadmap argues that as models become commoditised, differentiation shifts to the memory and context layer, which is now emerging as its own infrastructure category.

The Enterprise Context Layer - A Prescriptive View

The Enterprise context layer, from an opinionated perspective, consists of six stacked layers, plus a governance plane that spans them.

Enterprise Context Layer Stack

L1 Sources of truth Systems of record, warehouses, data lakes, documents, SaaS and APIs. The layer does not move them. It reads them where they already are.

L2 Access and ingestion plane MCP wrappers, CDC pipelines, event streams, document ingestion, and agent identity. This is the wrapping tier, and it has two jobs: context discovery, meaning what exists, and interconnect, meaning how it is reached.

L3 Living Semantic Core Knowledge graph, temporal ontology, entity resolution, and the governed metrics layer. Raw metadata becomes governed business meaning. A static ontology is stale before it ships, because the business it describes reorganises weekly while the semantic model waits for its quarterly review. The need is for a temporal ontology. Every fact carries a validity window. Facts are superseded, never silently overwritten, so the graph knows what was true when, what replaced it, and why. The definitions themselves are held as code: versioned, reviewed, promoted through environments like the software they now effectively are. Defined explicitly, versioned continuously.

L4 Context assembly & resolution runtime GraphRAG, vector stores, semantic SQL, context MCP endpoints, and policy with authorisation. Context is assembled, governed and delivered at query time, for the question actually asked. The design goal is to assemble the minimum sufficient context for this decision, not the maximum available context. The bottleneck in enterprise AI is the relevance; not retrieval volume. A person making a judgment call does not query everything the company has ever recorded. They draw on a compressed, relevance-weighted model of what matters now. The runtime is the mechanical version of that discipline: hybrid resolution across graph traversal, vector search and semantic SQL; entitlements applied at retrieval time so the assembled context arrives already permission-filtered; temporal validity applied so it arrives already time-filtered; then compression against the token budget of the model doing the reasoning. Every fact in the assembled context carries its provenance: source system, record and timestamp.

L5 Decision layer and context graph Decision traces, workflows, encoded policies and exceptions, and precedent. Everything below L5 stores what the enterprise knows. Nothing below it stores how the enterprise decides. And judgment, not knowledge, is how the decisions are made in real life: the call made on incomplete information, the exception granted under authority, the escalation that experience says is warranted even though the checklist does not. In every enterprise running today, that judgment is exercised thousands of times a day and retained nowhere. The record survives; the reasoning evaporates. The decision layer is where the reasoning stops evaporating. A decision trace records the inputs consulted, the policy applied, the exception invoked, the authority under which the call was made, the outcome, and the timestamp, written as graph structure rather than prose in a log. Workflows record the paths decisions actually follow, as distinct from the paths the process manual claims they follow. Encoded policies and exceptions turn rules into traversable graph elements, so that a constraint is something a query can hit rather than a paragraph someone must remember to consult. Precedent makes prior decisions retrievable as context for the next one. Importantly, this layer is not audit logging. Logs record what happened, for compliance, and are written to be forgotten; traces are written to be reasoned over. It is not case management, which governs work in flight and loses interest the moment a case closes. The distinction, in one word, is retrieval. The payoff from this layer comes in four forms: Guardrailing: constraints are enforced as graph traversal, so an agent physically cannot route around a policy it never sees. Observability: every automated decision has a traceable path through nodes, edges, constraints and authority. Evaluation: the traces are the evaluation dataset, generated as a by-product of operation rather than curated as a project. Self-learning: the graph improves from its own decision history. This is institutional memory made infrastructural.

The Write back loop Decisions made at the top of the stack, by agents and by humans alike, are written back into the decision layer as traces. Without the loop, the stack is a pipeline: context flows up, answers flow out, and everything learned in the act of deciding evaporates on contact with the air. With it, every adjudicated case enriches the layer that informs the next one, and the system compounds.

L6 Agent and application layer Copilots, custom agents, embedded AI, and BI and analytics. The consumers. They read business meaning, not raw tables, and they are the only layer most of the enterprise will ever see.

Governance plane Catalogue, lineage, observability, agent and model registry, evaluations and policy. This is a plane, not a stage. It spans L1 to L6 and runs through every layer. In this architecture every agent is a governed asset: registered, owned, versioned, entitled and retirable. The governance plane defines, L4 enforces. Policy written in the governance plane is enforced at resolution time, on every query, inside the runtime, not reviewed after the fact by a committee reading last month’s logs. Governance that exists only at design time is documentation. The same doctrine hardens evaluation from an option into a gate: an agent whose evaluations do not pass against the current state of the semantic core does not ship, and when a definition in L3 changes, the evaluations run again before anything that depends on it moves.

On Memory The decision layer is deliberately curated institutional memory. Agent memory is agent-scoped and learned through experience. Institutional enterprise context is enterprise-scoped and deliberately curated.

As it is evident that the enterprise context layer is emerging as the critical stack for AI success, multiple parties across the stack are accelerating in the race to own the domain. The Foundation model providers, data vendors, hyperscalers and practitioners are approaching the problem from their vantage point to bind a workable solution. As this landscape evolves, a few directions could emerge.

  1. Agent Evolution: Consolidation of the context layer into a durable “system of intelligence” tier, with the long-run end state framed as a deterministic and cognitive digital twin that understands the state of the business, captures rules and exceptions, supports human judgement, and teaches agents to improve, maturing over years rather than appearing as a single product.
  2. Standards Battle between Open Interchange vs Platform gravity: A standards battle over semantic portability between open interchange (OSI, MCP) versus platform gravity (Microsoft IQ, Palantir Ontology), for building federated foundational context that encodes how a company thinks, decides and acts.
  3. Memory as governed infrastructure: Memory maturing from frameworks into governed infrastructure.
  4. Governance hardening as a forcing function: Governance hardening as a forcing function, with increasing focus on memory manipulation protection as agents gain persistent state. Unified governance platforms will combine gateway and security functionality, accelerated by the EU AI Act timeline.
  5. Self-improving context: Contexts will evolve from agent experience rather than being hand-curated, which will make the context layer a learning system rather than a static integration layer.

The industry is now converging into a unified synthesis. The scarce asset in enterprise AI is no longer intelligence, but legible, governed, portable meaning, and the context layer is the architectural answer everyone is now racing to own.

References

"The Building Blocks of the Enterprise Context Layer" by Vinoth Haldorai is licensed under CC BY 4.0. You are free to share and adapt this material with attribution.

Related

  • Enterprise Context Layer: A reframe on legacy, modernization, and the AI decade

    For twenty years enterprises have relied on massive, multi-million bets that take many years to show value. Yet the modern data estate without business context cannot meaningfully let AI Agents enable faster business decisions at scale, identify patterns in real time, and act on context that no human can synthesize alone. Enter the Enterprise Context Layer. Instead of replacing your current estate, wrap it with a thin intelligent layer on top of it. It provides meaning to your data. It provides a way to make your legacy estate intelligent that compounds value.

  • How the permeation of Generative AI could impact the knowledge workplace

    Five shifts — from pyramids to pillars, commoditized early-stage creativity, decision governance, augmented collaboration, and trust as a function — that Generative AI could drive in the knowledge workplace.

  • The new paradigm - A "Platform-centric" approach to ERP Transformation

    Why the traditional market-first approach to ERP fails, and three mindset shifts — platform-first, continuous enrichment, and treating market variations as Lego blocks — that make ERP a strategic asset.

Comments