Explainability Requirements for AI Outputs in Critical Facility Design

AI-generated design recommendations now sit inside the most consequential decisions in data center delivery. A routing error that reaches procurement can cost a schedule month, and a schedule month can cost tens of millions in lost revenue. In this environment, an AI output you cannot trace to its reasoning is not a convenience. It is a liability the project cannot absorb.

Why data center design decisions have become harder to reverse

Data center facilities have crossed a threshold. A flawed AI-generated design recommendation is no longer a minor correction. It is a schedule and cost catastrophe. Ram Nagappan of Oracle Cloud Infrastructure points to the root cause: operators must now design for two fundamentally different AI patterns (large-scale training and distributed inference), each with its own layout, resilience, and network implications. A single AI-generated layout recommendation has to satisfy competing constraints at the same time.

The physical transformation compounds the problem. Varun Sakalkar of Google's datacenter technology and systems group describes rack densities moving from the 30–40 kW range to hundreds of kilowatts, with designs approaching the megawatt range. Each step up forces different electrical distribution, liquid cooling, and structural decisions. Those decisions interact across domains in ways the design team cannot fully see at the moment a recommendation is made.

MEP and controls now represent the majority of the total construction cost on AI-focused bids. Electrical distribution alone commands a large share. Utility transformer lead times run 30 to 36 months after order placement, and Tier 1 manufacturers quote 48–60 months. Procurement, not structural work, is now the true critical path. A one-month schedule slip on a large AI-optimized facility costs the owner tens of millions in lost revenue. At that scale, an unexplained routing or layout error that reaches procurement is not an annotation problem. It is a program event.

Why AI outputs in this environment cannot be taken on faith

Leofante et al., writing in AI Magazine, state the situation directly: most AI techniques used in energy and critical infrastructure management lack the transparency required to understand the rationale behind their decisions. That gap hinders their applicability in these settings. In critical-facility design, the gap blocks action rather than understanding alone. An engineer acting on an AI-generated power routing or layout recommendation without a traceable rationale cannot validate that the recommendation accounts for cooling interface assumptions, fiber routing constraints, or equipment lead-time dependencies. The recommendation is effectively unverifiable.

An unverifiable recommendation cannot be defended in an RFI. It cannot be handed off to commissioning. It cannot be traced back if it turns out to be wrong. It produces a liability gap at every downstream review point. Plevris and Hosamo (2025) argue that AI models influencing the design of load-bearing structures or safety assessments must provide clear reasoning so engineers can validate and trust their outputs. The principle applies directly to power, cooling, and structural decisions in critical facilities, where the same life-safety stakes attach to a bus duct run as to a floor slab.

One common objection is that engineers can apply judgment over any AI output. That stance fails at the scale of modern data center delivery. When MEP coordination spans thousands of cable paths, hundreds of rack positions, and dozens of interdependent submittal packages, human judgment applied to an opaque output is not verification. It is exposure.

What explainability actually means in an engineering workflow

The GLACIS XAI Guide frames explainability as the ability to answer four questions about a specific output: which inputs influenced this decision, how were inputs weighted, why this decision versus alternatives, and is the logic consistent and fair across similar cases. Each of those questions maps to a real engineering review checkpoint. They are not academic criteria. They are the questions an MEP engineer asks before signing a submittal.

Explainability is distinct from interpretability. A deep neural network is not interpretable. You cannot inspect its parameters directly. But post-hoc techniques such as SHAP or LIME can generate explanations for specific predictions, making the model explainable even if its internals remain opaque. In an engineering context, local explainability matters more than global interpretability. The question is not "how does this model think." It is "why this specific recommendation."

Four levels of explainability organize the landscape. Level one covers inherently interpretable models such as decision trees and rule-based systems. Level two covers post-hoc model-specific techniques such as attention mechanisms and saliency maps. Level three covers post-hoc model-agnostic techniques: SHAP, LIME, and counterfactual explanations. Level four covers example-based explanations. Deterministic rule-based systems used for compliance checking sit at level one. Their outputs are inherently traceable. Generative AI layout tools typically sit at level three and require explicit post-hoc explanation layers.

This distinction carries a clear design implication. Deterministic rules belong where precision and compliance are non-negotiable: power path validation, cooling zone assignment, cable tray capacity checks. AI belongs where flexibility matters, with a required explanation layer for any output that enters a formal workflow.

The legal baseline that already applies to critical infrastructure AI

The EU AI Act classifies AI used as safety components in the management and operation of energy and critical infrastructure as high-risk. It legally mandates the specific transparency properties that engineers have been asking for on functional grounds. EU AI Act Article 13 (Regulation 2024/1689) requires high-risk AI systems to be "designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users to interpret the system's output and use it appropriately." High-risk obligations under the AI Omnibus apply from 2 December 2027.

Leofante et al. confirm the legal framing: given the high-stakes nature of critical infrastructure systems, transparency may be an important legal requirement. The EU AI Act explicitly classifies AI applications in energy and critical infrastructure as high-risk and legally demands transparency that "allows appropriate traceability and explainability." The practical implication for project teams is direct. An AI tool used to generate or validate power routing, cooling layout, or cable paths in a data center may already be subject to these requirements. Documentation of explainability must exist before the output enters a formal design package, not after.

The Daimler Truck study offers a useful empirical anchor. In a regulated industrial context, researchers found that current requirements engineering practices provide limited systematic support for explainability requirements. Recurring failures appeared in elicitation, where conceptual ambiguity dominates. In specification, where testability is limited. And in validation, where criteria are fragmented and regulatory uncertainty persists. The data center industry faces the same structural gap.

Where cross-domain AI recommendations fail first: power, cooling, and cable

Power failure remains the leading cause of impactful outages. Alpha Controls' analysis drawing on Uptime Institute's 2025 findings names power as the primary cause for a majority of operators surveyed, with cooling second. That operational consequence originates in design-phase coordination failures. AI outputs that optimize within a single domain without surfacing their interface assumptions for adjacent systems create compounding risk, precisely because data center design is a cross-domain coordination problem rather than a single-system optimization.

Consider power routing. A generative AI tool that produces a power routing recommendation without explaining which cooling interface assumptions it holds leaves the MEP engineer unable to validate the recommendation against the cooling design. Those assumptions include CDU flow rates, thermal delta targets, and pump redundancy topology. Two correct-looking outputs can be mutually incompatible. The engineer cannot see the conflict until commissioning.

Liquid cooling adds a coordination requirement that AI models rarely surface. CDU telemetry, flow rate, temperature delta, pump status, and leak detection zones must be explicitly defined as integration requirements before design is finalized. An AI cooling layout that does not surface these constraints leaves the project team blind to a commissioning dependency.

One boundary demands particular care. Building automation must not absorb the safety-critical functions of generation and grid protection. Those functions remain with dedicated PLC, SCADA, protection, and microgrid-control systems. An AI recommendation that blurs this boundary by proposing consolidated control without explaining the safety-critical separation is not just unexplained. It is hazardous.

The cable and fiber routing domain compounds the problem further. A rack position change suggested by an AI layout tool cascades into power path, cooling zone, cable tray, and fiber route changes. If the AI does not explain which constraints drove the recommendation, the engineer cannot assess whether the cascade was correctly resolved or merely shifted.

How unexplained AI outputs break the RFI and submittal chain

RFIs in critical-facility projects are formal records. They determine design intent, establish liability, and drive procurement decisions. An RFI answer that cites "AI-generated routing" without traceable reasoning behind it is not an answer. It is a deferred dispute. Plevris and Hosamo establish the standard: developing explainable AI models is crucial for allowing engineers to justify decisions based on AI-generated outputs. In the RFI context, justification is not optional. It is the deliverable.

Four improvements apply across regulated engineering workflows: source tracing, clear justifications for tool-generated decisions, domain-specific adaptation, and compliance validation. Each maps to a specific RFI or submittal requirement. Which standard was applied. Which parameter drove the decision. Which equipment spec was used. Without those answers, the formal record has a gap that no downstream review can close.

The source of the problem often sits in unstructured data. Equipment data trapped in manufacturer PDFs and not extracted into the model means that an AI recommendation based on outdated or assumed spec values cannot be audited. If the spec extraction did not flow into the model, the RFI answer has no verifiable source. The governance gap is equally real. A majority of organizations plan to adopt agentic AI for infrastructure. Only a small minority have established governance foundations adequate for AI readiness. Most project teams are using AI outputs in formal workflows without the traceability infrastructure those workflows require.

Where the BIM-to-operations handoff inherits unexplained AI decisions

AI-generated design decisions that lack documented reasoning do not disappear at handoff. They become unexplained configurations in the DCIM, EPMS, and BMS, where operators must manage equipment they cannot fully audit. A DCIM-enabled digital twin is only as accurate as the data exchange architecture built during the construction phase. If AI-generated routing or layout decisions were not explained during design, they cannot be reconstructed at handoff. The asset record is incomplete by definition.

The operational systems each serve a distinct audience. BMS, EPMS, and DCIM must define which system owns each data point, which system triggers alarms, and how data flows between them. If an AI-generated design created integration assumptions that were never documented, operations teams inherit configuration dependencies they cannot trace. The BIM-to-DCIM handoff requires explicit schema alignment and a shared asset identifier strategy. If AI-generated equipment placements or cable routes were not tagged according to the operator's CMMS and DCIM schema during design, as-built reconciliation becomes a manual reconstruction project. It is not a structured handoff.

The economics compound. Retrofitting integration after handover is significantly more expensive and rarely complete. The cost of an unexplained AI decision compounds from design through commissioning through operations, not just at the point where the error was first made. The terminal state is what practitioners call the viewer-only BIM anti-pattern: models with no structured asset data, no WBS alignment, no operational schema. That pattern is the endpoint of a design process that treated AI-generated outputs as drawings rather than as structured, traceable design decisions.

What a workflow that requires explainability actually looks like in practice

The GLACIS XAI Guide establishes the governing principle: explainability must be designed into AI systems from the beginning, with techniques chosen based on regulatory requirements, user needs, and acceptable performance trade-offs. Treating it as a post-deployment checklist produces higher failure rates and costlier rework. The technique choice is itself a design decision. Deterministic rule-based systems are inherently traceable and belong where compliance and precision are non-negotiable: power path validation, cooling zone assignment, cable tray capacity checks. Post-hoc model-agnostic techniques such as SHAP or LIME are appropriate for AI-generated recommendations that require local explanation before entering a formal workflow.

In practice, an explainability-designed workflow has three components. First, a defined output format that includes the inputs that drove the recommendation, the constraints that were applied, and the alternatives that were rejected. Second, a formal review step where an engineer validates the explanation before the recommendation enters a submittal package. Third, a structured record that links the explanation to the asset in the DCIM schema. Each component is checkable. Each closes a specific gap identified earlier.

A connected workflow makes explanation a by-product of automation rather than a separate documentation effort. When a rack move cascades into power, cooling, and cable changes, the system surfaces which rules drove each downstream change. That architecture connects layouts, critical-systems engineering, cable routing, documentation, RFIs, and operations-ready handoff inside a single workflow. The Daimler Truck study found that fragmented validation (vague criteria and regulatory uncertainty) is among the cross-cutting failures in explainability engineering. The answer is defined acceptance criteria for AI outputs established at project kickoff, not informal review at closeout.