Your AI Pilot Worked. Now What?

There is a pattern I have watched repeat itself across almost every organization attempting AI transformation right now. The pilot succeeds. The demo impresses. Leadership greenlights the next phase. And then nothing happens.

Not because the technology failed. Because the organizational decisions around the technology were never made.

I have been building AI infrastructure inside Avenue Z for the past 18 months alongside a team of marketing engineers whose entire job is figuring out what actually works when you move AI from experimentation into production. We have made the wrong calls. We have rebuilt things. We have sat through demos that looked transformative and produced nothing when we tried to deploy them against real client work. The Glean Work AI Institute recently published research from 100 leaders working through exactly this problem, and what they found maps almost precisely to what we learned in the field.

The gap between AI experimentation and AI transformation comes down to a small set of decisions most organizations delay, delegate, or get wrong. This post is about those decisions.

The Pilot Trap

Arvind Jain, CEO of Glean, frames this plainly in the research: most enterprise AI pilots over the last two years had no clear reason for existing. They were launched because the capability looked interesting, not because they were tied to a specific business metric that someone was accountable for moving.

We fell into this ourselves. We ran agents that produced things people found interesting but that did not change how work got done. The measure of whether an agent is working is not whether it produces output. It is whether its output is now a required step in a workflow that did not need a human to trigger it.

That distinction matters more than it sounds. When you start a project as a pilot, you signal to your security team, your data team, and your business stakeholders that this is optional. People do not connect real systems to optional projects. They give you test data. They assign a junior resource. They do not clear the political path for real integration. And then you discover that your AI cannot answer real questions because it was never connected to real information.

Our marketing engineers have a simple test now before we take on an AI workflow build: can the person requesting it name the metric it will move and the workflow step it will replace? If not, it is not ready. That discipline is harder to maintain than it sounds, especially when the technology is genuinely exciting and stakeholders want to show progress. But the graveyard of impressive POCs inside most organizations is proof that excitement without accountability produces nothing at scale.

The pilot trap is not a technology problem. It is a commitment problem.

The Centralization Question

Once an organization decides to move beyond experimentation, the next decision is structural: where does AI live in the organization?

There are two failure modes, and most organizations drift toward one of them.

The first is full decentralization with no standards. Every team adopts whatever tools they prefer, builds whatever workflows they find useful, and optimizes for their own function. You get creative activity, shadow IT, and eventually a security incident or a compliance problem that forces a painful reset.

The second is full centralization with no autonomy. A center of excellence gets built, staffed with AI specialists, and tasked with building solutions for every department. The specialists do not understand the actual workflows of the business teams well enough to build things people want to use. Adoption stays low. The center of excellence becomes a cost center without demonstrable ROI.

We have experienced versions of both. Early on, individual practice leads at Avenue Z were making their own tool decisions. We ended up with overlapping platforms, inconsistent data access, and agents that could not talk to each other because they were built on different systems. When we centralized too aggressively in response, the marketing engineers became a bottleneck and adoption slowed because people were waiting for infrastructure they could not build themselves.

The answer is a design question about what gets centralized and what does not.

Centralize: security standards, data governance, model selection, vendor contracts, and the knowledge infrastructure that all AI tools run on top of. These require consistency. An AI system that respects data permissions in one department but not another is not enterprise-grade, it is a liability.

Decentralize: workflow design, use case identification, agent customization, and day-to-day learning about what the technology can do for a specific team’s work. These require proximity to the actual work.

At Avenue Z, this is how our Transformation Pod operates. Our marketing engineers own the central infrastructure. A 20 to 30 percent allocation into each service line ensures the people closest to client work are the ones identifying what to automate. Neither group can do the job of the other, and that division of responsibility took time to get right.

Stanford’s Bob Sutton, whose research informed the Glean report, offers a useful benchmark for what mature AI adoption eventually looks like: every team with an AI-native person the same way every team has a finance person. The central accounting function sets standards. Finance people embedded in each business unit apply those standards to local problems. When AI reaches that level of structural normalization, you know adoption is real.

The Expert-Novice Shift

One of the more counterintuitive findings from the research concerns expertise. The conventional assumption is that AI primarily benefits generalists, giving them capabilities that previously required specialists. That is partially true, but it misses the more important dynamic.

AI changes the value of expertise depending on where in the creative process you are.

When you are executing a proven path, deep expertise remains critical. An expert who has done something many times and understands the failure modes is not replaced by an AI that can approximate the output. The expert’s judgment about when the AI is wrong is what is valuable.

But at the front end of a problem, when you are exploring what the right path even is, expert knowledge can become a liability. Experts pattern-match to what has worked before. AI can surface options the expert would not consider because they fall outside their domain experience.

We have seen this in our own build. Our most experienced marketing leads sometimes have the hardest time using AI effectively on strategy work because they immediately recognize the output as a variant of something they already know. Our marketing engineers, who are closer to the infrastructure than to the marketing discipline, often catch problems with strategic assumptions that the domain experts miss precisely because they are not anchored to how things have always been done.

The practical implication: AI is most powerful in your organization when it is used to interrogate assumptions, not just accelerate execution. If your team is using AI to produce more content faster, that is a productivity gain. If your team is using AI to question whether the content strategy is right, that is a capability shift. Both are useful. Only the second one changes your competitive position.

The Flattening Org Chart

The research draws a useful distinction between heads-up work and heads-down work. Heads-down work is independent and focused. Heads-up work is collaborative, relationship-intensive, and requires human judgment in real time.

The argument is that AI primarily removes friction from heads-down work, which means organizations with a high proportion of heads-down work can afford to flatten their org charts. Organizations where heads-up work dominates cannot flatten as aggressively without overloading the people managing those relationships.

The more specific observation from Arvind Jain is that AI allows leaders to extend their effective span of control by making it possible to understand what their team is actually doing without requiring constant one-on-one time. A manager can understand the status of work across a team without needing a check-in for each person, because the information becomes ambient and current rather than dependent on someone’s discipline around updating a project management tool.

We track this directly in how our marketing engineers instrument client work. The goal is not to remove manager judgment from account decisions. It is to make sure the information that judgment requires is available without the coordination overhead that normally produces it. That changes what a client services lead can manage and at what span.

For marketing organizations, the implication is direct. Campaign performance, creative production status, account health signals: in most agencies these live in whatever system the relevant person last remembered to update. AI-powered infrastructure makes that information current and accessible without depending on someone’s habits around data hygiene. That is not a small operational change.

Speed: When to Push, When to Brake

The pressure to move fast with AI is real and legitimate. The organizations that figure out enterprise AI in the next 18 months will have a compounding advantage that is difficult to close. But speed without direction is expensive noise.

Sutton uses a race car analogy that holds up: the way to go fastest down the straightaway is knowing when to brake so you can make it through the turn. Organizations that have moved fastest on AI share a specific pattern. They made deliberate decisions about what not to do. They did not pursue 100 use cases. They identified five that were tied to their most important metrics, committed real resources, and built for production rather than demonstration.

The failure mode that wastes the most organizational energy is experimentation that generates internal enthusiasm and good demos but has no pathway to production. Our marketing engineers track this with a simple question: what workflow does this agent replace, and when does it replace it? If the answer is vague, the project is a demo, not a deployment.

We killed projects that were technically impressive because they could not answer that question. That discipline is uncomfortable in an agency environment where clients want to see momentum and leadership wants to show progress. But the alternative is a portfolio of interesting experiments that never change how revenue gets generated or how client work gets delivered.

What This Means for Our Clients

The organizations we work with at Avenue Z are at different stages of this journey. Some are still in pilot mode, running AI tools in isolated functions without connected infrastructure. Some have made the commitment to production but are discovering that the organizational decisions around centralization, governance, and workflow integration are harder than the technology selection.

What our team of marketing engineers has built inside Avenue Z is the proof of concept that these decisions are solvable and that the investment pays off. Our proprietary Z/OS platform and FFCI measurement product are outputs of exactly the organizational discipline described in this post: clear business outcomes, centralized infrastructure, decentralized workflow design, and the patience to kill projects that cannot demonstrate a path to production impact.

If you are a marketing or business leader working through these decisions, the technology is not the constraint. The questions worth spending time on are organizational: what are the five priorities, who owns them, what does success look like, and when does the pilot become a production system that changes how work gets done.

The Summary

• Start with business outcomes, not capabilities. The pilot that cannot name a specific metric it will move is not a strategy.
• Centralize governance, security, and knowledge infrastructure. Decentralize workflow design to the teams closest to the work.
• AI is most powerful when it expands the questions you ask, not just the speed at which you execute existing answers.
• Build for production from day one. Connect to real systems, real data, and real business processes.
• Control the rhythm. Identify the five priorities, commit fully, and be willing to kill projects that cannot demonstrate a path to measurable impact.
• AI flattens organizations by improving information flow, not by cutting headcount. The value is visibility and reduced coordination friction.

The organizations that internalize these decisions and act on them will look fundamentally different in 36 months. Not because they used more AI. Because they made the organizational choices that let AI actually change how they work.

Avenue Z is a hybrid marketing and PR agency with a dedicated team of marketing engineers who build AI-powered infrastructure for growth-oriented brands. If you want to talk through what this looks like for your organization, reach out.