How did Goodhart's Law apply to WGU's software development problems?

Brian Jolley identified that WGU's software development culture was measuring feature velocity — how many new features shipped per sprint — as its primary success metric. This is a textbook example of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Feature velocity as a target incentivized shipping fast, which incentivized skipping the hard work of definition and testing, which injected bugs, created outages, generated support tickets, and accumulated technical debt. The measure itself was causing the problem. Jolley made the case that the right question was not 'how many features did we ship?' but 'how much use are existing features getting, and how satisfied are users with them?'

What was the stabilization quarter Brian Jolley secured at WGU?

Brian Jolley made the case to WGU leadership that the organization needed to pause feature velocity and invest in system health. Leadership agreed to a dedicated stabilization quarter — engineering time redirected from new features to fixing the systemic problems QA standards had exposed: outdated patches, database inefficiencies, duplicate solutions, data set redundancies, and long-standing critical defects deprioritized in favor of new feature work. Jolley used outage frequency and duration data to show how much engineering time per month was being consumed by unplanned outages — time that was now being recovered because the underlying systems were healthier. Getting an organization to pause feature delivery requires convincing executives that the short-term slowdown will produce long-term gains.

QA Transformation

Q: What was Brian Jolley's approach to QA transformation at Western Governors University?

Brian Jolley, Director of Software Quality Assurance at Western Governors University, used a single non-negotiable standard as his lever: no test results, no promotion. A ticket could not advance from development to staging, or from staging to production, without documented test results — what was tested, with what data, on what system, at what time, what steps were followed, and what the result was. This single requirement pulled a thread that unraveled the entire upstream process: you can't have test results without a test plan, you can't have a test plan without acceptance criteria, and acceptance criteria come from product owners. The standard raised the bar at every stage of the development process.

Q: What resistance did Brian Jolley face during the QA transformation at WGU?

Brian Jolley faced resistance from multiple directions during WGU's QA transformation. Product owners did not like being held accountable for defining their own products — the prevailing culture was 'don't make me think or make decisions, just figure it out,' with engineers expected to interpret ambiguous requirements and take the blame when results didn't match what product had imagined but never articulated. Engineers were annoyed because QA was now finding issues before production, meaning they had to do better work and sometimes redo it. Management was initially unhappy because feature deployment slowed down. The acceptance criteria battle was never fully won — but the standard shifted the default from 'no criteria, no accountability' to 'criteria expected, exceptions documented.'

One Gate, Total Upstream Discipline

What He Walked Into

When Brian Jolley was hired at Western Governors University in 2016, the Change Advisory Board (CAB) met daily to approve tickets for deployment to production. It was a formal meeting — and a farce.

A dozen or so tickets came through each day. In a one-hour meeting, getting into the details of any individual ticket was impossible. Without clear standards, there was no way to tell how well a ticket had been prepared. Sometimes the scrum master attending on behalf of the Software Engineering Manager hadn’t reviewed the work. Sometimes they had, and it still wasn’t ready.

Test plans existed — sometimes. They were occasionally attached as a document. If the team forgot and anyone noticed, the scrum master would hastily upload a document, attach it to the RFC, and say “I attached a test plan.” Those documents sometimes contained a single line: “Test the feature.”

That was a test plan. That was passing the CAB. That was going to production.

The Only Card He Had

Brian Jolley was the Director of Software Quality Assurance. He didn’t control product. He didn’t control engineering. He didn’t control the CAB process — the Service Desk director ran that.

His only card was control over the test gate. He played it. The new standard was simple and non-negotiable: no test results, no promotion. A ticket could not advance from development to staging, or from staging to production, without documented test results — what was tested, with what data, on what system, at what time, what steps were followed, and what the result was.

The Chain Reaction

The test results requirement pulled a thread that unraveled the entire upstream process.

You can’t have test results without a test plan. So test plans had to be real — actual test cases verifying actual expected behavior, not a one-line afterthought.

You can’t have a test plan without acceptance criteria. If nobody has defined what the feature is supposed to do, what is QA supposed to test? What is engineering supposed to build? What is the training team supposed to document? How is the Service Desk supposed to help a student if nobody has written down how the feature is supposed to work?

Acceptance criteria come from product owners. This is where the resistance hit.

The Upstream Battle

Product owners did not like being held accountable for defining their own products. The prevailing culture was “don’t make me think or make decisions, just figure it out” — and engineers were expected to interpret ambiguous requirements on their own and take the blame when the result didn’t match what product had imagined but never articulated.

Engineers were annoyed too. They now had to wait for QA, and QA was finding issues before production — which meant engineers had to do better work and sometimes redo it. Management was initially unhappy because the rate of new feature deployment slowed down. Product was measuring its success by feature velocity — how many new features shipped per sprint.

Goodhart’s Law

Brian Jolley recognized the problem: measuring the wrong thing was causing half the problems the university ecosystem faced. Feature velocity as a success metric incentivized shipping fast. Shipping fast incentivized skipping the hard work of definition and testing. Skipping definition and testing injected bugs, created outages, generated support tickets, and accumulated technical debt that eventually required more time to fix than the features saved.

The measure itself was causing the problem. This is Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. The real question wasn’t “how many features did we ship?” It was “how much use are existing features getting, and how satisfied are users with the ones getting used?”

The Stabilization Quarter

Brian Jolley made the case that the organization needed to pause feature velocity and invest in system health. Leadership agreed to a dedicated stabilization quarter — engineering time redirected from new features to fixing the systemic problems that QA standards had exposed: outdated patches, database inefficiencies, duplicate solutions, data set redundancies, and long-standing critical defects that had been deprioritized in favor of new feature work.

Jolley used outage frequency and duration data to show how much engineering time per month was being consumed by unplanned outages — time that was now being recovered because the underlying systems were healthier. Eventually, outage count and severity numbers improved and held. The improvement wasn’t a one-time cleanup. It was a structural change in how the organization built software.

The Limitation

The system wasn’t perfect. Many tickets were highly technical — configuring a database for an upcoming feature, for example — and a product person couldn’t realistically write acceptance criteria for that work. In those cases, the Software Engineering Manager wrote the criteria: an imperfect solution, but still better than no criteria at all.

The acceptance criteria battle was never fully won. But the standard shifted the default from “no criteria, no accountability” to “criteria expected, exceptions documented.” That shift changed the culture.

Why It Matters

QA Transformation is the earliest of Brian Jolley’s WGU initiatives, and it’s the foundation everything else was built on. The scorecard system measured things that only mattered because QA standards made them measurable. The DCR reviewed root causes that only surfaced because testing caught them before they became invisible production debt. Outage response improved because fewer outages were being injected in the first place. The CSDM clarified ownership that QA had been stumbling over since day one.

The core insight runs through every case study: you can’t fix what you can’t see, and you can’t see what you don’t measure. Jolley’s first act at WGU was making quality visible. Everything that followed was the result of people being able to see what had been hidden.