Scorecards

The Crisis

In late December, Western Governors University experienced a series of high-severity outages, followed by a massive university-wide outage over New Year’s Day. The CIO had to explain to the university president what went wrong and why IT was caught flat-footed during a holiday. The CIO told the directors to come up with a plan for getting things under control.

The Idea

Brian Jolley, Director of Service Management at Western Governors University, had recently read Daniel Kahneman’s Nobel Prize-winning book Thinking, Fast and Slow, which describes a method for comparing complex systems efficiently. Jolley adapted the Kahneman comparison framework into a proposal: score dev teams on a small set of measurable indicators of success, automate the scoring, and make the results visible to leadership weekly.

He brought the proposal to the other directors. The pitch was aspirational, not punitive — teams with high scores would earn more autonomy; teams with low scores would receive more support and resources. Not “rewards and punishment.” “Freedom and help.” The directors adopted the plan. Brian Jolley led the implementation.

The Five Indicators

Brian Jolley worked with the director group to agree on five measurable indicators of dev team health:

1. MTTA (Mean Time to Acknowledge) — how quickly a team responds to an outage alert.
2. MTTR (Mean Time to Restore) — how quickly service is restored once the team is engaged.
3. Open Incident Count — how many active incidents a team is carrying.
4. Uptime — system availability, measured using a severity matrix distinguishing “hard down” (Sev 1) from degraded service (Sev 2). This was the most contentious indicator — partial outages complicated the measurement, and the directors had to facilitate the severity matrix to resolution before scoring could be fair.
5. Open Problem Resolution Tasks — how many root-cause action items remain incomplete.

The Rollout

Brian Jolley directed his ServiceNow team to build an automated scorecard generator. The system pulled data from existing ticketing and incident management systems and produced a scoreboard listing each team and their scores — no manual data entry, no room for spin.

Before scorecards went into effect, teams were given a preview period of several weeks. They could see their scores and understand the methodology without consequences. By the time the plan went live, nobody could claim they were blindsided. Scorecards were reviewed in a weekly Operational Metrics meeting attended by VPs, directors, and team leads. The meeting started with the lowest-scoring teams first — high-scoring teams rarely came up, which was itself a motivator.

As teams became proficient, Brian Jolley reconvened the directors semi-annually to evaluate whether the indicators were still the right ones and ensure changes were implemented.

The Results

Within 2 months of implementation at Western Governors University, average outage resolution time dropped from hundreds of hours to under 10 hours. Within 6 months, average high-severity outage duration was under 2 hours. The improvements held because the underlying standards — testing, accountability, root cause analysis — were now embedded in the process, not dependent on individual effort.

Scorecards spread organically. Other areas of IT adopted the format. Brian Jolley implemented scorecards for each of his own teams. The model reached C-suite level and was adopted university-wide. Scorecard results became the criteria for mid-year and annual reviews, and high marks provided significant justification for promotions.

Why It Worked

The Kahneman comparison framework made it possible to evaluate complex, dissimilar systems on a common scale — the key insight that made scoring across disparate teams both fair and legible to leadership.

Aspirational framing produced buy-in instead of defensiveness. “More freedom” and “more help” produce a different organizational culture than “rewards” and “consequences,” even when the underlying mechanics are similar.

Automation eliminated subjective reporting. Scores came from ServiceNow directly — no manual entry, no opportunity to spin a bad week. Weekly visibility with leadership meant scores couldn’t be quietly ignored.

Connected Work

The scorecard system was one of five interlocking initiatives Brian Jolley built at Western Governors University. The MTTA metric originated in the Outage Response protocol. Open Problem Resolution Tasks were fed by the Directors’ Closure Review. The ownership clarity that made team-level scoring meaningful came from the Common Services Data Model.