Measuring success in companies pivoting to agile

(*Source)

As a developer I know how annoying it can be to be measured, mainly because when this measurements information starts being shared, you start to get questioned on things that you otherwise would have been left alone to work in peace.
Having metrics is one of most harmful tools if you use them wrongly, and I'll try to share with you my insights and experiences on this matter.

What are we measuring

Most likely the organization that you work on, is already measuring something. The organizations that are making an effort into making the agile transformation, tend to emphasize initially in the values, principles, and practices within the scrum teams and their direct ecosystems, and let the teams evolve and accommodate while they keep on working. This is a good approach.
What usually gets overlooked by the scrum teams nonetheless, are the metrics that are kept -unless stated otherwise- to bring measurements on how we are going during this transition. Sometimes the ones with active "hands on" the project, fail to notice the importance of those, because they are not shared with the scrum teams:"it's just information that the managers gather to share the project status with the leadership area".

Things common to see as measurements are the indicators of: if we are on scope, on schedule, on budget. But these are remaining measurements of a once non agile organization, since for an agile one, you might realize the metrics might no longer add the value they did in the past.

How does that impact the organization 

 The first thing that can be noticed out of these indicators, are it's relatable likeness with what is known in project management as the iron triangle. 

By having this indicators we are measuring how good we are on each of the three dimensions, and as such those sound great, right?

Well, almost: Having the three indicators as things we need to keep fixed on a goal, is only reasonable if we start from the premise that there are no risks, no market changes, and the product -that we've turned into an extensive documentation- is perfect as it is, and perfection is the enemy of what's possible.
This analysis is worth another entry on its own (and it's interesting enough so I'll create a note to myself to post something on it in this blog).

Doing a recap on this, in a risk existing context, "aiming for all" might possibly get you a huge quality sacrifice at best, and a huge pile of code that does nothing yet, at its worst. The thing about this is that not all customers understand that.

Customers, in most cases, were used to work heavily in a detail of what's needed at the beginning, and after an agreed amount of time, they were assured they would receive the full product. Under agile that no longer is the case, and we, by being adaptive to change, end up agreeing together -customers and IT- what are the next set of small increments we would like to see.

By working together, sending the whole perfect requirement detail at the beginning does no longer make sense, since we accept the fact that we don't know what perfection looks like.

Something I came to a conclusion, and during my experience happened, is that these indicators trigger unwanted behaviors in the mid level management. I can easily remember three particular cases where the fear of altering the indicators generated miscommunications:

1. Product owner receives update from team, we need more time, decides not to communicate, since communicating will get nothing good.
2. Guy representative of IT, responsible to set contracts with internal customers gives the team to estimate a project, decides to communicate the customer a lesser duration than what the team told him it could be done, to make sure he gets that contract. Note that he's measured by the amount of contracts he gets.
3. Manager directly alters the communication to leadership on a regular report, to avoid difficult  back and forth communication with annoyed customers. Postponing a mid deadline to overlap with the final deadline. If we don't reach the new goal, now won't be their fault.

I'm sure you can think on a couple of your own.

In the three cases:

• The deadlines most likely won't be met.
• There's a fear to communicate truthfully, or a direct beneficial impact by altering the communication.

What's wrong with this people? They wouldn't be in the position they are now if they were unskilled or unintelligent. The fact is that they are working with what they have, a determined context where taking that decision makes sense to them, even more so, their behavior is very likely to be successful from their perspective.

Ok, and what can be done then? can we remove them from the organization and place them in a place where they can do no harm? Hypothetically and in an organization where you have the power to take that decision (usually you don't, not easy at least) you could do that, and place another person there. But, How long do you think would pass until this next person starts realizing they can do the same thing? And how likely it is for them not to actually repeat that behavior? 

From the tree diagram perspective, by taking the original person out, you are chopping down the problem, but you are not attacking the causes, so given a particular context, your problem could grow back. So even if you could rotate out (or teach that person to not do that) it could be that this solution does not reach far.

I would analyze further, Why are they acting this way? Because it's useful for them. Why is it useful for them? Because they're being measured by something that, with this behavior, they'll always land on their feet.

So here is where my groundbreaking proposal starts taking shape: What if we changed what is considered as success?

What are the risks of measuring

Conceptually it makes sense to consider the most important things for a project as the things we want to measure, but we need to be aware of the consequences of using measurements as a target. More than 40 years ago, an economist called Charles Goodhart said something paraphrased for simplicity purposes as "When a measure becomes a target, it ceases to be a good measure." and that is still valid for any measurement, not only in mid 70's british economy. People will stop doing a good job to focus on hitting the metrics they are being measured by.

(Source)

You might have your teams smiling for the picture, focusing no longer in a good, healthy or predictable product, but instead a good, healthy and predictable metric.

What should we measure

After all this you might be thinking that I'm against getting metrics, but no. I'm just against metrics that are reactive. Let's say we have an indicator called "Quality" for instance. This would be measured after an iteration happened, and the metric would be the outcome of that iteration. To have a semaphore on yellow or red would then imply that the quality achieved is less than the one expected, and we would be talking on past time, something that has happened already and we need to adjust for next time.
If this is evaluated with the money spent on the project so far -which is almost always the main measure unit for leadership-, it will speak to them in terms of "we spent on a project returning red/yellow indicator, which is far from what was promised". Awful, right?

If  we only measure the gas tank of the airplane we can't assume we are ok.
We put our trust in the ones living the now of the project, any other
one would provide slow answers to imminent definitions.

Under an agile environment you can still get metrics, but instead proactive metrics, values regularly and quickly updated that, by being aware, the ones generating them can take immediate action. Not the leadership, but the scrum team, and their closest stakeholders, in the same way the pilot and co-pilot would be driven by the plane's control board info.

The metrics should be the visualization of how we are going, and the baseline where the teams can start doing experiments in order to change them. This impact the teams start doing with the experiments are ideally positive, but reality will tell you that experiments can fail.

There's a catch here that not all the organizations really take seriously: You need people that fails forward. Wake up and realize this: Failure is simply a price we pay to achieve success. (I might talk more on this in upcoming entries).

How does that work

The metrics should then be for the exclusive use of the teams.
The metrics should be tangible things that maybe by themselves won't tell you a lot, and no one should align how good we are to a single metric.
Metrics can be grouped on a single dimension, to get what we can call an indicator, the operational teams define how the indicators are constructed for them.
Leadership should know how are the indicators going, and propose the dimensions they are interested in seeing.

So here's a take on how has the implementation been in my previous working experiences with a good amount of success as months went by (2/3 of the operational teams in a big company using the metrics with actual improvements): 

1. Leadership defines Policies/Analysis dimensions.
2. Team defines metrics that are related to each dimension.
3. Team validates the stability of those metrics. (If those are not, they'll look for other ones)
4. Team defines how will the metrics impact the indicator.
5. Team start experiments on Plan-Do-Check-Act cycles.

You can always say that you might find yourself in a big prescriptive organization that might not allow you to generate these changes, but if you truly believe that this might generate a positive impact in your organization, lose your fear of failure, and start failing forward. Even when you believe that you're not moving forward a lot, you can say that you've learned from the path you've walked. And it's more important the learning path itself, than actually achieving your goal. Most likely, as me, you will be able to capitalize your learnings multiple times in your career path.