Assessment is a skill that pays off in time

A while ago, I led a strategic project whose aim was to improve significantly the performance of certain use cases of a large enterprise application. The project looked particularly difficult given that nobody in the team had any experience with performance analysis and optimizations.

The project was organized according to the humane assessment philosophy. We started by identifying the right people for the team. We identified the use cases and agreed on a performance improvement target. Then we built a tool that helped us identify patterns of performance problems. We measured. We experimented. We measured again. We improved the tool with new detections based on the lessons learnt. And we repeated. At the end, we obtained for various use cases a speedup of 2 to 10 times in a much shorter amount of time than initially planned for.

At the end of the project, I gave a summary presentation to top management. They got excited with the results. A spreadsheet that shows 10 times speedup can easily produce that effect. Still, given that we only tackled a couple of dozen use cases, the question remained whether this effort was enough for improving the overall system problems.

I told them that while the spreadsheet might show current improvements, the most important gain is another one that cannot be quantified as easily: the team now knows how to approach a performance problem and it has the appropriate infrastructure to do it. As expected, this remark did not get much attention.

A couple of months down the road, we encountered another performance problem. It was a serious problem that threatened to bring the whole system to a halt. It happened unexpectedly, like all such problems. While various people showed signs of panic, one engineer took the problem, experimented with several setups and measured repeatedly. After a couple of hours, he concluded that the error can only come from one place. And he was right. He arrived to this conclusion exactly by following the simple learnings from the original performance improvement project. Based on his finding, the problem was fixed and deployed within a day.

It's easy to panic in crisis moments. And the panic is made even more likely if you do not see yourself capable of approaching the problem.

Investing in assessment is investing in knowledge. The effect of knowledge is hard to quantify especially when your tool is an Excel file, but its value becomes obvious when you see it in action.