Discussing assessment

A while ago I had a fruitful discussion with a smart executive about the necessity and nature of assessment in a software development context. Here are the notes for my part of the talking:

• Both the system and the organization optimize around what is in the feedback loop.
• Just like with laws, whatever the metrics, they should not be fixed forever.
• The law making mechanism is more important than the current set of laws.
• Instead of looking for the perfect metric now, it is better to invest in the mechanism that gets you the metrics you need based on the current situation.
• In the future, we will only get more data, not less. Thus, investing in the ability to analyze data in order to assess the situation will be more and more critical.
• Analyses always require interpretation to be useful.
• Interpretation is best made in the context of a previously formulated hypothesis and question.
• To be able to interpret analysis results, you need to control the data input, and the actual analysis.
• Analysis tools are important because without them we cannot grasp the vastness of data, but more important are the skills and know-how of approaching data, formulating questions and interpreting the results.
• The value of an analysis depends on the quality of data.
• It is often easier to make a metric look good, than to get the measured thing right. Thus, when you have subjective data, you want to ensure that those that create it believe in its value.