To make humane assessment practical, we need to be able to address the issue of the cost of building dedicated analysis tools.
First and foremost, an analysis tool is a piece of software. Like any piece of software, its building should follow a process, too. Ideally, this process should be fast and agile, but it is a process nevertheless and it includes activities like specification, estimation, design, implementation, testing, and even assessment.
An analysis can be complex but the right infrastructure can significantly alleviate the effort of building one. Moose is one such infrastructure designed exactly to support engineers build their own tools. It goes as far as to offer mechanisms for crafting throwaway analysis tools.
When the infrastructure is not enough, dedicate a buildup phase.
Infrastructure is important, but the analysis skill is critical. This skill is concerned with knowing what to look for and how to dig into data. The more know-how you possess, the cheaper and faster tool building is. This is the role of the assessment facilitators. The facilitators form the main development team of custom tools. Invest in them.
A tool can still cost too much to implement even when you have the right infrastructure in place. However, this does not mean that you have to give up.
One scenario in which this occurs is when the problem cannot be mapped on a tool because the input data might be too irregular. For example, suppose that we would like to identify the relationships between the user interface layer and the business logic layer. If we cannot identify the classes that belong in the two layers, we cannot solve the problem.
This is a typical sign for another problem that needs to be solved first. In this case, a restructuring of the system is required such that the rule to identify the classes that belong to each component becomes simple. For example, perhaps the package structure needs to be reshaped to reflect the components.
Another scenario is when the assessment question is too ambitious with respect to what can be achieved simply. In the end, an analysis infrastructure is still just a piece of software technology with the afferent limitations. It will make some analyses easy, and others less easy. However, it is often possible to approximate the problem to another one that can be implemented easier.
Assessment is not a one way street. In general, if a tool costs too much, look for two things: restructuring of the system, or refining of the problem.