Moose: custom analyses made easy

Moose is advertised as an open-source platform for software and data analysis. But, what does that mean more specifically?

A decade ago, Moose's goal used to be to provide ready made software analysis tools. The keywords that described its existence were language independent modeling, metrics and polymeric views. Since then, Moose underwent a major redefinition and it became a platform that helps engineers to craft new custom analyses easily. Ten years ago Moose advertised the amount of out-of-the-box analyses. Now, it focuses on how fast you can build your own.

Moose is designed for programmers, not for clickers. To get the most out of it, you have to program it. First, that means that you have to program in Pharo. Second, you have to learn the inner workings of Moose to understand how to use it.

This is actually less difficult than it sounds. Pharo is a beautiful language, and if you do not know it already, you will not be sorry for learning it. And Moose is rather small having less than 2000 classes and less than 150k lines of code. Let's take a quick tour.

The Moose internal structure is generically depicted by the schematics below.

Moose-architecture.png

The picture emphasizes two aspects:

  1. The analysis process is iterative. Once data imported, the input to the actual analysis is a model. Applying an analysis yields another model that can be further analyzed. This is somewhat similar to a pipes-and-filters design, only it involves more feedback and interaction possibilities.
  2. The analysis process is customizable. The job of dedicated engines is to help the engineer craft new importers, models and analyses.

This might sound like mumbo jumbo, but let us take a moment and consider how this diagram is instantiated through a couple of examples.

PetitParser

PetitParser is a parsing engine. Its goal is to help engineers craft parsers for various programming languages and data formats.

Petitparser.png

For example, to build a parser for SQLite, you can follow the official abstract grammar. In this grammar, the selectCore is one of the central productions and the specification looks like:

syntax diagram select-core

Using PetitParser, this becomes:

selectCore
     ^ select ,
          (distinct / all) optional ,
     (resultColumn separatedBy: $, asParser trim) ,
     (from , joinSource) optional ,
     (where , expression ) optional ,
     (groupBy , (orderingTerm separatedBy: $, asParser trim), ('HAVING' asParser caseInsensitive trim , expression) optional ) optional

Due to PetitParser's expressivity, the implementation is essentially as compact as the original abstract definition. This characteristic makes mapping abstract notations to actual implementation simple. But, PetitParser goes a step further than usual similar engines, and it also provides custom development tools. For example, it comes with a browser that lets the engineer navigate the structure of a parser, and debug the execution of a parser against a textual input.

In the picture below, you can see the definition of the selectCore production from the SQLite grammar. In the lower part, the programmer can enter a sample and debug the result. The debugger is interactive and shows which production matched which part of the input string (in our example, joinSource matches TABLE).

Petitparser-with-inspector

Furthermore, the browser offers other visual representations of the grammar. For example, in the picture below, the browser shows a graphical decomposition of the selectCore production. The picture is essentially the same as the picture shown in the abstract grammar.

Petitparser-graphical.png

Developing an analysis is a software development project, and it is subject to the same problems any software development project has. Given that Moose's aim is precisely to decrease the cost of analysis, having a dedicated environment covering the complete cycle of developing a parser is a key ingredient for decreasing the cost of building non-toy parsers.

You can learn more about PetitParser in the dedicated chapter from the Deep into Pharo book.

Fame, FAMIX, MSE

Fame is one of the two engines that provide the meta-modeling infrastructure of Moose. The other one is Magritte and it is mostly used for user interface purposes. Fame provides several things:

  • a meta-engine,
  • a compact serialization format (MSE) in which both models and meta-models can be serialized,
  • code generation possibilities, and
  • a Java implementation that is fully compatible with the Moose one. In particular, the latter allows us to easily populate a model from Java, export it to an MSE file, and then load it into Moose.

Fame.png

Fame is essentially used for all meta-models used in Moose, the most prominent being FAMIX - a language independent meta-model for modeling software programs. For example, FAMIX models entities like Method, Attribute, or Access.

Fame attaches meta-descriptions to implementation via pragmas. For example, BehaviouralEntity is annotated on the class side like:

FAMIXBehavioralEntity class>>annotation
     <MSEClass: #BehaviouralEntity super: #FAMIXContainerEntity>
     <package: #FAMIX>
     ^self

Properties are annotated in a similar fashion. For example, the property pointing to all Access entities that originate in a BehavioralEntity is annotated in the getter method:

FAMIXBehavioralEntityaccesses
     <MSEProperty: #accesses type: #FAMIXAccess opposite: #accessor>
     <multivalued> <derived>
     <MSEComment: 'Accesses to variables made by this behaviour.’>
     ^ accesses

Based on these annotations Fame constructs a meta-model that lives in parallel to the actual implementation and that offers extra information useful for scenarios like browsing or import-export.

An engine is not complete without at least a dedicated tool that helps you manage it. In particular, because with Fame you can define various meta-models implemented in the image via pragmas attached to the implementation classes, we need to keep track of what is available.

To this end, the Moose Meta Browser provides an easy way to keep an overview of the various meta-models available at any given time in the Moose image. The picture below shows the details of the BehaviouralEntity from FAMIX.

Metabrowser.png

But, FAMIX is more than just a simple meta-model for software systems. It is also a rich API for querying those models.

Let’s look at an example. Suppose you want to look for the clients of all classes that are annotated with a certain annotation.

annotatedClasses := model allClasses select: [:each | each isAnnotatedWith: 'AnAnnotation’].
annotatedClasses flatCollectAsSet: #clientTypes

Having a highly expressive API brings makes querying inexpensive. And when you combine this ability with visual and interactive tools, you get a different way of experiencing your systems.

Roassal

Roassal is an engine for crafting interactive graph visualizations. For example, the visualization from the picture above is built using Roassal.

Roassal.png

At its core, Roassal offers a fine grained object-oriented model for displaying, interacting and animating graphs. On top of this model, Roassal offers several convenient builders for mapping arbitrary objects onto a drawable graph, the most known being the MondrianViewBuilder (this simple layer replaced the old Mondrian engine).

To make visualization crafting easier, Moose offers tools to develop such visualizations. For example, the Roassal Easel provides an interactive editor in which developers can develop visualizations and preview them against a given piece of data.

For example, the picture below shows an example of a visualization script executed against a group of classes. In this particular case, the classes are actually from Moose itself and the visualization shows what annotations are used within those classes and how they are related to each other (these happen to be the annotation with which one can customize the GTDebugger - the Pharo debugger that is built with Moose technology).

If you want to reproduce it, you can use the script below:

view shape ellipse withoutBorder; fillColor: Color lightGray.
view nodes: classGroup.
view shape
     rectangle withoutBorder; fillColor: (Color blue alpha: 0.2);
     withText: #name.
view nodes: (classGroup flatCollectAsSet: [:each |
     each methods flatCollect: #annotationTypes]).
view edges: (classGroup flatCollect: #methods)
     fromAll: #annotationTypes
     to: #parentType.
view forceBasedLayout 

Although the logic behind the script is complicated (getting all annotations from a class, linking them with each class etc), the script is rather compact. Even if it is compact, you do not even have to write the complete script at once. Instead, you can preview the state of the visualization after each modification. Thus, you can iterate easily until you reach an acceptable result.

Roassal-easel.png

These visualizations are fully interactive, and developers can continue inspecting the objects behind the visual representations. This is particularly relevant for supporting iterative analyses.

You can learn more about Roassal in the dedicated chapter from the Deep into Pharo book.

Glamour

Visualizations are great, but analyzing complicated models requires browsing through multiple complementary views. To help you craft browsers fast and inexpensively, Moose comes with the Glamour browsing engine.

Glamour.png

The entire user interface of Moose is built with Glamour (yes, the pun is intended). For example, take a look at the code browser below. It shows a standard browser opened on a Java system.

Glamour-code-browser.png

The size of the code associated with this browser measures about 130 lines. This tiny compared with the functionality. For example, inside the same browser, you can scope the methods by the instance variables that they use, and you can navigate through the senders and implementors of a method. This is possible because Glamour is based on a novel model for capturing the essence of browsing at a higher level than that of user interface widgets.

The goal of Glamour is to let you build the workflow of a browser, while the more fancy rendering is delegated to other engines like Roassal. Glamour also comes with a dedicated editor that enables developers to test and preview their browsers.

For example, the picture below shows a browser that lets you query or select classes from a list, and preview their interaction as a System Attraction visualization. The browser is generated with the script at the bottom and previewed on top.

composer tabulator with: [ :t |
     t column: #list; column: #map.
     t transmit to: #list; andShow: [ :a |
          a list
               beMultiple;
               showOnly: 100;
               format: #mooseDisplayString;
               withSmalltalkSearch ].
     t transmit from: #list; to: #map; andShow: [:a :selected |
          a roassal
               painting: [:view |
                    FAMIXSystemAttraction new view: selected on: view ] ] ].
composer startOn: classGroup 

Glamour-editor.png

Perhaps interesting to note is that the editor itself was built with Glamour, too. Or the Finder interface that you can see below was also written in Glamour in a couple of hundreds of lines of code.

Browsers are important tools for understanding complicated models. Glamour makes the construction of such browsers inexpensive. You can learn more about Glamour in the dedicated chapter from the Moose Book.

Finder

All these engines can be combined in various forms. One interface in which most of them come together is the Moose Finder, the basic browser meant to help you navigate through models.

Finder.png

The browser is based on Glamour. To cope with any model, it generates dynamically the navigation paths based on Fame descriptions (for example, when selecting a class you can navigate to its methods). It also includes multiple Roassal-based visualizations that are specific to each entity type. Also, for color displaying source code with syntax highlighting, it uses a PetitParser-based parser.

For example, the screenshot below shows the finder open on a group of classes from a Java system. The pane to the right shows the result of the query entered at the bottom (each annotationTypes notEmpty retrieves all classes that have at least one class-level annotation). Furthermore, the preview is shown highlighted on a System Nesting Map.

Finder-with-query-and-visualization.png

Selecting one of the classes from the map, spawns another pane to the right. In our case, the browser shows the source code of the selected class with syntax highlighting.

Finder-with-visualization-and-source-code.png

The Moose Finder focuses on the iterative nature of analysis. Every instance of the browser captures one analysis flow where every pane represents a step in the analysis. This is particularly useful when the analysis requires multiple steps. Furthermore, the Finder is extensible, and developers can easily add presentations specific to their entities.

Other engines and tools

The engines and tools mentioned above are the most visible one that come with Moose. Yet, Moose offers more. To name a few:

  • Arki for building reports;
  • MooseAlgos for expressing various algorithms related to data mining, graphs or traversals;
  • The Glamorous Toolkit for supporting live programming in Pharo (including the GTInspector and GTDebugger);
  • GraphET for visualizing charts;
  • Various dedicated browsers;
  • Metanool for handling custom annotations on entities;
  • Multiple off-the-shelf software metrics and visualizations.

Summary

Moose is a platform that is made for programmers. It provides engines to express those analyses, and a rich development environment to actually craft them.

Working with Moose does imply a learning curve. However, this investment is worthwhile from two perspectives:

  1. Moose aims to be versatile enough so that you can address multiple scenarios with it. Its most obvious target is the analysis of source code models. However, the same tools can be used to reason about any objects, too.
  2. Moose makes humane assessment possible by bringing the cost of crafting custom tools close to zero. When tools cost almost nothing, the activity of humans can change dramatically.

But, Moose is more than the source code. It is also an active and open community. If you want to give it a try, just join us. Work with us. Play with us. Software engineering is still young and there are plenty of things to uncover.

And if someone asks you about Moose, you can tell them that Moose is all about:

Custom analyses made easy

Posted by Tudor Girba at 1 January 2014, 10:53 pm with tags analysis, moose, tooling link
|