Blog

Installing GToolkit

GToolkit holds a new generation of Pharo development tools based on Moose engines. Currently, it comes with three fully productive tools:

These tools work in Pharo 3.0. To use them, you have to first load the GToolkit code and then replace them in the image. You can achieve this in several ways.

Download the latest image

You can download the latest development image from the Jenkins build server: https://ci.inria.fr/moose/job/gtoolkit/lastSuccessfulBuild/artifact/gtoolkit.zip

The image is based on a Pharo 3.0 image and comes with the GToolkit already installed.

Install it from the command line

Command line interfaces are particularly useful for continuous integration job. GToolkit comes with a dedicated command line handler that lets you set up an image with a one liner. Here is an example of how it can look like in a Jenkins job:

wget --quiet -O - http://get.pharo.org/30+vm | bash
./pharo Pharo.image save $JOB_NAME
REPO=http://www.smalltalkhub.com/mc/Moose/GToolkit/main
./pharo $JOB_NAME.image config $REPO ConfigurationOfGToolkit --install=development
./pharo $JOB_NAME.image gtimagesetup

Load it from the image

Obviously, you can also load the code and install the tools from the image, too:

Gofer new
     smalltalkhubUser: 'Moose' project: 'GToolkit';
     configuration;
     loadDevelopment.
#GTImageSetupCommandLineHandler asClass new activate


Enjoy! And do not forget to send a mail to the Moose mailing list or the Pharo mailing list to let us know how it works for you.

Posted by Tudor Girba at 25 March 2014, 7:14 am with tags moose, pharo, tooling, assessment link
|

On analyzing the density of nil checks

How many nil checks are there in your system?

A recent paper entitled Does return null matter? published at CSMR-WCRE 2014 goes after this issue. The paper presents a systematic analysis of many systems over multiple versions and among others, it concludes that most systems oscillate between 20-40 nil checks for every 1000 lines of code, and that systems that go beyond the 40 limit seem to get in trouble.

I liked the paper. Null proliferation is a plague that deserves a closer examination. The paper conclusion looked quite intriguing, too. While not necessarily providing a direct path to action, answering this question for your system can still be educating. At the very least, it can make for a nice analysis exercise.

In the following I describe my experience of measuring the density of null checks for two systems and the lessons I learnt from it.

Checking a Java system

I first wanted to check a Java system I know. To detect the null checks, I could have relied on a deep abstract syntax tree traversal, but in my case, a simple text analysis was sufficient: I simply checked all occurrences of null checks using regular expressions on a FAMIX model of the system.

relevantClasses := model allModelClasses reject: [ :each | 
     each isAnonymousClass and: [ each isInnerClass ].
nullCheckTextPatterns := #('== null' '!= null' 'null ==' 'null !=‘).
numberOfNullChecks := relevantClasses sum: [ :each |
     (each sourceText allRegexMatches: ( '|' join: nullCheckTextPatterns)) size].

Note how the input to the measurement is a subset of all classes defined in the system. Because I was doing a plain text analysis, I needed to ensure that every line of code was traversed only once. Thus, I had to ignore both the anonymous classes (which are defined inside methods) and inner classes (which are defined inside classes).

Once the number of null checks available, I simply divided the amount of null checks to the total number of lines of code.

numberOfNullChecks / (relevantClasses sum: #numberOfLinesOfCode)

The result was 0.016.

Using average can be misleading when judging a distribution. Here is a map (drawn with the new CodeCity implementation from Richard Wettel) that offers a visual perspective: boxes represent classes grouped in meaningful packages; the height and the redness of a box grow with the amount of null checks; the base of the box grows with the amount of methods. The visualization reveals that there are clear hot spots throughout the system.

Java-codecity.png

Checking the Moose code

For comparison purposes, I wanted to find the answer for the Moose code as well.

Given that Moose is built in Pharo, and that in Pharo we wrap actual nil checks into more meaningful messages, I could not apply the same text-based analysis. Instead I needed a deeper analysis. But, before that, I first had to find what are the methods that actually do imply a nil check.

We know that typical nil checking methods have nil in their name. For example, #ifNil: is such a method. But, what are all the other methods?

A simple analysis provides the answer. I created a FAMIX model for the Moose code, and looked for all methods that have nil in their name:

model := MooseModel root allModels first.
(model allInvocations select: [:each | '*nil*' match: each signature ]) 
     collectAsSet: #signature

This resulted in a set of 25 methods:

a Set(#'morph:withTitleOrNilOf:(Object,Object)' #'ifNil:ifNotNil:(Object,Object)' #'allowNil()' #'ifNotNil:ifNil:(Object,Object)' #'ifNotNilDo:ifNil:(Object,Object)' #'propertyNamed:ifNil:(Object,Object)' #'useImplicitNotNil()' #'ifNotNilDo:(Object)' #'characterStyleOrNilAt:(Object)' #'ifNotNil:(Object)' #'concreteClassOrNil()' #'isEmptyOrNil()' #'useExplicitNotNil()' #'characterStyleOrNilIfApplying:(Object)' #'renderWithTitleOrNil:(Object)' #'usesImplicitAllNil()' #'findClassNamedOrNilFromFullName:(Object)' #'notNil()' #'ifNil:(Object)' #'useExplicitAllNil()' #'nilLiteral()' #'usesImplicitNotNil()' #'isNil()' #'selectedItemOrItemsOrNil()' #'allowAllNil()')

Of these, not all represent actual nil checks. A messages like #isNil clearly represents a nil check, but some others have other purposes. For example, #allowNil is used in Glamour as a setting. Thus, for the actual analysis, I had to disregard those messages manually.

Armed with the actual target message list, I could proceed to find all their usages within the code. On the same model I could do:

numberOfNullChecks:= model allInvocations select: [ :each |
     #(#'ifNil:ifNotNil:(Object,Object)' #'ifNotNil:ifNil:(Object,Object)' #'ifNotNilDo:ifNil:(Object,Object)' #'propertyNamed:ifNil:(Object,Object)' #'ifNotNilDo:(Object)'  #'ifNotNil:(Object)' #'isEmptyOrNil()' #'notNil()' #'ifNil:(Object)' #'isNil()' ) includes: each signature
     ].

Finding the density is now a simple arithmetic problem:

numberOfNullCheckssize / (model allMethods sum: #numberOfLinesOfCode) asFloat. 

The result? 0.014.

Comparing the two systems

Both systems had about the same ratio. This was highly surprising because when working with the Java system, I encountered more problems related to handling nulls than when working with Moose. I was intrigued.

After sleeping on it, I realized two things. First, the expressiveness of the Pharo language and libraries makes programs written in it more concise. For example, a simple collection selection in Pharo is a one liner, while in Java it requires at least three lines: new collection instantiation, a for loop, and adding to the collection. This issue would then imply that the same amount of nil checks would appear more dense in a Pharo code than in a Java one. Nevertheless, this issue is hard to capture automatically. One possibility would have been to relate the density to the number of statements rather than the number of lines of code, but this would have meant a different measure than the one proposed in the paper.

Second, there simply are more empty or irrelevant lines in Java than there are in a Pharo system. Let’s consider an example of a Java class:

public class A
{
     private X x;

     public X getX()
     {
          if (x == null)
          {
               x = this.computeX();   
          }
          return x;
     }

     public void setX(X x)
     {
          this.x = x;
     }
}

Computing the default number of lines of code, retrieves 19. However, there are several lines that are not that relevant for our computation:

  • there are several empty lines of code, and
  • there are several lines containing only a curly brace.

The corresponding code in Pharo would look like:

Object subclass: #A
     instanceVariableNames: 'x'
     classVariableNames: ' '
     poolDictionaries: ' '
     category: ' '
A>>x
     ^ x ifNil: [x := computeX]
A>>x: anX
     x := anX

In this case, all lines are relevant. Also, note how while x requires an full line in Java, in Pharo, this is part of the variables definition placed on one line. To make the comparison more reasonable, we would need to normalize the lines of code. To this end, I considered only:

  • the lines inside methods,
  • the lines that are not empty and that have more than a curly brace (Java) or square bracket (Pharo).

To compute this, I defined a new metric that only counted a line if it contained more than one non-space character:

relevantClasses := model allModelClasses reject: [ :each | each isAnonymousClass ].
numberOfRelevantLinesOfCode := relevantClasses sumNumbers: [ :each |
     each methodsGroup sumNumbers: [ :m |
          (m sourceText lines select: [ :line | 
               line trimBoth size > 1 ]) size ]]

Given that the granularity of the analysis changed from class-level to method-level, this time, I only ignored the anonymous classes, as they are part of the method. Inner classes are not excluded anymore because I needed to count their methods, too.

Interestingly, the metric worked out of the box for both Pharo and Java systems. (Also, please observe how the expressive Pharo string manipulation library makes the line checking almost trivial.)

After redoing the measurement I got:

  • Moose: 0.016
  • Java system: 0.028

In the case of Moose, the result is almost unchanged (0.016 instead of 0.014). However, for the Java system we get an almost double value (0.028 vs 0.016).

Lessons learnt

While analyses should be objective, subjective feelings still have an important place in the assessment process: they generate questions. In our case, I could have simply be content with the results and consider the two systems be similar, but my gut feeling told me otherwise. I questioned the underlying assumptions behind the original analysis, and got a significantly different result.

Of course, one danger with this approach is that you might end up constructing scenarios and gather facts just to confirm your feelings. This is one psychological issues of the pitfalls described in the Sources of Power book. It could be that this is what happened here. However, I am more confident in the later analysis than I am in the first one. The main reason behind my confidence is that I can easily describe the reasons for the improved analysis.

What’s more, I slept over the problem before interpreting it. That is always useful when you are unsure.

From a different point of view, a rather easy metric proved to produce significantly different results once I flipped a bit the interpretation. Tom deMarco famously said that you cannot control what you cannot measure. There certainly is truth to this, but I still feel the need to be more precise:

You cannot control if you do not know how you measure.
Tudor Girba

Posted by Tudor Girba at 14 March 2014, 7:11 am with tags spike, story, moose, pharo, assessment link
|

Dynamic exploration of a Postgres database with the GTInspector

Constructing and previewing queries over relational databases is typically done in dedicated DB client tools that are far away from the development environment. However, when working with relational data, querying is a common activity in software development, and thus it should be supported more prominently by the IDE.

Moreover, database clients are rather poor at doing anything non-textual. Thus, any time a graphical representation is needed, the developer typically resorts to exporting the data and using an external charting tool.

The GTInspector offers a simple set of extensions that brings together both of these aspects into one workflow. In this post, I demonstrate the way it works on a Postgres database, but a similar approach can be used for any other DB binding.

For the purpose of this exercise, I am using the World database, and the PostgresV2 Pharo implementation. To work through it, download the latest Moose image, load the PostgresV2 binding:

Gofer new
     smalltalkhubUser: 'PharoExtras' project: 'PostgresV2';
     configuration;
     load.
(#ConfigurationOfPostgresV2 asClass project version: '2.0-baseline') load

and the GTInspector extension for PostgresV2:

Gofer new 
     smalltalkhubUser: ‘Moose’ project: ‘GToolkit’;
     package: 'GT-InspectorExtensions-PostgresV2’;
     load.

The first step is to create a Postgres connection.

| conn |
conn := PGConnection new.
conn connectionArgs:
          (PGConnectionArgs
               hostname: 'localhost'
               portno: 5432
               databaseName: 'girba'
               userName: 'girba'
               password: '').
conn startup.
conn

Executing this gets you a connection object. The connection object is the entry point to starting any type of DB interaction via SQL. To make this smoother, the GTInspector offers a dedicated SQL presentation.

Connection.png

For example, the DB has a city table. Let’s inspect its contents:

select * from city

Executing the query (CMD+o) spawns a result object that can be viewed as a table.

City-result.png

The result table is obviously not static. Selecting a row spawns a preview of all values.

Values.png

Now, let’s consider a more concrete scenario. We want to get an idea of what are the largest cities in the world and to which continent the belong. For this purpose, we need a more complicated SQL statement:

select 
     city.countrycode,
     city.name as city name,
     city.population,
     country.lifeexpectancy as life,
     country.continent 
          from city left join country
          on city.countrycode=country.code

Evaluating it, gets us another result.

City-complex-result.png

For our problem we would benefit greatly from a visual representation. The simplest way to represent the data is a bar chart. To this end, we can use the Graph-ET engine that ships with Moose and that is integrated in the inspector:

| builder models |
models := self dataRows
          sorted: [ :x :y | (x atName: #population) > 
                            (y atName: #population) ].
builder := GETDiagramBuilder new.
builder horizontalBarDiagram
     models: models;
     modelLabels: [ :row | row atName: #cityname ];
     x: [ :row | row atName: #population ];
     if: [:row | (row atName: #continent) = 'Asia’]
          color: Color orange lighter;
     if: [:row | (row atName: #continent) = 'Europe’]
          color: Color red darker;
     if: [:row | (row atName: #continent) = 'North America’]
          color: Color blue darker;
     if: [:row | (row atName: #continent) = 'South America’]
          color: Color green darker.
builder view

Executing the above code within the context of the result object gets us a view object that offers a preview:

Chart.png

Describing this session through screenshots does not quite convey the dynamic experience that allows you to occasionally pick through data and come back to continue scripting. Perhaps the video below tells a better story:

All in all, this session involved multiple actions: querying a DB, previewing results, exploring code to learn the API, scripting a chart, and extending the inspector from within the inspector with a dedicated presentation (shown in the video only). All these are captured through a simple and consistent user interface that offered by the most basic tool available in a Pharo-based image: the inspector. This is not cosmetics. It is an essential redefinition of the I in IDE.

And if we are at it, the implementation of the Postgres specific inspector extensions consists of 36 lines of code.

Posted by Tudor Girba at 9 March 2014, 10:20 pm with tags pharo, moose, assessment, spike, story link
|

Beyond code reading

In a recent post on code reading, Peter Seibel notes how he went from seeing code as literature pieces to seeing code as specimen. He builds on this idea and proposes that developers practice the art of studying such specimen.

I concur with the idea that code is not literature. Actually, wait! Code is not even text. It only incidentally has a text shape, but that is not what defines it. Code is data.

The aforementioned post reports that developers consistently say: code reading is important and we should do it more. I am asking similar questions and I confirm that I get similar responses.

The author proposes a solution for the do it more predicament: hold code reading sessions in which samples of code are being scrutinized. This is a nice proposition and I can see its value. However, while reading code in the small can be educating, digesting code in the large is more important.

Going back to the code reading is important and we should do it more statement, I see the first part as being equally problematic. Why is it important? It is important because developers spend most of their time doing exactly that: reading code. And yet, we never talk about it. Hence, we never really learn and optimize. It's a classic elephant in the room situation.

We need is to start the conversation and learn from each other. The proposed code reading exercises have the potential of starting that conversation.

Yet, we have to be careful about how we frame the problem. If we look closer, we might notice that we do not even have a name for the activity of looking around the system to figure out what is going on. We keep saying code reading, but is where the problem is in the first place. Code reading describes only the means. The larger activity and goal remain unnamed.

Code reading is still going to be required no matter what, but it cannot remain the only option. Perhaps to get it out of our system, we should rename code reading to the-most-manual-way-to-dig-through-lots-of-data-having-a-textual-shape. We need a better name that does not tie us with a predefined solution and that sets the stage for considering other options. I call this assessment, and this site is dedicated to dissecting its various implications.

We need to have this conversation in software engineering. To my mind, it’s the most important endeavor we still have to undertake before calling software engineering a complete discipline. Building alternatives to reading code has been a research focus for a long time, but the conversation cannot remain in the academic ivory tower. It has to descend and include the engineers from the code trenches. Only when it will be relevant.

Let’s proceed.

Posted by Tudor Girba at 7 March 2014, 6:49 am link
|

Introducing the GTPlayground

One of the concepts that Pharo inherited from its Smalltalk ancestors is the Workspace.

The workspace is a little utility that lives next to the Inspector and it serves at least two large use cases:

  1. It lets you execute code snippets. This is particularly convenient for various things, such as loading code from a repository using Gofer scripts, or initializing objects from the image.
  2. It lets you initiate an investigation by inspecting the result of executing a piece of code.

The workspace is designed as a simple standalone code editor. For the first usage, it’s enough of a concept as the only thing you need is a place to write code. But, let us focus on the second case.

Suppose you want to investigate a file reference from the working directory. You start by typing:

FileSystem disk workingDirectory

You inspect the result, and continue in the inspector. This results in two unrelated windows that you have to manage. We have followed this workflow for 30 years. It’s time to rethink it.

Default-workspace.png

Let’s imagine a different situation in which you have an inspector already opened on FileSystem disk, and you execute self workingDirectory. Indeed, with the default inspector you end up with two unrelated windows, but with the GTInspector you get only one workflow.

Looking at these scenarios, it should follow that we should be able to integrate the workspace with the inspector. Enter (GT)Playground.

The GTPlayground (part of the GToolkit) is essentially a GTInspector in which the first pane is just one editor that has no binding for self. That’s it. Everything else works exactly as in an inspector.

Thus, if you press Cmd+o, you spawn the object to the right, like in the regular inspector. So, our example looks like:

Playground.png

This can look like a minor difference, but it can have some surprising consequences. First, you get a simple mechanism for previewing intermediary code while playing with it. Take a look at the video below:

But, more interestingly, it can play the role of a generic editor for all sorts of use cases. For example, Moose provides a dedicated editor, called easel, for developing Roassal visualizations. The new GTPlayground can offer a similar behavior without the need of a specialized editor. It simply takes advantage of the inspector being able to show a preview of the view object (and of course, the feature of remembering the last viewed presentation). This can be applied equally well to any kind of engines that offer a preview.

Playground-easel.png

The playground also remembers all snippets you are playing with (you can get a similar functionality in the default workspace). As soon as you close a playground, the code is stored and you can retrieve it from the dropdown menu from the top right. Thus, you can play without worrying for losing content.

The playground is a new take to an old problem. There are still things to improve (for example, create variables dynamically) but the difference is already significant.

Ah, before I forget: the current implementation has 68 lines of code.

Posted by Tudor Girba at 23 February 2014, 9:22 pm with tags moose, pharo, tooling, assessment link
|
<< 1 2 3 4 5 6 7 8 9 10 >>