Blog

Communicating Pharo 5.0

Pharo 5.0 is out. This was a great ride.

As it has become customary, I worked on capturing one aspect of the work around Pharo 5.0 in a visualization. Choosing only one prominent aspect was not easy this time. For example, the VM just got 30% faster. Just like that. And there is a new moldable debugger. And the reflection mechanisms just got a significant boost to make code instrumentation ridiculously inexpensive. And there are more. All these make for amazing stories, and we would have been happy with only one of these.

And that is exactly the most interesting story to tell. There used to be a time when one person could keep track of what is going on in the community. Not anymore. We have reached a tipping point at which there are just too many things that happen simultaneously. We have seen an amazing surge in the energy invested around Pharo, and this has been keeping up for some time now.

So, how should we capture this activity? Looking closer, we noticed that the attribution list included 100 distinct developers that have contributed code that entered in Pharo 5.0. Indeed, not all have contributed in equal measure, and some actually produced code a while ago. Still, when so many brains come together to build a common system, amazing things can happen.

100 contributors is a lot, especially when it happens in an open-source distributed environment, and when their effort is about modifying the core of an ecosystem. Enabling all these people to work in concert is a goal of Pharo. That is why Pharo is more than code. Pharo is a project of building an evolving community that reinvents software development.

To expose this effort I built the visualization that accompanied the official release announcement.

Pharo50-contributors.png

The picture depicts in red all authors of new code using automatically rendered glyphs based on a technique called VisualID. This is a new feature available in Roassal2 and was mainly created by Ignacio Fernandez and Alexandre Bergel. The rendering algorithm requires a way to retrieve the similarity of the rendered entities, and in our case, the similarity is determined by the prefixes of the packages that the different authors have touched. The authors are connected to the classes that they have touched using red edges, and the classes are connected to their packages using gray edges. The overall graph is laid out using a force-based layout.

The main goal of the visualization was to capture the activity of the authors. We can spot different clusters of people both due to location proximity and due to the similarity of glyphs. But, the most amazing thing to observe is that only one author is far apart from the rest (at the top), while all others are linked in one way or another. This is an amazing showcase of collaboration.

As is always the case, building this visualization was not a linear process. The overall effort spanned several days, and dozens of variations. Here are some.

Pharo50-contributors-tries.png

Building each of these did not take long. What took longer was to trim the data and find a balance that captured both the activity and the amount of work that made Pharo 5.0 happen. Building pictures is not difficult. However, building meaningful pictures requires multiple iterations and this is where a live rich environment plays a critical role. I leave it up to you to decide if the end result was worth that effort.

The code that produces the final visualization can be found at http://ws.stfx.eu/J2NFUA9TVOH7, and it also includes the loading of the Roassal visualization engine, the analysis of methods and the extraction and trimming of authors. You can simply paste it in a Spotter opened in a plain Pharo 5.0 image to reproduce the Playground as seen below.

Pharo50-contributors-playground.png

Posted by Tudor Girba at 16 May 2016, 9:00 am link
|

Pharo 5.0

Dear World,

The time has come for Pharo 5.0!

Pharo50-screenshot.png

Pharo 5.0 is our most significant release yet. Here are some highlights:

  • The PharoVM is now based on Spur, the new memory management, and it brings with it a 35% speedup!
  • A new unified foreign function interface (UFFI) replaced NativeBoost to provide a strong Spur-compatible framework for interfacing with the outside world.
  • The Glamorous Toolkit now includes the GTDebugger to offer a moldable infrastructure that allows the developer to customise the debugger deeply.
  • The underlying Reflectivity mechanism has reached maturity with multiple pieces coming together to empower developers to instrument their own systems. For example, we now have breakpoints implemented as just a simple extension of this mechanism.
  • QualityAssistant is now part of the image to provide live feedback during development.

These are just the more prominent highlights, but the details are just as important. We have closed 2446 issues in Pharo 5.0. Take a moment to go through a more detailed recount of the progress.

While the technical improvements are significant, just think of getting 30% faster out-of-the-box, still the most impressive fact is that the new code that got in the main Pharo 5.0 image was contributed by 100 people. Together we have touched 43% of the classes, and 20% of the methods. The following visualisation rendered with Roassal in Pharo 5.0 is dedicated to this effort. The picture shows the touched classes and packages in gray, the authors and the links to the changed classes in red, and, using an automatically generated visual id, you can spot authors that have worked on similar projects.

Pharo50-contributors.png

Pharo is more than code. It is an exciting project involving energetic people. We thank all the contributors of this release:

Abdelghani Alidra, Clara Allende, David Allouche, Nicolas Anquetil, Thibault Arloing, Jean Baptiste Arnaud, Mangesh Bendre, Clement Bera, Alexandre Bergel, Torsten Bergmann, Usman Bhatti, Vincent Blondeau, Johan Brichau, Camillo Bruni, Miguel Campusano, Damien Cassou, Nicolas Cellier, Danny Chan, Andrei Chis, Christopher Coat, Ben Coman, Bernardo Contreras, Gabriel Omar Cotelli, Tommaso Dal Sasso, Paul De Bruicker, Sean De Nigris, Christophe Demarey, Simon Denier, Marcus Denker, Martin Dias, John Dougan, Stephane Ducasse, Stephan Eggermont, Johan Fabry, Sergio Fedi, Cyril Ferlicot, Holger Hans Peter Freyther, Joshua Gargus, Tudor Girba, Thierry Goubier, Kris Gybels, Norbert Hartl, Thomas Heniart, Dale Henrichs, Nicolai Hess, Alejandro Infante, Henrik Johansen, Goran Krampe, Pavel Krivanek, Juraj Kubelka, Denis Kudriashov, Matthieu Lacaton, Laurent Laffont, Kevin Lanvin, Jannik Laval, Alexander Lazarević, Skip Lentz, Max Leske, Dave Lewis, Esteban Lorenzano, Sheridan Mahoney, Mariano Martinez Peck, Max Mattone, John McIntosh, Rene Meusel, Eliot Miranda, Henrik Nergaard, Marion Noirbent, Merwan Ouddane, Nick Papoulias, Nicolas Passerini, Alain Plantec, Guillermo Polito, Damien Pollet, Baptiste Quide, Andreas Raab (RIP), Alain Rastoul, Stefan Reichhart, Lukas Renggli, Mark Rizun, Michael Rueger, Valentin Ryckewaert, Ronie Salgado, Udo Schneider, Boris Spasojevic, Igor Stasenko, Roger Stebler, Serge Stinckwich, Aliaksei Syrel, Camille Teruel, Pablo Tesone, Yuriy Tymchuk, Peter Uhnak, Masashi Umezawa, Dion Stewart, Sven Van Caekenberghe, Jan Van De Sandt, Benjamin Van Ryseghem, Toon Verwaest, Franck Warlouzet.

(If you contributed with Pharo 5.0 development in any way and we missed your name, please send us a mail and we will add you).

Enjoy!

The Pharo Team

Posted by Tudor Girba at 13 May 2016, 6:39 pm link
|

Hunting leftover weak announcements with GTInspector

Yesterday, I paired with Andrei and joined the hunt for leftover weak announcements that tend to accumulate in Pharo images since recently. This is a problem because it leads to an overall slowdown of the system. There were already several valuable points made in the long thread and there was even a fix readily available provided by Max Leske.

The main issue seemed to be that the weak announcements were not garbage collected because of a VM problem even if their subscribers were nil. However, there also seemed to be indications that the code related to the Glamorous Toolkit might have been responsible for this problem, so we wanted to understand the issue to see if there was anything we could do to help, especially given that the very problems in the inspector seemed to turn the inspector not so useful for identifying the problem. This turned out to be an exciting 30 minutes assessment session. Let me get you through it.

In a Pharo 50662, we started by inspecting:

SystemAnnouncer uniqueInstance

The inspector readily showed us the 2602 subscriptions. From the start we saw that there were many for which the subscribers are nil, a point also raised by Peter:

Weak-1.png

Another thing we noticed is that there seemed to be a repetition of announcements that appeared with nil subscribers. To check this, we executed:

grouped := (SystemAnnouncer uniqueInstance
    subscriptions glmSubscriptions select: [ :each |
    each subscriber isNil ])
        groupedBy: [:each | each announcementClass name].

Indeed, there were only 4 announcements that were affected by the problem:

Weak-3.png

More interestingly, we noticed that we had an equal amount of subscriptions for each announcement type (i.e, 640). This suggested that these registrations came from the same place. To browse this possibility, we looked for all methods that reference all these classes:

grouped keys allButFirst
    inject: (SystemNavigation new
             allReferencesTo: grouped keys first asClass binding)
    into: [ :result :each |
        result intersection: (SystemNavigation new
                              allReferencesTo: each asClass binding) ].

We got 12 such methods:

Weak-5.png

We browsed a bit, and we saw that some of these methods registered other system announcements as well, such as ClassCommented, but those did not appear in our problem list:

Weak-6.png

Thus, this route did not seem to be the best way to understand the source of the problem. We turned a bit around and took another route. As, each subscription provides us the selector that is to be sent when the announcement gets triggered, we queried the system to give us those selectors:

groupedSelectors := grouped associations collect: [ :assoc |
    assoc value collect: [:each | each action selector ] as: Set].

In this case, too, we got for each distinct announcement exactly one selector.

Weak-7.png

So, we looked at all methods that send all of these selectors:

selectors := groupedSelectors flatCollect: #value.
selectors allButFirst
    inject: (selectors first senders)
    into: [ :result :each |
        result intersection: each senders ].

And we got exactly one method: PragmaCollector>>installSystemNotifications.

Weak-8.png

Essentially, this means that the responsible for creating the global subscriptions that should have been garbage collected was the PragmaCollector. Certainly, the problem was not related to the PragmaCollector, but we now knew that no code in the Glamorous Toolkit prevented these subscriptions to become garbage collected.

We manually applied the patch to finalize and eliminate these subscriptions:

(SystemAnnouncer uniqueInstance subscriptions glmSubscriptions copy select: [ :sub |
    sub class = WeakAnnouncementSubscription and: [
        sub subscriber isNil ]])
            do: #finalize.
Smalltalk garbageCollect.

We were not yet quite done because executing:

WeakAnnouncementSubscription allInstances select: [:sub | sub subscriber isNil]

told us that we have more than 2000 instances left to deal with. The first question we asked was where did all these come from. There needed to be some global variables that were keeping those around but we did not know which one given that SystemAnnouncer uniqueInstance was cleaned up. We applied the same type of analysis.

announcements := (WeakAnnouncementSubscription allInstances select: [ :each |
     each subscriber isNil ])
          asOrderedCollection groupedBy: [:each | each announcementClass ].

Again, we found that all of the announcements are global announcers about the state of the system, and that each of them appeared in equal quantities.

Weak-10.png

Like in the previous case, we again looked at the place where all these announcements are sent from:

announcements keys allButFirst
    inject: (SystemNavigation new
            allReferencesTo: announcements keys first binding)
   into: [ :result :each |
      result intersection: (SystemNavigation new
                            allReferencesTo: each binding) ].

We found 3 cases. The first two were using SystemAnnouncer uniqueInstance weak, so these could have not been the root. In the third one, we discovered a new announcer: SystemAnnouncer uniqueInstance private.

Weak-11.png

In the end, this reported problem was not related to GT. There were other problems related to the management of announcements, but those were other cases.

This session showed again how the quick connection between the runtime and code can dramatically speedup the amount and the depth of the questions we can ask our system. And this is critical for modern software engineering.

Posted by Tudor Girba at 29 March 2016, 7:50 am with tags story, assessment, gt link
|

Speaking at ArchConf San Diego, 2016

I will again be present at ArchConf. The spring edition is organized in San Diego during April 4-7. I will start with a 1 day tutorial on Steering agile architecture, and in the last day I will follow with four 90-minute talks on:

Posted by Tudor Girba at 23 March 2016, 11:59 am link
|

Solving real problems without reading code on the NFJS podcast

When I was at ArchConf 2015 in Fort Lauderdale, I had a conversation with Michael Carducci around the issue of how to solve real problems without reading code. As usual, I argued that code reading is the single most expensive activity in software development and that we ought to pay more attention to it.

This conversation appeared recently on the No Fluff Just Stuff podcast. I had a funny hoarse voice because I had a cold and the recording took place late in the evening after having given four long talks during that day.

Posted by Tudor Girba at 21 March 2016, 10:10 am link
|
<< 1 2 3 4 5 6 7 8 9 10 >>