Writing Pillar books with the GTInspector

Documentation is regarded as an important activity in the Pharo community. Besides commenting code, a significant documentation effort is being invested around books. For example, one of the ongoing such project is the Pharo for the Enterprise.

The book is written in Pillar, a wiki-like syntax extracted by Damien Cassou from the Pier content management system. The Pillar project offers transformers into several target formats such as LaTeX, html or markdown.

It’s quite a nice package. The only problem is that it offers no tool support for editing book contents, and as a consequence, people prefer using external editors that offer a minimal source highlighting. That is a no go for an environment that touts itself as being live.

I teamed up with Andrei and Jan to change this.

We needed several things to get this to work. First, we needed a text widget that can handle custom syntax highlighting. To this end, Andrei extended the Rubric text editor (authored by Alain Plantec) and its integration in Glamour to make it more suitable for the task.

Second, we needed the logic of producing the syntax highlighting. For this purpose we needed a parser. Pillar does come with its own hand-written parser, but the parser was only conceived for generating other formats and it does not come with any support for mapping parsed constructs to the original source. This is where the work on island parsing of Jan comes in. With an island parser we can match subsets of the input without needing to understand the whole input. This is particularly valuable when building a syntax highlighter. It’s a bit like regular expressions, only more powerful.

The end result looks like this:


We get as you type highlighting for headers, links, embedded files, notes, code etc directly in the GTInspector for each file that has a pier or pillar extension. Adding new constructs is as easy as extending the parser. For example, the embedding rule is specified as:

     ^ '+' asParser , file token , '+' asParser ==> [:res |
          GTPillarFileHighlightingCommand new
               token: res second]

Granted, the parser is not fast. In fact it is more than an order of magnitude slower than the hand-made parser that comes with Pillar. It would ideal for the Pillar parser to keep track of tokens as well, but until then, the island parser is cheap and it works reasonably given that the highlighting happens in the background.

Perhaps this does not look like much, but this is the first time when we can have a cheap way to manage custom highlighter in Pharo. Yet, the highlighting was actually not the end goal of this exercise. Let’s put it in perspective for a second. Highlighting is a necessary step to bring the effort of writing Pillar-based documentation inside Pharo, but once this is achieved, the interesting things happen when we start to take advantage of the live environment.

For example, the syntax highlighting of Pharo code is done by simply delegating to the code highlighter (i.e., Shout). This is easily possible exactly because we are inside Pharo.

Other opportunities open up as well. For example, a Pillar book is made of multiple files (one per chapter) placed in subfolders of the root book folder. These must then be linked from a pillar.conf file that is placed in the root folder. Given that the logic is entirely in Pharo, when we inspect the root directory of a Pillar book, the inspector can offer us a Pillar specific index. For example, the picture below shows the folder of The Moose Book that I started to rewrite using Pillar. Selecting a file from the index previews the highlighted content to the right.


Essentially, we get a cheap book editor. And when we are ready with editing, we can simply trigger the generation of the output directory from the inspector (see the play button on the top right).

One trouble when handling such books written in a language that is not visual is to manage the links to external pictures. To show the potential of a tight integration, the current solution comes with a simple tool support that provides a link for every embedded file (in our example, we have pictures/glamour-presentations-details.png). If the file cannot be found on the harddisk, the text becomes red. If the file is found on the harddisk, the text becomes a magenta link. Clicking on the link spawns the file object to the right offering a quick preview possibility.


To get this behavior, load the following code in a Moose 5.0 image, and inspect the directory of a Pillar book.

Gofer new
     smalltalkhubUser: 'Pier' project: 'Pillar';
Gofer new
     smalltalkhubUser: 'JanKurs' project: 'PetitParser';
     configurationOf: #PetitParserIslands;
#ConfigurationOfPetitParserIslands asClass loadDevelopment.
Gofer new
     smalltalkhubUser: 'Moose' project: 'GToolkit';
     package: 'GT-InspectorExtensions-Pillar';

The current solution is an exercise that opens up new possibilities. An easy application is that of enhancing the rendering of class comments. For example, when loading the aforementioned packages, you also get the ability of rendering class comments with Pillar syntax.


But, we can envisage bolder directions as well. One clear direction is to go towards more live book writing. For example, if we stay in the image, we can also manage all code snippets in real methods instead of having them copied as text in a book. Pillar could simply provide a way to embed a method like +method:Object>>#printString+ and this would embed the method code while still allowing the writer to preview the code. Yet another thing is to move towards eliminating static pictures. For example, most pictures in a technical book are some sort of screenshots. These screenshots could easily be produced with snippets that could be generated. Thus, the book would only include a way to embed the result of a piece of code like +object:aScriptThatGeneratesAGraphicalForm+.

Yet another direction is to bring the books inside the image and link them to the code they talk about. Imagine that every packages comes with a dedicated chapter that offers scenarios of usages, and each scenario links explicitly to methods. In this case, when browsing the methods in question, you can get relevant scenarios to read about.

I am certain there are many other ideas that will flourish by bringing liveliness to documentation. We just have to be bold. I invite you to join us.

Posted by Tudor Girba at 7 September 2014, 3:27 pm with tags tools, moose, pharo, analysis link