In a previous article we looked at how moldable tools can change the development experience when tracking down a bug. The bug in question was a duplicated behaviour in GTDebugger: triggering an action from the context menu of the stack triggered that action twice.
A few weeks after that bug was fixed, another interesting bug was reported for GTInspector, affecting the navigation feature of the inspector: when an object is selected in a view, the pane to the right is created four times, as if the user selected that object four times.
In this article we detail the workflow that we followed to understand and fix this bug. This workflow consists in treating software problems as data problems, and formulating and solving one hypothesis at a time. Hence, we do not try to gain knowledge just by reading code. Instead we use tools to query and visualize our data, the code. When an appropriate tools is not available we build it. We then use the gained insight to formulate the next hypothesis and iterate until we understand the cause of the bug, and are able to fix it. As much as possible, we make all hypotheses explicit.
Reproducing the bug
The bug in question only appers when the user selects an object within certain inspector views. For example, we reproduced the bug by creating and inspecting a SharedQueue object, and selecting any value in the 'Items' view. To get a visual indication of the bug, as the SharedQueue that we created contains integers, we added an #inform: message to the method Integer>>gtInspectorIntegerIn: creating the 'Integer' view. This way we can see that when selecting an object in the 'Items' view, the method is called four times:
However, the bug did not appear when selecting an object in the 'Raw' view of a SharedQueue object:
By looking at the code of SharedQueue>>#gtInspectorItemsIn:, we observe that it uses a Fast Table presentation. Given that the bug that we previously investigated was also related to Fast Table, we like to check first whether this bug is also related to Fast Table. One way to verify this hypothesis is to create an identical presentation for SharedQueue objects that does not use Fast Table to display a list, but the previous list renderer based on Pluggable Tree Morph. We can view the code of SharedQueue>>#gtInspectorItemsIn: and create the new view directly in the inspector:
When redoing the previous scenario using the newly created view, we can indeed confirm that the bug is not present; Integer>>gtInspectorIntegerIn: is called only once. Hence, this bug is again related to Fast Table.
Comparing a buggy and a correct scenario
Given that we have a correct and a buggy scenario, before going any further, we would like to quickly check if there are any differences between an execution using Fast Table and an execution using Pluggable Tree Morph. We hypothesis that the problem could be caused by the Glamour renderer for Fast Table not following the same logic as Pluggable Tree Morph. Glamour, is the browsing engine on top of which GTInspector is implemented. Glamour provides two distinct renderers for displaying lists, one for Fast Table and one for Pluggable Tree Morph. If we do not see a divergence in the call stacks of the two renderers, we can focus our investigation on the Fast Table renderer.
We could approach this task by opening two debuggers and then manually scrolling through the stacks to find a difference. However, a less manual and error-prone approach consists in creating a custom view that shows the two call stacks using a tree, highlighting the points where the stacks diverge.
To build this tool we first need to log the two stack traces. For that we can rely on the Beacon logging engine and replace the #inform: call from the method Integer>>gtInspectorIntegerIn: with a logging statement that records the stack of method calls (MethodStackSignal emit).
Integer>>gtInspectorIntegerIn: composite
<gtInspectorPresentationOrder: 30>
MethodStackSignal emit.
^ composite table
title: 'Integer';
display: [ | associations | "..." ]
"..."We then start the recording using MemoryLogger start, and trigger the two scenarios by selecting an item in the views 'Items (simple)' and 'Items'. Next, we stop the recorder and inspect the collected stacks using the GTInspector:
MemoryLogger instance recordings collect: #stack
We observe that we got five stacks. Four correspond to the buggy scenario, and one to the correct scenario. We also notice that there is a difference in the number of stack frames between the two scenarios.
Nevertheless, if we briefly look at one correct and one buggy stack trace, we notice that there are a lot of frames that are not related to Glamour, caused by how the two graphical widgets (FTTableMorph used by the Fast Table renderer and PaginatedMorphTreeMorph used by the Pluggable Tree Morph renderer) handle the selection of an element. To verify our hypothesis, we are only interested in those stack frames related to Glamour.
To build the custom view, we switch to the 'Raw' view of the inspector and use a Roassal script. First, we select a correct and a buggy stack trace and then remove all stack frames that are not related to Glamour. We rely on the fact that all Glamour classes have the prefix 'GLM'. Then, we add the filtered stack frames in one set, draw edges between consecutive entries, and arrange the graph in a tree. We also attach an index to methods for ensuring that multiple occurrences of the same method in the stack will have different entries in our view.
| view stacks |
view := RTMondrian new.
stacks := ({self first. self second}) collect: [ :aStack |
aStack select: [ :frame | frame methodClass name beginsWith: 'GLM' ] ].
stacks := stacks collect: [ :aStack |
aStack withIndexCollect: [ :aFrame :index |
index -> aFrame method ] ].
view shape label text: [:each | each value gtDisplayString truncate: 50 ] .
view nodes: (stacks flatCollectAsSet: #yourself).
stacks do: [ :aStack |
aStack overlappingPairsDo: [ :a :b |
view edges
connectFrom: [:x | b ]
to: [:x | a ] ] ].
view layout tree.
view.
Executing the script in place shows us the view in a new pane to the right directly in the inspector:
We immediately notice that most of the two stacks are identical, with some small differences at the top. We zoom in and slightly rearrange the top contexts to better understand what causes this difference:
In this case, we discover that the difference appears because the Glamour renderer for Fast Table and the Glamour renderer for Pluggable Tree Morph have a different way of propagating the selection. However, in the end, both renderers call the correct method GLMPresentation>>#selection:. Hence, the problem is most likely related to Fast Table. We use this newly gained insight to steer the focus of our investigation.
Comparing the four incorrect stack traces
Our next hypothesis is that in the four buggy stack traces the execution branches in a certain method because of a loop. Verifying this hypothesis requires a tree visualization that shows execution branches caused by the same method being sent to different objects. In the previous view, we only took into account methods. To build this new view, we also need to log the actual objects from the execution stack. We can do this be replacing the MethodStackSignal logger with ContextStackSignal logger in the method Integer>>gtInspectorIntegerIn::
Integer>>gtInspectorIntegerIn: composite
<gtInspectorPresentationOrder: 30>
ContextStackSignal emit.
^ composite table
title: 'Integer';
display: [ | associations | "..." ]
"..."
We then clear and start the MemoryLogger, and select an item in the buggy `Items` view. Inspecting the log shows the four stack traces:
We create next the tree vizualization using Roassal and execute it in place. We follow the same steps as for the previous visualization, this time without doing any filtering of the stack frames:
In the resulting view we can see that we have four stack traces because the execution branches in two places: an initial branch point that occurs only once, and a second one that occurs two times.
Next, we can zoom in and investigate each stack frame corresponding to a branch point in detail:
In each case we notice that the inspected announcer has some duplicated subscriptions. Having some knowledge of the Glamour renderer, we suspect that this should not be the case; each subscription should be registered only once. We use this insight to continue the investigation.
Reducing the scope
Now that we detected a possible cause for the bug, before going any further, we can devise a simpler example exhibiting the same behaviour:
GLMCompositePresentation new
with: [ :c |
c fastList
send: [ :anInteger | self inform: '#send:'. anInteger ] ];
openOn: (1 to: 42)After executing the code above and selecting an element in the list, we get the same four notifications as before.
Finding the cause of the duplication
To verify if those announcements need to be registered twice, we need to find the places where the registration happens. Searching for the methods that reference the class FTSelectionChanged (one of the duplicated announcements) we discover that it is called from the method GLMMorphicFTRenderer>>initializeAnnouncementForDataSource. The fact that the subscription is registered twice indicates that the method is called twice. We proceed as before, and instead of putting a breakpoint in the method, we add a Beacon logging statement, and only record events while we open the buggy browser.
GLMMorphicFTRenderer>>initializeAnnouncementForDataSource
ContextStackSignal emit.
"..."
RecordingBeacon new
runDuring: [
GLMCompositePresentation new
with: [ :c |
c fastList
send: [ :anInteger | self inform: '#send:'. anInteger ] ];
openOn: (1 to: 42) ]. We see that indeed there are two calls to #initializeAnnouncementForDataSource. As the log event recorded the entire stack, we can select and explore the two contexts making the call to the method #initializeAnnouncementForDataSource that registers the FTSelectionChanged announcement:
We observe that the first call is made from the method GLMMorphicFastListRenderer>>render:, and the second from the method GLMMorphicFastListRenderer>>dataSourceUpdated:. To understand the relation between these two methods within the execution, we again construct a tree with the two call stacks. We already have a visualization for this task, as we built it in a previous step.
We immediately see that the two stacks diverge in the method GLMMorphicFastListRenderer>>render:. This is actually the method that makes the first call to #initializeAnnouncementForDataSource. We can now navigate through the other call stack to understand why it was made. We can basically use the GTInspector as a port-mortem debugger!
Before moving forward, we need to clarify a relevant aspect regarding the design of the Glamour renderer for Fast Table. GLMMorphicFastListRenderer is the Fast Table renderer. The renderer creates a graphical widget (FTTableMorph), a data source and links them together. The widget displays the elements visually and the data source provides the elements that will be displayed.
We see in the previous view that the second call to #initializeAnnouncementForDataSource happens when the graphical widget FTTableMorph is initialized. The reason is that whenever the data source is changed within a FTTableMorph widget, the method #initializeAnnouncementForDataSource is called to link the graphical widget with the new data source. We can see this in the method GLMMorphicFastListRenderer>>#dataSourceUpdated: that is called whenever the data source is changed in the widget, as a result of the GLMDataSourceUpdated announcement (GLMMorphicFastListRenderer>>readyToBeDisplayed):
dataSourceUpdated: announcement
tableModel ifNotNil: [ self unsubscribeDataSource: tableModel ].
tableModel := announcement newDataSource.
self initializeAnnouncementForDataSource
If we look for methods referencing GLMDataSourceUpdated, we discover that the link between the announcement GLMDataSourceUpdated and the method #dataSourceUpdated: is created when the renderer (GLMMorphicFastListRenderer) is initialized:
initializeAnnoucementForPresentation: aPresentation
aPresentation when: GLMDataSourceUpdated send: #dataSourceUpdated: to: self.
aPresentation when: GLMContextChanged send: #actOnContextChanged: to: self.
aPresentation when: GLMPresentationUpdated send: #actOnUpdatedPresentation: to: self
At this point we gained a good understanding of the factors causing the bug. To summarize them: when a Fast Table view is created, the method GLMMorphicFastListRenderer>>render: instantiates a new data source and calls #initializeTableMorph. #initializeTableMorph creates a graphical widget and sets its data source. After the graphical widget is initialized GLMMorphicFastListRenderer>>render: calls #initializeAnnouncementForDataSource to properly set the announcements between the graphical widget and the data source. However, this method was already executed when the data source was set in a FTTableMorph widget in #dataSourceUpdated:. Hence, we can fix the bug by removing the explicit call to #initializeAnnouncementForDataSource from GLMMorphicFastListRenderer>>render:.
Documenting our finding
Now that we found and fixed the bug we can document our finding. To ensure that this bug will not happen in the future, the best solution is to rely on a test. We create a test that verifies that there are no duplicated subscriptions in an announcer. We then apply this test on the two announcers from Glamour that have duplicated subscriptions. This way, if at any point in the future a duplicated subscription is introduced, we will be notified and can check if the duplication makes sense or if it is a bug.
testNoDuplicateRegistrationsInFastTableRenderer
| table |
window := GLMCompositePresentation new
with: [ :c |
c fastList ];
openOn: (1 to: 42).
table := self find: FTTableMorph in: window.
self assertNoDuplicatedAnnoucementsIn: table announcer.
self assertNoDuplicatedAnnoucementsIn: table dataSource announcer. Building a toolset
Solving this bug was done through building custom tools. In this particular case, the tool consisted in a view for displaying stack traces using a tree. Initially, when we solved the previous bug related to Fast Table, we also built such a view, but then we did not know if we will even reuse that view (tool) so we threw it away. Now, we had to reuse the view again and make a few adaptations: filter stack frames using a condition and create edges based only on method calls. Hence, we can now spend 15 minutes more and transform this view into a tool that we can then reuse in the future, whenever we need to compare stack traces.
We transform this view into a tool by putting in into a dedicated class and adding an API for configuring it:
BeaconRTStackViews>>#executionTreeForContextSignals: aCollectionOfSignals
| stacks |
stacks := ((aCollectionOfSignals
select: [ :each | each isKindOf: ContextStackSignal ])
collect: #stack).
^ self executionTreeForContexts: stacks
select: [ :each | true ]
transform: [ :each | each ]BeaconRTStackViews>>#executionTreeForContexts: aCollectionOfStacks select: aFilterBlock transform: aTransformBlock
| view stacks |
stacks := aCollectionOfStacks collect: [ :aStack | aStack select: aFilterBlock ].
stacks := stacks collect: [ :aStack |
aStack withIndexCollect: [ :aFrame :index | aTransformBlock cull: aFrame cull: index ] ].
view := RTMondrian new.
view shape label text: [:each | each value gtDisplayString truncate: 50 ] .
view nodes: (stacks flatCollectAsSet: #yourself).
stacks do: [ :aStack |
aStack overlappingPairsDo: [ :a :b |
view edges
connectFrom: [:x | b ]
to: [:x | a ] ] ].
view layout tree.
^ viewWe are still not done. We invested effort into building this tool, however, if other developers are not aware that this tool exists they will not use it. Hence, we should invest a few more minutes to address this.
This view is applicable when we inspect a collection of Beacon events of type ContextStackSignal. We extend the GTInspector with a custom action for collections that is only applicable if the collection contains at least one Beacon event of that type. We further implement the action so that the view is opened in a new pane to the right, preserving thus the workflow in the same inspector window.
SequenceableCollection>>#gtInspectorActionExecutionTree
<gtInspectorAction>
^ GLMGenericAction new
title: 'View contexts execution tree';
category: 'Beacon';
action: [ :aPresentation |
aPresentation selection:
(BeaconRTStackViews new executionTreeForContextSignals: self) ];
condition: [ self anySatisfy: [ :each |
each isKindOf: ContextStackSignal ] ]Now, whenever somebody inspects a collection containing stack traces recorded with Beacon, she will be able to discover and open this view directly from the inspector:
Remarks
We started this session with the goal of understanding the cause of a bug and end it by adding a custom tool to our environment. We achieved this as we were able to rapidly create the necessary tool and easily incorporate it into the IDE. We could do this as Pharo is a moldable IDE where creating a new custom extension is just as easy as creating a test.
We say that Glamorous Toolkit (GT) is a moldable development environment. There were a couple of emails on the Pharo mailing list in the recent period that questioned the usefulness of moldability. More specifically, the questions were raised in relation to the newly introduced GTDebugger. The concept of a moldable debugger is new to any IDE, and for this reason we cannot compare with the standard IDE behavior.
Nevertheless, the question is certainly legitimate. Given that we cannot compare with other solutions, we can observe the impact of other moldable interfaces. If we take a step back, the same concept was applied to the GTInspector and GTSpotter before. We introduced these in Pharo 4.0. Let’s see what is the impact.
In Pharo 3, the EyeInspector offered a basic extension possibility, and the Pharo image shipped with 8 such extensions. Together with the introduction of the GTInspector, we shipped 138 extensions. One year later we have 165 in the core image. In the meantime there are many more extensions in external packages. For example, the Moose 6.0 image which is based on Pharo 5.0 ships with about 230 extensions.
Also, in Pharo 4, we shipped 92 extensions to GTSpotter. In Pharo 5, there are 122 extensions. Similarly, there are several more extensions in external packages. For example, in Moose 6.0 we have 135 and if we include the generic way of searching through models we have several hundred more (more about this in a future post).
The explosion of extensions shows that there is a need to have such extensions. This is a validation of a hypothesis put forward by the humane assessment approach a long time ago which started from the observation that context is key in software development, and as such, tools should take this context into account. This idea was first explored in the context of Moose and it stands at the very core of GT. You can see this embodied now in 3 distinct tools, and we will see more as we proceed with the project.
Now, why the difference between Pharo 3 and Pharo 4? First, the cost associated with an inspector extension went down from ~19 lines of code in a separate class to ~9 lines of code in one single method. Second, the value of the extension increased because of the interaction workflow that came with the GTInspector design.
Interestingly, after we introduced the GTInspector, almost all discussions were geared towards the Raw presentation because it introduced a new kind of interaction. Almost no email was about the extension mechanism. The same pattern happened with GTSpotter where messages focused almost exclusively on searching classes and methods. And rightly so, as the default behavior is what people see at first. We have exactly the same type of issues with the GTDebugger.
Now, in the default Pharo image, there are 3 different debuggers: the default one, the bytecode one and the SUnit one. In the Moose image there are 6 debuggers and there are a couple in outside packages. For example, here are two screenshots of two such debuggers: one of a PetitParser debugger, and one of a debugger debugging the update of the debugger.
The cost of these debuggers is measured in hundreds of lines of code. We will certainly not see as many debugger extensions as in the case of the inspector because the granularity is larger and because the cost is larger, but we will certainly see more custom debuggers.
We still need to learn more about how to reach the balance between extensibility and usability. We are at the beginning, but there is clear value in extensibility and we should not discard it as unimportant. The key here is the ability of creating extensions with low effort and this is unprecedented. Let’s put this in perspective: Eclipse started more than a decade ago with a plugin architecture. Right now, the Eclipse marketplace (marketplace.eclipse.org) has 1722 tools. Granted these are more extensions than we have and they are larger, but at the same time the community that builds those is several orders of magnitude larger than the Pharo one. Yet, we can already compete with this because of the radically low cost structures.
Until now we looked at quantitative plain data. Nevertheless, are these extensions actually affecting productivity qualitatively? In my experience they can have a significant impact, and this site and blog features multiple examples of how this is so. Furthermore, more recently I realized that my workflow has changed quite significantly and the amount of time I spend in the inspector is around 60%.
But, let me give you another perspective. I went around the world over the past year and I asked directly more than 1000 developers working in various languages if they agree that they read code for 50% or more of their time. The vast majority agrees (this is on top of research showing the same thing). Yet, when I ask them if they talk about it to find new ways of understanding systems, they acknowledge that they almost never do. This basically means that people are spending half of their budget on something they never talk about. These are just the direct costs, and many systems see some 80% of their overall effort spent in maintenance. Understanding systems is the single most expensive activity, but the industry does not approach this explicitly. And typical IDEs focus to a large extent on the active part of creating code. For example, is it not ironic how in all IDEs the reading happens in an editor? In Pharo, we look at this problem in a novel way and we have the chance of affecting business costs radically.
Moldability is a competitive advantage. This is why we would like to encourage people to play with these mechanisms and push the envelope of software engineering.
Dear World,
Pharo 4.0 is here!
Pharo is a pure object-oriented programming language and a powerful environment, focused on simplicity and immediate feedback.
Many things have changed in Pharo. Here are some highlights:
- Inspector/Playground/Spotter are new moldable development tools for inspecting, coding and searching objects.
- Slots model instance variables as first class entities and enable meta-programming on this level.
- ShoreLine reporter introduces a way to report system errors and collect statistics, that we will use for future improvements
- Dark theme.
These are just the more prominent highlights, but the details are just as important. We have closed 1700 issues in Pharo 4. Take a moment to go through a more detailed recount of the progress: ChangeLogs.
Pharo is improving on many fronts, but one of the most prominent changes is the addition of moldable tools for inspection and search. These tools provide extension mechanisms that allow every object to define ways in which it can be understood effectively. To provide an idea of the impact of the already existing extensions, the map below shows the Pharo classes grouped in packages, highlighting in red those parts of the system that have at least one such custom view coming with the main distribution. The spread of these extensions shows that moldability is powerful mechanism that can be used in many contexts.
Remember that Pharo is your platform. We thank all the contributors of this release:
Clara Allende, Jean-Baptiste Arnaud, Jean-Christophe Bach, Philippe Back, Clement Bera, Alexandre Bergel, Torsten Bergmann, Vincent Blondeau, Noury Bouraqadi, Santiago Bragagnolo, Johan Brichau, Sven Van Caekenberghe, Damien Cassou, Nicolas Cellier, Guido Chari, Dimitris Chloupis, Andrei Chis, Ben Coman, Bernardo Contreras, Tommaso Dal Sasso, Jan Van De Sandt, Christophe Demarey, Sean DeNigris, Marcus Denker, Martin Dias, Stephane Ducasse, Stephan Eggermont, Luc Fabresse, Johan Fabry, Hilaire Fernandes, Jerome Garcia, Tudor Girba, Thierry Goubier, Jigyasa Grover, Kris Gybels, Norbert Hartl, Dale Henrichs, Pablo Herrero, Nicolai Hess, Pavel Krivanek, Juraj Kubelka, Jan Kurs, Laurent Laffont, Jannik Laval, Kevin Lanvin, Max Leske, David Lewis, Diego Lont, Esteban Lorenzano, Tim Mackinnon, Attila Magyar, Esteban Maringolo, Stefan Marr, Max Mattone, Martin Mc Clure, Eliot Miranda, Alain Plantec, Guillermo Polito, Damien Pollet, Stefan Reichhart, Mark Rizun, Udo Schneider, Ignacio Sniechowski, Henrik Sperre Johansen, Igor Stasenko, Aliaksei Syrel, Ciprian Teodorov, Camille Teruel, Sebastian Tleye, Yuriy Tymchuk, Peter Uhnak, Andres Valloud, Sven Van Caekenberghe, Thomas Vincent, Jan Vrany, Martin Walk, Richard Wettel, Dmitri Zagidulin
And all those who contributed indirectly, by reporting bugs, participating in discussion threads and providing feedback.
Pharo 4.0 is another big step. And, the best is yet to come.
Enjoy!
The Pharo Team
We, Alex Syrel, Andrei and I, are actively working on the Glamorous Toolkit.
First, together with the integration in Pharo 4, we have received a significant amount of feedback that we took into account. Second, we continued building new functionality.
Improvements
The contextual menu now contains multiple actions explicitly (including cut/copy/paste).
Cmd+o changed to become Cmd+g, and the action is now called Go. The reason for the change is that in Pharo, Cmd+o is already mapped on global composite shortcuts for opening various tools. There is still an ongoing discussion to change it again to simply be triggered via the Inspect when inside the Playground or Inspector.
The default object inspection presentation changed its title from State to Raw. It seems that State got people confused, and we found that Raw better denotes that the presentation is showing the raw object structure.
Close-able inspector panes
A feature that people asked for is the ability of closing panes in the inspector. Now, the Pager user interface offers the possibility of closing the last pane. To make it feel comfortable, we also added an animation of sliding the rest of the panes to the right when one closes. For this feature to happen, Glamour now supports one presentation specific action that is rendered in the title bar.
One question that might arise is why make this action available only on the last pane? First, we do not want to allow closing arbitrary panes. This would be a reasonable request if the panes would not be causally connected. But, the inspector keeps track of the exact inspection session and removing arbitrary steps would break this contract. Second, we could have the action of closing a pane from the middle of a session to mean removing all panes to the right. This might conceptually work, but it would be a surprising effect for a newcomer. As a consequence, we offer the possibility of closing the only last pane.
Shrinking tab labels
One concern about the design of the inspector is that the tabs rendering does not scale if we have too many presentations. To address this problem, Alex Syrel worked along two directions. On the one hand, tab labels shrink when there is not enough space.
On the other hand, shrunk tab labels expand on mouse hover so that you can still read the contents.
Of course, the same mechanism also works on the object title. There are still things to improve. For example, when a label is even larger than the available tab space, you will not see the complete. A solution here would be to have the printout be rendered as a popup right on top of the tab. Nevertheless, the current solution is already pretty workable.
Instant caching
Remembering code changes when the text changes. In the previous version, the current code was remembered only when pressing Cmd+g. However, the real goal is to not lose anything that is typed, not just executed. So, now, the playground remembers the code on every text change.
To support this feature, Glamour now offers the possibility of hooking a custom action when a port changes. A simple example looks like this:
GLMCompositePresentation new
with: [ :composite |
composite rubricText
onChangeOfPort: #text act: [ :textPresentation |
Transcript cr; cr; show: textPresentation text ] ];
openOn: 'Type something and check the Transcript'On disk playground caching
Still related to remembering code, up to now, the playground remembered the code in memory. This was nice, but as soon as you moved to another image, the code was not available anymore. Furthermore, if the image crashes (it typically does not, but once in a while it does) or you forget to save, you again loose the contents in the playground.
To solve the problem, the playground now saves/loads the content to/from disk. By default it works with a folder named ''play-cache'', but one can choose a custom one via a setting.
Playground remote sharing
A not so seldom use case is that of prototyping a code snippet and then sending it around via pastebin or a similar service. Best would be to service this use case directly in the Playground. *Sven created a shared workspace>http://ws.stfx.eu* a while ago, and we shamelessly replicated the client side behavior in the Playground. Triggering the ''save to cloud'' icon publishes the contents of the workspace and then conveniently copies the url in the clipboard.
These are a couple of new features that we worked on recently. More will come soon. Stay tuned. Actually, don’t stay. Better use the tools and give us feedback.
Documentation is regarded as an important activity in the Pharo community. Besides commenting code, a significant documentation effort is being invested around books. For example, one of the ongoing such project is the Pharo for the Enterprise.
The book is written in Pillar, a wiki-like syntax extracted by Damien Cassou from the Pier content management system. The Pillar project offers transformers into several target formats such as LaTeX, html or markdown.
It’s quite a nice package. The only problem is that it offers no tool support for editing book contents, and as a consequence, people prefer using external editors that offer a minimal source highlighting. That is a no go for an environment that touts itself as being live.
I teamed up with Andrei and Jan to change this.
We needed several things to get this to work. First, we needed a text widget that can handle custom syntax highlighting. To this end, Andrei extended the Rubric text editor (authored by Alain Plantec) and its integration in Glamour to make it more suitable for the task.
Second, we needed the logic of producing the syntax highlighting. For this purpose we needed a parser. Pillar does come with its own hand-written parser, but the parser was only conceived for generating other formats and it does not come with any support for mapping parsed constructs to the original source. This is where the work on island parsing of Jan comes in. With an island parser we can match subsets of the input without needing to understand the whole input. This is particularly valuable when building a syntax highlighter. It’s a bit like regular expressions, only more powerful.
The end result looks like this:
We get as you type highlighting for headers, links, embedded files, notes, code etc directly in the GTInspector for each file that has a pier or pillar extension. Adding new constructs is as easy as extending the parser. For example, the embedding rule is specified as:
GTPillarParser>>embed
^ '+' asParser , file token , '+' asParser ==> [:res |
GTPillarFileHighlightingCommand new
token: res second]Granted, the parser is not fast. In fact it is more than an order of magnitude slower than the hand-made parser that comes with Pillar. It would ideal for the Pillar parser to keep track of tokens as well, but until then, the island parser is cheap and it works reasonably given that the highlighting happens in the background.
Perhaps this does not look like much, but this is the first time when we can have a cheap way to manage custom highlighter in Pharo. Yet, the highlighting was actually not the end goal of this exercise. Let’s put it in perspective for a second. Highlighting is a necessary step to bring the effort of writing Pillar-based documentation inside Pharo, but once this is achieved, the interesting things happen when we start to take advantage of the live environment.
For example, the syntax highlighting of Pharo code is done by simply delegating to the code highlighter (i.e., Shout). This is easily possible exactly because we are inside Pharo.
Other opportunities open up as well. For example, a Pillar book is made of multiple files (one per chapter) placed in subfolders of the root book folder. These must then be linked from a pillar.conf file that is placed in the root folder. Given that the logic is entirely in Pharo, when we inspect the root directory of a Pillar book, the inspector can offer us a Pillar specific index. For example, the picture below shows the folder of The Moose Book that I started to rewrite using Pillar. Selecting a file from the index previews the highlighted content to the right.
Essentially, we get a cheap book editor. And when we are ready with editing, we can simply trigger the generation of the output directory from the inspector (see the play button on the top right).
One trouble when handling such books written in a language that is not visual is to manage the links to external pictures. To show the potential of a tight integration, the current solution comes with a simple tool support that provides a link for every embedded file (in our example, we have pictures/glamour-presentations-details.png). If the file cannot be found on the harddisk, the text becomes red. If the file is found on the harddisk, the text becomes a magenta link. Clicking on the link spawns the file object to the right offering a quick preview possibility.
To get this behavior, load the following code in a Moose 5.0 image, and inspect the directory of a Pillar book.
Gofer new
smalltalkhubUser: 'Pier' project: 'Pillar';
configuration;
loadStable.
Gofer new
smalltalkhubUser: 'JanKurs' project: 'PetitParser';
configurationOf: #PetitParserIslands;
load.
#ConfigurationOfPetitParserIslands asClass loadDevelopment.
Gofer new
smalltalkhubUser: 'Moose' project: 'GToolkit';
package: 'GT-InspectorExtensions-Pillar';
load.The current solution is an exercise that opens up new possibilities. An easy application is that of enhancing the rendering of class comments. For example, when loading the aforementioned packages, you also get the ability of rendering class comments with Pillar syntax.
But, we can envisage bolder directions as well. One clear direction is to go towards more live book writing. For example, if we stay in the image, we can also manage all code snippets in real methods instead of having them copied as text in a book. Pillar could simply provide a way to embed a method like +method:Object>>#printString+ and this would embed the method code while still allowing the writer to preview the code. Yet another thing is to move towards eliminating static pictures. For example, most pictures in a technical book are some sort of screenshots. These screenshots could easily be produced with snippets that could be generated. Thus, the book would only include a way to embed the result of a piece of code like +object:aScriptThatGeneratesAGraphicalForm+.
Yet another direction is to bring the books inside the image and link them to the code they talk about. Imagine that every packages comes with a dedicated chapter that offers scenarios of usages, and each scenario links explicitly to methods. In this case, when browsing the methods in question, you can get relevant scenarios to read about.
I am certain there are many other ideas that will flourish by bringing liveliness to documentation. We just have to be bold. I invite you to join us.
