Constructing and previewing queries over relational databases is typically done in dedicated DB client tools that are far away from the development environment. However, when working with relational data, querying is a common activity in software development, and thus it should be supported more prominently by the IDE.
Moreover, database clients are rather poor at doing anything non-textual. Thus, any time a graphical representation is needed, the developer typically resorts to exporting the data and using an external charting tool.
The GTInspector offers a simple set of extensions that brings together both of these aspects into one workflow. In this post, I demonstrate the way it works on a Postgres database, but a similar approach can be used for any other DB binding.
For the purpose of this exercise, I am using the World database, and the PostgresV2 Pharo implementation. To work through it, download the latest Moose image, load the PostgresV2 binding:
smalltalkhubUser: 'PharoExtras' project: 'PostgresV2';
(#ConfigurationOfPostgresV2 asClass project version: '2.0-baseline') load
and the GTInspector extension for PostgresV2:
smalltalkhubUser: ‘Moose’ project: ‘GToolkit’;
The first step is to create a Postgres connection.
| conn |
conn := PGConnection new.
Executing this gets you a connection object. The connection object is the entry point to starting any type of DB interaction via SQL. To make this smoother, the GTInspector offers a dedicated SQL presentation.
For example, the DB has a
city table. Let’s inspect its contents:
select * from city
Executing the query (CMD+o) spawns a result object that can be viewed as a table.
The result table is obviously not static. Selecting a row spawns a preview of all values.
Now, let’s consider a more concrete scenario. We want to get an idea of what are the largest cities in the world and to which continent the belong. For this purpose, we need a more complicated SQL statement:
city.name as city name,
country.lifeexpectancy as life,
from city left join country
Evaluating it, gets us another result.
For our problem we would benefit greatly from a visual representation. The simplest way to represent the data is a bar chart. To this end, we can use the Graph-ET engine that ships with Moose and that is integrated in the inspector:
| builder models |
models := self dataRows
sorted: [ :x :y | (x atName: #population) >
(y atName: #population) ].
builder := GETDiagramBuilder new.
modelLabels: [ :row | row atName: #cityname ];
x: [ :row | row atName: #population ];
if: [:row | (row atName: #continent) = 'Asia’]
color: Color orange lighter;
if: [:row | (row atName: #continent) = 'Europe’]
color: Color red darker;
if: [:row | (row atName: #continent) = 'North America’]
color: Color blue darker;
if: [:row | (row atName: #continent) = 'South America’]
color: Color green darker.
Executing the above code within the context of the result object gets us a view object that offers a preview:
Describing this session through screenshots does not quite convey the dynamic experience that allows you to occasionally pick through data and come back to continue scripting. Perhaps the video below tells a better story:
All in all, this session involved multiple actions: querying a DB, previewing results, exploring code to learn the API, scripting a chart, and extending the inspector from within the inspector with a dedicated presentation (shown in the video only). All these are captured through a simple and consistent user interface that offered by the most basic tool available in a Pharo-based image: the inspector. This is not cosmetics. It is an essential redefinition of the I in IDE.
And if we are at it, the implementation of the Postgres specific inspector extensions consists of 36 lines of code.
In a recent post on code reading, Peter Seibel notes how he went from seeing code as literature pieces to seeing code as specimen. He builds on this idea and proposes that developers practice the art of studying such specimen.
I concur with the idea that code is not literature. Actually, wait! Code is not even text. It only incidentally has a text shape, but that is not what defines it. Code is data.
The aforementioned post reports that developers consistently say: code reading is important and we should do it more. I am asking similar questions and I confirm that I get similar responses.
The author proposes a solution for the do it more predicament: hold code reading sessions in which samples of code are being scrutinized. This is a nice proposition and I can see its value. However, while reading code in the small can be educating, digesting code in the large is more important.
Going back to the code reading is important and we should do it more statement, I see the first part as being equally problematic. Why is it important? It is important because developers spend most of their time doing exactly that: reading code. And yet, we never talk about it. Hence, we never really learn and optimize. It's a classic elephant in the room situation.
We need is to start the conversation and learn from each other. The proposed code reading exercises have the potential of starting that conversation.
Yet, we have to be careful about how we frame the problem. If we look closer, we might notice that we do not even have a name for the activity of looking around the system to figure out what is going on. We keep saying code reading, but is where the problem is in the first place. Code reading describes only the means. The larger activity and goal remain unnamed.
Code reading is still going to be required no matter what, but it cannot remain the only option. Perhaps to get it out of our system, we should rename code reading to the-most-manual-way-to-dig-through-lots-of-data-having-a-textual-shape. We need a better name that does not tie us with a predefined solution and that sets the stage for considering other options. I call this assessment, and this site is dedicated to dissecting its various implications.
We need to have this conversation in software engineering. To my mind, it’s the most important endeavor we still have to undertake before calling software engineering a complete discipline. Building alternatives to reading code has been a research focus for a long time, but the conversation cannot remain in the academic ivory tower. It has to descend and include the engineers from the code trenches. Only when it will be relevant.
One of the concepts that Pharo inherited from its Smalltalk ancestors is the Workspace.
The workspace is a little utility that lives next to the Inspector and it serves at least two large use cases:
- It lets you execute code snippets. This is particularly convenient for various things, such as loading code from a repository using Gofer scripts, or initializing objects from the image.
- It lets you initiate an investigation by inspecting the result of executing a piece of code.
The workspace is designed as a simple standalone code editor. For the first usage, it’s enough of a concept as the only thing you need is a place to write code. But, let us focus on the second case.
Suppose you want to investigate a file reference from the working directory. You start by typing:
FileSystem disk workingDirectory
You inspect the result, and continue in the inspector. This results in two unrelated windows that you have to manage. We have followed this workflow for 30 years. It’s time to rethink it.
Let’s imagine a different situation in which you have an inspector already opened on
FileSystem disk, and you execute
self workingDirectory. Indeed, with the default inspector you end up with two unrelated windows, but with the GTInspector you get only one workflow.
Looking at these scenarios, it should follow that we should be able to integrate the workspace with the inspector. Enter (GT)Playground.
The GTPlayground (part of the GToolkit) is essentially a GTInspector in which the first pane is just one editor that has no binding for
self. That’s it. Everything else works exactly as in an inspector.
Thus, if you press Cmd+o, you spawn the object to the right, like in the regular inspector. So, our example looks like:
This can look like a minor difference, but it can have some surprising consequences. First, you get a simple mechanism for previewing intermediary code while playing with it. Take a look at the video below:
But, more interestingly, it can play the role of a generic editor for all sorts of use cases. For example, Moose provides a dedicated editor, called easel, for developing Roassal visualizations. The new GTPlayground can offer a similar behavior without the need of a specialized editor. It simply takes advantage of the inspector being able to show a preview of the view object (and of course, the feature of remembering the last viewed presentation). This can be applied equally well to any kind of engines that offer a preview.
The playground also remembers all snippets you are playing with (you can get a similar functionality in the default workspace). As soon as you close a playground, the code is stored and you can retrieve it from the dropdown menu from the top right. Thus, you can play without worrying for losing content.
The playground is a new take to an old problem. There are still things to improve (for example, create variables dynamically) but the difference is already significant.
Ah, before I forget: the current implementation has 68 lines of code.
In a recent post on the Pharo mailing list, Jannik noticed that when opening the System Browser on a single package settings, it is really slow. What could the problem be?
Let’s see. I actually do not know anything about the internal implementation of the browser, but we can start from the line that came with the original report:
SettingBrowser new setViewedPackageNames: 'Settings-Network'; open.
To investigate the problem, we use the GTInspector support for message tally.
reveals a 10s delay until opening the window. Indeed, something looks strange.
Investigating the execution further shows that the problem is related to applying filtering in
Going a step back, we notice that the filter is being applied only when
viewedPackages is not empty. We get confident that we am on the right path.
At a deeper inspection, we see that
PackageOrganizer gets involved. That rings a bell: Pharo 3.0 comes with the
RPackage support exactly to fix the problems of
Thus, one thing to do is to use
RPackage. For this we need two things:
- modify the lookup of the package, and
- modify the setting of packages.
Armed with this knowledge, we can modify:
^ self settingReceiver
ifNotNil: [self settingReceiver class package]
| allViewed |
allViewed := Set new.
aText asString substrings
do: [:sub | (RPackageOrganizer default
ifNotNil: [:pkg | allViewed add: pkg]].
self changePackageSet: allViewed.
self changed: #getViewedPackageNames
Did it solve the problem? Re-running the snippet after the modifications, gets us a 1.8s result:
Great! It’s probably not the best we could do, but it becomes reasonable. Yet, is the job ready?
Not quite. We only modified things that were related to the exercised use case. What if there are other parts in the implementation that depend on the old
PackageOrganizer and related classes? Of course, we could run all the tests, but unfortunately there are no tests for this browser. Or we could just click around, but we would still not know if we clicked enough.
Or we could search for what we know already: all references to
PackageInfo should be replaced:
(RPackageOrganizer default packageNamed: 'System-Settings')
methodReferences select: [:each |
| ast |
ast := each method parseTree.
(ast references: PackageOrganizer name)
or: [ ast references: PackageInfo name ] ]
This gets us one method:
^ self methodClass ifNotNil: [:mc | PackageOrganizer default mostSpecificPackageOfClass: mc ifNone: ]
Looking at the senders, reveals that the method is actually not used (the most relevant user is the one we just wrote):
We remove the method. And we are almost there. There is still one thing left we could do. We could search for all messages that sent from the Settings Browser and that are implemented in
PackageOrganizer but are not implemented in
(RPackageOrganizer default packageNamed: 'System-Settings')
methodReferences select: [:each |
each method parseTree allChildren anySatisfy: [ :node |
node isMessage and: [
(PackageOrganizer selectors includes: node selector) and: [
(RPackageOrganizer selectors includes: node selector) not ] ] ] ] ]
This gets zero hits. But, repeating the same query for
RPackage gets us 24 candidate methods:
However, going a step further, we see that the problematic messages are actually not RPackage specific at all:
Now we are done.
This took about half an hour of work, and at the end of it we improved the performance of a piece of code we never saw before. We did this by systematically digging through the problem and by using appropriate analyses for performance and static checks. Along the way, code reading was used sparingly and driven by a hypothesis and custom tools.
The ability to not read code all the time is invaluable for getting productive results.
Smalltalk has traditionally been unfriendly with the file system. But, that does not have to remain like that. The trick is to see files as objects. Take a look.