Compiling and error reporting

During the last few days, I have been reinstating error reporting. Previously, EclipseFP would call GHC, collect its output in a Java string, and send that string off to Cohatoe (the now defunct bridge from Java to Haskell). A Haskell module would parse the GHC output using Parsec, return the result back to Java, where EclipseFP would do fancy things with the parsed output.

If this all sounds very roundabout, you’re right. We are already interfacing with Haskell in the form of the Scion server, so why not let it do the compiling and send us (EclipseFP) the result in a formatted way? Thoughts in this direction are under way, but currently hampered by a few factors:

• Scion talks with the GHC API, which is single-threaded. This means that during background compiles, every Scion-based function in the IDE would stop working. It should be possible to fix this by letting Scion itself do background compilation, while still servicing requests that do not need to communicate with the GHC API directly, but this requires much work inside the Scion server.
• Ideally, when a Cabal file is present, Scion should let Cabal do the building. Unfortunately, Cabal does not play well as a library, because errors will terminate the program. This has been fixed in the upcoming Cabal 1.8, to be released along with GHC 6.12.1 in September.
• When a Cabal file is not present, Scion should do something along the lines of “ghc –make”. However, the make functionality is currently still interwoven with the compiler itself. A consequence is that compilation results are only available after all modules have been compiled. A callback for progress reports is in the works for GHC 6.12.1.

In the light of all these obstacles (especially the first one, which cannot be solved by installing a HEAD build of GHC) I decided to keep the current way of calling GHC as an external process. However, this meant that the output parsing had to be rewritten in Java. Moreover, because compiling a large project might take a while,  gathering up all output and processing it all at once is not the best approach. My new parser works directly on the output stream from the GHC process, so all errors will appear live in the IDE as soon as GHC spits them out.

With that in place, I sprinkled on some UI code (oh, this makes it sound so easy…) and got things working again. Like error squigglies:

You can hover over the error to get the message. Note that an error marker is also shown in the left margin, and next to the scrollbar. This last one is very convenient if you want to jump to an error in a large source file. These are the kinds of features that Eclipse gives you for free, which is the reason why I chose to do Haskell in Eclipse in the first place. Unfortunately, an error marker on the icon of the editor’s tab is more difficult, so I left that out for now.

The Project Explorer now also shows error markers on files that do not compile:

Parents of files with errors also receive this marker. This makes it easy to locate problems in a large project.

Errors are, as usual, also reported in a special Problems view:

It would be better if only the first line were shown, and could be expanded to show the additional information, but for a first attempt this will do.

If parsing of GHC’s output might fail for some reason, an error will be reported in the Error Log (which contains Eclipse errors, not Haskell errors), and you can always see the raw output in the Console:

Very nice.

There is, however, one small problem currently. This is caused by a mismatch between GHC’s notion of building, and Eclipse’s. When building, EclipseFP will visit all files in the project, then invoke “ghc –make” on them. It suffices to call “ghc –make” only on the file containing the main function, and GHC will take care of the rest. But not only is this approach wasteful: it will also lead to error markers appearing and disappearing as the same file gets compiled (as a dependency) multiple times. My plan for solving this is to add a project option to specify the main module and the name of the main function, so that we can call GHC only on that module. Much more efficient.

Little rant on modularity

And now, a little rant. Eclipse is, first and foremost, a Java IDE. It tries very hard to be extensible and flexible, so the Java functionality itself, like everything else, is a plug-in, or rather a set of plug-ins called the JDT (Java Development Tools). Very good.

But unfortunately, there is a lot of code in the JDT that does quite generic things, like showing the error markers in the Project Explorer, or on the editor tabs. I often find myself digging through the JDT sources to figure out how the big guys do things, only to find out that they implemented quite common functionality in the JDT that would also benefit many other language-specific plug-ins. Other plug-in writers who want to use this functionality have a choice: either have their plug-in depend on the Java Development Tools, or reproduce all the code in their own plug-in.

Depending on the JDT plug-ins would be a shame, because we’re trying to build a Haskell IDE here. Even though most Eclipse users will have the JDT already installed, we do not want to have to package it with an eventual “Hasklipse” (I hope that name doesn’t stick…) distribution containing an Eclipse platform for Haskell development. Moreover, it is not even always possible to use the classes from the JDT directly: often, they subtly depend on some Java-specific treat.

The alternative, copying and pasting to your own plug-in, is therefore often the best option. It is not pretty, but it works, and it allows you to customize the code to fit your own plug-in better. But I sure wish it wasn’t necessary. For a poor Haskell plug-in developer, even more modularity would be very welcome.

Introducing the new client-server protocol

It may seem that progress on this project has been quite slow during the last few weeks. Partly, this is because progress has been quite slow during the last few weeks. Writing a lot of self-contained code is easy, but as soon as I have to interface with the Eclipse API, each line of code can easily take an hour of poking around in documentation. Another reason for the lack of visible progress is that I’ve spent a few days refactoring and reorganizing the preference pages for EclipseFP. However, these changes aren’t ready yet and are currently parked away in a branch. But I decided not to let work “under the hood” keep me from posting updates!

Nominolo (my mentor, and the author of Scion) and I decided that it was no good to have the Scion server speak different protocols for each type of client that it serves. There were already separate Emacs and vim protocols, each with a different set of accessible commands, and adding and maintaining additional protocols would quickly grow unwieldy.

We therefore settled on one, and only one, protocol that is easy to parse from nearly every language: JSON. As simple as possible, but not any simpler. I threw away my hacked Lisp parser and implemented the JSON-based protocol in EclipseFP.

There are several options for JSON parsing from Java. Two feasible ones referenced from the JSON homepage are org.json (or whatever its name is) and json-simple. org.json uses specialized objects to represent the different JSON types, whereas json-simple uses standard Java classes. Although this sounds simple and elegant indeed, I abandoned json-simple because it does not employ generics, and therefore still requires many typecasts. In fact, neither of the two libraries are very nice and type-safe (because of JSON’s dynamic nature, they cannot be!), but org.json provides many utility methods that save typing out explicit casts.

Apart from simplicity and well-definedness, JSON has another advantage over the previous protocol: it is now possible to test the Scion server through telnet.

Cosmetics

Since I was feeling creative today, I decided I would work on some cosmetic issues in EclipseFP that have been bothering me for a while. They make the whole thing look unprofessional and out of tune with the rest of the Eclipse environment. Luckily, Eclipse has a User Interface Guidelines document for me to follow.

I started with some icons. The Guidelines specify precisely how they should look. The icons used in the Package Explorer are now in the same style as their Java counterparts:

Note how the Haskell source files have a nice lambda icon, and the literate source file also has its own icon. The Haskell project itself has a subtle lambda overlay. (I see now that I missed the Libraries icon. Also, the text “[GHC]” should be dimmed. Made a note of these.)

Editors, too, get the new icons:

And finally, the one that I’m most proud of. I figured it would be nice to have the new official “lambda-and-bind” Haskell logo somewhere. And what better place for this than the icon of the Haskell Perspective? So I brushed up my pixel painting skills and drew a variant of the Haskell logo that blends in with the Eclipse environment:

It’s always there, subtle, but visible, recognisable and fresh. I like it.

Public code release

These instructions are outdated. See the new instructions!

My code is now in a sufficiently clean state that I can push it out to the world. For anyone who wants to give it a try, here are the instructions that should get you up and running. This assumes that you have GHC, cabal-install, darcs, and git. I tested only on Ubuntu 9.4, but there is no reason why it should not work on Windows as well.

1. Get my branch of the Scion sources:
   git clone git://github.com/ttencate/scion.git
   (It should also work with nominolo’s original sources.)
2. Build and install Scion, the library as well as the server program:
   cd lib
cabal install
cd ../server
cabal install
3. Get Eclipse Galileo (3.5 RC4 or later), the distribution named “Eclipse for RCP/Plug-in Developers”. Extraction equals installation.
4. Get my branch of the EclipseFP sources:
   darcs get http://code.haskell.org/~ttencate/eclipsefp/
darcs get http://code.haskell.org/~ttencate/cohatoe/
   (Although Cohatoe is probably going to be removed, this version of EclipseFP still depends on it. Build the Cohatoe server if you like; it segfaults on my system.)
5. Import all Eclipse projects from these repositories into Eclipse. There are over forty, so the Multi Project Import/Export plugin might come in handy.
6. Wait for the build to finish, then… Run!
7. In the new Eclipse window that (hopefully) pops up, set the location of the Scion server via Window, Preferences. If you installed it locally, it should be in ~/.cabal/bin/scion_server. (Eclipse does not understand the tilde, so give the full path to your home directory.)

It’s quite some work to get it all up and running, and I probably forgot a step or two. Making the process easier is not currently a priority, but I envision a complete Eclipse release that is easy to set up and has everything you need to get Haskelling quickly.

If you try these steps, and run into any problems, please let me know, so that I can help you out, and improve the instructions and maybe even the code at the same time.

“Open Definition” and code improvements

Work on this project has been slow until a few days ago, but now I’m back in business. I’m getting the hang of this Eclipse API thing, which makes everything go much smoother. Today I present to you: a more robust Scion client, a rudimentary “Open Definition” feature, and a configurable path to the Scion server program.

The improved client code for the Scion server works with a command queue. It spawns one thread that is tightly coupled to the server process. This thread waits for a command to arrive in the queue, then sends it off to the server and waits for the response. If the server crashes for some reason, it will be restarted and the command will be retried on the new incarnation. The original sender of the command (for example, a GUI element) specifies a timeout: if the command is not executed successfully within this time, the client gives up and the original sender is left to deal with the problem in whatever way it sees fit.

Then, for the mandatory screenshots, I built a feature similar to the “Open Declaration (F3)” feature that you might know from Java development in Eclipse. You can select an identifier and hit F3 (or go through the context menu or main menu), and you will jump to the place where the identifier is defined:

This feature is very rudimentary: it only works if the definition is in the current file; you have to select the identifier; and the Ctrl+click that Java users may know does not work yet.

I also added a Preferences page to configure the location of the Scion executable. Autodetection is in the works, but is not there yet. Automatically installing Scion via Cabal upon request would also be a very nice feature to have.

This preference page may be a trivial feature, but very important: now that the path to my home directory is no longer hard-coded, other people might actually be able to try my code too! Feeling adventurous? See the next post!

Client/server communication

Because of other obligations (mainly my Master’s thesis) I had a late start with this Summer of Code, but now it has really begun. Below are the first results, but first some background information.

To make Eclipse understand your Haskell source, a lot of parsing and processing code is needed. Fortunately, this code has already been written, in the form of the GHC compiler. Unfortunately, this code is written in Haskell, whereas Eclipse is written in Java. Part of the purpose of Scion is to overcome this problem. On one side, it communicates with the GHC API; on the other side, it communicates with whatever program is interested. This used to mean Emacs only, but now it also includes Eclipse!

There are essentially two ways to make Scion communicate with a non-Haskell program. One is through an intermediary C program. Haskell can call into C using FFI, and Java can call into C using JNI, so this is technically possible. However, it would be very painful and cumbersome.

The alternative is to communicate through some kind of intermediary format that is sent over a stream (say, a pipe or a socket). This is how Scion’s current Emacs frontend is implemented. When needed, Emacs launches a separate process (scion_server) that listens on a local TCP socket. Emacs connects to the socket and sends a command; the server responds in a similar, Lisp-like format. (In fact, the returned value is sent straight into the Lisp interpreter.)

To do a similar thing with Eclipse, I could either make the server speak another language that is easily parsed from Java (XML-RPC comes to mind), or make Eclipse parse the Lisp code. To get quick results, my mentor nominolo and I chose the latter route, so I hacked together some code to parse a small number of specific responses. The full Lisp parser will have to wait a day or two.

Add some code to start the server, connect to its socket, send and receive commands, and we’re all set. The screenshot below shows one of the many things that are now within reach: automatic type inference, in a tooltip, straight from Eclipse!

The current code is still full of TODO comments, lousy error handling and hard-coded configuration, but it works. At the moment it is necessary to save the file before the type information gets updated, but I hope that it will be possible to send changes to the server incrementally.

Over the next few days, I will replace my quick hacks by something better, improve the error handling, and make the whole thing more robust. And I should also really, really do something about the hard-coded absolute path that points to the Scion server in my home directory…