Endgame

It seems that the build targets feature I previously wrote about is somewhat more work than I expected, mainly in terms of integration with the Eclipse GUI. With my university classes starting soon, and some work that I’ve been pushing ahead of me throughout GSoC, I am unable to finish this in time.

Therefore, I’ve parked the build target feature in a branch and decided to officially put a stop to this project for the time being. I have updated the EclipseFP web site to reflect the work I did. I also rolled a release that you can pull in using the Eclipse update mechanism. The details are provided on the web site.

I cannot say that I’m fully happy with the results that I achieved. I had hoped to get more mileage out of the integration with Scion, but adding all these features through the Eclipse API turned out to be hairier than I expected. However, I do feel I have made useful contributions on which future work can build.

I would like to thank my mentor, Thomas Schilling a.k.a. nominolo, for his guidance and his work on Scion. I would also like to thank Leif Frenzel, without whom EclipseFP wouldn’t have existed in the first place. And finally, thanks to the folks at Google who made GSoC a reality.

For now, I am forced to other business that I’ve been neglecting for far too long. Does that mean that EclipseFP is going to disappear from the radar again? I think not. There is sufficient interest in Haskell IDEs to keep this project going, and somebody will come along and keep improving upon it. Such is the nature of open source. And that somebody… might actually be me.

Update: it isn’t me after all. Less than a week after I wrote this, JP Moresmau chimed in on the mailing list, reporting some bugs, offering to fix them, then continue from there. He’s now taken over development. If you want to try EclipseFP, I recommend his latest version from here!

Build targets

My previous post was three weeks ago. However, I’ve been occupied with other business for a week, and been down with the flu for another; so, in a way, this is still a “weekly update.” I’m actually surprised at how much I got done during this period.

Google Summer of Code is officially over, but my project isn’t. My original project proposal said that I would be adding “Cabal support” to EclipseFP during the second half of the project. Soon after I submitted the proposal, it turned out that I’d been looking at an ancient version of the code base, and that Cabal support was already very much there.

So, essentially, I worked on non-Cabal support instead. That is, being able to compile non-trivial, multi-module programs without the overhead of Cabal. For beginning Haskell programmers, having to deal with just the Haskell source file is less intimidating. It also makes it easier to quickly hack up a program without having to worry about Cabal (maybe adding a Cabal file later). For more advanced functionality, however, the use of Cabal will still be required; it is not my intention to completely replace it.

To get this done, I introduced the notion of build targets. A build target is either an executable file or a library, built from the source code. In the case of an executable, it has a main function. In the case of a library, it has a list of the modules to be included in the library.

The nice thing about build targets is that they provide a subset of Cabal’s functionality. So, if a project does have a Cabal file, the very same user interface can still be used, but instead of writing to the EclipseFP .hsproject file, it writes the build targets as executable/library stanzas to the Cabal file. (Although, currently, automated Cabal file manipulations are broken. They used to be done through Cohatoe, the Java-Haskell bridge, which was removed in favour of Scion.)

Internal representation of the build targets is in place, and they can be stored in the .hsproject file. I am currently working on the GUI for adding, removing and modifying build targets. Even after months of use, the Eclipse API is still daunting, and I keep needing new parts of it, so this progresses slowly. The next step will be to make the builder actually take them into account, but this should be a fairly simple task.

When all this is in place and tested, I will see whether the plug-in can be distributed through the Eclipse update mechanism. Maybe I’ll even roll a Haskell-specific Eclipse distribution that includes the EclipseFP plug-in and its dependencies, to make getting started easier than ever.

The Green Bar

Today I have nothing exciting to report. Still, things happened behind the scenes.

The code used to give 156 compilation warnings, mostly about the long-deprecated Eclipse preferences API. I reworked all preferences-related code to use the new API instead. Now only 9 warnings remain. All of these are quite valid indications of possible problems, but mostly of dead code, so I left these for the time being.

Also, some refactorings in the way the GHC compiler was being called caused the JUnit unit tests to fail. I had so far ignored the existence of these tests, but no longer. I fiddled around with the tests and the code until the bar was green again. Unfortunately, it was mostly spinning my wheels: no real bugs were uncovered by the tests.

These kinds of things are not the most exciting parts of software development. But they are necessary.

Although, looking back through the commit logs, I can also report one small new feature. It is now possible to create a new project in an existing source directory. This makes it much easier to import an existing Haskell source tree into EclipseFP. It’s not perfect yet (for example, it simply assumes that the source is in a directory named src), but I’ll improve it in the upcoming days.

More robust Scion client code

New post, new theme! With this wider theme, the screenshots will no longer fall off the edge.

Although I did not plan to “build one to throw away”, this largely became reality nonetheless. By now I’ve done so many refactorings on the Scion client code that hardly a single line of the original remains. But I’m finally happy with the result.

The Scion client previously used its own command queue, which ensured that commands were sent to the server one after another. After some investigation, I replaced that by Eclipse’s job scheduling mechanism. With proper scheduling rules, we can keep the scheduler from running two commands simultaneously, so it is as powerful as the home-brew command queue. If one command depends on the result of another, we can make the second job fall asleep and have the first wake the second up when it’s done. And it has several advantages over the old system:

• We can now set priorities on commands, so that interactive commands (such as computing the text of a tool tip) take precedence over longer-running commands (such as builds).
• Commands will automatically be shown in the progress report windows.
• The sender of a command can either run it asynchronously, or wait for the command to complete before continuing.
• Of course, it’s simply less code to maintain and have bugs in!

Secondly, I introduced a Scion “manager” class that ensures that there is always one Scion server running per project. If the server crashes, it is restarted automatically and brought back to the same state as its previous incarnation.

Thirdly, all significant errors are now logged to the Error Log window in Eclipse. There should no longer be a need to turn on error tracing to troubleshoot problems in the Scion client/server system.

Finally, if the server fails to start in the first place, this is probably because its executable program could not be found. Most likely this is because the plugin is used for the first time. Therefore, we pop up a helpful dialog:

Little things like this go a long way, making it easier to get up to speed with EclipseFP. In the same vein, I will start working on a “new project from existing source” feature that is well known from Java development in Eclipse. (EclipseFP is already able to import Cabal packages from a .tar.gz, but strangely not from a directory on disk!)

New build instructions

Because of many problems I experienced with Darcs, I converted the EclipseFP repository to Git. I really like the clean and simple model and UI of Darcs, and I’m sorry to see it go, but it was simply too slow and too unreliable. Also, I removed all dependencies on Cohatoe. Without further ado, here are the new and improved instructions in case anyone wants to test-drive my current work.

As usual, this assumes that you have GHC, cabal-install, and git. I tested only on Ubuntu 9.4, but there is no reason why it shouldn’t work on other Unix platforms. There may still be some issues on Windows.

1. Get Eclipse Galileo (3.5), the distribution named “Eclipse for RCP/Plug-in Developers”. Extraction equals installation.
2. Get the Scion source:
   git clone git://github.com/ttencate/scion.git
   (My master branch corresponds to nominolo’s multi_proto_server branch.)
3. Build and install Scion, the library as well as the server program:
   cd scion/lib
cabal install
cd ../server
cabal install
4. Get the EclipseFP source:
   git clone git://github.com/ttencate/eclipsefp.git
5. Import all Eclipse projects from these repositories into Eclipse. There are 26, so the Multi Project Import/Export plugin might come in handy.
6. Wait for the build to fail. Yes, that’s right. It will complain about missing files, because ANTLR generated some that Eclipse doesn’t yet know about. Select all projects, right-click and click Refresh, then wait for the build to succeed. (Thanks to Björn Peemöller for pointing this out!)
7. Hit Run, Run Configurations. Add a new launch “Eclipse Application” launch configuration. The default settings for this launch configuration should work, so just click Run. (Previous instructions were wrong; thanks to Axeman6 for pointing this out!)
8. In the new Eclipse window that (hopefully) pops up, set the location of the Scion server via Window, Preferences, Haskell, Scion. If you installed it in a fairly standard location, the Autodetect button should do the trick; otherwise, just Browse.
9. If something doesn’t work, turn on tracing. In the Run Configurations dialog, on the Tracing tab, you can enable tracing options for various plug-ins. These will output to the Console at the bottom of the host Eclipse window. To see the traffic between the Scion client and server, turn on the logs for net.sf.eclipsefp.haskell.scion.client.

A known issue is that you have to save a file before it gets loaded into Scion, so you don’t get type tooltips and go-to-definition until then. This will be resolved in the near future, maybe tomorrow. After nine consecutive days of work, I’m taking a day off now.

Again, please let me know if you tried this. Also let me know if something didn’t work, so I can improve it!

Compiling and error reporting

During the last few days, I have been reinstating error reporting. Previously, EclipseFP would call GHC, collect its output in a Java string, and send that string off to Cohatoe (the now defunct bridge from Java to Haskell). A Haskell module would parse the GHC output using Parsec, return the result back to Java, where EclipseFP would do fancy things with the parsed output.

If this all sounds very roundabout, you’re right. We are already interfacing with Haskell in the form of the Scion server, so why not let it do the compiling and send us (EclipseFP) the result in a formatted way? Thoughts in this direction are under way, but currently hampered by a few factors:

• Scion talks with the GHC API, which is single-threaded. This means that during background compiles, every Scion-based function in the IDE would stop working. It should be possible to fix this by letting Scion itself do background compilation, while still servicing requests that do not need to communicate with the GHC API directly, but this requires much work inside the Scion server.
• Ideally, when a Cabal file is present, Scion should let Cabal do the building. Unfortunately, Cabal does not play well as a library, because errors will terminate the program. This has been fixed in the upcoming Cabal 1.8, to be released along with GHC 6.12.1 in September.
• When a Cabal file is not present, Scion should do something along the lines of “ghc –make”. However, the make functionality is currently still interwoven with the compiler itself. A consequence is that compilation results are only available after all modules have been compiled. A callback for progress reports is in the works for GHC 6.12.1.

In the light of all these obstacles (especially the first one, which cannot be solved by installing a HEAD build of GHC) I decided to keep the current way of calling GHC as an external process. However, this meant that the output parsing had to be rewritten in Java. Moreover, because compiling a large project might take a while,  gathering up all output and processing it all at once is not the best approach. My new parser works directly on the output stream from the GHC process, so all errors will appear live in the IDE as soon as GHC spits them out.

With that in place, I sprinkled on some UI code (oh, this makes it sound so easy…) and got things working again. Like error squigglies:

You can hover over the error to get the message. Note that an error marker is also shown in the left margin, and next to the scrollbar. This last one is very convenient if you want to jump to an error in a large source file. These are the kinds of features that Eclipse gives you for free, which is the reason why I chose to do Haskell in Eclipse in the first place. Unfortunately, an error marker on the icon of the editor’s tab is more difficult, so I left that out for now.

The Project Explorer now also shows error markers on files that do not compile:

Parents of files with errors also receive this marker. This makes it easy to locate problems in a large project.

Errors are, as usual, also reported in a special Problems view:

It would be better if only the first line were shown, and could be expanded to show the additional information, but for a first attempt this will do.

If parsing of GHC’s output might fail for some reason, an error will be reported in the Error Log (which contains Eclipse errors, not Haskell errors), and you can always see the raw output in the Console:

Very nice.

There is, however, one small problem currently. This is caused by a mismatch between GHC’s notion of building, and Eclipse’s. When building, EclipseFP will visit all files in the project, then invoke “ghc –make” on them. It suffices to call “ghc –make” only on the file containing the main function, and GHC will take care of the rest. But not only is this approach wasteful: it will also lead to error markers appearing and disappearing as the same file gets compiled (as a dependency) multiple times. My plan for solving this is to add a project option to specify the main module and the name of the main function, so that we can call GHC only on that module. Much more efficient.

Little rant on modularity

And now, a little rant. Eclipse is, first and foremost, a Java IDE. It tries very hard to be extensible and flexible, so the Java functionality itself, like everything else, is a plug-in, or rather a set of plug-ins called the JDT (Java Development Tools). Very good.

But unfortunately, there is a lot of code in the JDT that does quite generic things, like showing the error markers in the Project Explorer, or on the editor tabs. I often find myself digging through the JDT sources to figure out how the big guys do things, only to find out that they implemented quite common functionality in the JDT that would also benefit many other language-specific plug-ins. Other plug-in writers who want to use this functionality have a choice: either have their plug-in depend on the Java Development Tools, or reproduce all the code in their own plug-in.

Depending on the JDT plug-ins would be a shame, because we’re trying to build a Haskell IDE here. Even though most Eclipse users will have the JDT already installed, we do not want to have to package it with an eventual “Hasklipse” (I hope that name doesn’t stick…) distribution containing an Eclipse platform for Haskell development. Moreover, it is not even always possible to use the classes from the JDT directly: often, they subtly depend on some Java-specific treat.

The alternative, copying and pasting to your own plug-in, is therefore often the best option. It is not pretty, but it works, and it allows you to customize the code to fit your own plug-in better. But I sure wish it wasn’t necessary. For a poor Haskell plug-in developer, even more modularity would be very welcome.

Introducing the new client-server protocol

It may seem that progress on this project has been quite slow during the last few weeks. Partly, this is because progress has been quite slow during the last few weeks. Writing a lot of self-contained code is easy, but as soon as I have to interface with the Eclipse API, each line of code can easily take an hour of poking around in documentation. Another reason for the lack of visible progress is that I’ve spent a few days refactoring and reorganizing the preference pages for EclipseFP. However, these changes aren’t ready yet and are currently parked away in a branch. But I decided not to let work “under the hood” keep me from posting updates!

Nominolo (my mentor, and the author of Scion) and I decided that it was no good to have the Scion server speak different protocols for each type of client that it serves. There were already separate Emacs and vim protocols, each with a different set of accessible commands, and adding and maintaining additional protocols would quickly grow unwieldy.

We therefore settled on one, and only one, protocol that is easy to parse from nearly every language: JSON. As simple as possible, but not any simpler. I threw away my hacked Lisp parser and implemented the JSON-based protocol in EclipseFP.

There are several options for JSON parsing from Java. Two feasible ones referenced from the JSON homepage are org.json (or whatever its name is) and json-simple. org.json uses specialized objects to represent the different JSON types, whereas json-simple uses standard Java classes. Although this sounds simple and elegant indeed, I abandoned json-simple because it does not employ generics, and therefore still requires many typecasts. In fact, neither of the two libraries are very nice and type-safe (because of JSON’s dynamic nature, they cannot be!), but org.json provides many utility methods that save typing out explicit casts.

Apart from simplicity and well-definedness, JSON has another advantage over the previous protocol: it is now possible to test the Scion server through telnet.

Cosmetics

Since I was feeling creative today, I decided I would work on some cosmetic issues in EclipseFP that have been bothering me for a while. They make the whole thing look unprofessional and out of tune with the rest of the Eclipse environment. Luckily, Eclipse has a User Interface Guidelines document for me to follow.

I started with some icons. The Guidelines specify precisely how they should look. The icons used in the Package Explorer are now in the same style as their Java counterparts:

Note how the Haskell source files have a nice lambda icon, and the literate source file also has its own icon. The Haskell project itself has a subtle lambda overlay. (I see now that I missed the Libraries icon. Also, the text “[GHC]” should be dimmed. Made a note of these.)

Editors, too, get the new icons:

And finally, the one that I’m most proud of. I figured it would be nice to have the new official “lambda-and-bind” Haskell logo somewhere. And what better place for this than the icon of the Haskell Perspective? So I brushed up my pixel painting skills and drew a variant of the Haskell logo that blends in with the Eclipse environment:

It’s always there, subtle, but visible, recognisable and fresh. I like it.

Public code release

These instructions are outdated. See the new instructions!

My code is now in a sufficiently clean state that I can push it out to the world. For anyone who wants to give it a try, here are the instructions that should get you up and running. This assumes that you have GHC, cabal-install, darcs, and git. I tested only on Ubuntu 9.4, but there is no reason why it should not work on Windows as well.

1. Get my branch of the Scion sources:
   git clone git://github.com/ttencate/scion.git
   (It should also work with nominolo’s original sources.)
2. Build and install Scion, the library as well as the server program:
   cd lib
cabal install
cd ../server
cabal install
3. Get Eclipse Galileo (3.5 RC4 or later), the distribution named “Eclipse for RCP/Plug-in Developers”. Extraction equals installation.
4. Get my branch of the EclipseFP sources:
   darcs get http://code.haskell.org/~ttencate/eclipsefp/
darcs get http://code.haskell.org/~ttencate/cohatoe/
   (Although Cohatoe is probably going to be removed, this version of EclipseFP still depends on it. Build the Cohatoe server if you like; it segfaults on my system.)
5. Import all Eclipse projects from these repositories into Eclipse. There are over forty, so the Multi Project Import/Export plugin might come in handy.
6. Wait for the build to finish, then… Run!
7. In the new Eclipse window that (hopefully) pops up, set the location of the Scion server via Window, Preferences. If you installed it locally, it should be in ~/.cabal/bin/scion_server. (Eclipse does not understand the tilde, so give the full path to your home directory.)

It’s quite some work to get it all up and running, and I probably forgot a step or two. Making the process easier is not currently a priority, but I envision a complete Eclipse release that is easy to set up and has everything you need to get Haskelling quickly.

If you try these steps, and run into any problems, please let me know, so that I can help you out, and improve the instructions and maybe even the code at the same time.