Luca's meaningless thoughts  

Translation of e-mails using Mutt

by Leandro Lucarella on 2012- 09- 24 09:45 (updated on 2012- 10- 02 09:58)
tagged e-mail, en, floss, gmail, google translate, mutt, python, release, script, software, translate - with 0 comment(s)


New translation script here, see the bottom of the post for a description of the changes.

I don't like to trust my important data to big companies like Google. That's why even when I have a GMail, I don't use it as my main account. I'm also a little old fashion for some things, and I like to use Mutt to check my e-mail.

But GMail have a very useful feature, at least it became very useful since I moved to a country which language I don't understand very well yet, that's not available in Mutt: translation.

But that's the good thing about free software and console programs, they are usually easy to hack to get whatever you're missing, so that's what I did.

The immediate solution in my mind was: download some program that uses Google Translate to translate stuff, and pipe messages through it using a macro. Simple, right? No. At least I couldn't find any script to do the translation, because Google Translate API is now paid.

So I tried to look for alternatives, first for some translation program that worked locally, but at least in Ubuntu's repositories I couldn't find anything. Then for online services alternatives, but nothing particularly useful either. So I finally found a guy that, doing some Firebuging, found how to use the free Google translate service. Using that example, I put together a 100 SLOC nice general Python script that you can use to translate stuff, piping them through it. Here is a trivial demonstration of the script (gt, short for Google Translate... Brilliant!):

$ echo hola mundo | gt
hello world
$ echo hallo Welt | gt --to fr
Bonjour tout le monde

And here is the output of gt --help to get a better impression on the script's capabilities:

usage: gt [-h] [--from LANG] [--to LANG] [--input-file FILE]
          [--output-file FILE] [--input-encoding ENC] [--output-encoding ENC]

Translate text using Google Translate.

optional arguments:
  -h, --help            show this help message and exit
  --from LANG, -f LANG  Translate from LANG language (e.g. en, de, es,
                        default: auto)
  --to LANG, -t LANG    Translate to LANG language (e.g. en, de, es, default:
  --input-file FILE, -i FILE
                        Get text to translate from FILE instead of stdin
  --output-file FILE, -o FILE
                        Output translated text to FILE instead of stdout
  --input-encoding ENC, -I ENC
                        Use ENC caracter encoding to read the input (default:
                        get from locale)
  --output-encoding ENC, -O ENC
                        Use ENC caracter encoding to write the output
                        (default: get from locale)

You can download the script here, but be warned, I only tested it with Python 3.2. It's almost certain that it won't work with Python < 3.0, and there is a chance it won't work with Python 3.1 either. Please report success or failure, and patches to make it work with older Python versions are always welcome.

Ideally you shouldn't abuse Google's service through this script, if you need to translate massive texts every 50ms just pay for the service. For me it doesn't make any sense to do so, because I'm not using the service differently, when I didn't have the script I just copy&pasted the text to translate to the web. Another drawback of using the script is I couldn't find any way to make it work using HTTPS, so you shouldn't translate sensitive data (you shouldn't do so using the web either, because AFAIK it travels as plain text too).

Anyway, the final step was just to connect Mutt with the script. The solution I found is not ideal, but works most of the time. Just add these macros to your muttrc:

macro index,pager <Esc>t "v/plain\n|gt|less\n" "Translate the first plain text part to English"
macro attach <Esc>t "|gt|less\n" "Translate to English"

Now using Esc t in the index or pager view, you'll see the first plain text part of the message translated from an auto-detected language to English in the default encoding. In the attachments view, Esc t will pipe the current part instead. One thing I don't know how to do (or if it's even possible) is to get the encoding of the part being piped to let gt know. For now I have to make the pipe manually for parts that are not in UTF-8 to call gt with the right encoding options. The results are piped through less for convenience. Of course you can write your own macros to translate to another language other than English or use a different default encoding. For example, to translate to Spanish using ISO-8859-1 encoding, just replace the macro with this one:

macro index,pager <Esc>t "v/plain\n|gt -tes -Iiso-8859-1|less\n" "Translate the first plain text part to Spanish"

Well, that's it! I hope is as useful to you as is being to me ;-)


Since picking the right encoding for the e-mail started to be a real PITA, I decided to improve the script to auto-detect the encoding, or to be more specific, to try several popular encodings.

So, here is the help message for the new version of the script:

usage: gt [-h] [--from LANG] [--to LANG] [--input-file FILE]
          [--output-file FILE] [--input-encoding ENC] [--output-encoding ENC]

Translate text using Google Translate.

optional arguments:
  -h, --help            show this help message and exit
  --from LANG, -f LANG  Translate from LANG language (e.g. en, de, es,
                        default: auto)
  --to LANG, -t LANG    Translate to LANG language (e.g. en, de, es, default:
  --input-file FILE, -i FILE
                        Get text to translate from FILE instead of stdin
  --output-file FILE, -o FILE
                        Output translated text to FILE instead of stdout
  --input-encoding ENC, -I ENC
                        Use ENC caracter encoding to read the input, can be a
                        comma separated list of encodings to try, LOCALE being
                        a special value for the user's locale-specified
                        preferred encoding (default: LOCALE,utf-8,iso-8859-15)
  --output-encoding ENC, -O ENC
                        Use ENC caracter encoding to write the output
                        (default: LOCALE)

So now by default your locale's encoding, utf-8 and iso-8859-15 are tried by default (in that order). These are the defaults that makes more sense to me, you can change the default for the ones that makes sense to you by changing the script or by using -I option in your macro definition, for example:

macro index,pager <Esc>t "v/plain\n|gt -IMS-GREEK,IBM-1148,UTF-16BE|less\n"

Weird choice of defaults indeed :P

Release: Status Area Display Blanking Applet 1.0 for Maemo

by Leandro Lucarella on 2012- 08- 05 09:54 (updated on 2012- 08- 05 09:54)
tagged en, floss, maemo, n900, release, sadba, software, status area display blanking applet - with 0 comment(s)

Finally 1.0 is here, and in Extras-devel! The only important change since last release is a bug fix that prevented display blanking inhibition from properly work in devices configured with a display blanking timeout of less than 30 seconds (thanks cobalt1 for the bug report).

For more information and screenshots, you can visit the website.

You can download this release (binary package and sources) from here:

But now you just might want to simply install it using the application manager.

You can also get the source from the git repository:

Please feel free to leave your comments and suggestions here or in the Maemo Talk Thread..

Release: Status Area Display Blanking Applet 0.9 (beta) for Maemo

by Leandro Lucarella on 2012- 07- 30 19:35 (updated on 2012- 07- 30 19:35)
tagged en, floss, maemo, n900, release, sadba, software, status area display blanking applet - with 0 comment(s)

Final beta release for the Status Area Display Blanking Applet. Changes since last release:

  • Show a status icon when display blanking is inhibited.
  • Improve package description and add icon for the Application Manager.
  • Add a extended description for display blanking modes.
  • Update translation files.
  • Code cleanup.

Also now the applet have a small home page and upload to Extras is on the way!

This is how this new version looks like:

You can download this 0.9 beta release (binary package and sources) from here:

You can also get the source from the git repository:

Please feel free to leave your comments and suggestions here or in the Maemo Talk Thread..

Release: Status Area Display Blanking Applet 0.5 for Maemo

by Leandro Lucarella on 2012- 07- 29 15:55 (updated on 2012- 07- 29 15:55)
tagged en, floss, maemo, n900, release, sadba, software, status area display blanking applet - with 0 comment(s)

New pre-release for the Status Area Display Blanking Applet. New timed inhibition button that inhibit display blanking for an user-defined amount of time. Also there's been some code cleanup since last release.

You can download this 0.5 pre-release (binary package and sources) from here:

You can also get the source from the git repository:

Please feel free to leave your comments and suggestions here or in the Maemo Talk Thread..

Release: Status Area Display Blanking Applet 0.4 for Maemo

by Leandro Lucarella on 2012- 07- 27 15:12 (updated on 2012- 07- 27 15:12)
tagged en, floss, maemo, n900, release, sadba, software, status area display blanking applet - with 0 comment(s)

New pre-release of my first Maemo application: The Status Area Display Blanking Applet. Now you inhibit display blanking without changing the display blanking mode. The GUI is a little rough compared with the previous version but it works. :)

You can download this 0.4 pre-release (binary package and sources) from here:

You can also get the source from the git repository:

Please feel free to leave your comments and suggestions here or in the Maemo Talk Thread..

Release: Status Area Display Blanking Applet 0.3 for Maemo

by Leandro Lucarella on 2012- 07- 26 07:51 (updated on 2012- 07- 27 15:13)
tagged en, floss, maemo, n900, release, sadba, software, status area display blanking applet - with 0 comment(s)

New pre-release of my first Maemo application: The Status Area Display Blanking Applet. Now you can pick whatever blanking mode you want instead of blindly cycling through all available modes, as it was in the previous version.

You can download this 0.3 pre-release (binary package and sources) from here:

You can also get the source from the git repository:

Please feel free to leave your comments and suggestions here or in the Maemo Talk Thread..

Release: Status Area Display Blanking Applet 0.2 for Maemo

by Leandro Lucarella on 2012- 07- 23 06:56 (updated on 2012- 07- 23 06:56)
tagged en, floss, maemo, n900, release, sadba, software, status area display blanking applet - with 0 comment(s)

Second pre-release of my first Maemo application: The Status Area Display Blanking Applet. No big changes since the last release just code cleanup and a bugfix or new features (depends on how you see it). Now the applet monitors changes on the current configuration, so if you change the display blanking mode from settings (or by any other means), it will be updated in the applet too.

You can download this 0.2 pre-release (binary package and sources) from here:

You can also get the source from the git repository:

Please feel free to leave your comments and suggestions here or in the Maemo Talk Thread..

Release: Status Area Display Blanking Applet 0.1 for Maemo

by Leandro Lucarella on 2012- 07- 15 17:09 (updated on 2012- 07- 15 17:09)
tagged en, floss, maemo, n900, release, sadba, software, status area display blanking applet - with 0 comment(s)

Hi, I just wanted to announce the pre-release of my first Maemo "application". The Status Area Display Blanking Applet let you easily change the display blanking mode right from the status menu, without having to go through the settings.

This is specially useful if you have a short blanking time when you use applications that you want to look at for a long time without interacting with the phone and don't inhibit display blanking by themselves (for example a web browser, image viewer or some GPS applications).

You can download this 0.1 pre-release (binary package and sources) from here:

You can also get the source from the git repository:

Here are some screenshots (the application is highlighted so you can spot it more easily :) ):

Please feel free to leave your comments and suggestions.

I'll upload the package to extras-devel when I have some time to learn the procedure.

Subtítulos para todos (y todas)

by Leandro Lucarella on 2012- 07- 14 15:37 (updated on 2012- 07- 14 15:37)
tagged es, floss, release, software, subdivx, subdivxget, subtitle, subtítulos - with 0 comment(s)

¡Viva Perón, carajo!

Podrido de buscar subtítulos a mano en, que es una paja enorme por la interfaz bastaaante pedorra, decidí, hace mucho tiempo, hacer un pequeño script para hacer el trabajo sucio por mí. Como muchas veces pasa, esa mugre fue creciendo hasta convertirse en algo relativamente decente, así que decidí publicarlo.

Si bien no estoy tan orgulloso de él como para hacer un release serio, subí el código a un repo git que pueden bajar, mejorar y mandar parches.

También pueden bajar el script directamente y usarlo (chmod +x de por medio). Está probado con Python 2.7 y 3.2.

Acá un screenshot del usage, que es la única documentación que existe sobre el script:

Usage: subdivxget [OPTIONS] QUERY [FILTER ...]

Download subtitles from searching the string QUERY. If FILTERs
are specified, only subtitles that matches all those filters are downloaded.
Filters have the format "X:fitler", where X is a field specification:
t=titulo, d=desc, a=autor, f=formato, c=comentarios, C=cds, F=fecha and
D=downloads. filter is a string that should be found on that field (case
insensitive). If the format specifier is not known (or there isn't one) the
filter string is looked in all the fields.

  -h, --help       show this help message and exit
  -l, --list-only  Don't download the subtitles, just list them
  -q, --quiet      Don't print progress messages

Ah, sí, me quedó mezcla de inglés y castellano, es que estoy acostumbrado a codear en inglés, pero dado que subdivx AFAIK sólo tiene subtítulos en castellano se complica.

Espero que a alguien le sea útil.

More adventures with the N900

by Leandro Lucarella on 2011- 12- 18 16:30 (updated on 2011- 12- 18 16:30)
tagged en, fix, floss, hardware, linux, maemo, n900, nokia, notification, skype, software, sound - with 0 comment(s)

OK, after I recovered my phone without needing to reflash once, I was even much closer to do it again because of a new problem.

After missing an appointment and arriving at work about 3 hours late, I realized my phone stopped reproducing sound and vibrating when an alarm was fired. At first I thought I put the alarm incorrectly but then I verified that the alarm was not working. I still got a popup with the alarm message, but no sound or vibration.

So... Time to debug the problem. After searching a lot, I couldn't find anybody with my same problem, I found similar, but not the same, so I decided to report a bug. I got a very fast but useless response. Great!

Making long story short, I finally found some IRC channels and mailing lists where I could find a more opensourceish support that the one provided in the forums and bugzilla. So I'm happy I finally found a place where you can talk to actual developers.

I commented my problem and just after a very trivial but extremely useful suggestion (installing syslogd), I could trace the origin or the problem and fix it (I just love you strace!).

I also had another problem, suddenly the skype calls stopped working. Again the syslog helped a lot. Unfortunately I didn't save the exact syslog error message, but it was something like:

GStreamer - Could not convert static caps "!`phmcadion/x-rtp,
media=(string)video, payload=(int)[ 96, 127 ], clock-rate=(int)[ 1,
2147483647 ], encoding-name=(string)MP4V-ES"

As the MIME TYPE looked like garbage, I just grep(1)ed the filesystem searching for that string, and I found some binary file at /home/user/.gstreamer-0.10/registry.arm.bin. I backed up the file, remove it, and everything started working again (the file was recreated but with a very different content).

I have no idea how the symlink or the gstreamer file got broken, except maybe because of the unexpected reboot because of the broken batterypatch, but still, is really strange.

Anyway... Lessons learned:

  • Maemo (Nokia) bugzilla is useless for getting help
  • Install syslogd to debug Nokia N900 problems
  • The maemo developers mailing list is your friend

Conclusion: Reflash my ass!

How to rescue your Nokia N900 without reflashing

by Leandro Lucarella on 2011- 12- 11 13:40 (updated on 2011- 12- 11 13:40)
tagged en, floss, hardware, initrd, linux, maemo, n900, nokia, rescue, software - with 0 comment(s)

I bought a Nokia N900 recently, a great toy if you like to have a phone with a Linux distribution that uses dpkg as package manager :)

Of course you can use it as an end user, and never find out, but as the geek I am, I had to hack it, and use the devel package repositories. Of course, with that comes the problems (and the fun! :D).

The last update of the batterypatch package came with a weird feature. The device rebooted itself each time it starts, leaving it in a restart loop that rendered the device unusable.

Searching for valuable information was not easy (thanks forums! You SUCK at organizing information... I miss mailing lists).

Anyway, I hope I can save some work to someone if you get in a similar situation, so you don't have to waste ours searching the Maemo Forums.

Nokia N900

First you will need a tool to flash the phone (it can do other things besides flashing it, I used the maemo_flasher-3.5_2.5.2.2_i386.deb file). You can also check some instructions on how to load a (very) basic rescue image (from Meego). The good thing is this image is an initrd that's loaded in MEMORY, so you don't loose anything if you tried, the device goes to it's previous state (broken in my case :P) after a reboot.

What this image can do is put the device in USB mass storage mode (the embedded MMC -eMMC- and the external MMC). I've done this to backup my eMMC data, which holds the MyDocs vfat partition and the 2 GiB ext3 partition used to install optional software. You can also put the device in USB networking mode, you can get a shell console (and reboot/power off the device), but I found that pretty useless (because you don't have any useful tools, the backlit is not turned on, so is really hard to see anything, and because the kayboard doesn't have the function key mapped, so you can't even write a "/").

The bad thing about this image, is you can't access to the root filesystem (wich is stored in another NAND 256MiB memory). I wanted to access it for 2 reasons. First, I wanted to edit some files that the batterypatch program created to see if that fixed the rebooting problem. And if now, I wanted to make a backup of the rootfs so I didn't loose most of my customizations and installed software.

I first found that a way to access the rootfs was to install Meego in a uSD memory, but for that I needed a 4GiB uSD. Also it looked like too much work, it has to be something battery and easier to just mount the rootfs and play around.

And I finally found it. It was the hardest thing to found, that's why I not only passing you the original link, I'm also hosting my own copy because I have the feeling it can disappear any time! :P

This image let's you do all the same the other image can, but it turns on the backlit, it has better support for the keyboard (you can type a "/") and it can mount the UBI root filesystem. Even more, it comes with a telnet daemon, so you can even do the rescue work remotely using USB networking ;)

You can see the instructions for some of the tasks, but here is how I did to be able to log in using telnet, which is not documented elsewhere that I know off. Once you have your image loaded:

  1. You have to activate the USB networking in the device: /rescueOS/

  2. Configure your host PC to assign an IP to usb0: sudo ip a add dev usb0 && sudo ip link set usb0 up

  3. Start the telnet daemon in the device: telnetd

  4. I couldn't find out the root password, and since the initrd root filesystem is read-only, so I did this to change the root password:

    cp -a /etc /run/
    mount --bind /run/etc /etc

    Now type the new root password.

  5. That's it, log in via telnet from the host PC: telnet and have fun!

With this I just could edit the broken files and saved the device without even needing to reflash it, but if you're not so lucky, you can just backup the root filesystem and reflash using this instructions (I didn't tested them, but seems pretty official).

Now I should probably have to try the recovery-boot package, if it works well it might be even easier to rescue the phone using that ;)

Bacap updates

by Leandro Lucarella on 2010- 10- 01 21:11 (updated on 2010- 10- 01 21:11)
tagged bacap, en, floss, self, software, update - with 2 comment(s)

Just in case you are a Bacap user (and a lazy one, that doesn't want to subscribe to the repository RSS feed :), you might be interested in a few recent updates:



by Leandro Lucarella on 2010- 07- 01 22:02 (updated on 2010- 07- 01 22:02)
tagged bacap, backup, en, floss, release, rsync, software - with 0 comment(s)

I've published my backup script (and named it Bacap, which is how it sounds when you pronounce "backup" in Spanish ;), as it suddenly became used by somebody else and received some patches =)

It has a simple home page for those wanting to see what is it about. I won't be doing formal releases thought, since is such a simple script, that would be overkill (just be thankful it has a git repo and a home page ;).

Google's Go will be part of GCC

by Leandro Lucarella on 2010- 01- 28 11:40 (updated on 2010- 01- 28 11:40)
tagged compiler, d, dmd, en, fsf, gcc, golang, google, software - with 0 comment(s)

Wow! Google's Go (remember there is another Go) programming language front-end for GCC has been accepted for merging into GCC 4.5.

Just when there was some discussion (started by Jerry Quinn [*]) in D on how the DMD front-end could be pushed to be merged in GCC too, but DigitalMars (Walter) doesn't want to give away the copyright of his front-end (they are exploring some alternative options though). Maybe the inclusion of Google's Go makes Walter think harder for a solution to the legal problems :).

[*]He reported a lot of bugs in the language specification because he was planning to start a new D front-end, which can be donated to the FSF for inclusion in GCC.

DMD beta

by Leandro Lucarella on 2010- 01- 27 21:01 (updated on 2010- 01- 27 21:01)
tagged beta, compiler, d, development model, dmd, druntime, en, phobos, software - with 2 comment(s)

After some discussion [*] in the D newsgroup about the value of having release candidates for DMD (due to the high number of regressions introduced in new versions mostly), Walter agreed to make public what he called beta versions of the compiler, which he sent privately to people who asked for them (like some Tango developers).

The new DMD betas are announced in a special mailing list (available through Gmane too). It seems like Walter want to keep the beta releases with some kind of secrecy, or only for people really interested on them (the zip files are even password protected! But the password is announced in a public mailing list, that doesn't make much sense =/). I think he should encourage people to try them as much as possible instead, but one step at the time, at least now people have a way to test the compiler before it's released.

I can say without fear that the experience has been very successful already, even when there is no DMD release yet that came from a beta pre-release, you can see in the beta mailing list that multiple regressions have been discovered and fixed because this new beta releases. I think the reliability of the compiler has been increased already. Is really interesting to see how the quality of a product increases proportionally to the level of openness and the numbers of eyes doing peer review.

The new DMD release should be published very soon, as all the regressions seems to be fixed now and big projects like Tango, GtkD and QTD compiles (a lot of focus on fixing bugs that prevented the later to compile has been put into this release, specially from Rainer Schuetze, who submitted a lot of patches).

So kudos for a new era in D, I think this is another big milestone for having a reliable compiler.

[*]I'm sure there was previos requests for having release candidates, I know I asked for it, but I can't find the threads in the archives =)


by Leandro Lucarella on 2009- 12- 03 11:56 (updated on 2009- 12- 03 11:56)
tagged bpython, curses, en, floss, interpreter, python, software - with 0 comment(s)

I'll just copy what the home page:

bpython is a fancy interface to the Python interpreter for Unix-like operating systems (I hear it works fine on OS X). It is released under the MIT License. It has the following features:

  • In-line syntax highlighting.
  • Readline-like autocomplete with suggestions displayed as you type.
  • Expected parameter list for any Python function.
  • "Rewind" function to pop the last line of code from memory and re-evaluate.
  • Send the code you've entered off to a pastebin.
  • Save the code you've entered to a file.
  • Auto-indentation.

Go nuts

by Leandro Lucarella on 2009- 11- 11 12:14 (updated on 2009- 11- 11 18:48)
tagged compiler, d, en, go, google, language, software - with 11 comment(s)

I guess everybody (at least everybody with some interest in system programming languages) should know by now about the existence of Go, the new system programming language released yesterday by Google.

I think this has a huge impact in D, because it's trying to fill the same hole: a modern high-performance language that doesn't suck (hello C++!). They have a common goal too: be practical for business (they are designed to get things done and easy of implementation). But there are still very big differences about both languages. Here is a small summary (from my subjective point of view after reading some of the Go documentation):

Go D
Feels more like a high-level high- performance programming language than a real system programming language (no asm, no pointer arithmetics). Feels more like a real close to the metal system programming language.
Extremely simple, with just a very small set of core features. Much more complex, but very powerful and featureful.
Can call C code but can't be called from C code. Interacts very well with C in both directions (version 2 can partially interact with C++ too).
Feels like a very well thought, cohesive programming language. Feels as a bag of features that grew in the wild.
FLOSS reference implementation. Looks very FLOSS friendly, with proper code review, VCS, mailing lists, etc. Reference implementation is not FLOSS. Not very FLOSS friendly (it's just starting to open up a little but it's a slow and hard process).
Supported by a huge corporation, I expect a very large community in very short time. Supported by a very small group of volunteers, small community.

I really like the simplicity of Go, but I have my doubts about how limiting it could be in practice (it doesn't even have exceptions!). I have to try it to see if I will really miss the features of more complex programming languages (like templates / generics, exceptions, inheritance, etc.), or if it will just work.

I have the feeling that things will just work, and things missing in Go will not be a problem when doing actual work. Maybe it's because I had a very similar feeling about Python (indentation matters? Having to pass self explicitly to methods? No ++? No assignment in if, while, etc.? I hated all this things at first, but after understanding the rationale and using then in real work, it works great!). Or maybe is because there are is extremely capable people behind it, like Ken Thomson and Rob Pike (that's why you can see all sort of references to Plan 9 in Go :), people that knows about designing operating systems and languages, a good combination for designing a system programming language ;)

You never know with this things, Go could die in the dark or become a very popular programming language, only time will tell (but since Google is behind it, I guess the later is more likely).

DMD frontend 1.051 merged in LDC

by Leandro Lucarella on 2009- 11- 07 15:08 (updated on 2009- 11- 07 15:08)
tagged compiler, d, dmd, en, ldc, merge, software - with 0 comment(s)

After 5 or 6 DMD versions with important regressions, LDC has just been updated to DMD's frontend 1.051. This brings a lot of bug fixes to the LDC world (DStress results are looking good! ;).

Lots of thanks to LDC guy for merging the new frontend =)


by Leandro Lucarella on 2009- 10- 25 18:41 (updated on 2009- 10- 25 18:41)
tagged es, movie, software, subdownloader, subtitle, tv - with 0 comment(s)

Subdownloader es otro gran invento para la gente que mirá TV en la compu. Como el nombre lo indica, este simpático programa facilita la tarea de bajar (y subir!) subtítulos utilizando un server que tenga una API compatible con

Particularmente útil es que los subtítulos los busca calculando un hash del archivo, así que se pueden olvidar de los problemas de bajar un subtítulo para una versión que no corresponda.

LLVM 2.6

by Leandro Lucarella on 2009- 10- 24 18:30 (updated on 2009- 10- 24 18:30)
tagged d, en, llvm, release, software - with 0 comment(s)

Just in case you're not that well informed, Chris Lattner has just announced the release of LLVM 2.6. Enjoy!

KLEE, automatically generating tests that achieve high coverage

by Leandro Lucarella on 2009- 10- 20 11:20 (updated on 2009- 10- 20 11:20)
tagged coverage, d, en, klee, llvm, software, test, vm - with 2 comment(s)

This is the abstract of the paper describing KLEE, a new LLVM sub-project announced with the upcoming 2.6 release:

We present a new symbolic execution tool, KLEE, capable of automatically generating tests that achieve high coverage on a diverse set of complex and environmentally-intensive programs. We used KLEE to thoroughly check all 89 stand-alone programs in the GNU COREUTILS utility suite, which form the core user-level environment installed on millions of Unix systems, and arguably are the single most heavily tested set of open-source programs in existence. KLEE-generated tests achieve high line coverage — on average over 90% per tool (median: over 94%) — and significantly beat the coverage of the developers' own hand-written test suites. When we did the same for 75 equivalent tools in the BUSYBOX embedded system suite, results were even better, including 100% coverage on 31 of them. We also used KLEE as a bug finding tool, applying it to 452 applications (over 430K total lines of code), where it found 56 serious bugs, including three in COREUTILS that had been missed for over 15 years. Finally, we used KLEE to cross-check purportedly identical BUSYBOX and COREUTILS utilities, finding functional correctness errors and a myriad of inconsistencies.

I have to try this...

pybugz, a python and command line interface to Bugzilla

by Leandro Lucarella on 2009- 10- 16 11:14 (updated on 2009- 10- 16 11:14)
tagged bugzilla, cli, d, en, pybugz, python, software - with 0 comment(s)

Tired of the clumsy Bugzilla web interface? Meet pybugz, a command line interface for Bugzilla.

An example workflow from the README file:

$ bugz search "version bump" --assigned

 * Using ..
 * Searching for "version bump" ordered by "number"
 101968 liquidx net-im/msnlib version bump
 125468 liquidx version bump for dev-libs/g-wrap-1.9.6
 130608 liquidx app-dicts/stardict version bump: 2.4.7

$ bugz get 101968

 * Using ..
 * Getting bug 130608 ..
Title : app-dicts/stardict version bump: 2.4.7
Assignee :
Reported : 2006-04-20 07:36 PST
Updated : 2006-05-29 23:18:12 PST
Status : NEW
Severity : enhancement
Reporter :
Priority : P2
Comments : 3
Attachments : 1

[ATTACH] [87844] [stardict 2.4.7 ebuild]

[Comment #1] : 2006-04-20 07:36 PST

$ bugz attachment 87844

 * Using ..
 * Getting attachment 87844
 * Saving attachment: "stardict-2.4.7.ebuild"

$ bugz modify 130608 --fixed -c "Thanks for the ebuild. Committed to

D and open development model

by Leandro Lucarella on 2009- 10- 15 17:09 (updated on 2009- 10- 15 17:09)
tagged compiler, d, development model, dmd, druntime, en, phobos, software - with 6 comment(s)


Long post ahead =)

I'm very glad that yesterday DMD had the first releases (DMD 1.050 and DMD 2.035) with a decent revision history. It took some time to Walter Bright to understand how the open source development model works, and I think he still has a lot more to learn, but I have some hope now about the future of D.

Not much time ago, neither Phobos, DMD nor Druntime had revision control. Druntime didn't even exist, making D 1 split in two because of the Phobos vs Tango dichotomy. DMD back-end sources were not available either, and Walter Bright was the only person writing stuff (sometimes not because people didn't want to, but because he was too anal retentive to let them ;). It was almost impossible to make patches back then (your only chance was hacking GDC, which is pretty hard).

Now I can say that DMD, Phobos and Druntime have full source availability (DMD back-end is not free/libre though), almost all the parts of DMD have the sources published under a source control system. The core team has been expanded and even when Walter Bright is still in charge, at least 3 developers are now very committed to D: Andrei Alexandrescu (in charge of Phobos), Sean Kelly (in charge of Druntime) and Don Clugston (squashing DMD bugs at full speed, specially in the back-end). Other people are contributing patches in a regular basis. There were about 72 patches submitted to bugzilla before DMD was distributed with full source (72 patches in ~10 years) , since then, 206 patches were submitted (that is, 206 patches in less than 8 months).

But even with this great improvement, there is much left to do yet (and I'm talking only about the development model). This is a small list of what I think it's necessary to keep moving to a more open development model:


The release process should be improved. Me and other people are suggesting release candidates. This will allow people to test the new releases to find any regressions. As things are now, releases are not much different from a nightly build, except that you don't have one available every night :). People get very frustrated when downloading a new version of the compiler and things stop working, and this holds back front-end updates in other compilers, like LDC (which is frozen at 1.045 because of the regressions found in the next 5 versions).

I think Walter Bright is suffering from premature releasing too. Releases comes from nowhere, when nobody expects them. Nobody knows when a new compiler version will be released. I think that hurts the language reliability.

I think the releases should be more predictable. A release schedule (even when not very accurate, like in many other open source projects) gives you some peace of mind.

Peer review

Even when commits are fairly small now in DMD, I think they are far from ideal. Is very common to see unrelated changes in a commit (the classic example is the compiler version number being bumped in an bug fix). See revision 214 for example: the compiler version is bumped and there are some changes to the new JSON output, totally unrelated to bug 3401, which is supposed to fix; or revision 213, which announces the release of DMD 1.050 and DMD 2.035, introducing a bunch of changes that who knows what are supposed to do (well, they look like the introduction of the new type T[new], but that's not even documented in the release changelog :S). This is bad for several reasons:

  • Reviewing a patch with unrelated changes is hard.
  • If you want to fold in a individual patch (let's say, LDC guys want to fold a bug fix), you have a lot of junk to take care of.
  • If you want to do some sort of bisection to find a regression, you still have to figure out which is the group of related changes that introduced the regression.

I'm sure there are more...

Commit messages lacks a good description of the problem and the solution. Most commit messages in DMD are "bugzilla N". You have to go to the bugzilla bug to know what's all about. For example, Don's patches usually comes with very good and juicy information about the bug causes and why the patch fixes it (see an example). That is a good commit message. You can learn a lot about the code by reading well commented patches, which can lead to more contributions in the future.

Commits in Phobos can be even worse. The commits with a message "bugzilla N" are usually the good ones. There are 56 commits that have "minor" as the commit message. Yes, just "minor". That's pretty useless, it's very hard to review a patch when you don't know what is supposed to do. Commit messages are the base of peer reviewing, and peer reviewing is the base for high quality code.

So I think that D developers should focus a lot more in commit message. I know it can sound silly at first, but I think I would be a huge gain with too little effort.

Besides this, commits should be mailed to a newsgroup or mailing list to easy peer review. Now it's a little hard to make comments about a commit, you have to post the comment in the D newsgroup or make the comment by personal e-mail to the author. The former is not that bad but it's not easy to include context and people reading the comment will probably have to open a browser and search for the commented commit. This clearly make peer reviewing more difficult when the ideal would be to encourage it. The private mail is simply wrong because other people can't see the comments.

Source control and versioning

This one is tightly related to the previous two topics. Using a good DVCS can make help a lot too. Subversion has a lot of problems with branching, which makes releases harder too (as having a branch for each release is very painful). Is bad for commit messages too, because there is no real difference in branches and directories, so know every commit is duplicated (both changes for DMD 1 and 2 are included). It's not easy to cherry-pick single commits either, and you can't fix you commits if you messed up, which leads to a lot of commits of the style "Woops! Fix the typo in the previous commit.".

I'm sure both the release process and peer reviewing can be greatly improved by using a better DVCS.

Easy branching can also lead to a more fast evolving and reliable language. Yes, both are possible with branches. Now there are 2 branches: stable (D1) and experimental (D2). D1 is almost frozen and people is seeing less and less interest on it as it goes old, and D2 is too unstable for real use. Having some intermediate can be really helpful. For example, it has been announced that the concurrency model proposed by Bartosz Milewski will be not part of D2 because there is not enough time to implement it, since D2 should be release fairly soon as Andrei Alexandrescu is writing a book that has a deadline and the language has to be finalized by the time the book is published.

So concurrency (as AST macros) are delayed to D3. D2 is more than 2 years old, so one should expect that D3 will be not available in less than 5 years from now (assuming D2 would take 2.5 years and D3 would take the same). This might be too much time.

I think the language should adopt a model closer to Python, where a minor language version (with backward compatible improvements) is release every 1 ~ 1.5 years. Last mayor version took about 8 years, but considering how many new features Python included in minor versions that's not a big issue. The last mayor version was mostly a clean up of old stuff/nasty stuff, not huge changes to the language.


I think the DMD back-end should have a better license. Personal use is simply not enough for a reference implementation of a language that wants to hit mainstream. If you plan to do business with it, not being able to patch the compiler if you need to and distribute it is not an option.

This is for the sake of DMD only, because other compilers (like LDC and GDC) are fully free/libre.


Some of the things I mention are really hard to change, as they modify how people work and imply learning new tools. But other are fairly easy, and can be done progressively (like providing release candidates and improving commits and commit messages).

I hope Walter Bright & Co. keep walking the openness road =)

Tucan {up,down}load manager for file hosting sites

by Leandro Lucarella on 2009- 10- 06 11:02 (updated on 2009- 10- 06 11:02)
tagged download, en, floss, python, software, tucan, upload - with 0 comment(s)

Meet Tucan:

Tucan is a free and open source application designed for automatic management of downloads and uploads at hosting sites like Rapidshare.

GDC resurrection

by Leandro Lucarella on 2009- 10- 05 11:31 (updated on 2009- 10- 05 11:31)
tagged compiler, d, en, floss, gcc, gdc, software - with 0 comment(s)

About a month ago, the GDC newsgroup started to get some activity when Michael P. and Vincenzo Ampolo started working on updating GCD. Yesterday they announced that they successfully merged the DMD frontend 1.038 and 2.015, and a new repository for GDC. They will be hanging on #d.gdc if you have any questions or want to help out.

So great news for the D ecosystem! Kudos for this two brave men! =)