How to install IronPython2 with Mono on Ubuntu

This is just an instructional post with the hope that it might help others in the future.

I spent a long time trying to get IronPython2 running on Mono. Now IronPython1 ships with Ubuntu, so

apt-get install ironpython

, if that's all that you want. Ironically, I don't really know the difference between IronPython1 and IronPython2, but "2 has got to be better than 1", right?

The good news is that its very possible, but you will have to step out of your comfort zones.

You will need:
Mono 2.4 (sources) and IronPython 2.0.1 (binary)

First build Mono (note that you need Ubuntu's Mono to bootstrap, so

apt-get install mono

) but make sure you put it somewhere special so it doesn't mess with the system-installed Mono. This advice is taken from an article on the Mono wiki.

$ tar xvjf mono-2.4.tar.bz2
$ cd mono
$ mkdir -p ~/opt/mono
$ ./configure --prefix=~/opt/mono
$ make && make install

So this is putting Mono in opt/mono in my home directory (which is where I like to put things, but you might like them somewhere else, like /opt).

Then you will need to write a script. Again modified from the link above.

#!/bin/bash
MONO_PREFIX=~/opt/mono
export DYLD_LIBRARY_PATH=$MONO_PREFIX/lib:$DYLD_LIBRARY_PATH
export LD_LIBRARY_PATH=$MONO_PREFIX/lib:$LD_LIBRARY_PATH
export C_INCLUDE_PATH=$MONO_PREFIX/include
export ACLOCAL_PATH=$MONO_PREFIX/share/aclocal
export PKG_CONFIG_PATH=$MONO_PREFIX/lib/pkgconfig
PATH=$MONO_PREFIX/bin:$PATH
PS1="[mono] \w @ "

You will source this script to mutate your environment so you can use your nice new Mono, so say you saved it as ~/mono-2.4-env-activate.sh:

source ~/mono-2.4-env-activate.sh

And it will do its busines and change your prompt so you know it:

[mono] ~ @ 

And once you have that you can test it:

[mono] ~ @ which mono
/home/ali/opt/mono/bin/mono
[mono] ~ @ mono --version
Mono JIT compiler version 2.4 (tarball Sat May 23 01:13:11 BST 2009)

And once you are happy it works, you just need to execute the IronPython2 executable.

[mono] ~/tmp @ unzip IronPython-2.0.1-Bin.zip 
[mono] ~/tmp @ cd IronPython-2.0.1/
[mono] ~/tmp/IronPython-2.0.1 @ mono ipy.exe 
IronPython 2.0 (2.0.0.0) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Looking good! And before you kill yourself trying to exit (it wants

Control-z

which is going to be fairly impossible):

>>> import sys
>>> sys.exit()
[mono] ~/tmp/IronPython-2.0.1 @ 

And there you have it. I am not sure exactly how I feel about IronPython on Mono, but it can't be a bad thing that it works. I will be investigating it more in the near future, so I may document what I am learning.

Quis experiet ipsas experientiae?

Who will test those tests?

A while ago I was invited by the lynch-mob in #twisted to write a post about what I meant when I said that "Unit tests are not real tests", and here it is.

Now, recently there has been much said about Unit-testing, and code coverage, and even sensationalist blog post titles, which in retrospect "Unit tests are not real tests" would have been. I won't talk about code test coverage here, because its not relevant, we should strive to get as much coverage as we can. And of course I believe that unit tests are a fundamentally important piece of development. Especially since I tend to use Python, which is prone to disaster if not tested correctly.

So, if unit tests are real tests, then why did I say they aren't? I deal mostly with application development. The user of the application is a human being. Now quite simply, the only true test of my application is the user interacting with it. I understand now that this is not really the case for library development, where the user is a piece of code, and I guess you can have true tests for it that are unit tests.

So how do we do it? Well, I am not sure what kind of testing this is, but I call it "Acceptance Testing". It is evidence that the application behaves as advertised. I guess it would come under headings like "Verification". We have a list of written tests that go something like:

Test	Expected Outcome	Actual Outcome
User double-clicks the icon	log in screen displayed
User enters date in the future	text entry goes red and user is informed

And they go on and on, we have hundreds of pages of these. As I was in a start-up consisting of one person, I used to do these myself. Now the test department does it.

But these tests are just another type of unit test! Yes they are, and probably purists would disagree, but look at them:

• You test every single unit of an application (but from the user perspective)
• When you find a bug, you write a new test to test for it, then fix it
• When you add a new feature, you write the tests first (TDD!)
  
• Anyone can run the tests

And then as an added bonus you can give them to the client to run, which some of our clients insist on. They differ from traditional acceptance tests as they are run on single acceptance from the client.

Can't you automate these? Yes you can, but that would again slip them into the realms of "non-real tests". As I said, a human user has to test the application in order to reflect the actual user of the software. I really love automation. I have hacked on kiwi's automated test-runner, played with various other tools. Again, like traditional code-based unit tests, they are great tools.

So, I should really amend my statement to the #twisted guys to something more like:

"Code-based unit tests are not the true tests for an application that interacts with a human user."

Glashammer, an alternative framework on Google AppEngine

In his blog, on Friday, August 29, 2008, Johnathan talked about Google AppEngine, and said

the "you can use any web framework you like, as long as it's django" attitude

That may have been true then, it probably isn't true now. Glashammer have been working for the last few weeks on getting the Glashammer framework of Werkzeug and Jinja2 running and easy on Appengine.

Firstly this wasn't hard. Glashammer is very free about what kind of data storage you use, so using Appengine's DataStore was straightforward. Some utility functions for running easily, add a few decorators for controlling what happens for authentication form redirection to limit views for certain users and we are pretty much done.

They have additionally tried to make it easier to install with a little script to get us started. So (since it seems a good start):

gh-admin quickstart_gae

This will generate a starter AppEngine + Glashammer application that is ready to go. It will also fetch all the dependencies for you and build them in the right places in your application for uploading to AppEngine. Now you can deal with that bit yourself with:

gh-admin quickstart_gae nodeps

You will be asked the name of the project directory, and this directory will be created. Within this directory, Glashammer will create components necessary for the application to be both a Glashammer application, and an AppEngine application.

These include an app.yaml with the necessary rules for static files, in the shared directory (handled by AppEngine) and the remaining paths to the WSGI Application built in main.py and the templates directory (handled by Glashammer).

At this point, I should probably just refer you to the respective documentations of Glashammer and AppEngine, but I can't resist a little teaser.

Let's have a look at the generated main.py module:

from glashammer import make_app
from glashammer.bundles import gae

TEMPLATES_DIRECTORY = 'templates'

# Main application setup
def setup(app):
    # add the gae init function
    app.add_setup(gae.setup_gae)

    # setup templates
    app.add_template_searchpath(TEMPLATES_DIRECTORY)

def main():
    gae.make_and_run_gae_app(setup)

if __name__ == '__main__':
    main()

That application doesn't actually do anything, because there are no rules for views, but we could add one:

from glashammer import Response

def do_home(request):
    return Response('Hello World')

and then add it in our application setup function:

def setup(app):
    ...
    app.add_url('/', 'main/index', do_home)

And we have a simple Hello World application.

You can use different components from Glashammer and Appengine, and sometimes you can even choose which bits you want from each. For example, I like using Glashammer's Sessions and Memcached interface, but AppEngine's Django forms.

The real advantages are in having Jinja2 along with hooks to add template filters and globals is ready to go, and Werkzeug with its lovely API. For me, this feels much nicer than the provided API in AppEngine.

There are no monkey-patches or other trickery and all dependencies run out of the box. And since it was released today, roll on over and grab it at http://glashammer.org/downloads.html.

What Twisted could learn from Kamaelia.

Two awesome things: Twisted, and Kamaelia. I will not compare them. They are different, and have different purposes, and as I said. Both are great.

I have recently been playing with Kamaelia for the first time. If you don't know what Kamaelia is check out http://www.kamaelia.org/ but I admit it does take some getting my head around. (Just like Twisted did, back in the day when I first came across it.) But persevere.

Kamaelia is a library for creating highly concurrent applications. And along the way it has reminded me of: Erlang, Tasklets and Twisted. It has a highly componentized approach where components communicate with eachother using inboxes and message passing. That's all I'll say about it here. I am sure I will blog more as I use it more.

Now the first thing I want to do with Kamaelia is be able to hook it up with GTK, and I have managed to do that, but it was incredibly easy. Why? Because Kamaelia is happy to run its scheduler in the background in a non-main thread.

from Axon.background import background
background().start()

So the remainder of the gtk application can just run:

gtk.main() # yay!

Now Twisted has a solution for GTK, using the GTK reactor, and it works perfectly well, but look at the difference. Between the two:

Kamaelia is a library, Twisted is a framework.

bitbucket.org Dream Project Hosting?

Well, I am always in search of new ways to host and serve code. Recently I have been using bitbucket.org's free service more and more.

The benefits as I see them are:

1. Mercurial hosting
2. Ability to branch and publish any number of branches (a la launchpad)
3. Plain web space for hosting generated documentation etc, with built-in wiki
4. User interface entirely geared around source code
5. I don't have to host it myself
   
6. Responsive and generous developers

Now if we were to compare this to some other things, we might have: (and please correct me if I am wrong).

	Mercurial	Publish many branches	Web space	Wiki	Hosted by someone else for free
Bitbucket.org	Yes	Yes	Yes	Yes	Yes
Launchpad.net	No	Yes	No	No	Yes
Google code	No	No	No	Yes	Yes
Self-hosted Trac	Yes	No	Yes	Yes	No

There are features that I would still really like, and the great developers are implementing them as we speak. The most exciting is the idea of an API where third-party applications could be written to do just about anything.

So Bitbucket.org, it's perfect for me, maybe it is perfect for you too.

The problem with jQuery

Ok, I'll start by saying that I love jQuery as much as the next developer, and every contributor has my eternal thanks, but there is a problem.

Plugin development goes crazy, in that many similar plugins are released, and often these are just modifications including other sets of modifications. There must be a better way, like isn't this what version control is made for?

Here is an excerpt of http://docs.jquery.com/Plugins#Forms


* Autocomplete by Dylan Verheul
Autocomplete with caching to limit server requests and other options.

* jQuery Autocomplete Mod by Dan G. Switzer
Modification of Autocomplete plugin with enhancements and bug fixes.

* Modified Auto-complete by Anjesh Tuladhar
Auto-complete plugin extended for auto-completing multiple words in the same text input

* Autocomplete by Jörn Zaefferer
Heavily improved Dylan Verheul’s initial plugin, integrating modifications by Dan G. Switzer and Anjesh Tuladhar.

* jQuery Autocomplete by Saurabh Periwal
Modification of Autocomplete plugin with enhancements of onItemSelect and Multiselect.

* Jeditable + Autocomplete by Ritesh Agrawal
Extended jeditable to include option for autocomplete. Also extended Dylan Verheul's autocomplete javascript to include option for having input token separator. Check the demo.

End of snippet.

Blogger Comment Spam - Deleting it

It seems over recent months that my blog gets comment spam. I imagine any bloggers out there experience the same thing and it is a bit of a pain.

I have three immediate problems with this and blogger.com.

1. Blogger doesn't notify me of all comments at the time they are posted. It notifies me of some, and I have of course configured it to notify me of all comments, but it seems to miss off about 70%. So not only do I not notice the spam, I also miss a bunch of legitimate comments. Please get it together Blogger! Ajax panel configuration is nice, but only if the core functions work.

2. Blogger should/could/might try to stop this spam before it happens. I am not guessing how, but then the company that runs Blogger.com are much brighter than me, and I am sure they have a solution.

3. The interface for browsing comments and deleting many at a time simply does not exist. This would make the task of sifting through, identifying, and delting spam much easier.

Now that I have had my grumble about it, I will offer my small solution. In praise of Google, they do provide a nice API and Python bindings to access all of their services and blogger is one of them. So I wrote a small script to go through all the comments, do a little bit of flagging on dodgy looking ones and offer you a chance of deleting them.

The script is uncommented, has no tests, and I don't plan in any way to maintain it or release it, but for those people suffering the same problems, I provide it here.

It is worth noting that the spam detection is really pathetic, and it could be vastly improved. I targetted it at my particular spam.


Full script available here

"""
(c) Ali Afshar 2008
MIT License
"""

import sys, getpass

from gdata import service


def get_details():
    email = raw_input('email: ').strip()
    password = getpass.getpass()
    return email, password


def create_service(email, password):
    blogger_service = service.GDataService(email, password)
    blogger_service.source = 'blogger_spam_killer'
    blogger_service.service = 'blogger'
    blogger_service.server = 'www.blogger.com'
    blogger_service.ProgrammaticLogin()
    return blogger_service


def get_all_blog_ids(svc):
    query = service.Query()
    query.feed = '/feeds/default/blogs'
    feed = svc.Get(query.ToUri())
    for entry in feed.entry:
        blog_id = entry.GetSelfLink().href.split("/")[-1]
        yield blog_id


def get_blog_comments(svc, blog_id):
    query = service.Query()
    query.feed = '/feeds/%s/comments/default' % blog_id
    query.max_results = sys.maxint
    feed = svc.Get(query.ToUri())
    for entry in feed.entry:
        yield entry


def get_all_comments(svc):
    for blog_id in get_all_blog_ids(svc):
        for comment in get_blog_comments(svc, blog_id):
            yield comment


def rank_comment(comment):
    words = 0
    for word in spamwords:
        words += comment.content.text.count(word)

    author = comment.author[0]
    has_uri = (author.uri is not None and
                # I figure no one who puts a URI would link to a blogger
                # profile. They would link to whatever they are spamming.
                'http://www.blogger.com/profile/' not in author.uri.text)
    print 'Spam words: %s' % words
    print 'Dodgy author uri: %s' % has_uri
    return bool(words) or has_uri


def delete_comment(svc, comment):
    svc.Delete(comment.GetEditLink().href)


def filter_all_comments(svc):
    for comment in get_all_comments(svc):
        print '--'
        t = comment.content.text
        print t[:70] + '...'
        print '...' + t[-70:]
        a = comment.author[0]
        print 'Author Info: ', a.name.text
        if rank_comment(comment):
            print '**** LOOKS DODGY'
        else:
            print '==== OK'
        s = raw_input('Delete? (y/N) ').strip()
        if s == 'y':
            print 'Deleting.'
            delete_comment(svc, comment)
        else:
            print 'Not deleting.'


# http://codex.wordpress.org/Spam_Words
spamwords = """
4u
adipex
advicer
...
""".strip().splitlines()


if __name__ == '__main__':
    em, pw = get_details()
    svc = create_service(em, pw)
    filter_all_comments(svc)

CouchDB with YUI Datatable

I am not kidding, this must be the easiest way to get a ajaxy table of results from a database. For those who haven't heard the buzz, CouchDB is a document-based database. I am not an expert in databases so I won't really comment about CouchDB itself. One incredible feature is that the database server is queryable over HTTP (with REST) and responds with JSON, which seems perfect for writing pure Javascript applications.

YUI probably needs no introduction. I think its the best JavaScript library for maintainability, and documentation. It sometimes seems a bit verbose, but I like verbose.

Note: You will have to get around the fact that you cannot make XHR requests to a server other than the server that the page lives on, either with a reverse proxy setup, or by hosting the files themselves in CouchDB. (I won't explain that here).

So, a basic request to a database to return all the documents goes something like this:

a GET to:

http://myserver/dbname/_all_docs

And that will return a JSON data structure. In our case, each document has a "title" attribute and a "description" attribute, and the "rows" member of the return JSON contains a list of these.

Our database is called "tickets" and you could imagine it is a ticket tracking application.

function list_tickets() {
        var myDataSource = new YAHOO.util.DataSource("/tickets/_all_docs");
        myDataSource.responseType = YAHOO.util.DataSource.TYPE_JSON;
   
        // Define the data schema
        myDataSource.responseSchema = {
            resultsList: "rows", // Dot notation to results array
            fields: ["id", "title","description"] // Field names
        };

         // Define how it will appear in the table
        var myColumnDefs = [
            {key:"id", label:"Ticket ID"},
            {key:"title", label:"Ticket Title"},
            {key:"description", label:"Ticket Description"}
        ];

        var myDataTable = new YAHOO.widget.DataTable("table_holder", myColumnDefs, myDataSource);
    }

Just make sure you have a <div id="table_holder"> somewhere in your document, because this is what will be replaced by the data table.

That's really it. No web server application, no PHP (or Python or Ruby or Perl). Just javascript talking to your database. Incredible.

Using Storm and SQLite in Multithreaded Web Applications

Pysqlite doesn't allow you to access the same connection from different threads. The pysqlite manual says: "SQLite connections/cursors can only safely be used in the same thread they were created in."

When using Storm (the ORM) with Werkzeug (the WSGI utility lib) we suffer from the problem that the Werkzeug reloader runs code in a thread. Ok this feature is not exactly important in a production environment, but I can't guarantee that whatever platform I will be deploying the application on will not be threaded, so database access should be proofed against this.

The solution? The Storm manual mentions that you should use a Store/connection per thread. Someone has already done this with the Middlestorm application, which provides a threadsafe store in the WSGI environ. Rightly or wrongly (since I really don't want to have to wait to have a WSGI environ to get the store instance), and I am not exactly sure this kind of thing should be middleware, but that is a debate for another day.

Looking at the code, it uses threading.local(), which is a thread-local attribute store. In other words, each thread will have its own values for the local object's attributes.

So a very simple implementation:

from threading import local

from storm.locals import create_database, store

class ThreadSafeStorePool(object):

 def __init__(self, uri):
     self._db = create_database(uri)
     self._local = local()

 def get(self):
     try:
         return self._local.store
     except AttributeError:
         self._local.store = Store(self._db)
         return self._local.store 

Using an instance of this class, and calling get() when you require the thread-local store (which can be easily hidden behind a property descriptor ensures that each thread has its own Store, and pysqlite stops complaining.

uri = 'sqlite:test.sqlite'

store_pool = ThreadSafeStorePool(uri)

# From one thread
store = store_pool.get()
# Will always return the same store in that thread

# From another thread
store = store_pool.get()

Et cetera.

Leopard Spaces Is Unusable

With Leopard, OS X finally has a bundled virtual desktop implementation. Pre-leopard I used a (now unmaintained) 3rd-party app called VirtueDesktops to get this feature, and while not perfect, it worked fairly well. It certainly didn't annoy me. Spaces, on the other hand, does annoy me, to the extent that I find it unusable.

Spaces SWITCHES MY DAMN DESKTOP WITHOUT ME ASKING IT TO. When I command-tab between applications, Spaces changes to the desktop with the active window. Now, at first when I encountered this feature, I thought "oh, that's pretty nice," and indeed it would be ok if it worked consistently. It doesn't, though. Usually Spaces switches correctly, but occasionally (say every 5 or 10 application-switches) it switches to the wrong desktop!

If you don't believe me, try this out: Open two terminals, and put them on different desktops. Now open a browser on one of these desktops. Command-tab between the browser and the terminal a few times, and see what happens.

This makes Spaces unusable for me. When I'm working on code, command-tabbing between the browser, the terminal and textmate, having my concentration broken by this frequent and frustrating bug is something I just can't put up with. So I've turned it off, and I'm either going to install virtuedesktops again or just learn to live without the luxury of virtual desktops.

Also, it doesn't play nicely with firefox (disappearing windows).