Toward a Secret Sky

Getting the command line arguments in Dart (vm)

2011-12-10T23:14:00.001Z

Another hidden gem in the Dart library today, can you tell I am trying to write a command line thingy? Reading the command line options is really easy.

void main() {
  Options opts = new Options();
  for (String arg in opts.arguments) {
    print(arg);
  }
}

Let us play:

$ dart opts.dart a
a

$ dart opts.dart a b c d
a
b
c
d

Nothing to add, I think, except check the source.

Comments on Google+.

OAuth2.0 with Dart in a web browser

2011-12-09T11:36:00.000Z

I was trying to access a Google GData API using Dart in a browser. More for fun than profit, but it posed a few challenges, so I thought I should document them.

The first step was to Authorize using OAuth2.0 which is the easiest way of Authorizing an application to use Google APIs. This is pretty trivial using server-side Dart, but I wanted to achieve the entire thing inside a web browser, and so this involves a different OAuth2.0 flow to the web-server flow that I am used to using, in this case the OAuth2.0 Client-side Application Flow.

The main difference with the web-server flow is that the entire OAuth2.0 dance occurs in a single step, i.e., there is no step to exchange a code for a token. This single step is performed by asking for a token in the initial redirect step with a response_type parameter of token.

Additionally, the client secret is not passed at any time, because this would not be safe information for the client application to give up to a web browser (since anyone could just read it).

The response is also different in that the token is returned in the URI fragment of the redirect, rather than as a URI parameter as would be returned in a web-server flow, so that after authorization the browser is redirected back to something like:

http://localhost:7890/DartOAuthTest.html#access_token=AHES6ZS2-t1ntxFsGsvfKbEFosG6nokTOrn9qwVysp_&token_type=Bearer&expires_in=3600

So, this is what we are aiming at:

OAuth2.0 Client Flow (via http://code.google.com/apis/accounts/docs/OAuth2.html)

Step 1: Generate the Authorization URI

String authEndpoint = 'https://accounts.google.com/o/oauth2/auth?';

String generateAuthUri(String client_id, String redirect_uri, String scope) {    
  List<List<String>> params = [
        ['client_id', client_id],
        ['redirect_uri', redirect_uri],
        ['scope', scope],
        ['response_type', 'token'],
        ['approval_prompt', 'force']
    ];
    return authEndpoint + generateQueryString(params);
}

// Hopefully, this will one day be handled in a library

String generateQueryString(List<List<String>> params) {
  List<String> parts = new List<String>();
  for (List<String> part in params) {
    String k = encodeComponent(part[0]);
    String v = encodeComponent(part[1]);
    parts.add(Strings.join([k, v], '='));
  }
  return Strings.join(parts, '&');
}

// Taken from Uri.dart

String encodeComponent(String component) {
   if (component == null) return component;
   return component.replaceAll(':', '%3A')
                    .replaceAll('/', '%2F')
                    .replaceAll('?', '%3F')
                    .replaceAll('=', '%3D')
                    .replaceAll('&', '%26')
                    .replaceAll(' ', '%20');
}

Ok, so I'm sorry about that as it's mostly doing things that a decent URI library will handle, volunteers anyone? But we can see the parameters we will pass, and the important ones are the client_id, the scope, and the redirect_uri. The client_id and the redirect_uri should be configured in the Google API Console, and the scope is specific to whichever API(s) you will be using. In our case, we are using the Google Documents List API, with a scope of

https://docs.google.com/feeds/

Step 2: Open a popup window and redirect to the Authorization URI

We are now firmly in the realm of the DOM, and so once we have our redirect_uri, we open a window and send the client to it:

void redirectAuth(String uri) {
  window.open(uri, 'authwin', 'width=600,height=400');
}

Very simple, but the key will be how we handle the redirect back to us in the next stage.

Step 3: Receive the redirect and extract the OAuth2.0 Authorization token

Once we have been redirected back to our application, we should use window.location to extract the Authorization token. Remember that this is in the fragment, and once again we would hope that Dart eventually has libraries to handle this extraction, but in our case, I did something nasty like:

String extractToken() {
  // eww
  return window.location.hash.split('&')[0].split('=')[1];
}

Which probably won't work for 1000 edge cases, but you get the idea.

Once we have this token we need to somehow message it back to the parent window, which I was slightly puzzled how to do. I got some advice on Google+ and that was to use window.postMessage. It is a pretty nifty way of handling it, as the child window can access the parent as window.opener, and so the call simply becomes:

void sendTokenAndClose(String token) {
  window.opener.postMessage(token, '*');
   window.close();
}

Which needs to be caught by the parent window by registering an event handler for the message event:

window.on.message.add(tokenReceived, false);

void tokenReceived(MessageEvent e) {
  // the token is in e.data
}

We are now ready to use our token to make API calls, probably using JSONP which is the topic of another blog post, but you can see an idea on how to do it here in the Dart mailing list.

Running a subprocess with Dart and reading its output

2011-12-08T22:54:00.001Z

Starting a foreign program, and reading its standard output stream is useful, and here is an example of listing the root directory.

void main() {
  Process p = new Process('/bin/ls', ['/']);
  p.start();
  StringInputStream stdoutStream = new StringInputStream(p.stdout);
  print(stdoutStream.read());
  p.close();
}

Notes:

Argument list does not include the executable name like it usually does
Use a StringInputStream to read an InputStream as a String
Close the process or the script will hang forever

Comments on Google+.

Reading a file as a String in Dart

2011-12-08T14:41:00.001Z

It took me a minute to work out how to read a file in Dart (vm), so here it is for posterity:

void main() {
  File f = new File('my_filename.txt');
  FileInputStream fileStream = f.openInputStream();
  StringInputStream stringStream = new StringInputStream(fileStream);
  print(stringStream.read());
  fileStream.close();
}

Notes:

A StringInputStream reads strings from an InputStream
A FileInputStream is an implementor of InputStream that is opened from files
Remember to close the FileInputStream

Comments on Google+

Things I love about Dart (so far)

2011-11-24T16:10:00.001Z

I've only been using Dart for a few hours in total, but already there are things to love about it. Forget the obvious "It's not JavaScript", because I don't hate JS, and if I did hate JS it would be because of the lack of decent structure (who decided prototypes were nice to use?). Certainly, I don't hate dynamically typed languages, and in fact, I quite like them, and have built my career on them. This article is about small things, I'll leave that huge stuff for the clever people.

1. I already know Dart without having learned a single thing:

Ever used Java or C#? Dart is essentially the same as those familiar languages that we learned at school. A class is defined like:

class Banana {
  String getColor() {
      return "yellow";
  }
}

See, this could be any language you already know, and to illustrate that nicely the syntax highlighter treats it as Java, and no one really cares.

2. Class members are public by default:

If you define an instance variable like this:

class Banana {
  int age;
}

then anyone using an instance of the Banana class is free to read and write to that variable. "OMG you just broke encapsulation!". No, I didn't, you are free to define private variables if you like by prefixing them with the _ (underscore) character (not sure how much I love that syntax, but still). Who in the Java world decided that exposing an attribute is a bad thing to do? It's just part of the external interface of a class, use it as such, and move on with life having escaped 1. writing setAge and getAge, and 2. feeling dirty about using public.

3. Initializing and defining instance variables:

class Banana {
  int age, length, width, ends = 2;
}

Look how I have defined the types of all those variables and actually initialized the value of one of them to 2 — a banana has two ends, usually. These can be assigned to something called "constant expressions", which I am a bit fuzzy about.

There is more; how annoying is the constructor pattern where you have a constructor that takes a number of variables, only to assign them to instance attributes: this.age = age or self.age = age etc.? It must account for something like 5% of all my code. Construct and assign, construct and assign. Dart does this nicely, look here:

class Banana {
  int age;

  Banana(this.age);
}

The above class will assign the passed age to the instance variable on construction.

4. Top-level attributes and multiple classes per file:

I love these in Dart, but mostly because I hate the lack of them in Java. C# somehow gets away with having namespaces, but Python does fine, and Dart follows the Python model. Want 27,000 classes in a single file? be my guest. Want to have some constants defined at the top level of a module instead of in a useless static class, also be my guest.

5. Factory constructors:

These are constructors that actually return new instances. Something like Python's __new__ although that is slightly painful to use in Python, and not something you would wish on someone you like. Dart does it properly with the use of the factory keyword to indicate that a constructor will return a new instance. It is still a constructor so when creating an instance, the new keyword is used; which differs from using a static method to achieve the same thing, as you would in Java, look how:

class Banana {
  Banana._internal();

  factory Banana() {
    return new Banana._internal();
  }
}

The _internal constructor here is a private named constructor which the Banana constructor calls. This is great for doing nasty things like caching instances transparently from the caller of the API.

6. It's not JavaScript:

Just kidding, you know I can't resist trolling! But anyway, try Dart, it's fun, refreshing, safe, and as I said, you already know it. Read more about idiomatic dart if you are interested in the advisable ways of doing things.

Comments on Google+

How to install IronPython2 with Mono on Ubuntu

2009-05-23T20:26:00.009Z

This is just an instructional post with the hope that it might help others in the future.

I spent a long time trying to get IronPython2 running on Mono. Now IronPython1 ships with Ubuntu, so

apt-get install ironpython

, if that's all that you want. Ironically, I don't really know the difference between IronPython1 and IronPython2, but "2 has got to be better than 1", right?

The good news is that its very possible, but you will have to step out of your comfort zones.

You will need:
Mono 2.4 (sources) and IronPython 2.0.1 (binary)

First build Mono (note that you need Ubuntu's Mono to bootstrap, so

apt-get install mono

) but make sure you put it somewhere special so it doesn't mess with the system-installed Mono. This advice is taken from an article on the Mono wiki.


$ tar xvjf mono-2.4.tar.bz2
$ cd mono
$ mkdir -p ~/opt/mono
$ ./configure --prefix=~/opt/mono
$ make && make install

So this is putting Mono in opt/mono in my home directory (which is where I like to put things, but you might like them somewhere else, like /opt).

Then you will need to write a script. Again modified from the link above.


#!/bin/bash
MONO_PREFIX=~/opt/mono
export DYLD_LIBRARY_PATH=$MONO_PREFIX/lib:$DYLD_LIBRARY_PATH
export LD_LIBRARY_PATH=$MONO_PREFIX/lib:$LD_LIBRARY_PATH
export C_INCLUDE_PATH=$MONO_PREFIX/include
export ACLOCAL_PATH=$MONO_PREFIX/share/aclocal
export PKG_CONFIG_PATH=$MONO_PREFIX/lib/pkgconfig
PATH=$MONO_PREFIX/bin:$PATH
PS1="[mono] \w @ "

You will source this script to mutate your environment so you can use your nice new Mono, so say you saved it as ~/mono-2.4-env-activate.sh:


source ~/mono-2.4-env-activate.sh

And it will do its busines and change your prompt so you know it:


[mono] ~ @

And once you have that you can test it:


[mono] ~ @ which mono
/home/ali/opt/mono/bin/mono
[mono] ~ @ mono --version
Mono JIT compiler version 2.4 (tarball Sat May 23 01:13:11 BST 2009)

And once you are happy it works, you just need to execute the IronPython2 executable.


[mono] ~/tmp @ unzip IronPython-2.0.1-Bin.zip 
[mono] ~/tmp @ cd IronPython-2.0.1/
[mono] ~/tmp/IronPython-2.0.1 @ mono ipy.exe 
IronPython 2.0 (2.0.0.0) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>>

Looking good! And before you kill yourself trying to exit (it wants

Control-z

which is going to be fairly impossible):


>>> import sys
>>> sys.exit()
[mono] ~/tmp/IronPython-2.0.1 @

And there you have it. I am not sure exactly how I feel about IronPython on Mono, but it can't be a bad thing that it works. I will be investigating it more in the near future, so I may document what I am learning.

Quis experiet ipsas experientiae?

2009-02-25T20:34:00.006Z

Who will test those tests?

A while ago I was invited by the lynch-mob in #twisted to write a post about what I meant when I said that "Unit tests are not real tests", and here it is.

Now, recently there has been much said about Unit-testing, and code coverage, and even sensationalist blog post titles, which in retrospect "Unit tests are not real tests" would have been. I won't talk about code test coverage here, because its not relevant, we should strive to get as much coverage as we can. And of course I believe that unit tests are a fundamentally important piece of development. Especially since I tend to use Python, which is prone to disaster if not tested correctly.

So, if unit tests are real tests, then why did I say they aren't? I deal mostly with application development. The user of the application is a human being. Now quite simply, the only true test of my application is the user interacting with it. I understand now that this is not really the case for library development, where the user is a piece of code, and I guess you can have true tests for it that are unit tests.

So how do we do it? Well, I am not sure what kind of testing this is, but I call it "Acceptance Testing". It is evidence that the application behaves as advertised. I guess it would come under headings like "Verification". We have a list of written tests that go something like:

Test	Expected Outcome	Actual Outcome
User double-clicks the icon	log in screen displayed
User enters date in the future	text entry goes red and user is informed

And they go on and on, we have hundreds of pages of these. As I was in a start-up consisting of one person, I used to do these myself. Now the test department does it.

But these tests are just another type of unit test! Yes they are, and probably purists would disagree, but look at them:

You test every single unit of an application (but from the user perspective)
When you find a bug, you write a new test to test for it, then fix it
When you add a new feature, you write the tests first (TDD!)
Anyone can run the tests

And then as an added bonus you can give them to the client to run, which some of our clients insist on. They differ from traditional acceptance tests as they are run on single acceptance from the client.

Can't you automate these? Yes you can, but that would again slip them into the realms of "non-real tests". As I said, a human user has to test the application in order to reflect the actual user of the software. I really love automation. I have hacked on kiwi's automated test-runner, played with various other tools. Again, like traditional code-based unit tests, they are great tools.

So, I should really amend my statement to the #twisted guys to something more like:

"Code-based unit tests are not the true tests for an application that interacts with a human user."

Glashammer, an alternative framework on Google AppEngine

2008-12-22T10:39:00.011Z

In his blog, on Friday, August 29, 2008, Johnathan talked about Google AppEngine, and said

the "you can use any web framework you like, as long as it's django" attitude

That may have been true then, it probably isn't true now. Glashammer have been working for the last few weeks on getting the Glashammer framework of Werkzeug and Jinja2 running and easy on Appengine.

Firstly this wasn't hard. Glashammer is very free about what kind of data storage you use, so using Appengine's DataStore was straightforward. Some utility functions for running easily, add a few decorators for controlling what happens for authentication form redirection to limit views for certain users and we are pretty much done.

They have additionally tried to make it easier to install with a little script to get us started. So (since it seems a good start):


gh-admin quickstart_gae

This will generate a starter AppEngine + Glashammer application that is ready to go. It will also fetch all the dependencies for you and build them in the right places in your application for uploading to AppEngine. Now you can deal with that bit yourself with:


gh-admin quickstart_gae nodeps

You will be asked the name of the project directory, and this directory will be created. Within this directory, Glashammer will create components necessary for the application to be both a Glashammer application, and an AppEngine application.

These include an app.yaml with the necessary rules for static files, in the shared directory (handled by AppEngine) and the remaining paths to the WSGI Application built in main.py and the templates directory (handled by Glashammer).

At this point, I should probably just refer you to the respective documentations of Glashammer and AppEngine, but I can't resist a little teaser.

Let's have a look at the generated main.py module:


from glashammer import make_app
from glashammer.bundles import gae

TEMPLATES_DIRECTORY = 'templates'

# Main application setup
def setup(app):
    # add the gae init function
    app.add_setup(gae.setup_gae)

    # setup templates
    app.add_template_searchpath(TEMPLATES_DIRECTORY)

def main():
    gae.make_and_run_gae_app(setup)

if __name__ == '__main__':
    main()

That application doesn't actually do anything, because there are no rules for views, but we could add one:


from glashammer import Response

def do_home(request):
    return Response('Hello World')

and then add it in our application setup function:


def setup(app):
    ...
    app.add_url('/', 'main/index', do_home)

And we have a simple Hello World application.

You can use different components from Glashammer and Appengine, and sometimes you can even choose which bits you want from each. For example, I like using Glashammer's Sessions and Memcached interface, but AppEngine's Django forms.

The real advantages are in having Jinja2 along with hooks to add template filters and globals is ready to go, and Werkzeug with its lovely API. For me, this feels much nicer than the provided API in AppEngine.

There are no monkey-patches or other trickery and all dependencies run out of the box. And since it was released today, roll on over and grab it at http://glashammer.org/downloads.html.

What Twisted could learn from Kamaelia.

2008-10-04T13:17:00.004Z

Two awesome things: Twisted, and Kamaelia. I will not compare them. They are different, and have different purposes, and as I said. Both are great.

I have recently been playing with Kamaelia for the first time. If you don't know what Kamaelia is check out http://www.kamaelia.org/ but I admit it does take some getting my head around. (Just like Twisted did, back in the day when I first came across it.) But persevere.

Kamaelia is a library for creating highly concurrent applications. And along the way it has reminded me of: Erlang, Tasklets and Twisted. It has a highly componentized approach where components communicate with eachother using inboxes and message passing. That's all I'll say about it here. I am sure I will blog more as I use it more.

Now the first thing I want to do with Kamaelia is be able to hook it up with GTK, and I have managed to do that, but it was incredibly easy. Why? Because Kamaelia is happy to run its scheduler in the background in a non-main thread.


from Axon.background import background
background().start()

So the remainder of the gtk application can just run:


gtk.main() # yay!

Now Twisted has a solution for GTK, using the GTK reactor, and it works perfectly well, but look at the difference. Between the two:

Kamaelia is a library, Twisted is a framework.

bitbucket.org Dream Project Hosting?

2008-07-24T15:55:00.005Z

Well, I am always in search of new ways to host and serve code. Recently I have been using bitbucket.org's free service more and more.

The benefits as I see them are:

Mercurial hosting
Ability to branch and publish any number of branches (a la launchpad)
Plain web space for hosting generated documentation etc, with built-in wiki
User interface entirely geared around source code
I don't have to host it myself
Responsive and generous developers

Now if we were to compare this to some other things, we might have: (and please correct me if I am wrong).

	Mercurial	Publish many branches	Web space	Wiki	Hosted by someone else for free
Bitbucket.org	Yes	Yes	Yes	Yes	Yes
Launchpad.net	No	Yes	No	No	Yes
Google code	No	No	No	Yes	Yes
Self-hosted Trac	Yes	No	Yes	Yes	No

There are features that I would still really like, and the great developers are implementing them as we speak. The most exciting is the idea of an API where third-party applications could be written to do just about anything.

So Bitbucket.org, it's perfect for me, maybe it is perfect for you too.

The problem with jQuery

2008-07-06T00:56:00.000Z

Ok, I'll start by saying that I love jQuery as much as the next developer, and every contributor has my eternal thanks, but there is a problem.

Plugin development goes crazy, in that many similar plugins are released, and often these are just modifications including other sets of modifications. There must be a better way, like isn't this what version control is made for?

Here is an excerpt of http://docs.jquery.com/Plugins#Forms

* Autocomplete by Dylan Verheul
Autocomplete with caching to limit server requests and other options.

* jQuery Autocomplete Mod by Dan G. Switzer
Modification of Autocomplete plugin with enhancements and bug fixes.

* Modified Auto-complete by Anjesh Tuladhar
Auto-complete plugin extended for auto-completing multiple words in the same text input

* Autocomplete by Jörn Zaefferer
Heavily improved Dylan Verheul’s initial plugin, integrating modifications by Dan G. Switzer and Anjesh Tuladhar.

* jQuery Autocomplete by Saurabh Periwal
Modification of Autocomplete plugin with enhancements of onItemSelect and Multiselect.

* Jeditable + Autocomplete by Ritesh Agrawal
Extended jeditable to include option for autocomplete. Also extended Dylan Verheul's autocomplete javascript to include option for having input token separator. Check the demo.

End of snippet.

Blogger Comment Spam - Deleting it

2008-05-17T08:21:00.004Z

It seems over recent months that my blog gets comment spam. I imagine any bloggers out there experience the same thing and it is a bit of a pain.

I have three immediate problems with this and blogger.com.

1. Blogger doesn't notify me of all comments at the time they are posted. It notifies me of some, and I have of course configured it to notify me of all comments, but it seems to miss off about 70%. So not only do I not notice the spam, I also miss a bunch of legitimate comments. Please get it together Blogger! Ajax panel configuration is nice, but only if the core functions work.

2. Blogger should/could/might try to stop this spam before it happens. I am not guessing how, but then the company that runs Blogger.com are much brighter than me, and I am sure they have a solution.

3. The interface for browsing comments and deleting many at a time simply does not exist. This would make the task of sifting through, identifying, and delting spam much easier.

Now that I have had my grumble about it, I will offer my small solution. In praise of Google, they do provide a nice API and Python bindings to access all of their services and blogger is one of them. So I wrote a small script to go through all the comments, do a little bit of flagging on dodgy looking ones and offer you a chance of deleting them.

The script is uncommented, has no tests, and I don't plan in any way to maintain it or release it, but for those people suffering the same problems, I provide it here.

It is worth noting that the spam detection is really pathetic, and it could be vastly improved. I targetted it at my particular spam.

Full script available here


"""
(c) Ali Afshar 2008
MIT License
"""

import sys, getpass

from gdata import service


def get_details():
    email = raw_input('email: ').strip()
    password = getpass.getpass()
    return email, password


def create_service(email, password):
    blogger_service = service.GDataService(email, password)
    blogger_service.source = 'blogger_spam_killer'
    blogger_service.service = 'blogger'
    blogger_service.server = 'www.blogger.com'
    blogger_service.ProgrammaticLogin()
    return blogger_service


def get_all_blog_ids(svc):
    query = service.Query()
    query.feed = '/feeds/default/blogs'
    feed = svc.Get(query.ToUri())
    for entry in feed.entry:
        blog_id = entry.GetSelfLink().href.split("/")[-1]
        yield blog_id


def get_blog_comments(svc, blog_id):
    query = service.Query()
    query.feed = '/feeds/%s/comments/default' % blog_id
    query.max_results = sys.maxint
    feed = svc.Get(query.ToUri())
    for entry in feed.entry:
        yield entry


def get_all_comments(svc):
    for blog_id in get_all_blog_ids(svc):
        for comment in get_blog_comments(svc, blog_id):
            yield comment


def rank_comment(comment):
    words = 0
    for word in spamwords:
        words += comment.content.text.count(word)

    author = comment.author[0]
    has_uri = (author.uri is not None and
                # I figure no one who puts a URI would link to a blogger
                # profile. They would link to whatever they are spamming.
                'http://www.blogger.com/profile/' not in author.uri.text)
    print 'Spam words: %s' % words
    print 'Dodgy author uri: %s' % has_uri
    return bool(words) or has_uri


def delete_comment(svc, comment):
    svc.Delete(comment.GetEditLink().href)


def filter_all_comments(svc):
    for comment in get_all_comments(svc):
        print '--'
        t = comment.content.text
        print t[:70] + '...'
        print '...' + t[-70:]
        a = comment.author[0]
        print 'Author Info: ', a.name.text
        if rank_comment(comment):
            print '**** LOOKS DODGY'
        else:
            print '==== OK'
        s = raw_input('Delete? (y/N) ').strip()
        if s == 'y':
            print 'Deleting.'
            delete_comment(svc, comment)
        else:
            print 'Not deleting.'


# http://codex.wordpress.org/Spam_Words
spamwords = """
4u
adipex
advicer
...
""".strip().splitlines()


if __name__ == '__main__':
    em, pw = get_details()
    svc = create_service(em, pw)
    filter_all_comments(svc)

CouchDB with YUI Datatable

2007-12-17T22:42:00.001Z

I am not kidding, this must be the easiest way to get a ajaxy table of results from a database. For those who haven't heard the buzz, CouchDB is a document-based database. I am not an expert in databases so I won't really comment about CouchDB itself. One incredible feature is that the database server is queryable over HTTP (with REST) and responds with JSON, which seems perfect for writing pure Javascript applications.

YUI probably needs no introduction. I think its the best JavaScript library for maintainability, and documentation. It sometimes seems a bit verbose, but I like verbose.

Note: You will have to get around the fact that you cannot make XHR requests to a server other than the server that the page lives on, either with a reverse proxy setup, or by hosting the files themselves in CouchDB. (I won't explain that here).

So, a basic request to a database to return all the documents goes something like this:

a GET to:

http://myserver/dbname/_all_docs

And that will return a JSON data structure. In our case, each document has a "title" attribute and a "description" attribute, and the "rows" member of the return JSON contains a list of these.

Our database is called "tickets" and you could imagine it is a ticket tracking application.


function list_tickets() {
        var myDataSource = new YAHOO.util.DataSource("/tickets/_all_docs");
        myDataSource.responseType = YAHOO.util.DataSource.TYPE_JSON;
   
        // Define the data schema
        myDataSource.responseSchema = {
            resultsList: "rows", // Dot notation to results array
            fields: ["id", "title","description"] // Field names
        };

         // Define how it will appear in the table
        var myColumnDefs = [
            {key:"id", label:"Ticket ID"},
            {key:"title", label:"Ticket Title"},
            {key:"description", label:"Ticket Description"}
        ];

        var myDataTable = new YAHOO.widget.DataTable("table_holder", myColumnDefs, myDataSource);
    }

Just make sure you have a <div id="table_holder"> somewhere in your document, because this is what will be replaced by the data table.

That's really it. No web server application, no PHP (or Python or Ruby or Perl). Just javascript talking to your database. Incredible.

Using Storm and SQLite in Multithreaded Web Applications

2007-11-27T19:45:00.000Z

Pysqlite doesn't allow you to access the same connection from different threads. The pysqlite manual says: "SQLite connections/cursors can only safely be used in the same thread they were created in."

When using Storm (the ORM) with Werkzeug (the WSGI utility lib) we suffer from the problem that the Werkzeug reloader runs code in a thread. Ok this feature is not exactly important in a production environment, but I can't guarantee that whatever platform I will be deploying the application on will not be threaded, so database access should be proofed against this.

The solution? The Storm manual mentions that you should use a Store/connection per thread. Someone has already done this with the Middlestorm application, which provides a threadsafe store in the WSGI environ. Rightly or wrongly (since I really don't want to have to wait to have a WSGI environ to get the store instance), and I am not exactly sure this kind of thing should be middleware, but that is a debate for another day.

Looking at the code, it uses threading.local(), which is a thread-local attribute store. In other words, each thread will have its own values for the local object's attributes.

So a very simple implementation:



from threading import local

from storm.locals import create_database, store

class ThreadSafeStorePool(object):

 def __init__(self, uri):
     self._db = create_database(uri)
     self._local = local()

 def get(self):
     try:
         return self._local.store
     except AttributeError:
         self._local.store = Store(self._db)
         return self._local.store

Using an instance of this class, and calling get() when you require the thread-local store (which can be easily hidden behind a property descriptor ensures that each thread has its own Store, and pysqlite stops complaining.



uri = 'sqlite:test.sqlite'

store_pool = ThreadSafeStorePool(uri)

# From one thread
store = store_pool.get()
# Will always return the same store in that thread

# From another thread
store = store_pool.get()

Et cetera.

Leopard Spaces Is Unusable

2007-11-05T13:10:00.000Z

With Leopard, OS X finally has a bundled virtual desktop implementation. Pre-leopard I used a (now unmaintained) 3rd-party app called VirtueDesktops to get this feature, and while not perfect, it worked fairly well. It certainly didn't annoy me. Spaces, on the other hand, does annoy me, to the extent that I find it unusable.

Spaces SWITCHES MY DAMN DESKTOP WITHOUT ME ASKING IT TO. When I command-tab between applications, Spaces changes to the desktop with the active window. Now, at first when I encountered this feature, I thought "oh, that's pretty nice," and indeed it would be ok if it worked consistently. It doesn't, though. Usually Spaces switches correctly, but occasionally (say every 5 or 10 application-switches) it switches to the wrong desktop!

If you don't believe me, try this out: Open two terminals, and put them on different desktops. Now open a browser on one of these desktops. Command-tab between the browser and the terminal a few times, and see what happens.

This makes Spaces unusable for me. When I'm working on code, command-tabbing between the browser, the terminal and textmate, having my concentration broken by this frequent and frustrating bug is something I just can't put up with. So I've turned it off, and I'm either going to install virtuedesktops again or just learn to live without the luxury of virtual desktops.

Also, it doesn't play nicely with firefox (disappearing windows).

Desktop Widgets with PyGTK, Launchpad Remote Control

2007-11-05T02:53:00.001Z

"""
Desktop Widgets with PyGTK, Launchpad Remote Control

We shall today be looking at a dumb hack I glanced upon while foolishly
reading the list of GDK Constants in the PyGTK documentation. This hack sets
the type hint of a GTK Window instance (the thing that tells the window
manager how to treat it) as a desktop window, and hence embeds the thing in
the desktop.

It even works quite well:

The window and widgets appear in the desktop

The widgets are raised when the "Show desktop" command happens

The widgets appear on all desktops (in Gnome and XFCE4, but not in KDE)

The widgets appear fully functional

So I got all excited and tried to think of an awesome use-case as the first
plugin for a new kind of desktop widget framework. One which couldn't care
less about eye-candy, and focussed on truly functional and useful
desktop-based applets. As a friend commented while I was discussing the idea:
"I am often hacking away in emacs when I decide that I want to know what the
weather is like, but I am too lazy to look out the window, so I use my weather
desklet."

And that is the point, these desklet things are absolutely useless in my
opinion. I thought and thought and I came up with a handful of barely useful
things, which are too dumb to even mention. I would be grateful if anyone has
any ideas.

Maybe I am the problem. I don't use a desktop at all. I turn the feature off
in XFCE4 (my preferred desktop environment) and don't have to suffer icons and
right click and things like this. (Incidentally the hack works on noremal
desktops, even preserving the original desktop functionality - so you can
click on icons and see the weather at the same time.

So my lack of real-world use-case discouraged me from attempting the
framework. But I would feel bad not to mention it here at least.

A nice simple thought was to use Malone's XMLRPC to read any numbered bug from
the launchpad and display it on the desktop. As I say this isn't a real world
application, since you would just fire up a browser. And then I realised that
you can only use the Malone XMLRPC for filing bugs.

So I guess it will be a less pleasant thing that just opens your browser at
the bug page. Hopefully Launchpad will add simple things like this to the API
sooner rather than later.

So, on to the process:

First let's get the Launchpad bit out of the way


"""

# Python standard library module that does portable web-browser opening
import webbrowser
# You could make the product optional
BASE_URL = 'http://bugs.launchpad.net/pida/+bug/%s'

def navigate_bug(number):
  """
  Open the web browser at the required URL
  """
  url = BASE_URL % number
  webbrowser.open(url)

"""
Now we can concentrate on the more interesting desktop embedding
"""

import gtk

# Create the user interface for the application
class BugEntry(object):
"""
A miniature user interface for entering a bug number.
"""
def __init__(self):
   # an entry, a button, and a box to put them in
   hb = gtk.HBox()
   self.entry = gtk.Entry()
   hb.pack_start(self.entry)
   button = gtk.Button(stock=gtk.STOCK_OK)
   hb.pack_start(button, expand=False)

   # Connect a signal callback for the button
   button.connect('clicked', self.on_button_clicked)

   # A window to hold it all
   self.window = gtk.Window()

   # Set the type hint on the window to make the window manager think it
   # is a desktop
   self.window.set_type_hint(gtk.gdk.WINDOW_TYPE_HINT_DESKTOP)
   self.window.add(hb)

   # Don't forget to size and move your window where you want it
   self.window.move(100, 100)
   # If you wan to resize it: self.window.resize(x, y)
   self.window.show_all()

def on_button_clicked(self, button):
   """
   Called when the button is clicked
   """
   # A sane implementation would catch errors in the conversion,
   # But we just call the web browser launcher function:
   navigate_bug(self.entry.get_text())


if __name__ == '__main__':
   # Instantiate the interface, and run the gtk main loop
   be = BugEntry()
   gtk.main()

"""
Ok it looks really ugly, but you can see the promise there, The main thing is
that it really really works.

If anyone can think of a decent thing to do with this, please let me know!
"""

You are accessing this page from a forbidden country.

2007-10-25T18:24:00.000Z

A friend from Iran recently made me aware of something at Google code:


http://pida.googlecode.com/files/PIDA-0.5.1.tar.gz
You are accessing this page from a forbidden country.

It's not for me to judge right from wrong, and of course I am not as clever as the policy makers at Google so I just can't comment. I just want to make people aware of this so that they can make informed decisions about where they host their projects.

Pycon UK

2007-09-10T10:10:00.000Z

Just a quick note to say that Pycon UK was great fun, and hopefully a great success. A big "thank-you" to the organisers and speakers.

The talk I enjoyed the most was about Fluidinfo, mostly because I am a sucker for someone with a real vision. http://fluidinfo.com/

Hopefully, I managed to interest 1 or 2 people in PIDA with a little lightning talk. We'll see.

Ali

Using threads in PyGTK

2007-08-09T17:41:00.000Z

Well, all this thought of asynchronicity is getting to me. But since we
mentioned how to use subprocesses and Twisted, I think it is only fair to talk a
bit about threads.

So, what will we be doing?

Create an abstraction for running threads in PyGTK
Use it

Now the thing which we will be wanting to run is a generator, call it
"take_really_long". And it will surely block, and might be a bit intensive, and
well we obviously need something asynchronous. A generator is a bit nicer to use
than a single plain long-running function, because we can update the user
interface every time it returns, making it much much easier to do things like
Progress Bars.

You must remember two things when using threads with PyGTK:

1. GTK Threads must be initialised with gtk.gdk.threads_init:


import gtk
gtk.gdk.threads_init()

2. Any code that modifies the UI (basically any code) that is called from
outside the main thread must be pushed into the main thread and called
asynchronously in the main loop, with gobject.idle_add:


import gobject, gtk

def set_progress_bar_fraction(fraction):
   progress_bar.set_fraction(0.2)

gobject.idle_add(set_progress_bar_fraction, 0.2)
gtk.main()

This Gobject/GTK "idle_time" is something like "as soon as the main loop can
reasonably get around to it".

So, to reiterate, when writing threaded code, I expect to see lots of calls to
gobject.idle_add.

Here is a silly example, that I urge you not to use, as we will concentrate on
abstracting out the boilerplate later.


from threading import Thread
import time
import gtk, gobject
gtk.gdk.threads_init()

class MainWindow(gtk.Window):
   def __init__(self):
       super(MainWindow, self).__init__()
       vb = gtk.VBox()
       self.add(vb)
       self.progress_bar = gtk.ProgressBar()
       vb.pack_start(self.progress_bar)
       b = gtk.Button(stock=gtk.STOCK_OK)
       vb.pack_start(b)
       b.connect('clicked', self.on_button_clicked)
       self.show_all()

   def on_button_clicked(self, button):
       self.count_in_thread(5)

   def count_in_thread(self, maximum):
       Thread(target=self.count_up, args=(maximum,)).start()

   def count_up(self, maximum):
       for i in xrange(maximum):
           fraction = (i + 1) / float(maximum)
           time.sleep(1)
           gobject.idle_add(self.set_progress_bar_fraction, fraction)

   def set_progress_bar_fraction(self, fraction):
       self.progress_bar.set_fraction(fraction)

w = MainWindow()
gtk.main()

I say this is silly, but it is not really, it's a nearly a real-world example,
and its probably how you would do it. Now imagine an application where you had
to do this a lot, it would eventually wear you down.

So we can move on to making an abstraction that allows us to forget about this
boilerplate and concentrate on integrating it smoothly with your code. This will
take the form of a class that is responsible for:

starting a generator in a thread
calling a callback with every result (yield) of the generator
possibly calling a callback on completion

And here it is (modified from PIDA work by Tiago Cogumbrerio: Thanks Tiago!):


import threading, thread
import gobject, gtk

gtk.gdk.threads_init()


class GeneratorTask(object):

   def __init__(self, generator, loop_callback, complete_callback=None):
       self.generator = generator
       self.loop_callback = loop_callback
       self.complete_callback = complete_callback

   def _start(self, *args, **kwargs):
       self._stopped = False
       for ret in self.generator(*args, **kwargs):
           if self._stopped:
               thread.exit()
           gobject.idle_add(self._loop, ret)
       if self.complete_callback is not None:
           gobject.idle_add(self.complete_callback)

   def _loop(self, ret):
       if ret is None:
           ret = ()
       if not isinstance(ret, tuple):
           ret = (ret,)
       self.loop_callback(*ret)

   def start(self, *args, **kwargs):
       threading.Thread(target=self._start, args=args, kwargs=kwargs).start()

   def stop(self):
       self._stopped = True

So this does all of those things, you pass the generator, the callback for each
result and the callback for completion to the constructor, and the arguments to
call the generator with to the start method. This way you can call the same
GeneratorTask instance many times.

For an example of its use, lets modify our version above:



class MainWindow(gtk.Window):
   def __init__(self):
       super(MainWindow, self).__init__()
       vb = gtk.VBox()
       self.add(vb)
       self.progress_bar = gtk.ProgressBar()
       vb.pack_start(self.progress_bar)
       b = gtk.Button(stock=gtk.STOCK_OK)
       vb.pack_start(b)
       b.connect('clicked', self.on_button_clicked)
       self.show_all()

   def on_button_clicked(self, button):
       GeneratorTask(self.count_up,
                     self.set_progress_bar_fraction).start(5)

   def count_up(self, maximum):
       for i in xrange(maximum):
           fraction = (i + 1) / float(maximum)
           time.sleep(1)
           yield fraction

   def set_progress_bar_fraction(self, fraction):
       self.progress_bar.set_fraction(fraction)

w = MainWindow()
gtk.main()

One thing we have done is convert the count_up method to a generator so that we
can use it. You will notice that there is no real impact on the code that would make you
think "oh nasty, threads". The idle calls are taken care of for you, and the
generator task takes care of calling your UI function for each result.

Spawning subprocess with PyGTK using Twisted

2007-08-08T15:33:00.000Z

Well, it is an age-old problem: How to schedule long-running tasks withing a GUI main loop (in our case PyGTK). There are a few ways:

Use Python's subprocess module and select on the pipe with gobject's io_add_watch
Use GTK's built in subprocess spawning abilities
Use Twisted

1 & 2 are reasonable approaches, and they both work. Of course 1 won't work on Win32. The only problem with both 1 & 2 is that they use gobject's polling functions to achieve asynchronicity. This is nice when we are forced in a PyGTK main loop, but really not nice when the application wants to run in command line mode, and we really want to be able to share the execution code between different UIs, including perhaps other toolkits.

Enter Twisted. We need to do two things with Twisted:

Make sure Twisted knows we are running with PyGTK
Launch the process

Making sure Twisted knows that we are running inside PyGTK is quite easy (though I imagine the implementation was painful). To do this, you must install the gtk2reactor before importing any other reactors like so:


from twisted.internet import gtk2reactor
gtk2reactor.install()
from twisted.internet import reactor

Ok, I did say it was pretty easy. Now all that you need to remember with this is that you should now run your main loop with reactor.run, and not gtk.main.

Now we should think about spawning our subprocess. We will do this by using reactor.spawnProcess, which looks like:


twisted.internet.reactor.spawnProcess = spawnProcess(self,
    processProtocol,
    executable,
    args=(),
    env={},
    path=None,
    uid=None, gid=None, usePTY=0, childFDs=None
)

The only non-normal thing here is the processProtocol paramterer. All the other paramteres are standard things for things like subprocess.Popen. The processProtocol instance should be an instance of twisted.internet.protocol.ProcessProtocol, and defines how data is read from the pipes constructed to spawn the subprocess.

You can just use ProcessProtocol without overriding, but that will do nothing useful, not even print the results, so here is an example with our own ProcessProtocol class.


import os
# Have you remembered to install the gtk2reactor?
from twisted.internet import reactor
from twisted.internet.protocol import ProcessProtocol

class EchoingProcessProtocol(ProcessProtocol):

    # Will get called when the subprocess has data on stdout
    def outReceived(self, data):
        print 'STDOUT:', data
    
    # Will get called when the subprocess has data on stderr
    def errReceived(self, data):
        print 'STDERR:', data

    # Will get called when the subprocess starts
    def connectionMade(self):
        print 'Started running subprocess'

    # Will get called when the subprocess ends
    def processEnded(self, reason):
        print 'Completed running subprocess'

# Spawn the process and copy across the environment
reactor.spawnProcess(EchoingProcessProtocol(), 'ls', ['ls', '-al'], env=os.environ)
reactor.run()

Now the subprocess will execute, and anything written to the child's stdout will be printed to the screen.

This may seem entirely unremarkable, but this is now ready to plug into a GUI. Since the callback outReceived is called inside the gtk main loop, it won't block and it will be called when necessary, so it may as well do something like:


def outReceived(self, data):
    self.text_view.get_buffer().insert(
        self.text_view.get_buffer().get_end_iter(), data
    )

Which would add the line to a textview control (called self.text_view).

Links:
Twisted How-to Processes
Twisted How-to PYGTK

Side-by-side presentation and video

2007-08-06T19:00:00.000Z

Well, it was a nice idea I believe, to place a video side-by-side with a Powerpoint (or one of it's inferior Open Source clones) presentation. The idea came when my boss showed me Microsoft Producer, which does exactly this (it's nice to have people who at least pay some attention to non-Open Source, especially when he pays you.).

Producer is a plugin for Powerpoint, with a simple User Interface, which involves importing a presentation, and a video. Setting the timings for the slides, placing them adjacently in an output it video.

It sounds simple enough, and I thought this functionality would be possible in some of the Open Source video editing equipment, but either: I knew for a fact that the software didn't have the capability, or: the software was impossibly difficult to use.

So, I had a think, and a play, and another play, and I came up with the plan:

Split the presentation into images: 1 slide per file
Split the video into frames of png files, and save the sound for later
And for each frame:

Create a new image with the video frame, and the slide frame placed next to eachother.

Join all the frames to make a new video, with the sound.

Ok, that shouldn't be too easy! So I started to investigate tools. I knew of MPlayer, and Mencoder, (and actually they were capable of performing this task), but I had a few problems, and eventually I got hold of Ffmpeg, which is clearly the best tool for this job.

FFmpeg is a universal encoder-toolbox, and although it has most of the good functionality disabled in standard distributions, you can rebuild it to be able to deal with most files. When running Ffmpeg, you choose an input file, set some options for it, and choose an output file, and set options on it, and it converts between the two. Pretty amazing.

Stage 1: Converting a video into frames:

This is an example command:


ffmpeg -r 25 -i myvid.avi -r 25 -s 320x240 frames/%09d.png audio.wav

This splits myvid.avi into its frames and its audio track. The frame names are specified as the frames/%09d.png formatting string. In our case, we place the frames in a directory called frames.

Amazingly, the filenames is all Ffmpeg needs to work out what is what. It understands that if it is going to output to a filename like the format string for frames, that it should use the png encoder, and place the files there. It knows that because I also gave a wav filename, that the audio output should go there.

The -r option sets the frame rate. Setting -r before and after the input file is a bit weird, but the first -r applies to the input file, and the second -r applies to the output file (in this case the png frames). This ensures that the frames come out at the correct frame rate, so that we can be vaguely sensible about recombining them later. Essentially, you should decide at this point what you want your final frame rate to be, since from this point onwards you will be stuck with whatever you use.

Stage 2: Convert a presentation to a set of frames.

Well, using open office across the UNO bridge with Python was a fairly painful task, considering how simple it could be. Openoffice will export a slide to a Jpeg, and it does it quite reasonably, however, there seems no way in which you can tell it to export more than one slide, or even which slide to export. I hope I am wrong about this, and so can refactor this hideous hack:

Some pseudo code:


open the presentation and count pages
for each page:
open the presentation
delete all pages except the page we want
export the page

It's pretty awful, I know, and very slow, and when you add on to that the fact that you have to somehow spawn a headless Open Office server instance to communicate with, it can get a bit muddled. But stick to your guns.

Stage 3: Creating a new image from the frames

So you have the frames from the video in a directory, and you have one frame for each slide in a presentation. To create the new image, we will need to know which slide applies to which particular frame of the video. We need a list of timings, those are the times when the slides change.

Now, since we know our frame rate is 25 frames per second, we can calculate exactly which slide should be placed against which video frame, and we should do it. I used PIL (Python Imaging Library) for this task because it was the first library I thought of. I actually did a similar thing using ImageMagik, but not spawning a new process for each operation was better.

So, using PIL, and say we want a final video that is 640x320, with the presentation and video each occupying half of the picture, we should create a frame for each of the frames in the original video.


import os
import Image

for i, file_name in enumerate(os.listdir('frames')):
    frame_path = os.path.join('frames', file_name)
    new_image = Image.new('RGBA', (640, 320)) # Make a new image 640x320
    frame_time = i * 25 # Where are we in the video
    slide_path = get_slide_for_time(frame_time) # Get the frame for the time
    slide_image = Image.open(slide_path) # Open the slide file
    video_image = Image.open(frame_path) # Open the video frame file
    slide_image.resize((320, 240), Image.ANTIALIAS) # Resize to the target size
    video_image.resize((320, 240), Image.ANTIALIAS)
    new_image.paste(slide_image, (0, 0)) # Put each iamge on the new target image
    new_image.paste(video_image, (0, 320))
    new_image.save(os.path.join('output_frames', file_name)) # Save the file

Now, we left get_slide_for_time up to your imagination, but you can imagine. We open each file, and "paste" them onto the target file. Then we save the file with the same file name in a different directory. This will really help us when we are reencoding the whole thing, since it is the syntax ffmpeg already knows.

Stage 4: Reencoding the whole thing

So we have a directory of files which are the final frames, we have a framerate that we have stuck to, we have an audio track, and we want an output file.

So again using Ffmpeg:


ffmpeg -r 25 -i output_frames/%09d.png audio.wav -ar 44100 myvideo.flv

This time we specify two input files, one of which is actually a file mask, and it encodes them into the resulting video. Don't forget to tell it the frame rate of the frame images (that's the first parameter), and to set an audio rate. The output file we selected is a Flash Video, which is easy to play online.

Then what?

Then we write a script that does this for us. Which I have done, and it does everything I personally need it to. And you can see it at The Google Code Project Page. There is also a brief summary of how to use it there.

Thoughts

This could be a useful educational tool. The problem of recording lectures with presentations when you have to zoom the camera in on the screen is just nasty.

Edit: the world needs more screenshots, a variation on what we can do:

Running a WSGI Application inside Twisted

2007-07-26T09:41:00.000Z

There are a few ways to run a WSGI application, and today I will comment on the Twisted approach. The magic happens from the twisted.web2.wsgi module, and is fortunately simple.

The main advantages I can think of when running with Twisted/Twistd is that:

All Python solution (if that really is an advantage)
Ability to run other twisted services. Did you want RPC, Email, FTP, remote shell etc for free?
Twisted authorization framework

Since the application in question has a very simple web part, and is predominately a remote GUI application, this seems the perfect solution.

First make a tac file, which you will be running with the old favourite:


twistd -ny mytac.tac

On to some code:


# This is my process for creating my WSGI application. You may have other methods.
from graelsite.www.application import make_app
from graelsite.www import config
wsgi_app = make_app(config)

# Now on to the real stuff
from twisted.application import service, strports
from twisted.web2 import server, channel, wsgi

application = service.Application('web2-wsgi') # call this anything you like
site = server.Site(wsgi.WSGIResource(wsgi_app))
s = strports.service('tcp:9876', channel.HTTPFactory(site))
s.setServiceParent(application)

And I honestly think that is the easiest way to deploy a WSGI application (outside the safety of Cherry, Paster, Werkzeug, etc.).

Note: Werkzeug is excellent, use it!

PyGTK, Py2exe, and Inno setup for single-file Windows installers

2007-07-16T02:20:00.000Z

I am lucky, as part of my job, I get to code PyGTK. The downside is of course
that I have to deploy these applications on Windows. Now these Windows users
(especially at the price they are paying for this stuff) don't install
dependencies, don't understand what a scripting language or interpreter is,
and frankly they should not have to.

Unfortunately, the list of dependencies for a PyGTK application (at the very
minimum) is Python, GTK, PyGTK, PyGObject, PyCairo. 5 installers! So very
early on in this project I found the way to give them exactly what they want
and deserve, a single-file executable installer. There are a few guides online
about how to achieve this, but none seem to work, and none are particularly
new.

How do we achieve this? The toolchain involved consists of two elements:

1. Py2exe website
2. Inno Setup website

NOTE: The documentation for all the tools used is extensive, I will not repeat everything in there.

Py2exe

Py2exe is a distutils extension that searches for Python modules that your
application uses, and other dependent libraries, and copies them all into your
dist/ directory. The pure Python stuff gets zipped up, and the DLLs (Windows
remember) get copied into the directory.

The dist directory would be runnable from say a CDRom at this stage, but of
course this is not enough, we need a single file installer (more later).
Making py2exe work is a bit of an art. There is lots of documentation at the
website linked above, so I don't need to repeat that. What I will do is paste
my setup.py file so that it can be copied if required (just don't tell my
employer).


setup(
name='myapp',
version='1.3',
packages=['myapplib'],
scripts=['myapp.py'],
windows=[
{
  'script': 'myapp.py',
  'icon_resources': [(1, 'pixmaps/cdm.ico')],
}
],
data_files=[
('data', [
  'data/cd_db.csv'
  ]
),
('glade', [
  'glade/main.glade',
  'glade/patient_editor.glade',
  'glade/prescriber_editor.glade',
  'glade/report_chooser.glade',
  'glade/supplier_editor.glade',
  'glade/user_editor.glade',
  'glade/pharmacy_editor.glade',
  ]
),
('pixmaps', [
  'tools/kiwi_requirements/validation-error-16.png',
  'tools/kiwi_requirements/plus.png',
  'pixmaps/s2icon16.png',
  'pixmaps/s2icon24.png',
  'pixmaps/s2icon32.png',
  'pixmaps/s2icon48.png',
  ]
),
('catalogs', ['tools/kiwi_requirements/dummy.txt']),
('plugins', ['tools/kiwi_requirements/dummy.txt']),
('resources', ['tools/kiwi_requirements/dummy.txt']),
('pixmaps/kiwi', ['tools/kiwi_requirements/dummy.txt']),
('backups', ['tools/kiwi_requirements/backups_go_here.txt']),
],
options = {
'py2exe' : {
  'packages': 'encodings, sqlalchemy',
  'includes': 'cairo, pango, pangocairo, atk, gobject',
},
'sdist': {
  'formats': 'zip',
}
}
)

Ok, I changed the name of the application. "myapplib" is the name of the
package for the application, and the top-level script that runs is called
"myapp.py".

Note: there are some specific things in there for packaging Kiwi. That is
probably the hardest task (but is not relevant here).

Now there are two tricky things about using py2exe, and they are dependencies,
and data files (the rest is just basic setup.py stuff). Dependencies can be
explicitly set in options['py2exe'], and this is the magical bit. It *should*
find these automatically I think, but sometimes it just doesn't. Sometimes
putting something in "packages" works, but putting it in "includes" doesn't,
and I have to admit that I am not well enough versed with py2exe and the
documentation to know the difference. So, if you want it to work, I would just
copy what I have above, and save yourself the time and pain of making it work.

Data files are the next problem. Eventually you will want this data file
copied firstly to your dist directory, and then secondly when building the
installer into the installer file, and then from there somewhere into Program
Files, where your application will find it. So the above data_files list
demonstrates how I do this.

These data files will be copied into the dist/ directory, and then will be available (once installed) to your application in the Program Files\MY_APP_NAME directory, which should be easy enough to find. Examples are available on request for interested people.

Inno Setup

Inno setup is an application that packages any sort of distribution of files
into a single installer and unpacks it wherever you want. There are oodles of
options, and it does a pretty good job, and it also has pretty good
documentation that will get you on your way.

The important line from my Inno Setup config file is this:


[Files]
Source: "..\dist\*"; DestDir: "{app}"; Flags: ignoreversion recursesubdirs createallsubdirs

This basically just dumps the entire dist directory into the package, and will
dump it back out in a similar structure on the other side, which is exactly
what we want.

GTK Data files

It is not sufficient to just make py2exe pull in the GTK DLLs for packaging
(which it does pretty successfully). GTK also requires a number of data files
which include themes, translations etc. These will need to be manually copied
into the dist directory so that the application can find them when being run.

If you look inside your GTK runtime directory (usually something like c:\GTK\)
you will find the directories: share, etc, lib. You will need to copy all of
these into the dist directory after running py2exe.

You can prune some of these directories. My main pruning action comes by
removing all the non-English translations (which saves a few megabytes in the
final bundle).

Since I am in to automation (as we will discuss later) I have zipped this lot
up and unzip it from a location after running py2exe.

Putting it all together

Now that we have discussed the tools used, and some specific details about
them, I shall present the specific process involved in the build on a clean Windows system:

Install GTK runtime
Install Python
Install all the dependent Python packages
Run py2exe
Copy GTK data files
Run Inno Setup
Install the application

Wow, that is a lot of steps, especially when you change one line of code on the day before a release. The solution is to use some scripts for automation.

The first "script" will provide a source bundle with everything I need in it to upload onto the Windows computer (since I develop everything on Linux). Luckily distutils has such a thing, and we can use the sdist command to setup.py to achieve it, as long as you have correctly specified the MANIFEST.in file. For added bonus points, you should (and notice setup.py above) make sure you are bundling into a zip, not a tar.gz (unless you have ways of unpacking the tar on Windows). This can be achieved with a command line flag --formats or in the setup.py options dict, or even in a setup.cfg file.

I manually unzip the thing on the other side.

The second script will install all the requirements: GTK, Python etc. To achieve this I have a directory in my source distribution that contains all the Windows binary installers. I had to build some of them (for Python modules) but this is as easy as:

python setup.py bdist_wininst

I change to the directory of installers and I run a batch script. Batch is a pretty evil language once you have used something as nice as Bash(!), but for our purposes it is simple. Here are the contents of that batch file, which lives in the same directory as all the installers:


call gtk-2.10.11-win32-1.exe
call python-2.5.1.msi
call pycairo-1.2.6-1.win32-py2.5.exe
call pygobject-2.12.3-1.win32-py2.5.exe
call pygtk-2.10.4-1.win32-py2.5.exe
call kiwi-1.9.15.win32.exe
call gazpacho-0.7.0.win32.exe
call py2exe-0.6.6.win32-py2.5.exe
call SQLAlchemy-0.3.4.win32.exe
call isetup-5.1.10.exe

There is some [ Ok ] clicking to be done, this won't run unattended, but this script alone has saved me hours in total. Perhaps there are flags for unattended install of these things.

Once the dependency installers are installed, I can set to creating the installer package which are steps 4,5,6 above. Again I automated these with a script. This time the script is in Python, because now we have Python installed! Here it is:


import os, sys, shutil, zipfile

BASE_DIR = os.path.dirname(__file__)
OUT_DIR = os.path.join(BASE_DIR, 'dist')
GTK_ZIP = os.path.join('tools', 'windows', 'gtk', 'gtk_to_copy.zip')
INNO_SCRIPT = os.path.join('tools', 'config-inno.iss')
INNO_EXECUTABLE = '"c:\\Program Files\\Inno Setup 5\\ISCC.exe"'


class Unzip(object):
   def __init__(self, from_file, to_dir, verbose = False):
       """Removed for brevity, you can find a recipe similar to this in the Cook Book"""


def delete_old_out_dir():
   if os.path.exists(OUT_DIR):
     shutil.rmtree(OUT_DIR)


def run_py2exe():
   # A hack really, but remember setup.py will run on import
   sys.argv.append('py2exe')
   import setup


def unzip_gtk():
   Unzip(GTK_ZIP, OUT_DIR)


def run_inno():
   os.system(INNO_EXECUTABLE + " " + INNO_SCRIPT)


def main():
  #Clean any mess we previously made
  delete_old_out_dir()
  # run py2exe
  run_py2exe()
  # put the GTK data files in the dist directory
  unzip_gtk()
  # build the single file installer
  run_inno()
  # prevent the windows command prompt from just closing
  raw_input('Done..')

if __name__ == '__main__':
  main()

That produces an installer in the top-level source directory which has the name of the value of the config key setup/OutputBaseFilename, which you can then test to install.

I advise that you remove GTK, Python and all the dependent modules before installing, just so you can be sure that it will work on a clean Windows system.

You may think that including GTK, Python and a whole load of other stuff might make for a really heavy distribution. Actually it's not, it's pretty light. For the application I am referring to we come out at just under 6 megabytes, 11 if you include all the translations. Granted, it is a huge amount when you consider how much non-library code we are using, but by today's standards, no one is going to be upset about having to download that.

Please drop me a line if you have any queries.

Possible future post: "Making Kiwi work with py2exe"

Latest PIDA Release - Explicit is better than implicit

2007-06-30T09:06:00.000Z

Well, congratulations to all the developers, we have just released PIDA 0.5. This release is probably the most significant in that we are finally happy with the core architecture.

Although not everyone was agreed on how to do it, I was quite strict and employed this famous rule of thumb:

Explicit is better than implicit

And honestly, it really is. Previous versions of PIDA were dogged by overuse of magic and abuse of the declarative class syntax (and the associated metaclass madness). This caused one major problem. Things that should have been happening at runtime were happening at class declaration time. Which is fine if you can get your brain around it, but really evil when you want to reverse these things.

As an example, PIDA services (and a service is like a plugin) can basically do anything to the application (a bit like the Eclipse plugin architecture - but we think better!). One example of this "doing anything" is to define global configuration options.

In PIDA 0.4 options where defined something like:


class MyService:
class Options:
    class Opt1:
        """Opt1 documentation"""
        type = STRING
        default = 'Opt1'

When the service was instantiated, some metaclass magic would convert the option definition to options objects which could be used elsewhere in the application. This kind of declarative style has been popularised with ORM's like SQLObject, Django and extensions for SQLAlchemy, and is pretty nice to use and read.

The problem with it is that once options have been created and magically sent off into the ether, they are there, and removing them involves some similarly evil trickery. For this reason services were almost impossible to load at runtime and moreover almost impossible to unload at runtime.

So for 0.5 we did away with the class declarative syntax (with some annoyance from some of the developers) and replaced it with something more like:


self.add_option(
name='Opt 1',
type=STRING,
default='Opt1',
doc = _('Documentation for opt1')
)

Ok, so this is a bit more annoying to write, but I think one can get used to it easily enough. Clever readers will notice that the documentation string is now translatable. And the major bonus is that because this is not happening in the ether, self.remove_option() is an obvious possibility.

Unfortunately, implementing this for the various pluggable components of PIDA: Options, Events, Commands, Actions, Contexts, Views required a complete rewrite. They say never rewrite an application, but we have done, and with great success.

So, since this blog is just an obvious advertisement for a release, please visit the website at http://pida.co.uk/, and download the latest version. The feature-improvement is huge, and we are one step closer to bringing the world the One True IDE (tm). We also have Emacs support in this release which should appeal to some people.

Custom PyGTK Widgets in Glade3: Part 2: Custom widget adaptors

2007-03-21T23:39:00.000Z

In the last post about custom glade widgets, we briefly discussed how to add your PyGTK custom widgets to glade-3 so that they can be used in a user interface designer. In this post, this shall be extended to include how you can create a custom adaptor for a PyGTK widget to define additional behaviour.

We shall be using a custom widget as an example, which I have called a "Service View". We plan that this widget has a content section, which is the main part, and also contains a close button at the bottom. We will use this widget as a general dockable view, kind of like a dialog which is a widget rather than a top-level window. This may seem utterly pointless, but it should be a useful component in an application that generates many different views in notebooks, like an IDE: Debugger, Terminal, Documentation Browser (etc). These can all share the same basic layout, but just replace the single main part of the widget.

First let's write a widget:


import gtk

class ServiceView(gtk.VBox):

__gtype_name__ = 'ServiceView'

def __init__(self):
   self.frame = gtk.Frame()
   self.pack_start(self.frame)
   self._bb = gtk.HButtonBox()
   self._bb.set_layout(gtk.BUTTONBOX_END)
   self._bb.pack_start(gtk.Button(stock=gtk.STOCK_CLOSE))
   self.pack_start(self._bb, expand=False)

So, in this widget, we want to be able to add a single child component inside the frame, self.frame, and also to be able to set the label of self.frame.

If we add this widget to Glade3 using a catalog (in pywidgets.xml) such as:


<?xml version="1.0" encoding="UTF-8"?>
<glade-catalog name="pywidgets" library="pywidgets"
domain="glade-3" depends="gtk+" language="python">
<glade-widget-classes>
<glade-widget-class
title="Service View"  
name="ServiceView"      
generic-name="serviceview">
</glade-widget-class>
</glade-widget-classes>
<glade-widget-group name="PythonWidgets"
title="Python Widgets">
<glade-widget-class-ref
name="ServiceView"/>
</glade-widget-group>
</glade-catalog>

and support code (in pywidgets.py) which simply imports the widget like so:


from serviceview import ServiceView

(where the module serviceview.py is in the python path, and contains the widget above)

Install the catalog and module file in the right places, and fire up glade-3. Create a window, and add our custom widget to it. Immediately you will be queried with an input dialog as to the required size of the widget. Once you have answered this, the widget will be created.

Problems? Yes, it doesn't work as we would like in the following ways:

The query asking us the size of the widget
The grey extra sections which you can add widgets to outside our frame
The inability to change the contents of the frame, and its label

All these problems stem from the same issue, that our widget is a subclass of gtk.VBox, and so will be treated as a VBox for the extent of its lifetime.

So how can we fix these things? We need to do two things. Firstly override some properties in the catalog file, and secondly create a custom adaptor. (We might one day be able to do all these things from the adaptor, but this is not possible at the moment, and it is one problem highlighted by the glade-3 developers, that they have a "hybrid API".)

Fixing the catalog

Don't forget to read the excellent documentation at this point. Widgets in glade, and in the default catalog offer the ability to launch input windows for queries. This is not entirely necessary, as a sane default could be chosen, and then changed as required in the normal way, but it could be useful in some circumstances I guess. In this case, because we are using a subclass of the Box widget, it will ask us what size we would like it to be. Size is not a real property of the Box, it is a virtual property, and helps Glade know how to display it. If we look at this part of the default catalog, we can see this happening:


<glade-widget-class name="GtkBox" fixed="True"
title="Box">
...
<properties>
<property save="False" query="True" id="size" default="3"
name="Number of items">
<spec>glade_standard_int_spec</spec>
<tooltip>The number of items in the
box</tooltip>
<verify-function>glade_gtk_box_verify_size</verify-function>
</property>
</properties>
...
</glade-widget-class>

Each property tag inside a properties tag inside glade-widget-class tag must have the id attribute, but the rest shown here are optional. The important one for us here is query="True" which causes the query box to be shown, and so we can replace that in our own catalog in a sort of a property "override". Note that save="False" is set on the property, because it is not a real property, only a virtual property, and so it will not be written to the glade XML file.

Not only do we want to disable the query box, but we want to:

Set the default value of this to zero, so none of the grey areas are shown and no extra widgets can be added to the Box
Hide or disable the size widget

Again looking at the documentation, we discover that the default value can be set with default="0" and the property can eithe be disabled with disabled="True" or hidden with visible="False".

So the class definition of our catalog becomes:


<glade-widget-class title="Service View"      
name="ServiceView"
generic-name="serviceview">
<properties>
<property id="size" query="False" default="0" visible="False"
/>
</properties>
</glade-widget-class>

Reinstall the catalog, create a window, and add our widget. We see that we are no longer asked for the starting size, and the size value does not appear in our property list. There are a few properties of the Box left that we don't really want here, and removing them is left as an exercise for the reader, and we shall move on.

Using a Custom Adaptor

The adaptor class should appear in your support code module, and should be referenced in your catalog, for example:


<glade-widget-class title="Service View" name="ServiceView" adaptor="ServiceViewAdaptor">
</glade-widget-class>

Which references a Class that registers the __gtype_name__ of ServiceView Adaptor. The class should inherit from glade.get_adaptor_for_type('SomeGTKType'), and for our case of a VBox subclass, we could use glade.get_adaptor_for_type('GtkVBox') as our adaptor superclass (which it was automatically using itself), like so:


import glade

class ServiceViewAdaptor(glade.get_adaptor_for_type('GtkVBox')):
   __gtype_name__ = 'ServiceViewAdaptor'

In the adaptor code, glade is a special Python module only available within runtime glade, and it contains the glade user interface bindings. You can get it's documentation at runtime if you want to have a look, but we will be playing with this stuff in subsequent posts.

So far we have not overridden anything, and our running code should behave exactly the same as it was previously, but with our named custom adaptor (which doesn't do anything).

Now we can start adding methods to our custom adaptor. In general, the available methods that will be called are of the form:


do_something(self, to_object, *args)

Where something is the action, and to_object is the created object in the user interface. An example of this will be our first method:


def do_post_create(self, obj, reason):
   # Create one of those nice-looking grey areas from Glade that mean that you can add   something to this widget
   obj.frame.add(gobject.new('GladePlaceholder'))

At this stage, if you like, you can install your catalog and support code, and test this widget out. You will see that the grey area appears correctly inside our frame. Now try to add something to it, like a label. You will notice that the label is just added to the end of the VBox, where we really don't want it. For this to be fixed we need to override our next few methods, the child methods:


def do_get_children(self, obj):
   """Called when glade wants the children of a widget"""
   return obj.frame.get_children()

def do_add(self, obj, child):
   """Called to add a child to the widget"""
   obj.frame.add(child)

def do_remove(self, sv, child):
   """Called to remove a child from our widget"""
   obj.frame.remove(child)

def do_replace_child(self, obj, old, new):
   """
   Called to replace the child of our widget.
   Note that this is what is called by glade when you delete,
   or cut a widget since it actually replaces the widget with a
   GladePlaceholder
   """
   obj.frame.remove(old)
   obj.frame.add(new)

The implementation of these methods is particularly straight-forward, and they achieve our goals. The trick is knowing what does what. The get_children method needs to be in place so that the Glade widget hierarchy appears correctly with the child widget. The do_add, do_remove, and do_replace methods are for exactly those things, adding, removing and replacing widgets.

If you install your catalog and support module once more, you will notice that now it actually works. The widget is added in the correct place, and the hierarchy appears correctly with a single child on your widget. You will have a few warnings from GTK about child properties, and these are silenced by writing child_property methods:


def do_child_get_property(self, obj, child, prop):
   """Called to retrieve a property value"""
   if prop in ['expand', 'fill']:
       return True
   elif prop == 'padding':
       return 0
   elif prop == 'position':
       return 0
   elif prop == 'pack-type':
       return gtk.PACK_START
   return True

def do_child_set_property(self, obj, child, prop, val):
   """Called to set a property value"""
   if prop in ['expand', 'fill', 'padding', 'pack-type']:
       pass

Because our widget is behaving like a real container (although just via the adaptor), it will get queried as to how to set child properties. Most of these are not important for us, because a Frame has very little in the way of packing and expanding, and since it is a Bin, it has no spacing. So we can just return simple values. Likewise we don't need to set anything there.

Trying It Out

So now we have our fully functional widget in Glade. Try it out. Make a new project, add our widget to a window, add an example widget to the container and then load it up with some code such as (for a file called 'test.glade'):


import gtk
import gtk.glade

# Make sure this type is registered with gobject so glade can load it
from serviceview import ServiceView

gtk.glade.XML('test.glade').get_widget('window1').show_all()
gtk.main()

How did it look? Terrible! Yes, the glade loader has put our child widget inside the VBox, rather than inside the Frame! Disaster! Except when you realise what the glade loader is doing. It is calling a method on the ServiceView object to add children.

Because it is a VBox subclass, this method behaves as it would for VBoxes (to add an object to the end). There is a slight complication. The add method that we have available in a gtk.VBox in Python is not what Glade is using. It actually uses the method that Glade uses too. To override the method that Glade uses, we need to implement the method do_add which is kind of like a way of saying "override the C add".

Once we have implemented this like so:


def do_add(self, child):
  self.frame.add(child)

The widget loads with the child in the correct place.

Discussion

For those who wish to play around with this system, you might try (now that you have implemented do_add in the widget itself, to remove the do_add method of the custom adaptor. And you will see that this does in fact work as we would want. You could furthermore use the GtkFrame adaptor as the superclass for our adaptor, and see that that has the correct effect.

This leads to a general question as to where you should implement your widget functionality, in your Widget, or in your Adaptor. We could override the entire Container Interface in our widget, and it would behave correctly without an adaptor, but that has the disadvantage of not being able to behave like a VBox any more.

In general the course you decide should be based on your own requirements, but it seems that at them moment, with Glade3 you will need a blend of:

Catalog for Glade3
Custom Adaptor for Glade3
Widget Methods for the Glade Loader

Still to come in this series of posts:

Custom widget editors
Controlling the Glade3 UI from Python

NB: The API is early and changing at this stage, and there is no emphasis on backward-compatibility, so be prepared to throw any code away!

Custom PyGTK Widgets in Glade3

2007-03-21T17:01:00.000Z

Glade 3, GTK User Interface Designer is really quite nice since release 3.1, since it does away with the gimp-like multi-window view and looks like a normal application. Another nice feature it has is being able to support widgets created in non-C (in our case PyGTK).

Components of a plugin:

A catalog file
A support module
Some icon pixmaps (optional)

We shall use, for our example, the Kiwi Hyperlink widget, (which I wrote). This widget is an EventBox subclass, so we shall illustrate turning off some of the additional properties that are not required in the user interface designer.

The catalog file

This is described in http://glade.gnome.org/docs/catalogintro.html and the subsequent pages of the documentation. Essentially there are two sections of the catalog:

The Widget definitions
The Catalog list

The Widget Definitions are like so:


<glade-widget-classes>
<glade-widget-class title="Hyper Link" name="HyperLink" />
</glade-widget-classes>

These are the most-often used attributes for the widget class, but there are a few others too that are outlined on the documentation page.

The catalog listing is much more basic, and less variable, and takes the following form:


<glade-widget-group name="KiwiWidgets" title="Kiwi Widgets">
 <glade-widget-class-ref name="HyperLink"/>

This lists all the widgets that are currently available. So far we have not strayed from the documentation, and this is exactly how you would add widgets that are non-PyGTK.

What you have to do to make glade realise that you are using PyGTK widgets is use this in your main document element for the catalog:


<glade-catalog name="kiwiwidgets" library="kiwiwidgets" domain="glade-3" depends="gtk+" language="python">

The language="python" does that for you, and in addition the library attribute "kiwiwidgets" defines the location of our support library, which will be called kiwiwidgets.py.

The catalog file should live with the other catalogs in the glade3 catalog directory, which is, on my system, at:

/usr/local/share/glade3/catalogs/kiwiwidgets.xml

You can turn various properties off from being displayed in the user interface designer using the <properties> tag in your widget-class tag. An example of this would be:


<property id="visible-window"
        visible="False" />
<property id="text"
        name="Text"
        translatable="True" />

Ok, so we showed you two properties here. The first is a property of Event Box which we don't want shown inside the UI Designer (users don't care), and that is achieved with a simple visible="False". The second shows how to modify a property. We want the text of this hyperlink widget to be translatable, and access all the translation hooks in Glade with it.

The support module

The support module is python code that goes with our widgets. In our basic example, we have not defined anything fancy like a custom adaptor, or even a custom editor which we could if we were feeling brave, but will probably leave that for another blog posting.

The support module should live in the glade module directory, and be named as defined in the catalog file. On my system this belongs here:


/usr/local/lib/glade3/modules/kiwiwidgets.py

the code of the module in this simple case contains only one thing, an import statement on the HyperLink class (since that is all that is needed to register it as a GObject Type.


from kiwi.ui.hyperlink import HyperLink

Glade3 takes care of the rest for us.

That is all for now. Other items that should come up later include:

Defining custom adaptors
Defining custom widget editors
Silencing glade queries for box subclasses
Controlling the glade UI from Python

And I shall address these if there is any interest.

NB: The API is early and changing at this stage, and there is no emphasis on backward-compatibility, so be prepared to throw any code away!

First Blog Post from the gblogger GUI

2007-03-19T03:05:00.001Z

The gblogger GUI lives! It features listing the blogs for a user, listing the posts for a blog, new blog posting, and blog editing. Wysiwyg is achieved using GTKHTML3, and I have barely formatted this post because formatted posts seem to give 400 posting errors!

The client lacks category management, and image uploading, which remain on the TODO list.

(Now I am tentatively pressing the save button...after which I will re-edit the blog in Blogger.com, and add some screenshots!)

Posting to Blogger.com from the command line

2007-03-16T15:18:00.000Z

Well, you can tell I am procrastinating working today (but you can forgive me after the PIDA GSoc Application was rejected as a mentoring organisation), but I have hacked up an early version of a library (for Python) to manage blogger.com posting.

I was becoming slightly upset with Blogger.com, since it does some insane things like puts BR tags in the middle of PRE elements, which meant the syntax highlighting script I wanted to use was going nuts. I also gradually realised that none of the blogging clients work with the "new Blogger.com" although for me (who has only been blogging a few months) it has been the "only Blogger.com".

So, with a quick search around for documentation from code.google.com, I was able to discover that Blogger.com uses the google GData API with a bit of Atom mixed in. I have no real ideas about these specifications, but http://code.google.com/apis/blogger/gdata.html explains it. Note that another document I found: http://code.blogspot.com/archives/atom-docs.html seems depracated, but I can't confirm that at all.

Python is my language of choice for these things, so I started hacking.

Step 1: Authenticate with Google Login stuff. This is explained in http://code.google.com/apis/accounts/AuthForInstalledApps.html but basically you need to:

Send a post request to https://www.google.com/accounts/ClientLogin
Set the service value of the data to "blogger" (this had me for a while)
Parse the body of the response for an "Auth" token
Include the header Authorization: GoogleLogin auth=yourAuthValue in all your requests from thereon in (not obvious from the authentication docs)

Once you have successfully logged in you will want to get some information. Firstly the posting API works on the blog ID, not the blog name. The ID appears in the source of the blog page in this tag <link rel="service.post" type="application/atom+xml" ... but you can also fetch it from the blog's feed from the blog's name (see the API documentation).

Once you have this ID, posting is a matter of using the ID with your auth token, and entering data in an XML format as the post's body. Not forgetting to set the correct content-type for your post to "application/atom+xml". If you have posted correctly, the server returns a 201 Created response.

I have spared you the boring details of making urllib/urllib2 work with these things, but I have written some sample code and a command line client which you can browse online here: http://pygbloggerlib.googlecode.com/svn/trunk/gblogger/client.py as well as accessing the googlecode page (no releases yet) here: http://code.google.com/p/pygbloggerlib/

An example:


>>> from gblogger.client import BlogClient
>>> import getpass
>>> passwd = getpass.getpass()
Password:
>>> c = BlogClient('my-test-atom-blog', 'aafshar@gmail.com', passwd)
>>> c.post('this is a test post', 'this is the body of the test post with some html')
HTTP Error 201: Created
>>>

You can actually see this post online (if I haven't deleted the test blog) here: http://my-test-atom-blog.blogspot.com/2007/03/this-is-test-post.html

Since I never have any spare time, my plans for this library are a bit shaky. It is a toy library in that there are no real tests, and failure conditions are not really dealt with properly. Ideally I will:

Write a PIDA plugin for blogging
Merge it into something like BloGTK, or write a miniature gui client of my own.

I have no idea how to publicize this to the Blogger.com community, but if any of you big Blogger admins or google developers are interested in spreading the word, (or silencing the word!) please let me know.

UPDATE:
(some more useful links)

http://code.google.com/apis/gdata/basics.html
http://www.google.com/base/api/demo/html/demo.html

UPDATE:

Well, editing now works in the library! I know this because I accidentally overwrote this post! Lucky for the unnoficial planet python it was still lying around there!

Using pexpect to control Django manage.py

2007-03-15T11:18:00.000Z

In the early stages of developing a Django application, I was deleting my sqlite database, and rerunning:


python manage.py syncdb

Every few minutes as the database models were changing. You might consider this some kind of evil thing to do, because we should all have our database models in concrete before writing any code but I am not that organised.

One thing that was really distressing me was the fact that each time I ran a syncdb, I had to enter details for my default admin user.

Username
Email address
Password
Repeart Password

This was fine for the first 400 times, then I got bored. The solution was to use pexpect. (There are probably command line options that you can give manage.py, but finding them out would have been more effort than this hack)

Pexpect, http://pexpect.sourceforge.net/ is a pure python module for running external commands and controlling them as if you were a user. It has good documentation, and is perfect for needs of this nature.

A quick browse through the excellent documentation, and we were away. Here is the code:


PASSWORD = '123'    # replace this with something good
EMAIL = 'aafshar@gmail.com'

import pexpect

# spawn the child process
child = pexpect.spawn('python manage.py syncdb')

# Wait for the application to give us this text (a regular expression)
child.expect('Would you like to create one now.*')

# Send this line in response
child.sendline('yes')

# Again wait for this, and send what is needed
child.expect('Username.*')

# A blank string for the default username
child.sendline('')

# And so it goes on
child.expect('E-mail.*')
child.sendline(EMAIL)
child.expect('Password.*')
child.sendline(PASSWORD)
child.expect('Password.*')
child.sendline(PASSWORD)

# This means wait for the end of the child's output
child.expect(pexpect.EOF, timeout=None)

And we are done. This runs the command, and enters the correct details without having to be interactive, and saving a load of time and typing.

I actually combined this script with an initial data-import setup script which loaded the database with the initial data, for added fruitiness.

Unit Testing PyGTK

2007-03-13T16:40:00.000Z

Unit testing a GUI has always been something that scares me, and scares other people too.

Because of this there are about a gazillion tools that behave in different ways, for example in Python, just have a look at:

http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy#GUITestingTools

(Since the whole world is obsessed with Web Applications, and this is becoming perversely true even in Python circles, some of these GUI Testing tools are actually web testing tools, but never mind that.)

How do I do it? Well, we may talk in another blog about one of those tools, the awesome hack that is kiwi.ui.test but it is slightly restrictive on what you want to do, and how you should do it. Specifically, it needs you to have every widget named (which is fine when testing Glade-created interfaces, but a pain when hand-building them).

Enter this one function that I found in the kiwi source. Kiwi is LGPL (whatever that means, but you should read the license if you are going to use it).


import time
import gtk

# Stolen from Kiwi
def refresh_gui(delay=0):
  while gtk.events_pending():
      gtk.main_iteration_do(block=False)
  time.sleep(delay)

Simple, and tiny, and it has the effect of turning your awful asynchronous run-events-when-it-wants-to User Interface into a simple synchronous thing.

How? Well simply it runs all the events that are waiting in the gtk event queue, as if you ran gtk.main() for a short while until it was done with everything.

The optional delay argument is a short delay that might prove useful, but I have yet to actually use it.

Let's look at an example of how to use this in a unit test.


from unittest import TestCase, main
import gtk

class MyView(gtk.VBox):

  def __init__(self):
      super(MyView, self).__init__()
      self._button  = gtk.Button('Click Me')
      self._label = gtk.Label()
      self.pack_start(self._button)
      self.pack_start(self._label)
      self._count = 0
      self._button.connect('clicked', self.on_button_clicked)

  def on_button_clicked(self, button):
      self._count = self._count + 1
      self._label.set_text('clicked %s times' % self._count)
    
class MyViewTest(TestCase):

  def setUp(self):
      self._v = MyView()

  def test_count(self):
      self.assertEqual(self._v._count, 0)
      self._v._button.clicked()
      refresh_gui()
      self.assertEqual(self._v._count, 1)

   def test_label(self):
     self._v._button.clicked()
     refresh_gui()
     self.assertEqual(self._v._label.get_text(), 'clicked 1 times')

if __name__ == '__main__':
  main()

And we run it:


Ran 2 tests in 0.013s

OK

Great!

Things we should note:

We never had to put our custom widget into a window
We never had to display it to the outside world
We can be sure asynchronous things like signal callbacks have happened
We can treat our UI as a purely synchronous application
We use unittest.TestCase, we don't need any fancy UI testing framework's subclasses.

Some clever people might try this test without the refresh_gui call, and may even notice that the tests pass, but that is because it is extremely simple in this example. Try it with more complicated stuff that is scheduled in gobject's idle time, and your results will start to become inconsistent.

Conclusion: Keep it simple! In a GUI Testing War, I would hedge this function against anything.

Thoughts on Django

2007-02-07T22:50:00.000Z

What follows are some thoughts on django written by Isaac Alston, a friend of mine from university. He originally wrote it as a facebook note, but after reading it I asked if I could re-post it here. I think he gives an enlightening point of view on the experience of a python newcomer beginning to use django.

Here it is:

This is a short review of my experience with Django (http://www.djangoproject.com). Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.

In November, I was informed that the company I am working for would be leaving PHP behind. We had been looking for a faster, and cleaner development tool for a while and were deciding between Ruby on Rails (http://www.rubyonrails.org/) and Django. I had a preference for Django immediately, since I could already program in Python, whereas Rails required Ruby skills.

After much debate and discussion, my company eventually chose to use Rails. My co-worker, flatmate and (dare I say it) friend, Maciej, is currently working on our latest project for a fish company with Rails, and has produced some spectacular results. I was determined that I could do the same with Python, and so (instead of learning Ruby) launched into the docs for Django. At about the same time, I was approached by my church and asked if I would be willing to develop a new website for them. I agreed to the suggestion, because I knew that the opportunity would allow me to learn Django at a slower pace, and in a low-pressure environment.

I was initially very impressed by Django. I have been coding with PHP for about three years now, and it's been slow, messy and difficult to work with at times. I was amazed at such 'auto-generation' features including the database API, and the Admin centre.

Three weeks into development however, Django's niggles and problems can be seen in full view. The main problem with Django is that its Admin centre is not flexible enough.

Django is in a constant state of flux because the developers are rewriting the forms model (the way in which forms on a website interact with the database). 'Newforms' as they are called, are going to replace 'old-forms'.

At present, the Admin panel still makes use of the old-forms model, and will not be rewritten with newforms for a while. This is the primary cause of it's inflexibility. For example, I wanted to perform some custom validation of uploads with the default Admin. All I wanted to do was check whether the file was of a 'music type' ie, mp3 or ogg or some such format. Yet it appears that this is either very difficult or impossible. I say this because I have received no replies from the Django mailing list.

It's very nice having the 'free' Admin there, but in the end, if one wants to build something really complex, one must develop a custom admin panel from scratch. This I suppose, isn't so terrible because that is what I was doing with PHP anyway; however, it is infuriating that I am forced to write additional code, when the current Admin is so close to being able to do it.

The current documentation recommends to build any new websites with the newforms model, and so I downloaded the development version and set to work. This was all well and good until I discovered that there is nearly no documentation for the newforms library. Indeed, as a result, when I set about adding a simple contact form to my site, it took three hours, THREE HOURS! This was because I was literally forced to browse the source code for Django, to try and figure out what was going on.

The third problem with Django, is that doing AJAX requires a little work (and of course, programmers are lazy). Rails has nice 'helper functions' which generate JavaScript for you, which Django lacks. This was the result of another huge debate that went on within the Django community, who didn't want to be seen as 'Rails copycats'. Anyway, the maintainers eventually decided that web developers should (and I paraphrase from a rant): "grow up and learn to use JavaScript properly", so for people like me who have managed to get by copy-pasting JavaScript all these years, it means we'll have to sit down and learn it. Why should we though? Rails developers aren't... ;-).

I think that Django will be great when it gets the newforms library fully implemented (in the Admin as well) and in the current release (rather than the development version which constantly changes). I think it'll be great when there's some more documentation for newforms. Until then, I'm probably going to move to Pylons (http://www.pylonshq.com/), which a friend has recommended and which seems to be more Rails-like, offering a lot more flexibility. It also offers the nice 'AJAX helper functions' which I so desire, and also seems to have a more powerful ORM (Object Relational Mapping) implementation than Django's. I could of course learn Ruby, but, well... yeah.

django newforms for models

2007-01-11T22:03:00.000Z

The way django does forms is in a state of transition. There's the old (or, depending on your point of view, current) way, manipulators, and the new way, newforms. newforms is only available in the svn version of django at present, and is still incomplete. Despite this, newforms is usable and provides several compelling advantages over the old way. The docs at present are a little thin (although improving quickly), but the unit tests are full of useful examples (the stuff towards the end is the most interesting, and shows off some of the flexibility of newforms).

One thing that would be nice, though, is a way of easily creating a form class that uses a model class's fields. If we have this, we can avoid needless and tedious repetition.

Django recently got a function that can be used for this, newforms.form_for_model. It's a start, but there are a few things I'd like it to do that it currently can't. I'd like to use the model's specified defaults, if any, as the default values for fields, and I'd like the help_text entries to be available in the fields for rendering. This saves having to specify these seperately in each form that uses the model fields. It'd also be nice to be able to specify declaritively additional fields to be used in the form and I'd like a way to easily override the default widgets for a field.

It turns out it's fairly quick and easy to write our own solution to this. I've gone with a metaclass because I like doing things declaritively, and django already provides a metaclass for specifying fields that we can conveniently inherit from to get our "extra fields" feature for free. So here it is:


from django.newforms import forms, widgets
from django.db.models.fields import NOT_PROVIDED

class DuplicateFieldError(Exception):
   pass

class MetaModelForm(forms.DeclarativeFieldsMetaclass):
   """Creates form fields from the model in the 'model' class attribute and
   adds them to the created form."""

   # The default custom widgets we will use.
   _default_widgets = {
       'TextField': lambda _: widgets.Textarea({'rows': '10', 'cols': '40'}),
       'ImageField': lambda _: widgets.FileInput()
   }

   # This can be overridden to provide custom widgets for particular model
   # fields. It should be a mapping from model field class names (ie
   # 'CharField') to callables that take one argument, the model field,
   # returning a Widget instance. See _default_widgets above.
   model_widgets = None
   model = None
  
   def __init__(cls, name, bases, attrs):
       super(MetaModelForm, cls).__init__(name, bases, attrs)
       if cls.model is None:
           # We don't need to do anything.
           return
       if cls.model_widgets is None:
           cls.model_widgets = {}
      
       cls._update_fields()

   def _update_fields(cls):
       """Update the form's fields with fields created from cls.model. Bits
       of this are stolen from newforms.form_for_model"""
       opts = cls.model._meta
       # Create cls._widgets from defaults + overrides
       widgets = cls._default_widgets.copy()
       widgets.update(cls.model_widgets)
       cls._widgets = widgets
       for f in opts.fields + opts.many_to_many:
           if f.name in cls.fields:
               # We wimp out and start crying rather than try to resolve this
               raise DuplicateFieldError('Field %s is duplicated in the '\
                                         'model' % f.name)
           # Attempt to create a form field for the model field.
           form_field = cls._create_field(f)
           if form_field:
               # If we successfully get a form field, update the Form's fields
               # attribute with the new field.
               cls.fields[f.name] = form_field
      
   def _create_field(cls, f):
       """Create a form field from the given model field if possible,
       otherwise return None."""
       if f.default == NOT_PROVIDED:
           formfield = f.formfield()
       else:
           # Take the inital field value from the model field's default.
           formfield = f.formfield(f.default)
          
       if formfield:       
           try:
               # Check if there is a given field-specific widget
               widget = cls._widgets[type(f).__name__](f)
           except KeyError:
               # If not, use the form field's default widget.
               widget = formfield.widget
           else:
               # If a custom widget was found, the field must be updated
               # with it. We check for extra attributes provided for this
               # widget by the field, and update the widget accordingly.
               extra_attrs = formfield.widget_attrs(widget)
               if extra_attrs:
                   widget.attrs.update(extra_attrs)
               formfield.widget = widget
  
           # Add help text. Until there is a better way. ;)
           formfield.help_text = f.help_text
       return formfield

I have tried to comment the code so it is clear what is going on. model_widgets is a dictionary of callables that take the model field as an argument, rather than just a bunch of widget instances. This is so aspects of the model can be taken into account when deciding the widget (or widget attributes) to use.

Say we have a model as follows:


class Person(models.Model):
   first_name = models.CharField(maxlength=20)
   last_name = models.CharField(maxlength=20)
   address = models.TextField(help_text='The home address of the person.')
   age = models.PositiveSmallIntegerField(default=21)

We can create a form like so:


>>> class PersonForm(forms.BaseForm):
...     __metaclass__ = MetaModelForm
...     model_widgets = {
...         'PositiveSmallIntegerField': lambda _: widgets.TextInput({'size': 3})
...     }
...     model = Person
...     date_of_birth = fields.EmailField()
...     email = fields.EmailField()
...
>>> form = PersonForm()
>>> print form
<tr><th><label for="id_date_of_birth">Date of birth:</label></th><td><input type="text" name="date_of_birth" id="id_date_of_birth" /></td></tr>
<tr><th><label for="id_email">Email:</label></th><td><input type="text" name="email" id="id_email" /></td></tr>
<tr><th><label for="id_first_name">First name:</label></th><td><input id="id_first_name" type="text" name="first_name" maxlength="20" /></td></tr>
<tr><th><label for="id_last_name">Last name:</label></th><td><input id="id_last_name" type="text" name="last_name" maxlength="20" /></td></tr>
<tr><th><label for="id_address">Address:</label></th><td><textarea id="id_address" rows="10" cols="40" name="address"></textarea></td></tr>
<tr><th><label for="id_age">Age:</label></th><td><input id="id_age" type="text" name="age" value="21" size="3" /></td></tr>
>>>

Defining model_widgets is optional, but I wanted to show that it works (notice that the "age" field has a size="3" attribute). This shouldn't be hard to modify or extend so that it does exactly what you want. One possibility I'm considering is copying validator methods from the model to the form.

I should mention field ordering. date_of_birth and email appear first because they were created first (before the metaclass methods ran). newforms uses an internal counter to keep track of the order in which fields are created, and by default renders them in order of creation. This isn't difficult to work around if you desire it. However, instead of working around it, I use a different method, taking advantage of the fact it is very simple to subclass BaseForm and customise the automatic rendering.

Tomorrow I will post a simple form subclass for rendering a form using fieldsets (specified in a similar way to django's auto-generated admin), including a custom row format-string that includes help_text. I'll link to it from here when I post it.

It's late here, though, and I have a lecture tomorrow at the disgustingly early time of 11am.

What is wrong with Berlios?

2006-12-29T11:01:00.000Z

Berlios hosting was once a great replacement for Sourceforge and others. But these days it seems we have more downtime than uptime. I don't blame them (I am sure they have their reasons why nothing is ever available) but I do understand that I need something better to host my open source projects with.

While we do bug tracking, and source-code hosting with Malone and Launchpad already, there is no provision on the Launchpad for hosting distribution files, and so we have had to painfully search for alternatives.

For now we are going to go with Google code. This is not perfect for our needs, but it will replace everything we need Berlios for except for hosting the PIDA website, which can be easily arranged elsewhere.

The main pain will be updating all the links around the web that point to the Berlios website. But that will come in time, and I am sure that PIDA users are bright enough to click a redirect link!

For now the new project page is:

http://code.google.com/p/pida/

And we also have a new PIDA web site with fancy new domain name at:

http://pida.co.uk

Sorry Berlios, you just aren't making the grade.

PIDA 0.4.0 Released

2006-12-10T12:04:00.000Z

After a slightly quiet spell (also known as an extended beta period) we have released PIDA 0.4.0. What kind of open source author would I be if I didn't push my own software on my blog!

My favourite feature on this release is built in pyflakes. When you save a Python file it runs Pyflakes on it for you and prettifies the output into a GUI list for you to browse through, click on, and jump to the line number. All in Vim, as usual.

Please try it, at http://pida.berlios.de/

More on Python Dates, adding time with a datetime.timedelta

2006-12-06T15:45:00.000Z

I needed a date 2 weeks in the future. I needed it quickly. After blogging about Python dates, I thought I had a clue how to do it. To my relief the first attempted path of least resistance just worked! Here it is:


>>> from datetime import date, timedelta
>>> today = date.today()
>>> two_weeks = timedelta(days=14)
>>> two_weeks_time = today + two_weeks
>>> today
datetime.date(2006, 12, 6)
>>> two_weeks_time
datetime.date(2006, 12, 20)
>>>

This was used to automatically populate a column in an an old database: which wanted to be an expected_date = order_date + two_weeks.

Oracle from Python in Linux

2006-11-22T15:25:00.000Z

Well, pretending to know everything can have its downsides. Like when the boss says "Could you hack up a script to enter custom data into our 3rd party invoicing software?". One choice is to admit that you would not know where to start. The other would be to attempt to muddle through. "Of course, let's have a look".

So an hour later we have discovered that the software in question uses an Oracle back-end: Excellent, this is going to be easy, Oracle is an relational database (I read once somewhere) and we learned SQL at university, so it can't be that hard.

So, step one: boot into windows, and use the command line client to do some stuff. Great! The SQL is coming back to me already. But this is no good. I can't use windows! The single desktop is beginning to upset me, and well we need it running on Linux so I can feel leeter than my coworkers.

So what will we be needing?

1. SQLPLUS for Linux (this the the command line client for Oracle). Should be easy enough, but oh man I don't have to install all of Oracle do I? No! Oracle 10 has this thing called "instantclient" which is just the basic client for connecting to an Oracle database.

We need to visit the Oracle site, register some stuff with them, and then download the instantclient for Linux-32 (in our case), and the addon with the sqlplus libraries and executables.

We unzip both of these files, and looking around, decide the best place to put them is in /usr/lib/instantclient (although in retrospect /usr/local/lib/instantclient might have been better).


$ ls /usr/lib/instantclient/
classes12.jar
glogin.sql
libclntsh.so.10.1
libnnz10.so
libocci.so.10.1
libociei.so
libocijdbc10.so
libsqlplusic.so
libsqlplus.so
ojdbc14.jar
sqlplus

Ok a few jar files, to make us feel unwell, but this is looking fine. Running sqlplus doesn't work, it can't find the library it requires. Enter a simple skill C people know about. Edit /etc/ld.so.conf (or put a file in /etc/ld.so.conf.d) and add the path of the libraries (in our case adding the single line /usr/lib/instantclient is enough, and then as root, run ldconfig.

And then run sqlplus


$ sqlplus

SQL*Plus: Release 10.2.0.2.0 - Production on Wed Nov 22 15:39:43 2006

Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.

Enter user-name:

Muahaha, that felt good didn't it! But we are not there yet (by a long chalk). Connecting to Oracle databases and providing the database description string is going to be a scary nightmare, discussed elsewhere no doubt. But here is a little trick we want to show you.

Sqlplus does not use gnu readline on Linux. I know, awful, but don't despair: there are some applications that wrap command line applications with readline, and so give you awesome things (mostly history spanning sessions). The application that I ended up using is called rlwrap and was as easy to install as using apt-get install rlwrap. Combine that with the following shell script in /usr/bin/sqlplus:


$ cat /usr/bin/sqlplus
#! /bin/sh
/usr/bin/rlwrap /usr/lib/instantclient/sqlplus $*

And not forgetting to make it executable, we have a working Oracle command line client. And all for the price of your company details and an email address. Dear Lord, please bless those generous people at Oracle!

2. Now to be able to access the database programmatically. A little snoop around and we discover the Python library cx_Oracle. The prefix cx_ may remind you of the application cx_Freeze (for building standalone executable Python scripts) and that is because it comes from the same people.

Ok first snag, it's not in Ubuntu. Ok, it is a contender for MOTU (Universe) but it's not there yet. That's fine I thought, I'll build it. Using the usual python setup.py build and Eeeeek! It needs Oracle to build against. I should have thought that it might, and here I am wondering if I should download and install all 3 cds worth of Oracle. That is going to take a day or so, and I just haven't got the time.

For the second time today, we have to ask the Lord to thank the nice people at Oracle, because we can get away with just having the SDK for our already installed instantclient! It is available from the same place, and it unzips into the same place you unzipped instantclient, adding a subdirectory called sdk. This is nearly enough to allow building the python library. I say nearly because it searches for libclntsh.so, and we only have libclntsh.so.10.1, and the fix is as easy as symlinking one to the other:


/usr/lib/instantclient$ sudo ln -s libclntsh.so.10.1 libclntsh.so

And then build the python bindings, remembering to set the ORACLE_HOME environment variable:


$ ORACLE_HOME=/usr/lib/instantclient/ python setup.py build
$ ORACLE_HOME=/usr/lib/instantclient/ sudo python setup.py install

And I think we have a working cx_Oracle library. Let's check:


$ python -c "import cx_Oracle"
$

No news is certainly good news! The library comes with excellent documentation, and you will be on your way in no time. See, pretending to know everything sometimes forces you to work things out that you were clueless about!

Python Dates

2006-11-17T09:51:00.000Z

At first glance the datetime module in the python standard library (full documentation available at http://docs.python.org/lib/module-datetime.html appears as an insane mish-mash of things.

The types it makes available are date, time, datetime and timedelta. My initial confusion stemmed from the fact that the module name is the same as one of the type names (datetime and datetime.datetime) and this is one of the only times when I have thought that naming types in camel case is a good idea: datetime.DateTime might have been a saner approach. Nevertheless, we continue:

A datetime.time is a time, a datetime.date is a date, and a datetime.datetime is a date with a time. A datetime.timedelta is a difference between any of these times. Simple enough.

Usage of these types can be made easier by using their "alternative constructors" which are a set of class methods, and can in some cases be more useful than the actual constructor. For example:


>>> from datetime import date
>>> date(2000, 1, 1)
datetime.date(2000, 1, 1)
>>> date.today()
datetime.date(2006, 11, 17)

The actual constructor has the signature: date(year, month, day), but the alternate constructor that is very useful for us is the date.today() call, which returns today's date (as in the example).

The datetime type is similar to the date type but also comes with time information. Here the alternative constructor that I find most used is not today(), but now(). For example:


>>> from datetime import datetime
>>> datetime.now()
datetime.datetime(2006, 11, 17, 10, 1, 49, 913312)

The actual constructor here would want year, month, day and optional extra arguments for hour, minute, second and microsecond. This particular constructor (now) would be really useful for things such as SQLObject schemas, like:


>>> from sqlobject import SQLObject, DateCol
>>> class MySchema(SQLObject):
...     date_of_event = DateCol(default=datetime.now)
...
>>>

SQLObject has this nice feature of calling a callable default value, and in this case creating that object would give it a date_of_event attribute representing the time and date it was created. A really useful pattern.

So we come to the time type. And remember we are datetime.time, not time.anything - the time module is something else entirely, and where this starts to become a bit more painful. datetime.time accepts (as you would expect) hour, minute, second, microsecond. But amazingly has no alternative constructors like the other types to return the current time. To do that we need to perform a small amount of trickery:


>>> from datetime import datetime
>>> datetime.now().time()
datetime.time(10, 20, 46, 720865)

Oh I know it's a bit ghastly, but it works. Now it wouldn't quite work if you wanted to use it for an SQLObject schema as above, and you would need to use a function or inline lambda, like so:


time_of_event = TimeCol(default=lambda: datetime.now().time())

To make sure the datetime object was called at the right time. Hoops are only fun if you have to jump through them!

Shall we leave it there, or can we stretch ourselves to a short discussion of timedeltas? If we keep it simple. A timedelta is a difference between two dates, or datetimes, but not two times. And the difference has to be calculate with the same types, for example:


>>> dt1 = datetime(2000, 1, 1)
>>> dt2 = datetime.now()
>>> delta = dt2 - dt1
>>> delta.days
2512

Apparently that is the number of days since the turn of the millennium. Sounds good to me.

Rat, more easy PyGTK dialogs

2006-11-16T09:09:00.000Z

We have been indulging in some Kiwi-love recently. You may have read the recent blog about easy PyGTK dialogs with Kiwi. After a chat with the author of another PyGTK helper library (called rat) and available at http://python-rat.berlios.de/, who also happens to be a good friend of mine, I have decided to blog a similar example of dialogs using rat.

Rat has a similar set of dialogs (a few extra, and a few missing). We shall start with a simple error dialog:


>>> from rat.hig import error
>>> error('You made a mess in your trousers',
...               'Or maybe the mess was already there',
...                title='A big mess!')

A similarly simple API, and it gives us a similarly HIG-compliant error dialog:

Rat does not contain a yes/no dialog, but it does contain an ok/cancel dialog. Usage
is again very quick and simple:


>>> from rat.hig import ok_cancel
>>> from gtk import RESPONSE_OK
>>> r = ok_cancel('Are you sure you want to destroy something?',
...            'This Could be as evil as eval!',
...            title='Please ensure...')
>>> r == RESPONSE_OK
True

The resultant dialog does everything you want it to, and can be used for the same purpose as the Kiwi yes/no dialog.

Mmmm, HIG-love!

The extra dialog I want to have a look at is something only Rat has. This is the "Save Changes" dialog. It is designed for any editing application, where the user may wish to close the application while there are unsaved files or buffers e.g. a text editor, or graphical editing program. A small example:


>>> from rat.hig import save_changes
>>> files_to_save, response = save_changes(
...     ['foo.txt', 'bar.txt', 'baz.txt'],
...     title='You asked to quit the application')
>>> files_to_save
['bar.txt', 'baz.txt']

Amazing! And to show you how wonderfully beautiful it looks:

You will notice that of the 3 files passed to the dialog call, we selected two of them, and these are the two returned in the files_to_save variable. Lovely.

I will conclude here that although there are two ways (Rat and Kiwi) of performing similar tasks, and choice is always good, each library contains things that the other doesn't. The maintainers of each library know eachother well. Isn't it time that they collaborate and join into a single library? We, the PyGTK developers will be the ultimate winners!

Kiwi, Pain free PyGTK dialogs

2006-11-15T14:21:00.000Z

Since we are in love with Kiwi, we will mention another amazing thing it has done for us. HIG Dialogs. HIG are the (Gnome) Human Interface Guidelines, a set of guidelines for creating accessible user interfaces. I wish we could ignore these things, but unfortunately we can't. You can read about them here.

Kiwi has a set of dialogs ready for us that conform to these guidelines. They are presented in the form of funcitons that you can call, and which give you a response of some kind. Let's have a look:


>>> from kiwi.ui.dialogs import error
>>> error('You screwed something up',
...    'Or maybe it was already screwed to begin with')

Yes, it is as simple as that. None of that boilerplate code. And the dialog itself actually looks quite nice:

The error dialog is not the only one available of course. You have error, info, save, open, and my favourite: yesno


>>> from kiwi.ui.dialogs import yesno
>>> from gtk import RESPONSE_YES
>>> resp = yesno('Are you sure you want to install this virus?')
>>> resp == RESPONSE_YES
True

Again, nice and quick. You will find yourself adding these dialogs all over the place, maybe even in too many places. But it's all good, as long as the user knows what is what.

Kiwi proxy widgets, a common widget API

2006-11-13T16:03:00.000Z

While we are on the notion of Kiwi, the PyGTK helper library, we should be discussing the proxy widgets. They offer a certain set of advantages over their plain PyGTK counterparts:

They have a common signal emission API for change in content
They offer a common API for setting and reading their value.
They offer validation, and coercion
They contain a few general improvements over the plain PyGTK widgets

What does this mean?

Simply it means you can make any data widget (eg Entry, CheckButton, Combo etc), you can connect its signal "content-changed", you can read the data with widget.read(), and set the value with widget.update().

If you have read any of the previous blogs about adapting Zope.schema fields to widgets, and you are alert, you will notice that this is amazingly useful. It means that we can define an interface for a general InputWidget type, something like:


class IInputWidget(Interface):

def read():
"""Read the value in the widget"""

def update(value):
"""Set the value of the widget"""

def connect_content_changed(callback):
"""Set a callback to get called when the widget value is changed by the user"""

Once we have done this we can essentially forget about what widget we are using, and what type of field it is representing. A dream come true for the "program for an interface" paradigm.

At the same time, these widgets do have some downsides:

They are bound to an arbitrary concept of a "model" and an attribute within that model
They insist on having a data-type set

The good thing is that the downsides can be faked by setting the appropriate PyGTK properties. Hurrah!

Let's look at a basic example:


from kiwi.ui.widgets.entry import ProxyEntry

e = ProxyEntry()
# Fake those things that Kiwi requires
e.set_property('data-type', str)
e.set_property('model-attribute', 'fake')

# Test reading and writing
e.update('I am the starting value')
assert e.read() == 'I am the starting value'

Now imagine you had a lot of these, and that they were all automatically adapted from a zope schema (as discussed before). These widgets could all be readable using something like:


widgets[name].read()

Amazing! One might be a CheckButton, returning a boolean value. One might be a Date chooser returning a datetime, one might be an entry returning a str or a unicode. Complete automatic UI generation that things like Django would be jealous of!

PyGTK lists made easy with Kiwi

2006-11-13T00:00:00.000Z

Making nice lists is an utter pain with PyGTK. GTK+ does you the pleasure of separating the model and the view. I guess it is a reasonable idea in theory (don't they always teach you to separate the model and the view?) and allows you to have alternative views of the same model, and allows you to extend the implementation in any sort of way. BUT, to repeat: in most cases it is an utter pain. Even the simplest list becomes a mess of boilerplate code.

Anyone who has done a decent amount of PyGTK programming will have written their own implementation of model+view mixed together, but the best one I have used is from Kiwi. If you have never used it, Kiwi is a framework that lessens some of the painful aspects of PyGTK development. In fact, it's so useful, let's call it a library.

Creating a pythonic view of a list of objects is easy. Let's have a look at some code:


from kiwi.ui.objectlist import ObjectList, Column

class Fruit(object):
def __init__(self, name, is_red):
self.name = name
self.is_red = is_red


fruits = ObjectList(
 [
   Column('name', title='Fruit Name'),
   Column('is_red', title='Is the fruit red?',
          data_type=bool, radio=True)
 ]
)

fruits.append(Fruit('Apple', True))
fruits.append(Fruit('Melon', False))

Yes it is really that easy. Add a bit of code to put it in a window:


import gtk
w = gtk.Window()
w.add(fruits)
w.show_all()
gtk.main()

And you are ready to go. If you try to achieve the same thing with pure PyGTK, you will write at least 10 times as much code.

Zope in a gui, fields as widgets

2006-11-05T22:05:00.000Z

This is a long quest, and at this stage we have objects, which implement interfaces, and we have schemas that reflect the interfaces, and are helpers for creating views for the objects. This is all well and good, but what happens on a widget level in a gui?

Well we can give you a simple example. The view is an adaption of the Schema, and which then has its content set (see the last blog about this topic). The view can be built by adapting each of the schemas fields into a gui element, exactly as you would build a web form.

Say you have an attribute defined as a zope.schema.Text, and you wanted an editor field. Well simply, using pygtk:


import gtk
from zope.interface import implements, Interface, Attribute
from zope.component import adapts, provideAdapter
from zope.schema import Text

class IWidget(Interface):
def write(value):
    """display the value"""

class TextWidget(gtk.Entry):
adapts(Text)
implements(IWidget)

def __init__(self, field):
    self._field = field
    super(TextWidget, self).__init__()

def write(self, value):
    self.set_text(value)

provideAdapter(TextWidget)

Now everything is set up to adapt the field to a widget.


name = Text(title=u'The name of something')
print IWidget(name)

Gives you


<TextWidget object (GtkEntry) at 0xb715f16c>

Yay! Now you have everything you need to create a gui from a zope object.

In search of a zope gui, why the zope web way might not be the best way

2006-11-05T21:17:00.000Z

So in our last post, we saw how you could use the Zope interface architecture to adapt objects to views, and how Zope does it. The ultimate goal of this exercise is to have a gui way of representing Zope. This is all fine, but we have to consider that perhaps in a gui, one view per object may not be the best method.

Creating a gui repeatedly for the same type of object, just to change its contents is just wrong.

In our example, we had a multi-adapter for IObject, IRequest, which was a view object with a render method. No doubt perfect for web requests, but not for a gui which lives in a loop, is governed by events and can hide lots of stuff.

What it seems we actually need is an adapter for ITypeOfObject, which is creaeted without an object, and can then have the content set and reset.

Enter zope.schema

zope.schemas are interfaces and have all the interface properties. They do have additional niceties, in that you can define attributes as constrained field types, for example Int, and in zope all of these field types are adaptable into html form elements. In our gui case we want to adapts them to widgets. The schema can be the adapted object when the view is created, and the view can have some kind of content setter which is reposnible for updating the widgets in the view. It is worth noting that this approach may become more used with increasing developments in the Javascript world.

So, first we define ISchema, and an implementor of ISchema. It is important to note here that we have left how zope uses schemas. They are Interface subclasses, and although this has advantages, I fell slightly ill-at-ease with it. We will instantiate the Schema object and we will worry about iterating the fields in it later.


from zope.interface import implements, Interface, Attribute, classProvides
from zope.component import adapts, provideAdapter

class IAnimal(Interface):
    name = Attribute("""The name of the Animal""")

class ISchema(Interface):
    pass

class AnimalSchema(object):
    """This is a schema"""
    implements(ISchema)
    name = Text(title=u'The Name')

And now a something to implement the IAnimal/IAnimalSchema. The schema and the interface are essentially the same, and you might ask why not just use the interface. Well, simply because being an Interface subclass is a royal pain. It imposes lots of bizarre restrictions, and since we really want to push it a bit later on, we should depart from that now.


class Animal(object):
 """An animal"""
 implements(IAnimal)
 def __init__(self, name):
   self.name = name

And now the view stuff. This view will be setup once, and then available to have its content changed.


class IView(Interface):
 
 def render(self):
   """Render visually"""

 def set_content(self, content):
   """Update the view to reflect the content"""

class AnimalView(object):

 implements(IView)
 adapts(AnimalSchema)

 def __init__(self, schema):
     self._schema = schema

 def render(self):
     print 'Our schema is %s' % self._schema

 def set_content(self, content):
     print 'we have new content %s' % content.name

provideAdapter(AnimalView)

Now let's run it.


if __name__ == '__main__':
   view = IView(AnimalSchema())
   view.render()
   for i in map(Animal, [u'lion', u'tiger', u'leopard']):
       view.set_content(i)

And you get:


Our schema is <__main__.animalschema>
we have new content lion
we have new content tiger
we have new content leopard

Although this is just a command line output, you can see that the stages have been fulfilled. The gui can be setup in the initial phase by drawing all the widgets, and then the values can be set at later points, asynchronously.

Using multi-adapters a la Zope

2006-11-05T19:06:00.000Z

As a brief discussion (and I should mention that this has nothing to do with a web application), you have an object and you want to look at it. Imagine you have gone the whole way with this concept and you have arrived at Zope. Zope contains mechanisms for definition, storage and manipulation of objects, all using Python.

So you have an interface:


from zope.interface import Interface, Attribute

class IFruit(Interface):
    name = Attribute("""The name of the fruit""")

Ok, it's simple. Now you write a class that implements this, still simple.


class Fruit(object):
   implements(IFruit)
   def __init__(self, name):
      self.name = name

Pretty good. You can store that in an object database like ZODB if you like, but we are going to think how to view it.

A view (in Zope terminology) is something that adapts (multi-adapts) a "context" and a "request". We don't really understand what a "context" is, so we are going to call it an "object". A request can be something like "edit" or "view".

So of course we need an interface for views to adhere to, and an interface for a request to adhere to. This is where interfaces stop being documentation niceties, and start being implementation nasties.


class IView(Interface):
   def render():
       """Display what needs displaying"""

class IViewRequest(Interface):
   user = Attribute("""The user that made the request""")

So imagine we implement a simple view (now remember this is nothing to do with a web application). The view will display Fruit objects.


from zope.component import adapts, provideAdapter, getMultiAdapter

class FruitView(object):
   implements(IView)
   adapts(IFruit, IViewRequest)

   def __init__(self, fruit):
       self._fruit = fruit

   def render(self):
       print self._fruit.name

# not forgetting to register the adapter with zope.component
provideAdapter(FruitView)

And also a request implementation that does nothing


class ViewRequest(object):
   implements(IViewRequest)
   user = None

And we are ready to go:


view = getMultiAdapter((Fruit('banana'), ViewRequest()), IView)
view.render()

gives you


banana

Launchpad and Malone: Pain free bug-tracking

2006-05-12T04:34:00.000Z

Launchpad is a open source software management system being provided by Canonical (makers of Ubuntu). It has lots of services and features, and we'll just be scratching the surface today. The services it provides are somewhat like sourceforge. Launchpad is comprised of sub-systems, each of which seems a fully capable application.

Malone is the name of the bug tracker included with Launchpad. Malone is just the best bug tracker I have ever used. I am no expert, but I have really grown to detest using Trac and Bugzilla. The main reason being constant switching back and forth from lists to bugs and then back again, and oh yikes, it makes me dizzy. Malone seems to get around this with subtle use of javascript, and obsessive attention to user workflow.

From sniffing around the development crew of Launchpad (as one does!) I found some really well known and scary names. Experts from all departments, including a decent number of GUI expert programmers. The user interface seems really important to them, and it shows in the end product.

Well done everyone involved! At last, I look forward to managing bugs, not just fixing them. And at last my users have a pleasant way of reporting. And as a pleasant bonus, it is all hosted for me.

(links in text for more information)

Blogger doesn't look elite enough

2006-05-12T03:59:00.000Z

Well, this is just a test post really, but all this css makes my head hurt. We want a nice old-fashioned looking site. Even the minimal template is far too much.

(this is a test, don't get excited)