Toward a Secret Sky

Python's collections module has some of the most consistently useful collection data structures you will need for everyday programming. Here's one I didn't know about: collections.Counter  (Python 2.7 only!) It is designed to keep "tallies" or count instances of something. The example will make it all clear: from collections import Counter cars = Counter() # I see one go past, it is red cars['red'] += 1 # And a green one cars['blue'] += 1 # etc This is pretty much like a defaultdict with an integer value, but it is convenient and neat, with some useful constructors. Don't you think?

"Inotify is a Linux kernel subsystem that acts to extend filesystems to notice changes to the filesystem, and report those changes to applications." [Citation Needed] . You can use this service from Python using Twisted to watch a directory and its contents. Twisted is perfect for this as you likely want to be doing a number of other things at the same time, for example, making an HTTP request every time a change is noticed. The code is so monstrously simple, I will just paste it: from twisted.internet import inotify from twisted.python import filepath class FileSystemWatcher(object): def __init__(self, path_to_watch): self.path = path_to_watch def Start(self): notifier = inotify.INotify() notifier.startReading() notifier.watch(filepath.FilePath(self.path), callbacks=[self.OnChange]) def OnChange(self, watch, path, mask): print path, 'changed' # or do something else! if __name__ == '__main__': from tw...

The use-case is this: # Two instances of the same class x = A() y = A() x in {y: 1} # True So we want to be able to check an instance's presence inside a dict, set, or anything that hashes an instance for a key. You need two things: __hash__ method which returns an integer __eq__ method which tests equality against another instance and returns a boolean __hash__ method returns an integer which is used as the hash. I didn't want to invent a hash method, so instead I used the hash of a tuple, which seems a reasonable hashable type to use, but you could use an integer or string or anything. I wanted the tuple contents to be attributes of the instance, with the added side effect that the instance would pass the test: x is (1, 2) # True Here it is: def __hash__(self): return hash((self.name, self.age, self.location)) That is not enough though, if you implement __hash__ you must also implement __eq__, which is simple enough: def __eq__(self, other): return h...

You may know that the Google API Python Client is built on httplib2 . This is a reasonable general choice, but the tight coupling is unhelpful in situations where a different HTTP library, or an entirely different approach to network programming should be used. An example of this is Twisted . Aside: I won't be going on about how awesome Twisted is, but let's just take it for granted that it is so awesome that I could not write this application without it. Httplib2 is blocking, and that makes it incompatible with being run inside the Twisted reactor. Fortunately we are only the latest person to have this problem, and a solution exists: twisted.internet.threads.deferToThread api_call = drive.files().list() def on_list(resp):   for item in resp['items']:     print item['title'] d = deferToThread(api_call.execute) d.addCallback(on_list) A blocking call will be called in a thread and will callback on the returned deferred when it is done. I apprec...

Who will test those tests? A while ago I was invited by the lynch-mob in #twisted to write a post about what I meant when I said that "Unit tests are not real tests", and here it is. Now, recently there has been much said about Unit-testing, and code coverage , and even sensationalist blog post titles , which in retrospect "Unit tests are not real tests" would have been. I won't talk about code test coverage here, because its not relevant, we should strive to get as much coverage as we can. And of course I believe that unit tests are a fundamentally important piece of development. Especially since I tend to use Python , which is prone to disaster if not tested correctly. So, if unit tests are real tests, then why did I say they aren't? I deal mostly with application development. The user of the application is a human being. Now quite simply, the only true test of my application is the user interacting with it. I understand now that this is not really the ca...

In his blog, on Friday, August 29, 2008 , Johnathan talked about Google AppEngine , and said the "you can use any web framework you like, as long as it's django" attitude That may have been true then, it probably isn't true now. Glashammer have been working for the last few weeks on getting the Glashammer framework of Werkzeug and Jinja2 running and easy on Appengine. Firstly this wasn't hard. Glashammer is very free about what kind of data storage you use, so using Appengine's DataStore was straightforward. Some utility functions for running easily, add a few decorators for controlling what happens for authentication form redirection to limit views for certain users and we are pretty much done. They have additionally tried to make it easier to install with a little script to get us started. So (since it seems a good start): gh-admin quickstart_gae This will generate a starter AppEngine + Glashammer application that is ready to go. It will al...

Two awesome things: Twisted , and Kamaelia . I will not compare them. They are different, and have different purposes, and as I said. Both are great. I have recently been playing with Kamaelia for the first time. If you don't know what Kamaelia is check out http://www.kamaelia.org/ but I admit it does take some getting my head around. (Just like Twisted did, back in the day when I first came across it.) But persevere. Kamaelia is a library for creating highly concurrent applications. And along the way it has reminded me of: Erlang, Tasklets and Twisted. It has a highly componentized approach where components communicate with eachother using inboxes and message passing. That's all I'll say about it here. I am sure I will blog more as I use it more. Now the first thing I want to do with Kamaelia is be able to hook it up with GTK, and I have managed to do that, but it was incredibly easy. Why? Because Kamaelia is happy to run its scheduler in the background in a non-main thre...

It seems over recent months that my blog gets comment spam. I imagine any bloggers out there experience the same thing and it is a bit of a pain. I have three immediate problems with this and blogger.com. 1. Blogger doesn't notify me of all comments at the time they are posted. It notifies me of some, and I have of course configured it to notify me of all comments, but it seems to miss off about 70%. So not only do I not notice the spam, I also miss a bunch of legitimate comments. Please get it together Blogger! Ajax panel configuration is nice, but only if the core functions work. 2. Blogger should/could/might try to stop this spam before it happens. I am not guessing how, but then the company that runs Blogger.com are much brighter than me, and I am sure they have a solution. 3. The interface for browsing comments and deleting many at a time simply does not exist. This would make the task of sifting through, identifying, and delting spam much easier. Now that I have had my grumbl...

Pysqlite doesn't allow you to access the same connection from different threads. The pysqlite manual says: "SQLite connections/cursors can only safely be used in the same thread they were created in." When using Storm (the ORM) with Werkzeug (the WSGI utility lib) we suffer from the problem that the Werkzeug reloader runs code in a thread. Ok this feature is not exactly important in a production environment, but I can't guarantee that whatever platform I will be deploying the application on will not be threaded, so database access should be proofed against this. The solution? The Storm manual mentions that you should use a Store/connection per thread. Someone has already done this with the Middlestorm application, which provides a threadsafe store in the WSGI environ. Rightly or wrongly (since I really don't want to have to wait to have a WSGI environ to get the store instance), and I am not exactly sure this kind of thing should be middleware, but that ...

""" Desktop Widgets with PyGTK, Launchpad Remote Control We shall today be looking at a dumb hack I glanced upon while foolishly reading the list of GDK Constants in the PyGTK documentation. This hack sets the type hint of a GTK Window instance (the thing that tells the window manager how to treat it) as a desktop window, and hence embeds the thing in the desktop. It even works quite well: The window and widgets appear in the desktop The widgets are raised when the "Show desktop" command happens The widgets appear on all desktops (in Gnome and XFCE4, but not in KDE) The widgets appear fully functional So I got all excited and tried to think of an awesome use-case as the first plugin for a new kind of desktop widget framework. One which couldn't care less about eye-candy, and focussed on truly functional and useful desktop-based applets. As a friend commented while I was discussing the idea: "I am often hacking away in emacs when I decide that I want to...

Well, it is an age-old problem: How to schedule long-running tasks withing a GUI main loop (in our case PyGTK). There are a few ways: Use Python's subprocess module and select on the pipe with gobject's io_add_watch Use GTK's built in subprocess spawning abilities Use Twisted 1 & 2 are reasonable approaches, and they both work. Of course 1 won't work on Win32. The only problem with both 1 & 2 is that they use gobject's polling functions to achieve asynchronicity. This is nice when we are forced in a PyGTK main loop, but really not nice when the application wants to run in command line mode, and we really want to be able to share the execution code between different UIs, including perhaps other toolkits. Enter Twisted. We need to do two things with Twisted: Make sure Twisted knows we are running with PyGTK Launch the process Making sure Twisted knows that we are running inside PyGTK is quite easy (though I imagine the implementation was painful). To do this, ...

Well, it was a nice idea I believe, to place a video side-by-side with a Powerpoint (or one of it's inferior Open Source clones) presentation. The idea came when my boss showed me Microsoft Producer, which does exactly this (it's nice to have people who at least pay some attention to non-Open Source, especially when he pays you.). Producer is a plugin for Powerpoint, with a simple User Interface, which involves importing a presentation, and a video. Setting the timings for the slides, placing them adjacently in an output it video. It sounds simple enough, and I thought this functionality would be possible in some of the Open Source video editing equipment, but either: I knew for a fact that the software didn't have the capability, or: the software was impossibly difficult to use. So, I had a think, and a play, and another play, and I came up with the plan: Split the presentation into images: 1 slide per file Split the video into frames of png files, and save the sound for ...

There are a few ways to run a WSGI application, and today I will comment on the Twisted approach. The magic happens from the twisted.web2.wsgi module, and is fortunately simple. The main advantages I can think of when running with Twisted/Twistd is that: All Python solution (if that really is an advantage) Ability to run other twisted services. Did you want RPC, Email, FTP, remote shell etc for free? Twisted authorization framework Since the application in question has a very simple web part, and is predominately a remote GUI application, this seems the perfect solution. First make a tac file, which you will be running with the old favourite: twistd -ny mytac.tac On to some code: # This is my process for creating my WSGI application. You may have other methods. from graelsite.www.application import make_app from graelsite.www import config wsgi_app = make_app(config) # Now on to the real stuff from twisted.application import service, strports from twisted.web2 import server, channel, w...

I am lucky, as part of my job, I get to code PyGTK. The downside is of course that I have to deploy these applications on Windows. Now these Windows users (especially at the price they are paying for this stuff) don't install dependencies, don't understand what a scripting language or interpreter is, and frankly they should not have to. Unfortunately, the list of dependencies for a PyGTK application (at the very minimum) is Python, GTK, PyGTK, PyGObject, PyCairo. 5 installers! So very early on in this project I found the way to give them exactly what they want and deserve, a single-file executable installer. There are a few guides online about how to achieve this, but none seem to work, and none are particularly new. How do we achieve this? The toolchain involved consists of two elements: 1. Py2exe website 2. Inno Setup website NOTE: The documentation for all the tools used is extensive, I will not repeat everything in there. Py2exe Py2exe is a distutils extension that searches...

Well, congratulations to all the developers, we have just released PIDA 0.5. This release is probably the most significant in that we are finally happy with the core architecture. Although not everyone was agreed on how to do it, I was quite strict and employed this famous rule of thumb: Explicit is better than implicit And honestly, it really is. Previous versions of PIDA were dogged by overuse of magic and abuse of the declarative class syntax (and the associated metaclass madness). This caused one major problem. Things that should have been happening at runtime were happening at class declaration time. Which is fine if you can get your brain around it, but really evil when you want to reverse these things. As an example, PIDA services (and a service is like a plugin) can basically do anything to the application (a bit like the Eclipse plugin architecture - but we think better!). One example of this "doing anything" is to define global configuration options. In PIDA 0.4 opt...

In the last post about custom glade widgets, we briefly discussed how to add your PyGTK custom widgets to glade-3 so that they can be used in a user interface designer. In this post, this shall be extended to include how you can create a custom adaptor for a PyGTK widget to define additional behaviour. We shall be using a custom widget as an example, which I have called a "Service View". We plan that this widget has a content section, which is the main part, and also contains a close button at the bottom. We will use this widget as a general dockable view, kind of like a dialog which is a widget rather than a top-level window. This may seem utterly pointless, but it should be a useful component in an application that generates many different views in notebooks, like an IDE: Debugger, Terminal, Documentation Browser (etc). These can all share the same basic layout, but just replace the single main part of the widget. First let's write a widget: import gtk class ServiceView...

Glade 3, GTK User Interface Designer is really quite nice since release 3.1, since it does away with the gimp-like multi-window view and looks like a normal application. Another nice feature it has is being able to support widgets created in non-C (in our case PyGTK). Components of a plugin: A catalog file A support module Some icon pixmaps (optional) We shall use, for our example, the Kiwi Hyperlink widget, (which I wrote). This widget is an EventBox subclass, so we shall illustrate turning off some of the additional properties that are not required in the user interface designer. The catalog file This is described in http://glade.gnome.org/docs/catalogintro.html and the subsequent pages of the documentation. Essentially there are two sections of the catalog: The Widget definitions The Catalog list The Widget Definitions are like so: <glade-widget-classes> <glade-widget-class title="Hyper Link" name="HyperLink" /> </glade-widget-classes> These ar...

Well, you can tell I am procrastinating working today (but you can forgive me after the PIDA GSoc Application was rejected as a mentoring organisation), but I have hacked up an early version of a library (for Python) to manage blogger.com posting. I was becoming slightly upset with Blogger.com, since it does some insane things like puts BR tags in the middle of PRE elements, which meant the syntax highlighting script I wanted to use was going nuts. I also gradually realised that none of the blogging clients work with the "new Blogger.com" although for me (who has only been blogging a few months) it has been the "only Blogger.com". So, with a quick search around for documentation from code.google.com, I was able to discover that Blogger.com uses the google GData API with a bit of Atom mixed in. I have no real ideas about these specifications, but http://code.google.com/apis/blogger/gdata.html explains it. Note that another document I found: http://code.blogspot.co...

In the early stages of developing a Django application, I was deleting my sqlite database, and rerunning: python manage.py syncdb Every few minutes as the database models were changing. You might consider this some kind of evil thing to do, because we should all have our database models in concrete before writing any code but I am not that organised. One thing that was really distressing me was the fact that each time I ran a syncdb, I had to enter details for my default admin user. Username Email address Password Repeart Password This was fine for the first 400 times, then I got bored. The solution was to use pexpect. (There are probably command line options that you can give manage.py, but finding them out would have been more effort than this hack) Pexpect, http://pexpect.sourceforge.net/ is a pure python module for running external commands and controlling them as if you were a user. It has good documentation, and is perfect for needs of this nature. A quick browse through the exc...

Unit testing a GUI has always been something that scares me, and scares other people too. Because of this there are about a gazillion tools that behave in different ways, for example in Python, just have a look at: http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy#GUITestingTools (Since the whole world is obsessed with Web Applications, and this is becoming perversely true even in Python circles, some of these GUI Testing tools are actually web testing tools, but never mind that.) How do I do it? Well, we may talk in another blog about one of those tools, the awesome hack that is kiwi.ui.test but it is slightly restrictive on what you want to do, and how you should do it. Specifically, it needs you to have every widget named (which is fine when testing Glade-created interfaces, but a pain when hand-building them). Enter this one function that I found in the kiwi source. Kiwi is LGPL (whatever that means, but you should read the license if you are going to use it). import time i...