Calling the Google Drive API and other Google APIs asynchronously with Twisted

You may know that the Google API Python Client is built on httplib2. This is a reasonable general choice, but the tight coupling is unhelpful in situations where a different HTTP library, or an entirely different approach to network programming should be used. An example of this is Twisted.

Aside: I won't be going on about how awesome Twisted is, but let's just take it for granted that it is so awesome that I could not write this application without it.

Httplib2 is blocking, and that makes it incompatible with being run inside the Twisted reactor. Fortunately we are only the latest person to have this problem, and a solution exists:

twisted.internet.threads.deferToThread

api_call = drive.files().list()

def on_list(resp):
  for item in resp['items']:
    print item['title']

d = deferToThread(api_call.execute)
d.addCallback(on_list)
A blocking call will be called in a thread and will callback on the returned deferred when it is done. I appreciate that no one* is 100% happy with this solution. "Argh, threads!", "Argh, async!" But it is a testament to the Greatness of Twisted that it has this sort of facility to play well with other, less flexible, systems.

*Did I mention? I am 100% happy.

Please comment on this post in Google+

Popular posts from this blog

PyGTK, Py2exe, and Inno setup for single-file Windows installers

I am lucky, as part of my job, I get to code PyGTK. The downside is of course that I have to deploy these applications on Windows. Now these Windows users (especially at the price they are paying for this stuff) don't install dependencies, don't understand what a scripting language or interpreter is, and frankly they should not have to. Unfortunately, the list of dependencies for a PyGTK application (at the very minimum) is Python, GTK, PyGTK, PyGObject, PyCairo. 5 installers! So very early on in this project I found the way to give them exactly what they want and deserve, a single-file executable installer. There are a few guides online about how to achieve this, but none seem to work, and none are particularly new. How do we achieve this? The toolchain involved consists of two elements: 1. Py2exe website 2. Inno Setup website NOTE: The documentation for all the tools used is extensive, I will not repeat everything in there. Py2exe Py2exe is a distutils extension that searches...
Read more

Kiwi proxy widgets, a common widget API

Image
While we are on the notion of Kiwi , the PyGTK helper library, we should be discussing the proxy widgets. They offer a certain set of advantages over their plain PyGTK counterparts: They have a common signal emission API for change in content They offer a common API for setting and reading their value. They offer validation, and coercion They contain a few general improvements over the plain PyGTK widgets What does this mean? Simply it means you can make any data widget (eg Entry, CheckButton, Combo etc), you can connect its signal "content-changed", you can read the data with widget.read(), and set the value with widget.update(). If you have read any of the previous blogs about adapting Zope.schema fields to widgets, and you are alert, you will notice that this is amazingly useful. It means that we can define an interface for a general InputWidget type, something like: class IInputWidget(Interface): def read(): """Read the value in the widget"""...
Read more

Using threads in PyGTK

Well, all this thought of asynchronicity is getting to me. But since we mentioned how to use subprocesses and Twisted, I think it is only fair to talk a bit about threads. So, what will we be doing? Create an abstraction for running threads in PyGTK Use it Now the thing which we will be wanting to run is a generator, call it "take_really_long". And it will surely block, and might be a bit intensive, and well we obviously need something asynchronous. A generator is a bit nicer to use than a single plain long-running function, because we can update the user interface every time it returns, making it much much easier to do things like Progress Bars. You must remember two things when using threads with PyGTK: 1. GTK Threads must be initialised with gtk.gdk.threads_init: import gtk gtk.gdk.threads_init() 2. Any code that modifies the UI (basically any code) that is called from outside the main thread must be pushed into the main thread and called asynchronously in the main loop, w...
Read more