Spawning subprocess with PyGTK using Twisted

Well, it is an age-old problem: How to schedule long-running tasks withing a GUI main loop (in our case PyGTK). There are a few ways:

1. Use Python's subprocess module and select on the pipe with gobject's io_add_watch
   
2. Use GTK's built in subprocess spawning abilities
   
3. Use Twisted
   

1 & 2 are reasonable approaches, and they both work. Of course 1 won't work on Win32. The only problem with both 1 & 2 is that they use gobject's polling functions to achieve asynchronicity. This is nice when we are forced in a PyGTK main loop, but really not nice when the application wants to run in command line mode, and we really want to be able to share the execution code between different UIs, including perhaps other toolkits.

Enter Twisted. We need to do two things with Twisted:

1. Make sure Twisted knows we are running with PyGTK
   
2. Launch the process
   

Making sure Twisted knows that we are running inside PyGTK is quite easy (though I imagine the implementation was painful). To do this, you must install the gtk2reactor before importing any other reactors like so:

from twisted.internet import gtk2reactor
gtk2reactor.install()
from twisted.internet import reactor

Ok, I did say it was pretty easy. Now all that you need to remember with this is that you should now run your main loop with reactor.run, and not gtk.main.

Now we should think about spawning our subprocess. We will do this by using reactor.spawnProcess, which looks like:

twisted.internet.reactor.spawnProcess = spawnProcess(self,
    processProtocol,
    executable,
    args=(),
    env={},
    path=None,
    uid=None, gid=None, usePTY=0, childFDs=None
)

The only non-normal thing here is the processProtocol paramterer. All the other paramteres are standard things for things like subprocess.Popen. The processProtocol instance should be an instance of twisted.internet.protocol.ProcessProtocol, and defines how data is read from the pipes constructed to spawn the subprocess.

You can just use ProcessProtocol without overriding, but that will do nothing useful, not even print the results, so here is an example with our own ProcessProtocol class.

import os
# Have you remembered to install the gtk2reactor?
from twisted.internet import reactor
from twisted.internet.protocol import ProcessProtocol

class EchoingProcessProtocol(ProcessProtocol):

    # Will get called when the subprocess has data on stdout
    def outReceived(self, data):
        print 'STDOUT:', data
    
    # Will get called when the subprocess has data on stderr
    def errReceived(self, data):
        print 'STDERR:', data

    # Will get called when the subprocess starts
    def connectionMade(self):
        print 'Started running subprocess'

    # Will get called when the subprocess ends
    def processEnded(self, reason):
        print 'Completed running subprocess'

# Spawn the process and copy across the environment
reactor.spawnProcess(EchoingProcessProtocol(), 'ls', ['ls', '-al'], env=os.environ)
reactor.run()

Now the subprocess will execute, and anything written to the child's stdout will be printed to the screen.

This may seem entirely unremarkable, but this is now ready to plug into a GUI. Since the callback outReceived is called inside the gtk main loop, it won't block and it will be called when necessary, so it may as well do something like:

def outReceived(self, data):
    self.text_view.get_buffer().insert(
        self.text_view.get_buffer().get_end_iter(), data
    )

Which would add the line to a textview control (called self.text_view).

Links:
Twisted How-to Processes
Twisted How-to PYGTK