Jacks_Depression

Jacks_Twisted Web Subprocess

Posted: 2012-04-15 17:30:55

One of the things I like about the Twisted framework is it's wide variety of functionality. You can serve web pages, have cron jobs and run processes all from the same reactor without any of them blocking. I've been using twisted mainly for serving webpages but recently I threw together a website for showcasing my photos. I decided to cut a lot of corners by running image processing programs directly on the operating system rather then hooking into python libraries.

The conversions worked pretty easy, mostly because they where fire and forget. Anything that needed to be converted was added to a queue. And when the conversion was done, the file was placed where it needed to be. Errors where logged. Pretty simple.

Then I started looking into the idea of having a pre-conversion thumbnail. So I could tag it before it was generated. In this case though, I did not want this thumbnail written to disk. The less files to manage the better.

This was more tricky for two reasons.

The first problem actually wasn't that hard to solve. Thankfully the smart people working on the Twisted project built in the functionality I wanted just by simply returning the number 1.

from twisted.web import resource

class myresource(resource.Resource):
    def render(self, request):
        return 1 # Not done yet

This puts the burden on you to write the data yourself and close the connection when done. So here is a good example of how that would be done.

from twisted.web import resource
from twisted.internet import reactor

class myresource(resource.Resource):
    def render(self, request):
        self.request = request
        reactor.callLater(10, self.latercall)
        return 1 # Not done yet

    def latercall(self):
        self.request.write('All done!')
        self.request.finish()

This way, you can wait on what ever you need while the differed call takes its time.


The other problem was that I could not find a single program to produce the output I wanted. However, I could pipe one into another program that would give me what I needed. And since I don't see any built in piping functions in Twisted, I had to do the pipe myself. The desired functionality would look something like this.

dcraw -e -c IMG_0003.CR2 | convert - -resize 256x256 jpg:-

The `dcraw' program would extract the thumbnail from my raw file (cr2) using the `-e' option. And it would write it to standard output using the `-c' option.

The `convert' program would then take that input represented by the `-' argument. It would then resize it and write it in jpeg format to the standard output represented by the `jpg:-' argument.

Here is the final example of how it all goes down. Be sure to close your standard input to the second program, or it won't know when to start processing the image.

from twisted.web import resource
from twisted.internet import reactor, protocol

class Myresource(resource.Resource):
    def render(self, request):
        request.setHeader('Content-Type', 'image/jpeg')
        dcpros = DcrawProtocal()
        dcpros.start(request, 'picture_file.cr2')
        return 1 # Not done yet

class DcrawProtocal(protocol.ProcessProtocol):
    def start(self, request, path):
        self.data = ''
        self.request = request
        processargs = ('dcraw', '-e', '-c', path)
        reactor.spawnProcess(self, 'dcraw', processargs)

    def outReceived(self, data):
        self.data += data

    def processEnded(self, reason):
        if reason.value.exitCode < 1:
            inprocess = InlineConvertProcess()
            inprocess.start(self.request, self.data)
        else:
            print 'Dcraw failed'
            self.request.finish()


class ConvertProtocal(protocol.ProcessProtocol):
    def start(self, request, data):
        self.data = data
        self.result = ''
        self.request = request
        processargs = ('convert', '-', '-resize', '256x256', 'jpg:-')
        reactor.spawnProcess(self, 'convert', processargs)

    def connectionMade(self):
        self.transport.write(self.data)
        self.transport.closeStdin()

    def outReceived(self, data):
        self.result += data
   
    def processEnded(self, reason):
        if reason.value.exitCode < 1:
            self.request.write(self.result)
        self.request.finish()