Null Disquisition

WebSockets in Python

Since the dawn of AJAX, web developers have longed for persistent server-side connections. For a while Comet was hailed as the bastion of “server push”, but deep down we knew it was just a hack. Now finally, years later, we have an API and a protocol being standardized for socket connections between the browser and the server – aptly named, WebSockets.

WebSockets are bi-directional communication channels that run on single TCP sockets allowing communication between the client and the server. Since they behave like regular INET sockets, we should be able to easily implement them with existing tools. However, when I was looking for example implementations in Python, I didn’t find anything that quite satisfied me.

Python sockets module

Now don’t make the mistake of thinking I’m a systems programmer. I have never written a low-level network application like this, and in fact this is my first time playing with sockets or select in Python. The root of all of this is the WebSocket itself, which is just a socket.

import socket
websocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
websocket.bind(("localhost", 9999))
websocket.listen(5)

That’s all you need to get the WebSocket up and running. Granted, it’s not very useful since you can’t connect to it (no handshake), but it’s a WebSocket nonetheless. When a client connects to the socket, it initiates the handshake with the following

GET / HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: localhost:9999
Origin: file://
Sec-WebSocket-Key1: x   d3L703 2  {63 k  L1( 90
Sec-WebSocket-Key2: ^    14   +40Z7R<12om I8  0[

??????????????

And expects a response in a similar form:

HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: file://
WebSocket-Location: ws://localhost:9999/
Sec-Websocket-Origin: file://
Sec-Websocket-Location: ws://localhost:9999/

??????????????

The “?” are random bits used in the challenge/response part of the handshake. Interesting note: In addition to failing to do the Challenge/Response, Chrome looks for the “Websocket-X” headers, while Safari (correctly) looks for the “Sec-Websocket-X” headers.

Here’s my full standalone WebSocket server: http://gist.github.com/512987

I won’t delve into the details of the implementation, namely because I’m sure it’s suboptimal. I was pretty happy with Challenge/Response piece. I read the spec from IETF and implemented it, nice and simple. Aren’t open standards great? I ended up having to do the handshake because Safari 5 won’t let you use a WebSocket otherwise.

Stay tuned for everyone’s favorite asynchronous demo: a chat program!

-David

Making Python's pickle safe(r)

Warning pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

import hmac
import hashlib
import base64
from cPickle import dumps
 # The unsigned pickled data
string_to_sign = dumps({'foo':"bar",'spam':"eggs",'the answer':42})
 # The signature object
signature = hmac.HMAC(key="my application's super secret key",
    msg= string_to_sign, digestmod=hashlib.sha256)
 # The signed string: store this
signed_string = string_to_sign + base64.encodestring(signature.digest())

import hmac
import hashlib
import base64
from cPickle import loads
 # Break up the signed string into message and signature
signature = signed_string[-45:]
message = signed_string[:-45]
 # Calculate the signature of the message
msg_sig = hmac.HMAC(key="my application's super secret key",
    msg= message, digestmod=hashlib.sha256)
 # See that it matches the given signature
assert base64.encodestring(msg_sig.digest()) == signature

API Functional Testing with Python

Recently, at work we have written a totally badass XML API for clients to interface with our data (sorry no public side yet). After some gentle reassuring (and some not-so-gentle arm twisting), I convinced my boss-man we could do this in Python with AWS on the back-end. We settled on the Turbogears 2.0 meta-framework using Amazon S3/SimpleDB. The whole experience was very educational for many reasons - one, we had never using something besides MySQL for a data store, two, we had never used a Python framework before, and three, we had never really developed an app with a proper set of tests. That final point, testing, is the subject of this entry. Py.Test, from the vaingloriously-named "py" module, is my unit testing framework of choice (I have written about it before). It provides a convenient way to collect tests and to write generative tests (which are super useful) for unit testing. After getting a few sets of unit tests rolled out for our API, we recognized that we would need some higher level tests - so called functional, or acceptance tests. ### Functional Tests Functional tests describe high-level tests that rely on the interaction of many components of the system, whereas a unit test will only test smaller, lower level components. For example, one (very high-level) functional test for an XML API would be to see that the resulting XML is well-formed. The well-formedness of an XML response from an API request is dependent on several components of the system. It requires proper request parsing, validation, error handling, template rendering, et al. A more typical test might be to see that the number of items returned by the API does not exceed a user-provided maximum, i.e., if the user requests http://api.example.com/?[request params]&max_count=10, no more than 10 results are shown. Now, how to go about running these tests. The number of functional testing frameworks is too great to mention (here's a bunch), but one that is well known and widely used is Selenium. It is written in Java and can do some pretty fancy stuff. However, one big drawback of Selenium is it's weight. It's heavy - it is Java after all, and requires a client server (whether you sacrifice your own cycles or a remote server). For the simple functional tests we were writing, it was completely overkill. After searching around for a Python functional testing framework (or at least something lighter than Selenium), it occurred to me that I could just use the test-collecting abilities of Py.Test plus some additional libraries. And that's what we did. ### Bottom Line Mix together PyXML, Urllib2, and Py.Test and you have a pretty powerful (and portable) testing suite in Python. PyXML extends the built-in 'xml' module with some really nice packages including an XPath parser which I love. ### Exempli Gratia Consider an API that has a "users" noun, and just one verb "show". We will allow one optional parameter order_by and one required parameter max_count. An valid URL would look like http://api.example.com/users/show?max_count=10&order_by=date. We'll start by creating the class that will contain the tests, and writing a function to get an XML doc given some url parameters.

import urllib2
from collections import defaultdict
from xml.dom import minidom
from xml import xpath
class TestUserNoun:
        def get_xml_doc(self,url_params):
                url = "http://api.example.com/users/show?"
                url += "max_count=%(max_count)s&order_by=%(order_by)s"
                url_p = urllib2.urlopen( url % defaultdict(str,url_params) )
                doc = minidom.parseString( url_p.read() )
                url_p.close()
                return doc

class TestUserNoun:
        ...
        def test_user_count(self):
                # Test several values of max_count
                counts = (5,10,15,20)
                def count_users(n):
                        # Test that the number of results returned is less than or equal to n
                        doc = self.get_xml_doc({'max_count':n})
                        user_count = len( xpath.Evaluate('/xpath/expr',doc.documentElement) )
                        assert user_count 
And you get the idea - one can write tests ad nauseum (although I'm not sure if there's such a thing as too many tests). Of course neither of these tests will work since the XPath expressions are not valid - I didn't really feel like spelling out a whole XML schema just for this example. There are plenty of good XPath tutorials out there. The basic idea here is you want to test all of your request parameters for the API to see a number of things: 

* Does the controller handle the requests properly? What about missing/extra parameters?
* Are errors handled properly?
* Is the resulting XML valid? This is implicitly done by parsing the XML document
* Does the resulting data correspond to the request parameters? This one will require the most tests to be written - don't forget about generative tests!

A powerful test suite means a robust application. When you have a nice set of tests, you can push your code with confidence - and believe me, that is a very rewarding and relieving feeling. Writing this API has been an extremely rewarding experience, and probably the most educational thing I've done programming-wise since I wrote a cross-browser javascript event library like 5 years ago.

So go forth, programmer - embrace testing and empower yourself.

-David

Weekend Project - CloudCached

A friend and I have been bouncing around the idea of a caching system that ran on Amazon's cloud for a while now. Basically something like memcached, but without the (very real) limitations of physical memory or the need of a whole server. Sure, it's hard to beat the speed of memory-level read access, but I think the appeal of a distributed, limitless cache might outweigh the slowdown. ### Idea Provide an interface for storing/retrieving serialized data on S3 Pretty simple idea, pretty simple implementation. Thanks to the S3 interface provided by [Boto](http://code.google.com/p/boto/ "Boto rocks!"), things were a lot easier. I'm going to keep this open source under the MIT license. You can check out the code on [GitHub repository](http://github.com/mumrah/cloudcached/tree/master "CloudCached on GitHub") - please feel free to fork, improve, submit, etc. ### Overview A quick walkthrough of the code will reveal truly how simple this is. The Client class provides basic CRUD methods for interfacing with S3: __put__, __get__, __update__, __delete__. The put and update methods store a timestamp as the "expires" header for the file to keep track of cache expiration. Also these two methods write a "type" header to the meta-data so CloudCached knows how to de-serialize the file.

class Client:
"Here's the class schema"
        def get(self, key)
        def put(self, key, value, time_to_expire=3600, replace=False)
        def update(self, key, value, time_to_expire=3600)
        def delete(self, key)

CloudCached Benchmarks (10 runs)
        --------------------------------------------------------
        Test                                  |        Average (s)                | Total (s)  
        --------------------------------------------------------
        GET integer                         |        0.0283360004425        | 0.283360004425
        GET string (32 byte) |        0.0315794944763        | 0.315794944763
        GET string (512KB)         |        0.1265994787220        | 1.265994787220
        PUT integer                         |        0.0650457143784        | 0.650457143784
        PUT string (32 byte) |        0.0563205003738        | 0.563205003738
        PUT string (512KB)         |        0.1773290872570        | 1.773290872570
        --------------------------------------------------------

Python unit testing super fun time

There's a weird thing that happens after a long night of mind-blowing back-breaking coding. Well, hacking in this case. Every time I stay up late working really hard on something, I feel compelled to blog/tweet/emote about my experience so others might feel sympathy/compassion for me. Even though I'm dizzyingly tired, and have to get up in ~5 hours, I cannot deny this urge to massage my ego. So tonight I bring to you the joy of unit testing in Python. I've been using py.test, and loving it. It extends the basic functionality of Python's built-in module, unittest (which is really not that bad). The main improvements are in the simplicity of writing the tests. Py.test supports unit testing on methods, classes, even whole modules. Here's your first test

def test_iszero():
        assert 1==0

def isEven(x):
        assert x%2==0
def test_evenNumbers():
        n = [1,2,3,4,5,6]
        for x in n:
                yield isEven,x

Python static class members and You

After getting yelled at for not grading my student's homework, I decided to ignore the threatening emails and continue doing what I feel like. Undergrads, know this: TAs don't really care about you - sorry. I was debugging some code built on top of my awesome HTMLParser, and kept having a really frustrating problem. Some of my class variables were not getting reset during the __init__ call. So I poke around and after a while discover (buried in my libraries)

class Foo:
    a = True
    b = []
    c = []
    def __init__(self):
                ""

class Foo:
    a = "Hello"
print Foo.a
>> Hello

class Foo:
    a = []
    def __init__(self):
        print self.a
        self.a.append(1)
f = Foo()
f = Foo()
f = Foo()
>> []
>> [1]
>> [1,1]

a = [1,2,3,4]
        b = a
        c = a[:]
        b[0] = 5
        c[0] = 6
        print a
        print b
        print c
        >> [5,2,3,4]
        >> [5,2,3,4]
        >> [6,2,3,4]

HTMLParser, not for the faint of heart

class MyParser(HTMLParser):
    def handle_starttag(self,tag,attrs):
        if tag == "a":
            print "Found link:",attrs
    def handle_startendtag(self,tag,attrs):
        if tag == "img":
            print "Found image:",attrs
parser = MyParser()
parser.feed(rawhtml)

linepos = []
charpos = 0
for line in self.html.split("\n"):
        self.linepos.append(charpos)
        charpos += len(line)
parser = MyParser(linepos=linepos)

def getcharpos(self):
    return self.linepos[self.lineno-1] + self.offset

1d Fokker-Plank equation

As promised, I bring pretty pictures. The past few days I've been working on a solution to the 1d diffusion equation with a drift term, better known as the Fokker-Planck equation.

Sexy, I know. Anyhow, I finally worked out the Python code to get it rolling (literally!). The test system I did has periodic boundary conditions and an initial condition of a sharply-peaked Gaussian (a = 20). I'll spare the details and jump to the fun part.

Here's the Python code that made it happen (scipy and matplotlib required).

-David