Null Disquisition

Recently, at work we have written a totally badass XML API for clients to interface with our data (sorry no public side yet). After some gentle reassuring (and some not-so-gentle arm twisting), I convinced my boss-man we could do this in Python with AWS on the back-end. We settled on the Turbogears 2.0 meta-framework using Amazon S3/SimpleDB. The whole experience was very educational for many reasons – one, we had never using something besides MySQL for a data store, two, we had never used a Python framework before, and three, we had never really developed an app with a proper set of tests. That final point, testing, is the subject of this entry.

Py.Test, from the vaingloriously-named “py” module, is my unit testing framework of choice (I have written about it before). It provides a convenient way to collect tests and to write generative tests (which are super useful) for unit testing. After getting a few sets of unit tests rolled out for our API, we recognized that we would need some higher level tests – so called functional, or acceptance tests.

Functional Tests

Functional tests describe high-level tests that rely on the interaction of many components of the system, whereas a unit test will only test smaller, lower level components. For example, one (very high-level) functional test for an XML API would be to see that the resulting XML is well-formed. The well-formedness of an XML response from an API request is dependent on several components of the system. It requires proper request parsing, validation, error handling, template rendering, et al. A more typical test might be to see that the number of items returned by the API does not exceed a user-provided maximum, i.e., if the user requests http://api.example.com/?[request params]&max_count=10, no more than 10 results are shown.

Now, how to go about running these tests. The number of functional testing frameworks is too great to mention (here’s a bunch), but one that is well known and widely used is Selenium. It is written in Java and can do some pretty fancy stuff. However, one big drawback of Selenium is it’s weight. It’s heavy – it is Java after all, and requires a client server (whether you sacrifice your own cycles or a remote server). For the simple functional tests we were writing, it was completely overkill. After searching around for a Python functional testing framework (or at least something lighter than Selenium), it occurred to me that I could just use the test-collecting abilities of Py.Test plus some additional libraries. And that’s what we did.

Bottom Line

Mix together PyXML, Urllib2, and Py.Test and you have a pretty powerful (and portable) testing suite in Python. PyXML extends the built-in ‘xml’ module with some really nice packages including an XPath parser which I love.

Exempli Gratia

Consider an API that has a “users” noun, and just one verb “show”. We will allow one optional parameter order_by and one required parameter max_count. An valid URL would look like http://api.example.com/users/show?max_count=10&order_by=date.

We’ll start by creating the class that will contain the tests, and writing a function to get an XML doc given some url parameters.

import urllib2
from collections import defaultdict
from xml.dom import minidom
from xml import xpath
class TestUserNoun:
    def get_xml_doc(self,url_params):
        url = "http://api.example.com/users/show?"
        url += "max_count=%(max_count)s&order_by=%(order_by)s"
        url_p = urllib2.urlopen( url % defaultdict(str,url_params) )
        doc = minidom.parseString( url_p.read() )
        url_p.close()
        return doc

N.B., you can create a specific User-Agent with urllib2 if so desired, and defaultdict is used so we don’t have to check if the incoming dict (url_params) has everything we need for the url string.

Now we can start writing some tests

class TestUserNoun:
    ...
    def test_user_count(self):
        # Test several values of max_count
        counts = (5,10,15,20)
        def count_users(n):
            # Test that the number of results returned is less than or equal to n
            doc = self.get_xml_doc({'max_count':n})
            user_count = len( xpath.Evaluate('/xpath/expr',doc.documentElement) )
            assert user_count <= n
        for c in counts:
            yield count_users,c
    def test_order_by_date(self):
        # See that each item is older than the previous one
        doc = self.get_xml_doc({'max_count':10,'order_by':"date"})
        items = xpath.Evaluate('/xpath/expr',doc.documentElement)
        # Get the date of the first item
        last_date = xpath.Evaluate('@date_attr',items[0])
        # Compare the date of each item to the previous one
        for item,i in zip(items[1:],range(len(items[1:]))):
            item_date = xpath.Evaluate('@date_attr',item)
            assert item_date <= last_date
            last_date = item_date

And you get the idea – one can write tests ad nauseum (although I’m not sure if there’s such a thing as too many tests). Of course neither of these tests will work since the XPath expressions are not valid – I didn’t really feel like spelling out a whole XML schema just for this example. There are plenty of good XPath tutorials out there. The basic idea here is you want to test all of your request parameters for the API to see a number of things:

• Does the controller handle the requests properly? What about missing/extra parameters?
• Are errors handled properly?
• Is the resulting XML valid? This is implicitly done by parsing the XML document
• Does the resulting data correspond to the request parameters? This one will require the most tests to be written – don’t forget about generative tests!

A powerful test suite means a robust application. When you have a nice set of tests, you can push your code with confidence – and believe me, that is a very rewarding and relieving feeling. Writing this API has been an extremely rewarding experience, and probably the most educational thing I’ve done programming-wise since I wrote a cross-browser javascript event library like 5 years ago.

So go forth, programmer – embrace testing and empower yourself.

-David