All Unkept
Posted in: Django  —  11 November 2010

Use JSON instead of YAML for Django test fixtures

Today I learned that JSON is much faster than YAML for use in Django fixtures. I sped up the rather slow test suite in an app of mine by nearly a factor of 2 by switching to JSON (the test suite seems to be dominated by time spent parsing fixtures). Here is a handy script I wrote to convert the fixtures:

#!/usr/bin/env python
#
# yaml2json.py

import datetime
import os
import sys
import simplejson
import yaml


class JSONEncoder(simplejson.JSONEncoder):
"""
JSONEncoder subclass that knows how to encode date/time and decimal types.
"""

DATE_FORMAT = "%Y-%m-%d"
TIME_FORMAT = "%H:%M:%S"

def default(self, o):
if isinstance(o, datetime.datetime):
return o.strftime("%s %s" % (self.DATE_FORMAT, self.TIME_FORMAT))
elif isinstance(o, datetime.date):
return o.strftime(self.DATE_FORMAT)
elif isinstance(o, datetime.time):
return o.strftime(self.TIME_FORMAT)
elif isinstance(o, decimal.Decimal):
return str(o)
else:
return super(JSONEncoder, self).default(o)


def main(fname):
assert os.path.splitext(fname)[1] == ".yaml"

with file(fname) as fp:
d = yaml.load(fp)

outname = os.path.splitext(fname)[0] + ".json"
with open(outname, "wb") as fp:
fp.write(JSONEncoder(indent=' ').encode(d))

if __name__ == '__main__':
main(sys.argv[1])

Pass the filename as the first argument, something like this:

for TMP in $(find . -name '*.yaml'); do yaml2json.py $TMP; done

(and you can add git/hg/bzr/svn add and remove commands into that line too).

You will need to update the fixtures attribute in your tests if you specified the '.yaml' extension.

Comments §

§ On 15 November 2010, EtienneRobillard wrote: 957
Nice code! I do also like using JSON for fixtures, but sometimes
YAML allow very fast schema changes due to being easily maintainanble by hand!

just me 2 cents!

Cheers,

Etienne



§ On 24 November 2010, Dave wrote: 965
Question and a comment. Were you using the Python or C YAML parser? PyYaml can be set up to use libYaml, which makes parsing much faster.

Comment: first line should be "#!/usr/bin/env python" to be compatible with virtualenv.

§ On 25 November 2010, luke wrote: 966

@Dave:

Fixed the first line, thanks.

With regards to libYAML parser, but I really don't know how to check quickly. Doing "import yaml; print yaml.__with_libyaml__" returns True, but I don't know what that means.


§ On 17 March 2011, dickeny wrote: 1002
as a standard python script, i suggest using 'import json as simplejson'.
and, with json module. the "indent=' '" should change to "indent=4".

Thanks for your script

Add comment

Format:

  • Javascript has to be on to get past my spam protection, and cookies, and there is a delay, sorry for any inconvenience!
  • I reserve the right to moderate comments.