All Unkept
Posted in: Python, Web development, Django  —  December 28, 2007 at 10:51 PM

AJAX validation with Django, newforms and MochiKit

by Luke Plant

I present below some example code for using Django, newforms and MochiKit to do AJAX validation, in case anyone else is doing the same, complete with a live demo. My aim is to non-intrusively add AJAX validation to a form without duplicating any validation logic.

(BTW, complete code for the relevant bits is available at the bottom. This code will not run out-of-the box, but should show you all the relevant parts of the solution. Note also that most of the code presented is straightforward Django 'newforms', and not AJAX related).

Before we get going, some context: the Christian Camps in Wales web site has a forum system which allows people to create polls. I rewrote this recently to use newforms, mainly to keep up to date with Django, and also to try out some AJAX.

I first had to write a 'Form' subclass which encapsulates the logic for validating and processing the form. As it happens, the form is based on a model 'Poll', but it requires more fields. In particular, this single form is used to create all the PollOption objects that are attached to the Poll. This is handled by a simple textarea field, into which all the options are entered on separate lines. In Django, we implement this using a custom 'Field' that uses the 'Textarea' widget and adds the relevant validation.

The PollOptionListField looks like this:

class PollOptionListField(forms.CharField):
    widget = widgets.Textarea
    def clean(self, value):
        """Parses a string containing multiple lines of text,
        and returns a list of poll options or raises ValidationError"""
        value = super(PollOptionListField, self).clean(value)
        l = filter(lambda opt: len(opt) > 0, map(string.strip, value.split("\n")))

        if len(l) == 0:
            raise forms.ValidationError(u"At least one option must be entered")

        max_length = PollOption._meta.get_field('text').max_length
        if len(filter(lambda opt: len(opt) > max_length, l)) > 0:
            raise forms.ValidationError(u"Options may not be more than %s chars long" % max_length)

        return l

(Yes, there are a couple of places I could have used list comprehensions there. I'm afraid I find that I think in 'filter' and 'map', and find these so much more expressive than the idiomatic Python).

I need to create a Form that subclasses ModelForm (so that we get some fields autogenerated for us), and adds some others. I also don't want to use the default widgets for the 'Voting starts' and 'Voting ends' fields, so I end up with this:

class CreatePollForm(cciwforms.CciwFormMixin, forms.ModelForm):
    voting_starts = forms.SplitDateTimeField(widget=widgets.SplitDateTimeWidget,
                                             label="Voting starts")
    voting_ends = forms.SplitDateTimeField(widget=widgets.SplitDateTimeWidget,
                                           label="Voting ends")
    polloptions = PollOptionListField(label='Options (one per line)')

    class Meta:
        model = Poll
        fields = ['title', 'intro_text', 'outro_text', 'voting_starts', 'voting_ends', 'rules', 'rule_parameter']

    def __init__(self, *args, **kwargs):
        instance = kwargs.get('instance', None)
        initial = kwargs.pop('initial', {})
        if instance is not None:
            initial['polloptions'] = '\n'.join(po.text for po in instance.poll_options.order_by('listorder'))
        kwargs['initial'] = initial
        super(CreatePollForm, self).__init__(*args, **kwargs)

You may be wondering about 'CciwFormMixin' -- I'll come onto that. You can also see that the 'PollOptionListField' requires some initial setup work which is needed when editing an existing Poll i.e. it fills the textarea with the relevant info from the PollOptions. This is added to the constructor of the form.

(One problem with the code above is that you don't have as much control over the order of the fields as you need. Perhaps this will be sorted out at some point, although a nice method of doing it isn't obvious. But you can do this:

CreatePollForm.base_fields.keyOrder =
   ['title', 'intro_text', 'polloptions', 'outro_text', 'voting_starts', 'voting_ends', 'rules', 'rule_parameter']

Please don't tell the core devs I did that, I'm not sure if it's allowed :-)

Now for the main view function. It handles both 'Edit' and 'Create' mode, which means that it is mapped from two different sets of URLs, and it is passed a 'poll_id' when in edit mode, and the code is slightly more complex than a typical newforms example. It is decorated with 'member_required', which will handle redirection to a login screen etc. It also includes some bits and pieces that you can't understand outside the context of my application. (Each poll is attached to a forum, and forums can be in different places, so there is code to handle that. 'standard_extra_context' is a function which returns some standard stuff for my pages, etc).

@member_required
def edit_poll(request, poll_id=None, breadcrumb_extra=None):
    if poll_id is None:
        suffix = 'add_poll/'
        title = u"Create poll"
        existing_poll = None
    else:
        suffix = '/'.join(request.path.split('/')[-3:]) # 'edit_poll/xx/'
        title = u"Edit poll"
        existing_poll = get_object_or_404(Poll.objects.filter(id=poll_id))

    cur_member = get_current_member()
    if not cur_member.has_perm(Permission.POLL_CREATOR):
        return HttpResponseForbidden("Permission denied")
    if existing_poll and existing_poll.created_by != cur_member:
        return HttpResponseForbidden("Access denied.")

    forum = _get_forum_or_404(request.path, suffix)
    c = standard_extra_context(title=title)

    if request.method == 'POST':
        form = CreatePollForm(request.POST, instance=existing_poll)

        if request.GET.get('format') == 'json':
            return HttpResponse(utils.python_to_json(form.errors),
                                mimetype='text/javascript')

        if form.is_valid():
            new_poll = form.save(commit=False)
            new_poll.created_by_id = cur_member.user_name
            new_poll.save()

            if existing_poll is None:
                # new poll, create a topic to go with it
                topic = Topic.create_topic(cur_member, new_poll.title, forum)
                topic.poll_id = new_poll.id
                topic.save()
            else:
                # It will already have a topic associated
                topic = new_poll.topics.all()[0]
                topic.subject = new_poll.title
                topic.save()

            update_poll_options(new_poll, form.cleaned_data['polloptions'])

            return HttpResponseRedirect(topic.get_absolute_url())
    else:
        if existing_poll:
            form = CreatePollForm(instance=existing_poll)
        else:
            today = datetime.today()
            today = datetime(today.year, today.month, today.day)
            form = CreatePollForm(initial=dict(voting_starts=today))

    c['form'] = form
    c['existing_poll'] = existing_poll
    return render_to_response('cciw/forums/edit_poll.html',
                              context_instance=RequestContext(request, c))

The interesting part for the purpose of this code is the JSON handling code. Very simply, if it is a POST request and has the 'format=json' in the request query string, then just do validation and return the result as JSON, otherwise carry and do the processing. Those 2 lines are the only bit of server side logic required to enable JSON validation on this form.

For completeness, I should show the 'python_to_json' function that I use to do this. Due to the i18n framework, it is slightly complicated:

from django.utils import simplejson
from django.utils.functional import Promise
from django.utils.encoding import force_unicode

class LazyEncoder(simplejson.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Promise):
            return force_unicode(obj)
        return obj

json_encoder = LazyEncoder(ensure_ascii=False)

# This can handle contents of newforms.Form.errors
def python_to_json(obj):
    return json_encoder.encode(obj)

At this point, I will introduce CciwFormMixin. This is a class that implements the rendering of a form in a standardised format. This format is important, since client side we are going to have to do DOM manipulation to display and clear validation error messages, and that is going to have to be aware of the structure of the page. At the moment, I don't have a very clever way of doing this -- it is the only place in this solution where there is duplication of information. To minimise the problem, we try to make the form rendering as simple as possible, and rely on CSS to do most of the formatting. The formatting I've chosen is basically the same as BaseForm.as_p(), with some slight modifications, mainly to include some 'id's to make life easier later, and some 'class' attributes to simplify styling.

from django.utils.html import escape
from django.newforms.forms import BoundField
from django.utils.encoding import force_unicode
from django.utils.safestring import mark_safe

class CciwFormMixin(object):
    """Form mixin that provides the rendering methods used on the CCIW site"""
    def as_p(self):
        "Returns this form rendered as HTML <p>s."

        ## Remember to change cciwutils.js standardform_ functions if the
        ## HTML here is changed
        normal_row = '<p id="%(divid)s" class="%(class)s">%(label)s %(field)s%(help_text)s</p>'
        error_row = u'<div class="validationErrorTop">%s</div>'
        help_text_html = u' %s'
        normal_class = 'formrow'
        error_class = 'formrow validationErrorBottom'

        top_errors = self.non_field_errors() # Errors that should be displayed above all fields.
        output, hidden_fields = [], []
        for name, field in self.fields.items():
            bf = BoundField(self, field, name)
            bf_errors = self.error_class([escape(error) for error in bf.errors]) # Escape and cache in local variable.
            if bf.is_hidden:
                if bf_errors:
                    top_errors.extend(['(Hidden field %s) %s' % (name, force_unicode(e)) for e in bf_errors])
                hidden_fields.append(unicode(bf))
            else:
                if bf_errors:
                    output.append(error_row % force_unicode(bf_errors))
                    cssclass = error_class
                else:
                    cssclass = normal_class
                if bf.label:
                    label = escape(force_unicode(bf.label))
                    # Only add the suffix if the label does not end in
                    # punctuation.
                    if self.label_suffix:
                        if label[-1] not in ':?.!':
                            label += self.label_suffix
                    label = bf.label_tag(label) or ''
                else:
                    label = ''
                if field.help_text:
                    help_text = help_text_html % force_unicode(field.help_text)
                else:
                    help_text = u''
                output.append(normal_row % {
                        'errors': force_unicode(bf_errors),
                        'label': force_unicode(label),
                        'field': unicode(bf),
                        'help_text': help_text,
                        'class': cssclass,
                        'divid': "div_id_%s" % bf.name
                        })
        if top_errors:
            output.insert(0, error_row % top_errors)
        if hidden_fields: # Insert any hidden fields in the last row.
            str_hidden = u''.join(hidden_fields)
            output.append(str_hidden)
        return mark_safe(u'\n'.join(output))

The template file contains a bit of javascript (explained later), and some fairly standard HTML stuff. The 'form.as_p' uses the method above to do its rendering.

{% extends "cciw/standard.html" %}
{% block content %}

<script type="text/javascript" src="{{ pagevars.media_root_url }}/javascript/MochiKit/MochiKit.js"></script>
<script type="text/javascript" src="{{ pagevars.media_root_url }}/javascript/cciwutils.js"></script>
<script type="text/javascript">
<!--

// DEBUG:
//createLoggingPane();

var editpollformname = 'editpollform';

function get_input_change_handler(control_name, control_id) {
        function on_input_change(ev) {
                d = cciw_validate_form(editpollformname);
                d.addCallbacks(standardform_get_validator_callback(control_name, control_id),
                                standardform_ajax_error_handler);
        };
        return on_input_change;
}

connect(window, 'onload',
        function(ev) {
                // Add event handlers to everything that can be validated.
                add_form_onchange_handlers(editpollformname, function(input) {
                        return get_input_change_handler(input.name, input.id);
                });
        }
);


//-->
</script>

{% if existing_poll %}
<p>Edit the poll details below.  Please note, you can either:</p>
<ul>
  <li>Change the text on existing options (votes for that option will
  be preserved)</li>
  <li>Remove one or more option</li>
  <li>Add one or more option</li>
</ul>
<p>But do not do more than one of the above at once!</p>
{% else %}
<p>Enter the poll details below.  A new topic will be
created for this poll.</p>
{% endif %}

{% if form.errors %}
        <div class="userError">
        Please check your input:
        </div>
{% endif %}

<form action="" method="post" id="editpollform">

<div class="form">
{{ form.as_p }}
</div>

<div><input type="submit" name="submit" id="submit" value="Submit" /></div>
</form>

{% endblock %}

In terms of javascript, we need to add event handlers to all the editable inputs in the form. 'add_form_onchange_handlers' does this for us -- it accepts the id of the form to handle, and a second parameter which is a function that itself returns an event handler when passed an input control. This way, the function can return a different handler for each control, which is what we need: when the use has changed the 'title' field, the whole form will be validated, but the only errors we will flag up are errors in the 'title' field.

The 'on_input_change' function is therefore called every time a field is changed. This calls 'cciw_validate_form' (see below), which returns a MochiKit 'Deferred' object. Once we have added callbacks to this object, it will asyncronously do the AJAX request, and then call the control-specific validation functions that we have generated.

The rest of the Javascript (apart from MochiKit, of course) is here:

// A lot borrowed from Dojo, ported to MochiKit

function inArray(arr, value) {
        return findValue(arr, value) != -1;
}

function defaultFormFilter(/*DOMNode*/node) {
        // Used by encodeForm
        var type = (node.type||"").toLowerCase();
        var accept = false;
        if(node.disabled || !node.name) {
                accept = false;
        } else {
                // We don't know which button was 'clicked',
                // so we can't include any as an element to submit
                // Also can't submit files
                accept = !inArray(["file", "submit", "reset", "button", "image"], type);
        }
        return accept; //boolean
}

function encodeForm (/*DOMNode*/formNode, /*Function?*/formFilter){
        //summary: Converts the names and values of form elements into an URL-encoded
        //string (name=value&name=value...).
        //formNode: DOMNode
        //formFilter: Function?
        //    A function used to filter out form elements. The element node will be passed
        //    to the formFilter function, and a boolean result is expected (true indicating
        //    indicating that the element should have its name/value included in the output).
        //    If no formFilter is specified, then defaultFormFilter() will be used.
        if((!formNode)||(!formNode.tagName)||(!formNode.tagName.toLowerCase() == "form")){
                throw new Error("Attempted to encode a non-form element.");
        }
        if(!formFilter) { formFilter = defaultFormFilter; }
        var enc = encodeURIComponent;
        var values = [];

        for(var i = 0; i < formNode.elements.length; i++){
                var elm = formNode.elements[i];
                if(!elm || elm.tagName.toLowerCase() == "fieldset" || !formFilter(elm)) { continue; }
                var name = enc(elm.name);
                var type = elm.type.toLowerCase();

                if(type == "select-multiple"){
                        for(var j = 0; j < elm.options.length; j++){
                                if(elm.options[j].selected) {
                                        values.push(name + "=" + enc(elm.options[j].value));
                                }
                        }
                }else if(inArray(["radio", "checkbox"], type)){
                        if(elm.checked){
                                values.push(name + "=" + enc(elm.value));
                        }
                }else{
                        values.push(name + "=" + enc(elm.value));
                }
        }

        // now collect input type="image", which doesn't show up in the elements array
        var inputs = formNode.getElementsByTagName("input");
        for(var i = 0; i < inputs.length; i++) {
                var input = inputs[i];
                if(input.type.toLowerCase() == "image" && input.form == formNode
                        && formFilter(input)) {
                        var name = enc(input.name);
                        values.push(name + "=" + enc(input.value));
                        values.push(name + ".x=0");
                        values.push(name + ".y=0");
                }
        }
        return values.join("&") + "&"; //String
}


function add_form_onchange_handlers(formname, mk_input_change_handler) {
        // Summary: Adds 'onchange' handlers to all inputs in a form
        // formname: name of the form in the DOM
        // mk_input_change_handler: when called with one of the
        //   form elements, returns a handler to be connected to
        //   that element.
        var inputs = $(formname).elements;
        for (var i = 0; i < inputs.length; i++) {
                if (defaultFormFilter(inputs[i])) {
                        connect(inputs[i], 'onchange', mk_input_change_handler(inputs[i]));
                }
        }
}

function cciw_validate_form(formname) {
        // Summary: do AJAX validation of the form, using normal conventions,
        // returns a MochiKit Deferred object.
        var data = encodeForm($(formname));
        var d = doXHR("?format=json",
                        {
                                method:'POST',
                                sendContent: data,
                                headers: {
                                  "Content-Type": "application/x-www-form-urlencoded"
                                }
                        }
        );
        return d;
}

function django_normalise_control_id(control_id) {
        // Summary: returns the id/name that corresponds to
        // the whole Django widget.  For MultiWidgets,
        // this strips the trailing _0, _1 etc.
        return control_id.replace(/^(.*)(_\d+)$/, "$1");
}

// standardform_* functions depend on the HTML in CciwFormMixin
function standardform_get_form_row(control_id) {
        var rowId = 'div_' + control_id;
        var row = $(rowId);

        if (row != null) { return row; }

        logError("Row for control " + control_id + " could not be found.");
        return null;
}

function standardform_display_error(control_id, errors) {
        var row = standardform_get_form_row(control_id);
        if (row == null) {
                return;
        }
        if (!hasElementClass(row, "validationErrorBottom")) {
                // insert <ul> before it
                var newnodes = DIV({'class':'validationErrorTop'},
                                UL({'class':'errorlist'},
                                    map(partial(LI, null), errors)));
                row.parentNode.insertBefore(newnodes, row)
                addElementClass(row, "validationErrorBottom");
        }
}

function standardform_clear_error(control_id) {
        var row = standardform_get_form_row(control_id);
        if (row == null) {
                return;
        }
        if (hasElementClass(row, "validationErrorBottom")) {
                removeElementClass(row, "validationErrorBottom");
                // there will be a previous sibling
                // which holds the error message
                removeEmptyTextNodes(row.parentNode);
                row.parentNode.removeChild(row.previousSibling);
        }
}

function standardform_get_validator_callback(control_name, control_id) {
        var control_name_n = django_normalise_control_id(control_name);
        var control_id_n = django_normalise_control_id(control_id);
        function handler(req) {
                var json = evalJSONRequest(req);
                logDebug("JSON: " + req.responseText);
                var errors = json[control_name_n];
                if (errors != null && errors != undefined) {
                        standardform_clear_error(control_id_n);
                        standardform_display_error(control_id_n, errors);
                } else {
                        standardform_clear_error(control_id_n);
                }
        };
        return handler;
}

function standardform_ajax_error_handler(err) {
        logError("Err " + repr(err));
}

The function 'encodeForm' is borrowed from Dojo and adapted for use with MochiKit. It does the same thing that a browser does when submitting a form. This is important: since I'm using POST for my AJAX call, Django's CsrfMiddleware will require the 'csrfmiddlewaretoken' to be present or the request will be rejected. encodeForm will automatically include it for us, so we will get through. For this particular example, there are no security concerns and a GET request could be used, but that would require additional code paths to think about, with additional chances of mistakes and information leaks occuring via the JSON path.

Other functions of note: the 'standardform_get_form_row', 'standardform_display_error' and 'standardform_clear_error' functions are built for use with the formatting that CciwFormMixin provides. They use some of the nice utilities that MochiKit provides for creating and manipulating DOM elements.

'django_normalise_control_id' is required to work with MultiWidgets: widgets like SplitDateTimeWidget produce more than 1 input control for each field, with names like 'field_name_0', 'field_name_1' etc. To find the row that these controls belong to, and the field in form.errors that we should be looking for, we have to strip off the '_0' bit. (This bit is, of course, slightly fragile with respect to changes in Django's internals).

OK, I guess you want to see it in action. Well, go to:

http://www.cciw.co.uk/website/forum/add_poll/

You can log in as 'guest', with password 'guest'. Try changing the date to something invalid, or setting the title and then clearing it. (Please don't stray from that forum, BTW, since this is the live site. I will clear out things created by 'guest' every now and then.) In case it goes offline, here is a screenshot.

OK, there was a fair amount of code there. The good news is:

  • Once you have something like this in place, adding AJAX validation to your other newforms is a few lines (2 lines of Python, a few lines of Javascript to hook up the event handlers).
  • For validation, we have completely obeyed DRY. We have derived most of the form validation from the database level contraints, and the validation that exists server side is reused client side
  • I already wrote most of the code for you!

Thanks to MochiKit, you get browser compatibility for free. I developed using Konqueror (and Firefox for some debugging), and Internet Explorer and Opera just worked.

As promised, all the source code (these are extracts from my live code for the purpose of this blog post):

Comments §

blog comments powered by Disqus