The problem

Many programs build up sentences using bits - often a template into which different things might be substituted. However, the things you substitute into a sentence can change the sentence, and vice-versa, in ways that are not anticipated by the programmer.

For example, plurals. In English, you might try code like this:

if n == 1:
    return "I have 1 pig"
else:
    return "I have %s pigs" % n

Localising these strings gives problems, because the rules for how to create plural forms is different in every language.

This specific problem is generally considered 'solved' by the use of gettext, but many more exist.

For example, we have another problem as soon as we start substituting nouns:

"Delete selected %s?" % object_name

Various attributes about the noun could affect the sentence. In French, the adjective "selected" needs to agree in gender with the noun being substituted in. So you cannot lookup the translations for "Delete selected %s" and for object_name separately. (This is a real example picked from Django source code).

Further, depending on how the sentence uses the noun, the form of the noun might need to change. For example, the noun might appear in the accusative position for a given sentence and language, which requires a different form of the noun to be used compared to the nominative form.

Several other examples of this appeared in Django ticket 11688. One proposed solution on that ticket would require a huge amount of knowledge and effort on the part of Django programmers, and almost certainly would not work anyway.

This post is an attempt to come up with a better solution, or at least kick start discussion. I haven't been able to find any solutions to this problem online, and most people seem to be just using gettext, which is a 95% solution — and maybe that is good enough for most people.

[Update 2013-02-19 - ‘Richard’ pointed me to Locale::Maketext article, which has in essence a similar approach to what I've done here]

Assumptions and simplifications

We will assume that a sentence is a composable unit of meaning, such that sentences can be translated independently. So, if in language A we have sentences 1 and 2, in that order, we can translate these into language B by translating sentence 1 and sentence 2 independently, and putting them together in the same order.

This is, no doubt a simplification. In some languages, the two sentences might make more sense if re-ordered, or combined, or split in various ways. Indeed, some languages may not have a truly equivalent concept of 'sentence' at all.

However, we have to do something, and this is a reasonable approximation.

Requirements

We need a powerful way of defining sentences in a given human language. It must be powerful enough that the person doing the translation can do anything they need, without the programmer needing to be aware of all the things in the language that will cause difficulty.

So, we'll start with a full programming language, and chop out the things we shouldn't need.

We shouldn't need side effects - translation should be a pure function. So we'll use a purely functional programming language without side effects.

We need something fairly readable, because translators are going to have to use it. It should be as close as possible to declarative in style.

Pattern matching seems like a great fit for some of our needs.

Possible solution

Given the above requirements, let's start with a Haskell-like pure functional language, whose pattern matching will be extremely helpful. It will obviously have IO removed, and no type signatures (but that won't stop us inferring them and being able to statically type-check the code). Everything else will be borrowed directly from Haskell, so that I can avoid having to make up my own syntax and semantics.

If the concept works, we can argue about better or simpler syntax for some constructs, or helper functions that aren't part of the Haskell prelude.

Hopefully, we will find a relatively small subset of Haskell that is needed to give us all the power we need to solve this problem - a subset small enough that we could guarantee non-termination ideally, to avoid problems with translations created by malicious agents.

This will be an example based exploration.

Let's assume that every sentence can be generated by a function. The function will take as parameters all the substutions that are needed, and return the translated string.

So, suppose we have the English sentence "I have some pigs". For every different language we need, we would have a translation file which contains the function iHaveSomePigs, which in this case takes zero parameters. So for French:

iHaveSomePigs = "J'ai des cochons"

(The mapping between the English sentence "I have some pigs" and the function name iHaveSomePigs hasn't been defined, and we'll skate over that detail for now).

If we have a variable number of pigs, for French we might have:

iHaveNPigs 0 = "Je n'ai pas de cochon"
iHaveNPigs 1 = "J'ai un cochon"
iHaveNPigs n = "J'ai " ++ show n ++ " cochons"

For English we could do this:

iHaveNPigs 1 = "I have 1 pig"
iHaveNPigs n = "I have " ++ show n ++ " pigs"

(For those unfamiliar with Haskell, the way that pattern matching works is that the first definition that matches the arguments is used. Since n is not a literal, but a variable, it can match any argument.)

We can cope with more complicated rules, such as those used in Polish, perhaps something like this:

iHaveNFiles n = "Mam " ++ show n ++ " " ++ pluralize n "file"

plurals "file" = [ "plik"
                 , "pliki"
                 , "plików"
                 ]

pluralize n word = plurals word !! pluralForm n

pluralForm n
  | n == 1                                                                        = 0
  | n `mod` 10 >= 2 && n `mod` 10 <= 4 && (n `mod` 100 < 10 || n `mod` 100 >= 20) = 1
  | otherwise                                                                     = 2

Note that the complex logix in pluralForm and pluralize only has to be defined once. Adding more words simply requires additional plurals lines. It's not the nicest syntax, but could probably be improved, and it's pretty easy to copy.

Let's add in gender, using the sentences "Delete this %s?" (singular) and "Delete selected %s?" (plural). We can use guards:

deleteThisThing thing
    | isMasculine thing = "Supprimer ce " ++ singular thing ++ "?"
    | otherwise         = "Supprimer cette " ++ singular thing ++ "?"

    -- (Ignoring the problem with 'ce' followed by vowel for now...)

deleteSelectedThings thing
    | isMasculine thing = "Supprimer les " ++ plural thing ++ " sélectionnés"
    | otherwise         = "Supprimer les " ++ plural thing ++ " sélectionnées"

isMasculine thing = elem thing [ "pig"
                               , "man"
                               -- anything else masculine
                               ]

singular thing = pluralForm 1 thing
plural   thing = pluralForm 2 thing

pluralForm 1 "pig" = "cochon"
pluralForm 2 "pig" = "cochons"

pluralForm 1 "man" = "homme"
pluralForm 2 "man" = "hommes"

Note that the only thing required by this system is that the functions deleteThisThing and deleteSelectedThings exist. Everything else is at the freedom of the translator, and better ways of defining any of these functions are possible.

Of course, it isn't expected that a translator would be able to produce this by himself/herself. However, once the basic logic has been set up, this syntax is readable enough that a translator could easily add more of the same. Lines like:

pluralForm 1 "pig" = "cochon"

are actually pretty readable. The lack of parentheses in Haskell function calls is also a bonus (though, as I said earlier, exact syntax could be debated). This is not really that much harder than editing a .po file if you are just wanting to add more of the same.

Also, we've got flexibility. If we really don't care about getting the gender right, we can just do "sélectioné(e)s" and be done with it.

Let's make it harder - we'll add case. I'll use NT Greek as an example, because it has nouns that decline with case (and I don't know any similar modern languages well enough). I'm going to introduce an enum for the different cases, using data for now, and for the different genders. I could also do the same for number ("Singular" and "Plural"), but just using 1 and 2 seems easier.

Our sentence will be "You like the %s.". For this in Greek, we need to choose the accusative singular form of the thing we pass in. We also need to pick the word for "the" (the definite article) which matches the gender and number of the noun, and it has to match the accusative case too. So, if we pass in a masculine word, we need the singular accusative masculine definite article (having fun yet?):

data Case = Nominative | Accusative | Genitive | Dative
data Gender = Masculine | Feminine | Neuter

youLikeTheThing thing = "φιλεις "
                        ++ definiteArticle 1 Accusative (genderOf thing)
                        ++ " "
                        ++ accusativeSingular thing ++ "."

accusativeSingular thing = nounForm 1 Accusative thing

nounForm 1 Nominative "book" = "βιβλιον"
nounForm 1 Accusative "book" = "βιβλιον"
nounForm 1 Genitive   "book" = "βιβλιου"
nounForm 1 Dative     "book" = "βιβλιω"

nounForm 2 Nominative "book" = "βιβλια"
nounForm 2 Accusative "book" = "βιβλια"
nounForm 2 Genitive   "book" = "βιβλιων"
nounForm 2 Dative     "book" = "βιβλιοις"

genderOf "book" = Neuter
genderOf "man"  = Masculine
-- etc

definiteArticle 1 Nominative Masculine = "ο"
definiteArticle 1 Accusative Masculine = "τον"
definiteArticle 1 Genitive   Masculine = "του"
definiteArticle 1 Dative     Masculine = "τω"

definiteArticle 1 Nominative Neuter    = "το"
definiteArticle 1 Accusative Neuter    = "το"
definiteArticle 1 Genitive   Neuter    = "του"
definiteArticle 1 Dative     Neuter    = "τω"

-- feminine etc

-- definiteArticle 2 (plurals) etc.

Of course, you can easily define shorter aliases to avoid some typing here, and there may be better ways to generate the tables, though as written above they are pretty readable, and should be familiar to anyone who knows Greek.

The function youLikeTheThing here is no longer very readable, although it could be much worse. Some kind of substitution syntax/function could be used.

The code above actually works, BTW, and it actually ran first time I tried - the only correction I needed to make its output correct was to add a space after the definite article. You just need to put it in a file test.hs, add the following line:

main = putStrLn $ youLikeTheThing "book"

and do:

$ runhaskell test.hs

There is not a type signature in sight, but you have compile time guarantees. This is all a testimony to the clarity of Haskell's syntax.

The features of Haskell we've used are:

• functions
• simple pattern matching on numbers and strings
• guards
• data statements, limited to union types of nullary constructors i.e. effectively enumerated values. We could use a keyword enum for clarity.
• string concatenation
• lists
• a few arithmetic and logical operators

We haven't used recursion. I can imagine circumstances where it might be useful, but if deemed too risky, you could add some rules that would disallow it (e.g. by requiring a function mustn't call itself directly, and must only call functions that exist prior to it in the source code, to avoid mutual recursion.) This would be helpful to ensure termination.

You might also want a module system, to be able to pull in some common definitions and functions for a given language, for consistency across different projects.

This whole approach has the advantage of being able to refine and special case as much as you want. Take the sentence "you like the %s": suppose that if the thing is a human being e.g. "man" or "woman", you need to use a completely different verb. Then you just add a special case first:

isAPerson "man"   = True
isAPerson "woman" = True
isAPerson n       = False

youLikeTheThing thing
    | isAPerson thing = ...
-- fall through to the normal case here

In the other direction, if you just don't have the time to care about any of this, you can just use a really simple (and often wrong) formula:

youLikeTheThing thing =  "φιλεις τον " ++ greek thing

greek "book" = "βιβλιον"

Notice that the programmer of the main project does not know anything about plural forms, gender, case etc., or put any of that into the source code. The only thing he/she would do is call a function with all the things to be substituted. We could have some mapping from English strings to function names, or we could just use the function name as a string, e.g. from a Python project we might call the function like so:

prompt = translate("doYouWantToDelete", n, object_name)

This would call the translation function doYouWantToDelete with the parameters n and object_name.

As a refinement, we can provide a version which will work when the whole localisation machinery is turned off i.e. we allow the programmer to provide their own version of the translation function which returns the default language:

prompt = translate("doYouWantToDelete", n, object_name,
                   lambda n, object_name: "Do you want to delete these %s %s(s)" %
                                          (n, object_name))

As before, the provided function can be correct or simplistic as desired for English.

Feedback

There are a few questions in my mind:

1. Would a solution like this work for the languages you know? What additional features would be needed to cope with other human languages?

2. Is this vaguely practical? Could you get translators to be able to edit code like this? If not, and only programmers would be able to do this, are there enough programmer-translators to make it a viable solution, at least for some big projects?

   I'm aware that the string concatenation gets ugly fairly quicky, and some kind of interpolation might be needed (including the ability to call functions within that interpolation). With that in place, I think you could achieve a reasonable level of readability.

   A translation tool could also have language-specific templates to quickly insert the code for common forms.

3. Is it possible to have a simpler language that would still be able to cope with the examples here?

   The examples I've come up with suggest to me that you need a full programming language, and that attempting to start from the other direction (e.g. build up from the current gettext approach) will produce a monstrosity.

   gettext already does a 95% job, and we are at the point of diminishing returns. So if we are going to try to tackle the final bit, we need to err on the side of enough power to get all the of that 5%, rather than put a lot of effort in and discover we've only arrived at 96%.

   You also cover the case of having a client who insists that the program should output "cet homme" and not "ce homme" - while it might make your translation file ugly, you've got the power to be able to do it if you want.

Comments?

---
invoice: 34843
date   : 2001-01-23
bill-to:
    given  : Chris
    family : Dumars
    address:
        lines: |
            458 Walkman Dr.
            Suite #292
        city    : Royal Oak
        state   : MI
        postal  : 48046
---
invoice: 34844
date   : 2001-01-24
bill-to:
    given  : Pete
    family : Smith
    address:
        lines: |
            3 Amian Rd
        city    : Royal Oak
        state   : MI
        postal  : 48047

Python

Here is my Python solution:

import yaml
import json
json.dump(list(yaml.load_all(open('invoices.yaml')))[0]['bill-to'],
          open('address.json', 'w'))

Notes: I didn't have to consult docs once. This isn't just due to my familiarity with Python — it's also the fact that I can fire up IPython and go:

In [1]: import yaml
In [2]: yaml.<TAB>

and get a list of likely functions. I can then go:

In [3]: yaml.load_all?

and get help, or go:

In [4]: yaml.load_all??

and get the complete source code of the function/method/class/module, in case I need it.

Haskell

Now for the Haskell version. First, a disclaimer: I'm much less experienced in Haskell than in Python. I did manage to write my blog software in Haskell at one point, but I don't use Haskell on anything like a daily basis, and I do use Python that much.

I first need to parse YAML. I've got a choice of packages. Unlike in Python, for a library like this, the choice you make is likely to have a big impact on the code you write — switching to a different (perhaps faster) package won't be just a case of changing an import, as we will see. The choice of packages represents the fact that even designing how this thing should work in terms of API and data structures is not straightforward in Haskell, and represents a much bigger commitment, and therefore problem, for the library user. In Python, while there are a few API choices (like supporting streaming or not, potentially), mostly it's pretty obvious how the library should work.

Looking on Hackage, I first find the 'yaml' package. The first line of the Data.Yaml API docs reads:

A JSON value represented as a Haskell value.

(Yes, you read that right). This doesn't look good. The whole file has stuff about JSON, not YAML, with no indication why I want to be using JSON values, not YAML. But I have a go anyway, perhaps it was deliberate.

When trying to use the decodeFile function, I get an error about needing a type signature, due to the way decodeFile is defined:

decodeFile :: FromJSON a => FilePath -> IO (Maybe a)

There are lots of instances of FromJSON to choose from, but I have to know in advance the type of data. And it looks like I've got data that isn't going to fit into any of those types, because it involves heterogenous collections. [Correction in comments, see below].

I gave up and tried another package - Data.Yaml.Syck.

First try:

import Data.Yaml.Syck

main = do
  d <- parseYamlFile "invoices.yaml"
  print d

This works - well, I've got some kind of parsing going on, at least. It looks like I've got some YamlNode datastructure, and the top thing is an EMap (it looks like it has only parsed the first document, which is worrying, but doesn't matter given my requirements, so I'll ignore that). But how do I get data out?

OK, let's try yaml-light - it wraps HsSyck and has some easier utility functions, like lookupYL.:

lookupYL :: YamlLight -> YamlLight -> Maybe YamlLight

That expects the lookup key to be a YamlLight, so I need to create one from a string, somehow. The docs show how to turn a ByteString into a YamlLight node, and I need to pass in a String, which from previous experience requires doing something like pack from Data.ByteString.

My program so far:

import Data.Yaml.YamlLight
import Data.ByteString.Char8 (pack)
import Data.Maybe

main = do
  d <- parseYamlFile "invoices.yaml"
  print $ fromJust $ lookupYL (YStr $ pack "bill-to") d

Which gives this output:

YMap (fromList [(YStr "bill-to",YMap (fromList [(YStr "address",YMap (fromList [(YStr "city",YStr "Royal Oak"),(YStr "lines",YStr "458 Walkman Dr.\nSuite #292\n"),(YStr "postal",YStr "48046"),(YStr "state",YStr "MI")])),(YStr "family",YStr "Dumars"),(YStr "given",YStr "Chris")])),(YStr "date",YStr "2001-01-23"),(YStr "invoice",YStr "34843")])

Now I have to dump to JSON. From a Python perspective, all I want is a function that can take some ‘native values’ and dump them to JSON, like the Python json.dump function. But every piece of data in my data structure is wrapped in things like YStr and YMap.

In addition, though I can see the structure of my data in front of me, the requirements I've been given don't make guarantees that it will stay the same, just that it can be converted to JSON. I need a routine that will convert anything YAML to the equivalent in JSON, where that is possible.

It looks like I could create a JSON instance for YamlLight, so that the encode function I want to use (which dumps JSON to a string) could take YamlLight as an input directly. I end up with this:

import Data.Yaml.YamlLight (parseYamlFile, lookupYL, YamlLight(..), unStr)
import Data.ByteString.Char8 (pack, unpack)
import Text.JSON (JSON(..), encode, JSValue(..), toJSString, toJSObject)
import Data.Maybe (fromJust)
import Data.Map (toList)

instance JSON YamlLight where
  showJSON yml =
    case yml of
      YStr bs -> JSString $ toJSString $ unpack bs
      YMap m -> JSObject $ toJSObject $
                map (\(y1, y2) -> (unpack $ fromJust $ unStr y1, showJSON y2)) $
                toList m
      YSeq ymls -> JSArray $ map showJSON ymls
      YNil -> JSNull

main = do
  d <- parseYamlFile "invoices.yaml"
  writeFile "address.json" $ encode $ fromJust $ lookupYL (YStr $ pack "bill-to") d

This works, and I'm sure there are other solutions. If I were cleverer, and knew Haskell better, I could perhaps write a cleverer, shorter solution, which would also be proportionately more difficult for someone else to understand, so I'm not particularly interested in making this code shorter, as it does the job.

But this illustrates why some people like dynamically typed languages. The fact that you can implement a variant data type in Haskell (such as YamlLight or JSValue) doesn't mean much, because these data types are not used everywhere, and therefore you have multiple competing ones that you've got to convert between. If you did have a single variant datatype that was used everywhere... you'd have a dynamically typed language, in effect.

The strictness of the type system gave rise to a choice of libraries and APIs that made my life harder, not easier. I then had to write glue code to marshall between the dynamic types used by the two libraries I needed. [Edit: or, as it turned out, I need to know where to find it, possibly in the form of already written type class instances, or how to get the compiler to write it for me]

Some people might still prefer the Haskell version. It has some nice properties, like the fact that compiler has checked that it can indeed convert any YAML object into JSON — you'd get a warning if you missed a case. One response to that might be that if the two types didn't happen to match so well — for instance if the YAML library started supporting date/time objects — this benefit would disappear. If you need to avoid all possible problems up front, Haskell will help you out more. Python, on the other hand, will allow you to avoid spending time thinking about theoretical problems which may never happen in reality.

But there are always runtime errors that you could come across, even in Haskell — for example, if you want to convert this to cope with non-ASCII documents, the compiler can't point out all the places you need to fix, and if you forget one you could still get a runtime exception, or worse, silent data corruption.

So, in my opinion, this is a case where dynamic typing shines, and the ability to implement dynamic typing on top of static typing simply doesn't give you the benefits you get in a language that embraces dynamic typing to its core.

There are, incidentally, some interesting developments in Haskell that might allow the possibility of running programs that aren't quite typed correctly, as long as you don't encounter the type errors in practice. This could counter some of the points I've raised — see this interview with Simon Peyton-Jones , from 27:45 onwards.