Generated Dictionaries

Thinking about generated dictionaries in steno while I was making Quartz.

Background

Steno theories are traditionally abstract, and not very well specified. They're essentially a theory author's ideas of how they want things to work in their system. Many theories were not really "designed" at all, and were just made up over time to suit the creator's needs. This has worked fine for professionals for hundreds of years, but is a mismatch for creating theories that we want to share around in the hobbyist space.

Theory Creation

Once you've designed a theory, you can't use it yet, as you have no dictionary. You now have to undergo the endeavor of manually adding hundreds of thousands of entries into a JSON dictionary. This leads to dictionaries with a few qualities:

• They're very large, and take for freaking ever to make
• They couldn't ever be "complete" or consistent, as a theory specifies how any word could be written, and you'll never be able to add them all
• They aren't inherently well documented, they're just mappings of outlines to translations with no reasoning as to why things are the way they are
• They aren't really maintainable - any large theory change could requiring updating tens of thousands of existing entries, and you have no real way of knowing what entries have been affected. This, in turn, means that any dictionary that's been around for a while will be filled with vestiges - entries that used to make sense under the old theory rules, but now do not. There's no way to filter these vestiges out either, as they hide amongst the rest of the normal entries
• They're inevitably filled with conflicts that can only be discovered by using the dictionary and running into them

The underlying problem of these dictionaries is that they have no presence of their theories - they exist completely on their own.

This disconnect is a real problem for steno, as full-English steno theories are not simple. Re-implementing a basic fingerspelling dictionary might only take a few minutes, but re-implementing an entire theory dictionary could take months or years.

Programmatic Dictionaries

There is a type of dictionary available in that follows exactly the rules you give it, and that is a programmatic dictionary.

In Plover, this takes the form of a Python dictionary, and can be used with the plover-python-dictionary plugin.

Dictionaries are functions - they take an outline as input and return a translation as output. This is called "lookup", and Python dictionaries expose it directly via the lookup(outline) function, which returns a Plover translation.

This function is enough to implement any theory. Systems Shrimple, my Pinchord, and even full-English theories like StenoBee (https://github.com/Tink-Bell/StenoBee) are implemented as Python dictionaries.

A plain lookup function requires theory rules to be represented as a series of steps in a program, and the theory itself is entirely encoded in this form.

So why aren't more theories written in Python?

Well, Python isn't particularly well suited for representing steno theories. You can encode the rules of a theory in a Python function, but you're all on your own. Reifying a steno theory in Python is genuinely difficult problem, and the benefits of programmatic dictionaries are not immediately clear when given an empty lookup function to write.

Generated Dictionaries

This is where dictionary generation comes in. Dictionary generation is the process of turning a theory into a dictionary.

Rather than writing the dictionary yourself, you let the generator build the dictionary from a set of theory rules that you define. These rules make up the reified representation of your theory

This allows you to directly turn a reified theory into the final dictionary. It creates a direct relationship between theory and dictionary: dictionaries are derived from a theory, rather than trying to satisfy a theory.

This theory representation allows you to specify what a theory about, and how it works, in a way that can be easily observed and changed over time.

The benefits of dictionary generation don't have to be speculated, as such a tool does exist: Froj (https://github.com/StenoHarri/Froj)!

Froj takes a reified theory and spits out a JSON dictionary for use in Plover or Javelin.

• Eventually will write about: Questions of Generated Programmatic Dictionaries / Context Awareness / Programmatic Dictionaries

The content on this page is licensed under CC BY-SA 4.0