The concerns that nag me about the booming universe of Design Tokens are all issues @davereid and I encountered when we built and supported the Token system for Drupal. I’m still trying to suss out how much of that is “learning from experience” vs “applying the wrong lessons.”
The token system for Drupal was originally conceived around '07 or so. Its ecosystem was evolving to give site builders composable pieces (“image-displayer” and “grid-display” and “pager”) rather than drop-in functionality (“forum”, “gallery”, etc).
Around then we built one of the first large-scale production sites that used Drupal’s early “click-together tools” instead of pure code: Sony/BMG Music’s artist site platform, which allowed them to spin up new interactive community sites for each artist in just a couple of days.
What we realized: lots of strings that could be hard-coded in a tailored app (like the text of the ‘Next Post’ button in a forum)… were impossible to pre-plan, because the “forum-ness” and “post-ness” was a consequence of per-site configuration rather than planned functionality.
Now, more and more of those click-together tools supported admin-configurable labels! …But often the labels (or alerts, or destination links, or…) were derived from the context of the page the user was on. Like, “Read [title of post]” instead of “Read next post.”
Handling it as a special case once or twice is fine, but as we built out a complex site using these click-together tools, and planned to scale that site to 5, 10, 50 artist’s web sites… well, it was obvious a system was needed.
The result was an API baked into the CMS and publishing tool that exposed loads of values like “the name of the site” and “the name of the author of the current piece of content” and “the type of post being viewed” as strings like [post-type] and [site-name] and [author-name].
Site managers could write emails like: “Welcome to [site-name], [user-name]!” or configure a link to read: “View more [post-type] from [author-name]”, and developers just had to say, “Here, tokenize that string” and it all worked. Within 2y or so it was the most-installed plugin.
Now, for anyone who got suckered into reading this because I mentioned design tokens, I should be clear that our big concern was related to microcopy rather than the spacing, color, and typography standards that often nudge teams towards adopting design tokens.
But the two problem spaces have a remarkable amount overlap and address a similar core challenge. Namely, when you have a complex system that combines lots of reusable bits, consistency is easier if you use consistently-named placeholders for stuff that appears everywhere.
And almost immediately it was a success. Lots of other things inside the system (site notification emails, alerts and error messages, etc) turned out to benefit from a standardized token infrastructure. More plugin module authors exposed their data in token form, too. Yay! [but]
The problem success brought was that the number and precision of tokens exploded. Token names like [post-field-image-sizex] and [author-lastlogin-timestamp] abounded. Just finding the right token was a chore, and many tokens needed multiple variations for different contexts.
Under the hood, every single token was just a hard-coded string name that we assigned a particular value to — and slowly but surely, it was turning into a hard-coded version of every piece of data in the system.
We rebuilt it to make it a bit smarter — tokens got ‘smart’ internally: [post:author:name:plain] wasn’t just a hard-coded string with a value, but a command to get the author of the current post and format the author’s name as plaintext.
But then people started bolding conditional logic onto it, so tokens like [if:post:type:is:forum:then:author:name:plain:else:…] and someoen built a workflow automation system that used tokens to drive business logic and BOY HOWDY
What I eventually came to grips with was that no matter how simple the “core” use case I imagined was, I was building a system that encoded meaning in word-like symbols, and asked humans to compose complicated things out of them.
It wasn’t a language in the formal sense, but it had language-like problems. And I’d only ever thought about the vocabulary itself — how to make a dictionary of tokens and meanings. Not the grammar of how they’d combine, or how to deal with the semantics of ambiguous context.
Now, bringing this back around to the world of design tokens, I think there are a lot of differences between the token-system in Drupal and the way Design Tokens are approached. Different problems, different pressures! …But.
The challenges I see in a lot of design token conversations mirror what we faced trying to map a rapidly-expanding cloud of tiny properties to a set of increasingly long and convoluted strings. We needed concepts like “types” and “formats” and “allowed combinations”.
Because what we were making wasn’t a dictionary. What we were making was a language for talking about the bits of information in the site’s content. Same with Design Tokens — they provide a language for talking about design decisions in a precise and parse-able way.