This document attempts to answer the question "Why is There a TiddlyWeb?" It was originally written to frame the thinking @cdent was doing to create a presentation called Tiddlers on the Web.
The short answer is "to get tiddlers on the web, with good behavior".
The long answer is long and covers many concepts:
There is Nothing New Under The Sun
First, it is important to state right up front that TiddlyWeb and TiddlySpace are not particularly unique. They are examples of code that presents some stuff on the web. The internet world is full of that. At a general level there is nothing new here. What is special are the choices made with regard to the details of the system and how these choices impact the character of the tools and the services built with them. As we'll see, TiddyWeb is about putting tiddlers on the web. We'll also see that tiddlers are just resources on the web.
So what are these choices? What are the opinions that drove, drive and color tiddlers?
Wikipedia says "[[Stigmergy|http://en.wikipedia.org/wiki/Stigmergy]] is a mechanism of indirect coordination between agents or actions". The canonical example is of ants leaving trails of pheromones to guide other ants to a food source.
In the information world of humans, resources on the web act, whether we mean them to or not, as artifacts that indirectly coordinate action. While that action can be quite concrete (such as a project plan and task list) it can also be rather more abstract: learning and serendipitous discovery.
I assert that serendiptious discovery is hugely valuable. An unexpected learning is far more important than confirming something we already thought or verifying an assertion. Serendipity is where new ideas are born.
Information resources on a public network act as stigmergic structures for:
- Individuals (who need a reminder)
- Groups that need a pseudo-objective third party (the resources) to help guide activity.
- Nascent groups that are accreting around ideas.
For these structures to be any good, they need to be easy to create, easy to find and easy to use.
Networked information resources make explicit the cybernetic system in which a group (implicit or explicit) operates, making feedback more available and more transparent.
You've probably heard something like this before.
When the web came about I remember thinking, "cool we get another chance to put publishing back in the hands of people". I did not think "we're having an information revolution". If you've read James Gleick's //The Information// every innovation in the distribution of information is hailed as a revolution, but they are all rather the same: A new medium expands opportunities for the creation and distribution of messages, reaching larger populations. The apparent revolution is a product of speed and scale, not behavior.
The 20th century has a history of pushing these new media out to the masses and then consolidating them back to central or corporate control. How the 21st century will be remembered remains to be seen: The corporatization and centralization of media alongside the tendency towards mediocre, lowest common denominator small talk could lead to the devaluing of learning and intellectual exchange. There are, of course, bright lights but many are simply the result of statistics: If there is a huge amount of information, inevitably some of it will be excellent.
When TiddlyWiki came to life it was pretty good at providing information management and publishing for a single person. It was not excellent at doing the same for groups, or on the public internet. TiddlyWeb was developed to fix those gaps.
Engelbart, Wicked Problems and Granularity
Much of the thinking that drives TiddlyWeb has historical antecedents in the writing, thinking and research of Doug Engelbart.
Engelbart is generally remembered for being the inventor of the mouse but he has done much more: He orchestrated the mother of all demos which showed off precursors for pretty much everything we think of as modern computing.
Engelbart was convinced that human intellect could be augmented with proper training and tools. He believed augmentation was necessary to ensure the continued survival and success of the species in the face of increasingly complex problems, especially so-called wicked problems.
Engelbart's augmentation was built around tools used to enhance our ability to create, access and use information. These tools were especially important in enhancing what he eventually called 'C' activities:
- A: The things you do as the core goal: Make widgets.
- B: The things you do to get better at A.
- C: The things you do to get better at B.
When developers get together to talk about software development techniques, they are having a sort of 'B' activity. They are meeting to share information about how they do what they do, so they can get better at it. If they also talk about how useful these gatherings are, "how come we don't do them more often" and "how can we make them better" then they are also having something of 'C' activity.
At an abstract level 'C' activities can be described as getting better at using information. Getting better at using information means bootstrapping ourselves with tools that not only allow us to use information better but also think about the creation of still better information tools.
Innovation, at its core, can be described as new ways of using and understanding information: Exposure to information and juxtaposition of information breeds new thought.
New understandings of information come from making comparisons.
To compare things we must be able to access them and if that access is to be efficient we must be able to access at a granular level. The things being accessed must be small and identified at that small level. We don't point people at a book of the bible, we point them at chapter and verse.
In TiddlyWiki speak, the tiddler is the granular piece. To make it truly accessible, to make it //useful// information we must give it a URI. We must make it a first class resource on the web. That's what TiddlyWeb does.
TiddlyWeb is a Python-based WSGI application that provides:
- an HTTP API for storing and retrieving tiddlers
- an interface for persisting those tiddlers to a store
- an interface for presenting those tiddlers in various representations (e.g. HTML, JSON, plain text, TiddlyWiki, Atom)
- a rendering subsystem for turning various types of tiddlers into HTML
- entities known as bags which provide a namespace, authorization and/or topical domain for a collection of tiddlers
- entities known as recipes which assemble (and optionally filter) bags to create collections of tiddlers from multiple bags
- self-hosted authentication or pluggable integration with other auth systems
Within TiddlyWeb a tiddler is modeled as an object with:
- a title
- the bag in which it is contained
- a modifier
- a modified timestamp
- a list of tags
- a dictionary of arbitrary key value pairs
- an optional type
- a text
- a revision id
(Tiddlers do not have truly unique and persistent identifiers to keep them aligned with the ethic of naming things that comes from wiki culture. While true identifiers were mooted as a really cool things to have, it was decided not to use them as that would have distracted from the (original) purpose. It is possible now to hack such identifiers into the system if needed.)
If a tiddler has no type it is assumed to be aligned with TiddlyWiki's notion of a tiddler: a block of text, most likely in TiddlyWiki syntax or a JavaScript plugin for use in TiddlyWiki. Otherwise type is a MIME type. With textual types {{{text}}} is a unicode string, otherwise bytes.
//A tiddler is a revisioned web resource with some useful metadata. A tiddler is agnostic with regard to content.//
But First some History
The tiddler concept, as used in TiddlyWeb and TiddlySpace comes from TiddlyWiki, which was created by Jeremy Ruston in 2004. TiddlyWiki was an early example of quite complex DOM manipulation and JavaScript in the browser. It caused quite a bit of buzz and became a successful open source project.
A few years later BT bought Osmosoft, ostensibly to better understand open source communities and techniques.
Jeremy has said that he created TiddlyWiki as a way to help him write: The tiddlers can operate as separate chunks that can be moved around, re-used, separately edited. Tiddlers started as documents and became many things, including plugins which could impact the behavior of their container, the TiddlyWiki file itself.
In 2002 I co-founded a think tank called Blue Oxen Associates. We were born out of a mailing list devoted to talking about Doug Engelbart's Unfinished Revolution. I found the mailing list because I was a recovering system administrator taking a break to get a masters degree. I studied Engelbart, Information Organization, Knowledge Representation, Human Computer Interaction and related topics. I connected with Eugene Kim, the other founder, because we both had a "let's not just talk, let's also make some stuff" attitude.
The core issues we thought about were:
- Hard stuff is hard.
- Some hard stuff is so hard that we must collaborate to solve it.
- Collaboration itself is hard.
- Collaboration needs to be augumented with tools.
Blue Oxen set out to pursue doing and researching augmented high performance collaboration, establishing what became known as the Blue Oxen Way.
We needed to bootstrap our information processing. We wanted to more effectively access, use and re-use information.
So we built a wiki. Called PurpleWiki. With Purple Numbers. These were a riff on the HIDs and NIDs in Engelbart's various information systems (NLS, Augment, Hyperscope). They provide granular addressability to chunks of content.
Once those addresses become persistent it becomes possible to have granular reuse through transclusion.
At its height Purple Numbers were able to integrate the wiki, multiple blog engines, mailing list archives and an IRC bot into a self referential and transcluding network of granular content. Each system was able to refer to and quote from the other by reference, not by copy.
I eventually had to find a real job and took the purple toolset with me. The new job was software development. We used it to augment our collaboration. It was awesome.
So where's Purple now? It died the death of most interesting software systems: It was cool for the initiated and trained and pretty much inscrutable for the unmotivated. You can see it living on today when ¶ shows up when you mouse over some headlines in documentation (often Python related). That's because Simon Willison spread them around. He got them from Tim Bray who got them from me.
That was in the spring of 2004.
Church of Purple Hymn
In September of 2004 I started work at Socialtext and tried to spread the church of purple, but it didn't take: Too weird, too much work for people.
And then I saw TiddlyWiki, with tiddlers, and started stealing some ideas from there, primarily whole page transclusion: Hacking granularity into Socialtext by being able to assemble complex pages from smaller, simpler parts that could be reused in multiple places.
In summer of 2006 we started on what we called the REST API for Socialtext. It would not have had the Roy Fielding stamp of approval, so it would be better to call it the HTTP API for Socialtext, but we called it REST to highlight that it wasn't SOAP, the other API we had just finished, created to interoperate with Sharepoint and, ew, ick.
We designed the HTTP API with a few very important things in mind:
- We wanted to be sure that it imposed no constraints on what people what might do with it.
- We knew that meant the API itself should be very constrained, very general.
- Therefore it should be of, about and for the web.
Four years later somebody on twitter said:
Best REST API I've developed for? Socialtext, hands down. It just does the right stuff and doesn't complain.
The first client of the HTTP API was TiddlyWiki with which a product called Socialtext Unplugged was created. This allowed a Socialtext wiki to "fill" a TiddlyWiki and be taken offline, edited, and have changes sync back. The API and TiddlyWiki's adaptor mechanism mutually informed one another's development. Jeremy and I got to know one another.
Meanwhile TiddlyWiki had become quite popular and the notion of "server-sides" was developed. These were ways of hosting a TiddlyWiki on a web server in a fashion that still allowed editing. For most this meant a single POST of the entire content of the TiddlyWiki back to a small server that would rewrite the TiddlyWiki file. TiddlyWikis got ~URIs.
This is very useful and cool but is flawed. It assumes that the primary entity of import in the TiddlyWiki system is the TiddlyWiki itself: the entire TiddlyWiki. This is too constraining. The important part of TiddylWiki is the tiddler. The little chunk. The piece to be reused.
I left Socialtext near the end of 2007 to join up with the Osmosoft gang to give tiddlers ~URIs. To get them on the web.
TiddlyWeb was born and grew from just an HTTP server for tiddlers into:
- A library for manipulating tiddlers on the web.
- An infrastructure for creating and managing plugins which provide tiddler on the web related functionality.
TiddlySpace is built on top of TiddlyWeb providing enhanced sociability and greasing some of the rough spots involved in multi-user handling of lots of tiddlers. That is, like purple numbers, TiddlyWeb was a bit too weird, too much work for people.
Why a Tiddler Matters
As imagined in TiddlyWeb, a tiddler is a piece of content (//any// kind of content) with a title, a bag, a small number of pre-defined attributes, and zero to many key value pairs. In a very real way a tiddler on the web is a mildly enriched //anything// on the web. A resource with improved affordances.
A tiddler can still be a wiki page, or it can be a program, or an image, or a complex data structure. With a properly configured Storage Interface that data structure can be indexed, thus making TiddlyWeb an able member of the NoSQL crowd (of the document oriented ilk).
You get the web, plus you get storage, editing, access control, a common format, serializations, and extensive tools.
Why Open Source Matters
Intially TiddlyWeb was oriented toward being the perfect "reference" server-side for TiddlyWiki: other implementations would be created that use the same HTTP API, but optimize for different situations or environments. Being a reference impacted the design and architecture:
- The code needed to be transparent and easy to read. I chose Python. I was coming down from a few years on Perl so it was the obvious choice for me. There were concerns at the time that this would mean it would not get used by non-technical users, who more commonly had access to PHP. The hope at the time was that someone would do a PHP port if the API was proven useful.
- The architecture should be effectively layered and extensible to allow use in lots of different situations (e.g. different storage and authentication scenarios).
- The implementation of HTTP handling should be relatively pure, to set a good example.
- TiddlyWeb would need entities that represented containership and composition of tiddlers. Thus bags, recipes and filters.
- Tiddlers were to have names, not identifiers to preserve their character. This was a contentious decision.
Because there was a desire for the code to be clear about what was happening to the HTTP request and response, straight WSGI was used, without frameworks such as Django or request objects such as WebOb or Werkzeug. These abstract away important details.
I set to work. I made sure that I had good tools for testing (py.test, wsgi-intercept) and took a test driven approach while being willing to throw away anything that didn't work. There was a fair bit that could be borrowed from the Socialtext setup:
- notions of how to do content negotiation
- multiple representations of resources
- clean, clear and cool ~URIs
but more importantly there were several things which we didn't do but wanted to so I did them in TiddlyWeb:
These turned out to be of critical importance. A default TiddlyWeb installation is simple with low requirements. That same code base can be built up to support thousands of users and hundreds of thousands of tiddlers with the right extensions and plugins. That's how we get TiddlySpace, wiki-data and ILGA.
TiddlyWeb was initially built working to an abstract goal. I tried to have good YAGNI behavior, but taken too far that would have resulted in nothing being created: There was no active customer of the concept. Once an initial working prototype was available other people were able to help get it built in the right direction.
This is when it started becoming clear that a lot of our early assumptions were incomplete or wrong. The original belief was that TiddlyWeb would be used as a backend to TiddlyWiki-based applications (called verticals) allowing TiddlyWeb to remain relatively simple. Turns out this was only partly the case. People wanted to be able to use TiddlyWeb as a web app development framework with tiddlers (not TiddlyWiki) as the main data store and point of entry. The flexible serialization system meant that rather complex things could be done with one or many tiddlers with ~UIs created and managed from the server. An early example was ILGA.
All this development was happening in full view of the TiddlyWiki open source community. As they and the other Osmosoft developers tried things, the many eyes made bugs shallow.
Eventually TiddlyWeb came to a bit of standstill. It was sufficiently complex to setup and then to manage that it did not penetrate beyond a savvy but fairly small audience. In particular many existing TiddlyWiki users who might have been interested in using TiddlyWeb did not have access to the resources to make it go. At Osmosoft we decided to create TiddlySpace to make tiddlers on the web easier to access by creating a multi-user environment where people could curate and share tiddlers and reuse (through inclusion) tiddlers from other people.
Again the initial goal was to make a TiddlyWiki based interface and again this proved to be constraining. TiddlySpace has made it clear that in the tiddly universe the tiddler is the thing. To some extent this is simply a matter of physics: a TiddlyWiki is a large single entity containing lots of other things. It is big and slow to load, just one thing, yet non-atomic. It is complex. A tiddler is just one thing, but one thing that can come in many forms.
The vast majority of use on TiddlySpace is done from within TiddlyWiki, but the (IMHO) innovative work is done with tiddlers and simple JavaScript based applications, often presented as ~SPAs: Single Page Applications. A tiddler operates as the data entity, the viewer of those entities, and as documents. Many different JavaScript tools have been used in the process: jQuery, Backbone, Bootstrap, etc.
TiddlySpace caught on pretty well, re-energizing TiddlyWeb development. For the first time large numbers of requests for large numbers of entities were being made. TiddlyWeb was shown to be slow and needed to be fixed. The large user base provided the data necessary to do accurate measurement and optimize those parts of the code that matter. This mostly meant turning naive Python into slightly more interesting Python (e.g. lists into generators) but also hardware, database and cache tweaking.
Additional features were created as well:
Eventually TiddlySpace became fast and functional enough that it is now a useful place to do web development in addition to web publishing. Either with TiddlyWiki or without. The tiddler operates as a nice augmented container for //something// on the web. It is quick and easy to make stuff and reference stuff. A web with some improved webby affordances.
Why in Public
There is a TiddlySpace installation on BT's internal network, but primary development targets the public service at http://tiddlyspace.com/ using commodity hosting. This provides a few benefits.
- Deploy multiple times every day if desired. Change anything anytime.
- Diversity of use and audience.
- Widespread contributions.
- Infrastructure is cheap to acquire.
What's Bad
There's still a gap in the TiddlyWeb community between the people who work on or near the core and people who just want to use it. There are few people who bridge that gap providing documentation, tutorials, bug reports and administrative tools.
What's Good
TiddlyWeb plus TiddlySpace provides a platform for doing rapid development of web-based information sharing. For BT it was especially useful in the creation of ORC: Openreach Cauldron. TiddlySpace made it possible to ignore the backend and focus on frontend concerns, getting something in front of users quickly and experiment and work in short iterations. Before needing to establish any infrastructure with Operate.
We had similar opportunities at a recent hothouse, experimenting with ways for communications providers to communicate with open reach engineers. We made quite functional prototypes in two short afternoons, demonstrating concrete technology that people could interact with to more fully refine and examine their requirements.
Demos
The following demonstrate some of the different capabilities of TiddlyWeb and TiddlySpace.