This is an updated version of a plugin tutorial originally written in the early days of TiddlyWeb, published on @cdent's tumblr.

In in the early days of TiddlyWeb I was asked by some members of the Osmosoft crew to give a little demo or tutorial of creating a server-side plugin for TiddlyWeb. That went pretty well. It's since been suggested that writing something here explaining how it fits together would be useful. This document is the result.

TiddlyWeb plugins are, at their most basic, Python modules which are imported into the TiddlyWeb process when it starts up. They are imported when the web server starts or the twanager command line tool is used. A TiddlyWeb instance is made (explicitly) aware of available plugins by having their names added to the tiddlywebconfig.py in the root of the instance. system_plugins are used by the web server, twanager_plugins by (as you might have guessed) twanager.

We're going to create a plugin called jinx.py that starts out as a simple hello world, and incrementally adds support for a variety of features.

Please follow along at home if you are inclined.

''Note'': If you are familiar with using virtualenv then feel free to use it in this case. If you do the use of sudo below is not required.

Install TiddlyWeb

The first thing you need to do is install TiddlyWeb or make sure you have the latest version (at the time of this writing the latest version is 1.4.0). Make sure you have Python 2.5, 2.6 or 2.7. Get Python's pip and then:

sudo pip install -U tiddlywebwiki

This will install TiddlyWeb and all the extra modules that it needs, including the package that manages the presentation and handling of TiddlyWiki files. If you are on a legacy operating system, you may not have sudo, in which case you need to be able to write to your disk in whatever way is required for that. If you do not want to install TiddlyWeb in its default location, you have several options which you can explore for yourself. Tools like Python's virtualenv may be useful.

Create an Instance

TiddlyWeb comes packaged with its own web server, so you can run it without needing to configure Apache or something similar. It will work other servers, but it doesn't have to. Any single TiddlyWeb data store and server combination is called an instance. Find a suitable location on your filesystem for an instance and create it with the twinstance tool that comes with tiddlywebwiki:

twinstance jinx

This creates a directory named jinx, and puts within it a basic tiddlywebconfig.py file and a store directory. Change into the jinx directory, we'll do the rest of our work from there:

cd jinx

Confirm the Server

Let's confirm the server is working:

twanager server

This should start up a server running at http://0.0.0.0:8080/ . If there are errors (likely on some windows machines) it may be that using 0.0.0.0 does not work on your system or port 8080 is in use. Try instead:

twanager server 127.0.0.1 8888

which will start a server at http://127.0.0.1:8888/

Go to the URL using your browser, and see links to recipes and bags. Click around, explore. When you are convinced that things are working, type Ctrl-C in your terminal window to kill the server. Because the web server is multi-threaded it make take some time and multiple taps of Ctrl-C to get things to shut down.

Hello World

When a plugin is imported by TiddlyWeb the controlling code will call a function called init(config) in the plugin module. config is the current value of tiddlyweb.config, which has a bunch of useful information in it that your plugin may need or want. A plugin does not have to pay attention to config but it does need to define the method. Let's create a file called jinx.py in the current directory (the TiddlyWeb instance directory). In that file put:

def init(config):
    pass

Edit tiddlywebconfig.py so that it looks something like this:

config = {
    'secret': 'bf88b86XXXXXXXXXX01952a1c6639140912f28b6',
    'system_plugins': ['jinx'],
    'log_level': 'DEBUG',
    'css_uri': 'http://peermore.com/tiddlyweb.css',
}

Setting log_level to DEBUG will cause some verbose logging to tiddlyweb.log, which will help us know if we are doing things correctly. The css_uri setting makes the output from the server a bit more pretty than the default. If you don't like me knowing that you are using that CSS file, just copy it to a location of your own. Start the server, stop the server, and look in tiddlyweb.log:

twanager server
Ctrl-C
cat tiddlyweb.log

Somewhere in there you should see:

DEBUG    attempt to import system plugin jinx

Huzzah! You've created your first plugin! Hmmm, it seems to do nothing, let's fix that.

A common plugin need is to add a new URL to the HTTP interface that TiddlyWeb presents. We're going to add one at /jinx. Add to jinx.py so it now looks like this:

def jinx(environ, start_response):
    start_response('200', [])
    return ['hello world']


def init(config):
    config['selector'].add('/jinx', GET=jinx)

If you start the server (twanager server) and go to http://0.0.0.0:8080/jinx you should see hello world as plain text. If not, review the steps up to here. It's important to get this part right, or none of the rest of this will work.

There's a lot going on in this small amount of code, so we'll pause here to cover what's going on.

TiddlyWeb is a WSGI application, or more correctly a collection of WSGI applications. WSGI is a specification for making portable and interoperable web tools. It defines a simple contract for how the tools get and return information that makes it possible to stack them up and achieve a great deal of flexibility while keeping code separated and concise.

Selector is a WSGI application used by TiddlyWeb to dispatch urls patterns to code. The line

    config['selector'].add('/jinx', GET=jinx)

adds the /jinx URL to the existing dispatch rules, pointing to the method jinx when there is a GET request. Selector expects the things it dispatches to also be WSGI applications. A WSGI application is a callable (a function or class that can be executed) with a specific signature. We see that signature in the definition of the jinx method:

def jinx(environ, start_response):

environ is a dictionary containing HTTP request headers, other request information and anything upstream WSGI applications have chosen to inject into the dictionary. start_response is how we set response codes and headers:

    start_response('200', [])

A WSGI application must return a list or generator of strings:

    return ['hello world']

So what we've done is pass the HTTP environ into the jinx function, done nothing with it, set a 200 response code but no headers, and returned hello world. Let's make it so we are sending HTML instead. Change two lines:

start_response('200', [('Content-Type', 'text/html; charset=UTF-8')])
return ['<html><body><h1>hello world</h1></body></html>']

Start the server back up and have a look at http://0.0.0.0:8080/jinx.

Above we set our Content-Type to text/html and our status code to 200 with a start_response call like this:

start_response('200', [('Content-Type', 'text/html; charset=UTF-8')])

It's quite likely we're going to make that very same start_response call pretty often, so there is some code you can use so you don't have to write it. Install the tiddlywebplugins.utils package to get some useful decorators and other methods:

sudo pip install -U tiddlywebplugins.utils

You can see some basic documentation for the available functions by:

pydoc tiddlywebplugins.utils

Let's use do_html and entitle to get rid of some common code. To the top of the file add:

from tiddlywebplugins.utils import do_html, entitle

change the jinx function so it looks like this:

@do_html()
@entitle('Hello World')
def jinx(environ, start_response):
    return ['<h1>hello world</h1>']

Start the server (note that in the default configuration of TiddlyWeb, for every code change you need to stop the server and then start it again), go to /jinx, and view the source of the generated page. Where did all that extra stuff come from? do_html() makes the response Content-Type be text/html. entitle does the rest. It uses HTML header and footer information provided by the default HTML serialization. Advanced TiddlyWeb plugins can modify or replace the serialization to extensively customize the HTML output, but we won't get into that here.

Private Hello World

'User GUEST' is showing the currently active user. If no login has happened, then the current user is GUEST. Let's require a user at /jinx by making this change to the code:

@do_html()
@entitle('Hello World')
@require_any_user()
def jinx(environ, start_response):

Start the server. Whoops, there's been an error:

NameError: name 'require_any_user' is not defined

We need to make sure we have the necessary code. Change the import line to look like this:

from tiddlywebplugins.utils import do_html, entitle, require_any_user

Now start the server. Reload /jinx. You should see a form asking for username and password. Whoops, no users exist. In your instance create a user:

twanager adduser <username> <password>

Go back to the form, enter the username and password and submit the form. Eventually you'll get back to the jinx page and where it used to say 'GUEST' it will now say the name of the user you created.

require_any_user checks to see if there is a user other than GUEST and if not, asks TiddlyWeb to redirect to its challenger system. Challengers are tools for authenticating a user.

Personalized Hello

If /jinx is private it's not really the world we are saying hello to, is it? Let's have TiddlyWeb say hello to you:

    username = environ['tiddlyweb.usersign']['name']
    return ['<h1>hello %s</h1>' % username]

When you restart the server, and reload /jinx it should now say "hello ". Woot! Personalization.

tiddlyweb.usersign is a data structure containing information about the current user. The current user is extracted from the current HTTP request by a thing called a "Credentials Extractor". TiddlyWeb can support multiple extractors. The interface is described in tiddlyweb.web.extractors. An extractor WSGI application is called before the selector dispatch application, which means every request is scanned for user information.

Personalized Custom Greeting

"Hello" isn't always what you want to hear. Let's change things so the message being sent is based on the incoming URL. Make the following changes (make sure you keep the lines beginning with @):

def jinx(environ, start_response):
    username = environ['tiddlyweb.usersign']['name']
    message = environ['wsgiorg.routing_args'][1]['message']
    if not message:
        message = 'Hello'
    return ['<h1>%s %s</h1>' % (message, username)]


def init(config):
    config['selector'].add('/jinx[/{message:segment}]', GET=jinx)

If (after restarting the server) you go to /jinx you'll get the old hello message, but if you go to something like /jinx/Greetings you'll see a different message.

Selector lets you put both required and optional named parameters in the URL. These are then accessible in environ at the wsgiorg.routing_args key. Read the selector documentation for more on how to take advantage of this.

Note that in modern TiddlyWeb there is a helper function to get at the routing args: get_route_value. See the core docs with:

pydoc tiddlyweb.web.util.get_route_value

or read get_route_value.

Where's TiddlyWeb

Okay, this is all nice, but where does TiddlyWeb fit into all of this? What about Tiddlers? The basic functionality of TiddlyWeb is the presentation and storage of Tiddlers, Bags and Recipes. The default URLs let you view what's stored by TiddlyWeb in various ways, and store more stuff. Additional URLs allow you access to the "stuff" in different ways. Let's extend the code we have now to list all the Bags on the system in addition to our greeting. Make the body of the /jinx method be the following:

username = environ['tiddlyweb.usersign']['name']
message = environ['wsgiorg.routing_args'][1]['message']
if not message:
    message = 'Hello'
store = environ['tiddlyweb.store']
bag_names = ', '.join([bag.name for bag in store.list_bags()])
return ['<h1>%s %s</h1>\n<h2>%s</h2>' % (message, username, bag_names)]

Now, in addition to your greeting, you should also see "common, system". store is a reference to the StorageInterface. It is set earlier in the WSGI stack. A store provides the interface to getting data in and out of the TiddlyWeb "database". list_bags() lists all the bags in the system, without doing any permissions checks. It is but one of several ways to get at data in the store. Others will be explored later. See pydoc tiddlyweb.stores for more info.

Our management of HTML is starting to get really annoying. This looks like a job for templates.

Templating TiddlyWeb

The current template engine of choice is jinja2. It has a straightforward syntax and doesn't overreach. You should be able to:

sudo pip install -U jinja2

We're going to make a very simple template for our greeting. entitle is still being used, so we only need to worry about the guts of our output. Create a directory called templates

mkdir templates

and in that directory create a file jinx.html with the following contents:

<h1>{{ message }} {{ name }}</h1>

<h2>We have some bags:</h2>

<ul>
    {% for bag in bags %}
    <li>{{ bag.name }}</li>
    {% endfor %}
</ul>

To the top of jinx.py add:

from jinja2 import Environment, FileSystemLoader
# if you are using windows you may need to uncomment the following
# if there are template not found errors
# template_dir = os.path.join(os.path.dirname(__file__), 'templates')
template_env = Environment(loader=FileSystemLoader('templates'))

and change the jinx() method body to:

username = environ['tiddlyweb.usersign']['name']
message = environ['wsgiorg.routing_args'][1]['message']
if not message:
    message = 'Hello'
store = environ['tiddlyweb.store']
bags = store.list_bags()
template = template_env.get_template('jinx.html')
return template.generate(message=message, name=username, bags=bags)

template_env provides us with the mechanism to read in a template file from something, in this case from disk. generate populates the template with the passed data and returns it in a structure suitable for WSGI responses.

For a real plugin it is recommend that you use the tiddlywebplugins.templates package which encapsulates some of the above work. It isn't used in the above examples to make it clear how things operate.

End Piece

That's an introduction to TiddlyWeb plugins. Here's all the code:

from jinja2 import Environment, FileSystemLoader
template_env = Environment(loader=FileSystemLoader('templates'))

from tiddlywebplugins.utils import do_html, entitle, require_any_user

@do_html()
@entitle('Hello World')
@require_any_user()
def jinx(environ, start_response):
    username = environ['tiddlyweb.usersign']['name']
    message = environ['wsgiorg.routing_args'][1]['message']
    if not message:
        message = 'Hello'
    store = environ['tiddlyweb.store']
    bags = store.list_bags()
    template = template_env.get_template('jinx.html')
    return template.generate(message=message, name=username, bags=bags)


def init(config):
    config['selector'].add('/jinx[/{message:segment}]', GET=jinx)

Let's review what we poked at:

1. system_plugins in tiddlywebconfig.py
2. adding functions to be called by the Selector dispatch map
3. helpful routines, like require_any_user from the tiddlywebplugins.utils package
4. using arguments from the URL path
5. getting at data in the TiddlyWeb store
6. adding support for templates

In other tutorials we can look at other things you can do, such as:

1. logging
2. adding commands to twanager, the command line tool
3. listing and sorting tiddlers
4. using tiddlers to store arbitrary content
5. parsing URL query strings
6. modifying the environment then calling existing TiddlyWeb code
7. user roles
8. accepting uploaded content on the server
9. adding to the WSGI stack

If topic is of particular interest, please leave a comment asking for more info.