Charlie Harvey

Snipp.IO: A pastebin made out of Haskell, Yesod and mongodb

I’ve recently been learning about the Haskell-based Yesod web framework. This is about my experience of building a simple pastebin with Yesod, mongodb and Haskell, which took a couple of evening sessions and a Saturday afternoon.

screen shot of snipp.io pasteboin

Why Yesod?

The first question to answer is why choose Yesod as a framework. There is certainly no lack of frameworks out there, even other Haskell frameworks. There were a number of reasons why I was interested in Yesod in particular.

  • I wanted to play with something in Haskell. Regular readers will be all too aware of my continuing quest to learn as much Haskell as possible. But I wasn’t really up for developing my own framework
  • Being made out of Haskell and therefore compiled and functional, Yesod has a very good performance, asynchronicity and scalability characteristics. Though recent releases seem to have slowed since the node.js beating 2011 benchmarks
  • The documentation and community seemed up to scratch and there was a Yesod book. From O’Reilly, which usually indicates that the book will be up to scratch (in contrast to apress or packt)
  • I wanted to see how much a pure functional approach made web development harder or easier

The next question was, of course, why mongodb? Well that was as simple as me not having played much with mongodb and wanting to faff with some cool tech.

Getting set up

I found Michael Snoyman’s introductory screencast contained probably 70% of what I needed to grok to build Snipp.IO.

Like most frameworks, you can start with a scaffolded site. In Yesod’s case you type $ yesod init Rather than accepting commandline parameters Yesod asks you a few questions to get you set up — application name, your name and what persistence layer you want to use. I chose mongodb, as I said above. Then I could hop into the project directory and start the development server$ cd snippio && yesod develVisiting localhost:3000 now brings up an "hello world" program as you might expect. The templates are based on Twitter bootstrap which seemed fine for now.

I hopped straight in to the data models, which of course used the mongodb persistence layer. Yesod uses a very nice little DSL to represent the models. I can imagine it may get hairier when we wanted to model more complex relationships, but my needs were simple, I just created a snip like thisSnip title Text content Textarea created UTCTimeI also needed to add an apikey field for the User model. This meant people who had logged in having access to a long string that they could use to authenticate themselves when creating snipps from the commandline. I had no particular desire to do this with passwords or something and all the authentication was made out of OpenID.

After models come routes. Unlike frameworks like Catalyst, Yesod takes the approach of defining routes centrally in a single file, config/routes. Another nice clean DSL is used to map URLs and params to functions. The result is very clean and I felt ratehr intuitive./static StaticR Static getStatic /auth AuthR Auth getAuth /favicon.ico FaviconR GET /robots.txt RobotsR GET /about.html AboutR GET / HomeR GET /index.html SnipsR GET POST /snip/#SnipId SnipR GET /s SnipsPlainR POST /s/#SnipId SnipPlainR GETThat was it. Routes with a /#SomethingId can take id params. Routes that have a POST must implement a post method in the Handler. Static and Auth routes are a bit special.

Handlers

Yesod takes a fat-controller approach to its design, which makes a lot of sense for a functinal framework. Behaviour mostly resides in Handlers rather than in Models as it might in a more OO framework like Rails or Django. In fact Snipp.IO has only a soingle handler which implements all the functions needed by the routes configuration above. I shall just select a couple to give you an idea of how they look.

We shall look at the about page first of all, as it is a simple static page that uses what Yesod calls a widget to display some text. You can see how the live page looks at Snipp.IO’s about page. Here is the code (excluding the templates) that makes that page.getAboutR :: Handler RepHtml getAboutR = do defaultLayout $ do setTitle "Snipp.IO: A pastebin sort of thingy" $(widgetFile "about") Yes that really is it. The GET AboutR in the routes knows to call getAboutR, so that is what we call the function. getAboutR has type Handler (because it is an handler) RepHtml (because it returns an HTML representation of its widgets). Reading the function body from right to left we create a widget from a file about, set a title for our page and feed both the widget and the setTitle to defaultLayout. I should explain defaultLayout. Like most frameworks Yesod allows you to have a site layout (known as site wrappers in some frameworks).

Let’s look at a more complex route, one that implements GET and POST /index.html SnipsR GET POSTThe index page points to the SnipsR handler, and expects to support GET and POST. So we will have to make getSnipsR and postSnipsR functions. As you can see on Snipp.IO, what you see is different for authenticated and anonymous users. In fact we require people to log in to be able to POST to SnipsR.

getSnipsR is a bit more involved than getAboutR, we get the id of the user who perhaps logged in, and look her up in the database. This time we also create a form widget using generateFormPost and only then do we render everything. getSnipsR :: Handler RepHtml getSnipsR = do muser <- maybeAuthId snips <- runDB $ selectList [] [LimitTo 10, Desc SnipCreated] (formWidget, enctype) <- generateFormPost snipForm defaultLayout $ do setTitle "Snipp.IO: A pastebin sort of thingy" $(widgetFile "snips") So what is snipForm? Just another function that we use to create the form we will use to allow authenticated users to add new "snips". Here it is.snipForm :: Form (Snip) snipForm = renderBootstrap $ Snip <$> areq textField "Title" { fsName = Just "t" } Nothing <*> areq textareaField "Content" {fsName = Just "c"} Nothing <*> aformM (liftIO getCurrentTime) -- the "created at" datestamp

And of course we will need to do something when someone sends a POST by submitting the form. As I mentioned that function is called postSnipsR and it looks like thispostSnipsR :: Handler () postSnipsR = do ((result, _), _) <- runFormPost snipForm (_) <- case result of FormSuccess snip -> do snipId <- runDB $ insert snip setMessage "Snip imported" redirect $ SnipR snipId _ -> do setMessage "Bad input of some description" redirect SnipsR return () So, we "run" the POSTed form, and if it is successful (meaning it validates), we insert the new snip into mongodb, set the success message and resirect to the SnipR (note singular) handler to display the imported snip. Otherwise we set an error message and return to the form.

Widgets and Layouts and Shakespeare … oh my!

At this point it is worth talking a little about widgets, which are an important idea in Yesod, and one of its most interesting features. Normally there are components that you might want to use again and again in your site. They have some HTML, some CSS and maybe some Javascript. Now, you could just include them where you need them, which would be fine, but means that you will have script and style tags sprinkled through your HTML. Which is ugly and can be slow. Yesod allows you to create these discrete packages of CSS, HTML and Javascript but boshes them together at compile time to make nice, clean, cachable HTML. You can define widgets inline or in a template file.

Snipp.IO uses the commandline example in a couple of places, I implemented it as a widget so that I could change it in one place but have it updated everywhere. In my handler I make a simple function cliUsage of type widgetcliUsage :: Widget cliUsage = $(widgetFile "cliUsage")Now my handler templates can interpolate the cliUsage widget as if it were an include. In the templates/cliUsage.hamlet file I write the following<pre> <code> \$ echo hello snippio | \ curl -L -d t='my snip' \ -d k=[apikey] \ --data-urlencode c@- @{SnipsPlainR}Hamlet is the first of 4 templating languages named after characters in Shakespeare that Yesod uses to build templates. You can use indentation rather than tag closing to build your HTML. Variable interpolation is supported as is route interpolation which we use here to get the route of the SnipsPlainR handler — @{SnipsPlainR}. The \s preserve the spacing which is not a feature you would need much outside pre formatted blocks.

Now if I had wanted to add some custom css to the cliUsage widget I could have used the next "Shakespearean" templating language, Cassius. Again, it allows us to use semantic indentation, which simplifies CSS somewhat obviating the need for {} so we can write body font-family: 'PT Serif', serif font-size: 18px background: url("/static/img/billie_holiday.png") repeat scroll 0% 0% transparent color: #333 line-height: 1.375 min-height:85%Cassius also supports variable interpolation. So you could set a variable darkBlue and writeh1 color:#{darkBlue}There is a second CSS templating language Lucius that keeps CSS’s braces if you prefer that way of doing it. I haven’t played with it yet.

The final templating language is for javascript and is called Julius. Again, I have not played with it as yet. It implements variable interpolation and not much else.

I mentioned layouts before. The idea is that there is a lot of "chrome" on most sites which remains the same. Yesod has two "levels" of layout — a wrapper, which holds the main features of the page like the header and footer block, script imports and all that, and the layout proper which defines by default a layout for the page content. This seemed sensible enough and was all I needed to make Snipp.IO. The Hamlet files live in the templates folder again, here is the trivial default-layout.hamlet$maybe msg <- mmsg <div #message>#{msg} ^{widget}As you can see, the mmsg (same idea as Rails flash) is a Maybe type. If it is present, show it. Then show the widget, whatever happens.

The dafult-layout-wrapper.hamlet file was a lot bigger. Here is a section from the top to help give the idea$newline never \<!doctype html> \<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en"> <![endif]--> \<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en"> <![endif]--> \<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en"> <![endif]--> \<!--[if gt IE 8]><!--> <html class="no-js" lang="en"> <!--<![endif]--> <head> <meta charset="UTF-8"> <title>#{pageTitle pc} <meta name="description" content=""> <meta name="author" content="Charlie Harvey"> So, don’t bother showing newlines, then it looks like a standard bit of HTML except we preserve some spacing and interpolate the pageTitle variable — which gets set in the handler up above.

Solid Foundation

Not just a blatant excuse to listen to a beautiful song by the Congos, honest. Yesod means foundating in hebrew, and there is a very important file called Foundation.hs where you can noodle with the various Yesod typeclasses. This is where I got authentication working as I wanted and so on. The coverage in the book is a bit scattered because it takes the approach of illustrating examples with complete snippets rather than showing the scaffolded code. The #yesod IRC channel helped as did looking at the haskellers source code on github.

Deployment

I first thought I was going to put Snipp.IO on heroku. As I was being all hipster with mongodb it sort of made sense. However, I got a bit stuck compiling for that platform as my laptop couldn’t run the vm image that one would usually use to compile the code with. So in the end I deployed to OX4 which meant recomiling, making a shell script and adding an entry in the inittab on the OX4 virtual machine. I then set Apache or nginx to proxy requests. I have high hopes for openshift, but not much time, so I will see how this goes. Here is the launch script, runsnipp.sh#!/bin/bash cd /home/charlie/snipp exec dist/build/snipp/snipp Production +RTS -N 2>&1>sniplogAnd I added this entry to autorun the script, in my inittabz1:2345:respawn:/path/to/runsnipp.sh Finally the relevant bit from my Apache config was as follows ServerName snipp.io ServerAlias www.snipp.io <Proxy *> Order deny,allow Allow from all </Proxy> ProxyRequests Off ProxyPass / http://localhost:3000/ keepalive=On

Conclusions

Yesod did feel productive. Despite being compiled, the toolset didn’t feel massively more laggy to develop on than using other do other frameworks with built in servers — a worry when you need to recompile even for css tweaks. I have not worked much with static typing on the web and I was expecting it to be more painful than it was. Having a decent type system there gives one a good level of confidence that XSS attacks and such are going to be rarer. And the behaviour-orientated style of Yesod contrasts nicely with the data-centric nature of many other frameworks.

I thought the stongest ideas in Yesod included the widget concept. This is the sort of generalizable approach to specific problems that means that functional programming is enjoying something of a renaissance at the moment. The templating systems are all very nice to work with and covered my use cases perfectly.

Yesod is a smaller framework than Catalyst or Django, and there is not as huge a community out there to support you. The documentation is all of good quality, though perhaps not as complete as I would have liked. I would also have liked more cutandpasteable examples, because I am quite lazy. But the friendly irc and the fact that there are some codebases on gthub pretty much made up for those issues.

Overall I would say that Yesod is definitely suitable for a fair number of projects, particularly where you aren’t contending with huge and tangly data models. I imagine something more objecty would be better for that type of work. The performance is fine and though it looks like one could scale up quite nicely. I reckon I might have a crack at node.js next. Until then …


Comments

  • Be respectful. You may want to read the comment guidelines before posting.
  • You can use Markdown syntax to format your comments. You can only use level 5 and 6 headings.
  • You can add class="your language" to code blocks to help highlight.js highlight them correctly.

Privacy note: This form will forward your IP address, user agent and referrer to the Akismet, StopForumSpam and Botscout spam filtering services. I don’t log these details. Those services will. I do log everything you type into the form. Full privacy statement.