Web development is a big part of programming today. I say this with no facts, no figures, and absolutely no way to back that claim up. I say it because, well, I am a web developer and I want everything to be about me.

Unfortunately, when people speak about LISP, it is rarely in the context of using it to create websites, which is a tremendous shame as I believe that 1) LISP is a fun language to write, 2) LISP is a quick language to build with, and 3) LISP has superpowers. These features make LISP an incredible tool which (to me) surpasses offers such as Rails and Laravel.

Today, we’re going to create a small website which has a home page, a page which responds to URL parameters, and an API route which performs calculations for us. It’s not going to be the next VC-funded unicorn but it’ll hopefully be enough to demonstrate the building blocks which could be used to build elaborate, complex applications.

The full code for this can be found at git.sr.ht/~smcn/lisp-for-webdev.


Dependencies

Hunchentoot

Hunchentoot is a web server written in LISP. It has a great developer experience and comprehensive documentation. It also makes sense; when you get a feel for it, it’s very intuitive.

Spinneret

Spinneret is used to generate HTML in a LISP-y way.

For example:

<div class="content">
	<p>This is a paragraph!</p>
</div>

can be written as:

(:div :class "content"
	(:p "This is a paragraph!"))

Isn’t that so much nicer?

Jonathan

Jonathan is the best JSON parser/encoder that I’ve found. I’ve used all of the big ones but nothing comes close, in my personal experience. to-json and parse will do 99.999% of what you require, or your money back.


Getting started

Almost every single web framework that you’ve ever used has started with you opening a port for the application to listen on. Hunchentoot is no different in this regard. Glancing over this quickly, we’re going to define a global variable to hold the connection, and then we’ll define three functions to start the connection, to stop the connection, and to restart the connection.

;;;; smcn-io-webdev.lisp

(defvar *connection*)

(defun start-connection (&optional (port 8080))
  "Start the Hunchentoot connection on PORT."
  (format t "Starting connection~%")
  (setf *connection* (hunchentoot:start
		               (make-instance 'hunchentoot:easy-acceptor
			                          :port port)))
  (format t "Connection started~%"))

(defun stop-connection ()
  "Kill the Hunchentoot connection."
  (format t "Stopping connection~%")
  (hunchentoot:stop *connection*)
  (setf *connection* nil)
  (format t "Connection stopped~%"))

(defun restart-connection (&optional (port 8080))
  "Restart the Hunchentoot connection on PORT."
  (format t "Restarting connection~%")
  (when *connection*
    (stop-connection))
  (start-connection port)
  (format t "Connection restarted~%"))

If we open up a REPL and run (start-connection) this will bind the port 8080 to LISP. As easy as that!

Navigating to a browser and opening localhost:8080 will show:

Hunchentoot default page

Let’s quickly redefine that using Hunchentoot’s define-easy-handler function.

;;;; home-page.lisp

(hunchentoot:define-easy-handler (home-page :uri "/") ()
  "Hello world!")

If we evaluate that, open the browser again, and refresh, we’ll see “Hello world!”. Amazing! I mean, not amazing, but progress! We can see that define-easy-handler wants to return a string. So, let’s see if we can send some HTML to the browser.

;;;; home-page.lisp

(hunchentoot:define-easy-handler (home-page :uri "/") ()
  ;; set the content-type of the response object to text/html
  (setf (hunchentoot:content-type*) "text/html")
  "<h1>Hello world!</h1>")

Re-evaluate that, refresh the page, and voila! ✨ formatting

Not very LISP-y, though. Also, it’ll become wildly unwieldy almost immediately. Enter Spinneret. Spinneret has this really handy macro with-html-string which we can use to generate HTML strings using LISP-y forms. We can finally rewrite the above function as:

;;;; home-page.lisp

(hunchentoot:define-easy-handler (home-page :uri "/") ()
  (setf (hunchentoot:content-type*) "text/html")
  (spinneret:with-html-string
    (:h1 "Hello world!")))

Getting into gear

That was a lateral move. Let’s make it interesting. We’re going to create our own mini-framework using macros.

A macro is a way of generating code. When LISP sees that you’re using one, it’s going to expand your s-expression into the desired form. This means that we can avoid writing a lot of boiler plate, and we get to define a language tailored for our domain.

First of all, we want a function to define a controller, which listens for requests on a specific route then sends the incoming request to a specific function depending on the request method (GET, POST, PUT, PATCH, DELETE).

;;;; framework.lisp

(defmacro defcontroller (name uri &body body)
  `(hunchentoot:define-easy-handler (,name :uri ,uri) ()
     ;; DEFINE-EASY-HANDLER binds the incoming request object to a
     ;; variable called *request*.
  
     ;; With Hunchentoot, if a method ends in *, it means that it'll
     ;; accept the *request* variable implicitly. Meaning we don't
     ;; need to provide it with an argument.
     (let ((req-method (hunchentoot:request-method*)))

       ;; BODY is a list of lists where the first element is a symbol
       ;; representing a request method and the second is the function
       ;; which receives the request object.
	   ;;
       ;; i.e. ((:GET  (lambda (r) (print "GET request")))
       ;;       (:POST (lambda (r) (print "POST request"))))
	   ;;
	   ;; We'll use a COND to route the request.
       (cond
   	     ,@(mapcan
   	        (lambda (arg)
   	          (destructuring-bind (method handler-func) arg
   	     	    `(((eq req-method ,method)
   	     	       (funcall ,handler-func *request*)))))
   	        body)
   
	     ;; If the request isn't handled by a function, return 
		 ;; 405: Method Not Allowed
   	     (t (setf (hunchentoot:return-code*) hunchentoot:+http-method-not-allowed+)
   	        "Method Not Allowed")))))

The above macro isn’t too hairy, you’re essentially creating a define-easy-handler function which inspects the request method and calls the relevant function with the incoming request. Let’s rewrite our home page to use it.

;;;; home-page.lisp

(defcontroller home-page-controller "/"
  (:GET (lambda (*request*)
            (setf (hunchentoot:content-type*) "text/html")
       	    (spinneret:with-html-string
	          (:h1 "Hello world!")))))

If we call macroexpand-1 on the controller, it’ll show us the generated code:

(define-easy-handler
  (home-page-controller :uri "/")
  nil
  (let ((req-method (request-method*)))
    (cond ((eq req-method :get)
           (funcall (lambda (*request*)
                      (setf (content-type*) "text/html")
                      (with-html-string (:h1 "Hello world!")))
                    *request*))
          (t
           (setf (return-code*) +http-method-not-allowed+)
           "Method Not Allowed"))))

Re-evaluate the controller, refresh the page, and it should look the same!

I’m not loving that lambda, though. We can do better. Let’s create a request handler.

;;;; framework.lisp

(defmacro defhandler (name &body body)
  `(defun ,name (*request*)
     ,@body))

This macro creates a function which accepts the Hunchentoot request. This may seem unnecessary but just go with it for now. Okay, let’s use it.

;;;; home-page.lisp

(defcontroller home-page-controller "/"
  (:GET #'home-page-content))

(defhandler home-page-content
  (spinneret:with-html-string
    (:h1 "Hello world!")))

Hey! It’s working! And it’s pretty! But it’s not perfect. I don’t love that we’re embedding HTML in handlers. We want to separate logic as much as possible.

Let’s create a new type of function which defines templates.

;;;; framework.lisp

(defmacro defpage (name &body body)
  `(defun ,name ()
     (setf (hunchentoot:content-type*) "text/html")
     (spinneret:with-html-string
       (:doctype)
       (:html
	    (:head
	     (:title "The Worlds Best Hello World!"))
	    (:body
	     ,@body)))))

Okay, that was easy enough. We’ve created a function which sets the content-type and then builds the scaffolding for a HTML page. As we’re creating a function which returns a string, we can call (home-page) at the end of the handler, but that’s a little bit too ambiguous for me. I would prefer to know when I’m rendering a page. So, let’s add some syntactic sugar:

;;;; framework.lisp

(defmacro render (page)
  `(funcall ,page))

Using this, our home-page.lisp now looks like:

;;;; home-page.lisp

(defcontroller home-page-controller "/"
  (:GET #'home-page-content))

(defhandler home-page-content
  (render #'home-page))

(defpage home-page
  (:h1 "Hello world!"))

Not too shabby! It’s all there, it all makes sense, and it all works!

Stepping it up a bit

However, that’s still elementary. You could achieve the same result with NGINX and static files. Let’s add a little bit of interactivity with web pages. I mean, not loads, but some. “Some” is a stretch, too, honestly. We’ll create a page that says hi to a URL parameter.

Let’s replicate the basic controller > handler > page flow that we used above.

;;;; say-hi.lisp

(defcontroller say-hi-controller "/say-hi"
  (:GET #'say-hi-handler))

(defhandler say-hi-handler
  (render #'say-hi))

(defpage say-hi
  (:h1 "Hi there!"))

Easy enough, right? If you evaluate these and then go to localhost:8080/say-hi, you’ll see the above in action.

You may be noticing an issue in the code. We want the handler to get the URL parameter and then pass it to the page, but doing so means that we’ll need to rewrite the defpage and render macros to accept zero or more arguments.

To side step this potentially ugly code, we can make use of LISPs dynamic scoping. We can define a variable at the top level, give it a default value, then shadow it (temporarily/lexically redefine it). The benefit of this is that defpage doesn’t need to worry about accepting any arguments and can focus on what it does best: giving us some pretty1 HTML.

;;;; say-hi.lisp

;; Default value of the *name* global variable
(defvar *name* "there")

(defcontroller say-hi-controller "/say-hi"
  (:GET #'say-hi-handler))

(defhandler say-hi-handler
  (let* ((url-param (hunchentoot:get-parameter "name"))

	     ;; use the URL parameter if is set, otherwise fall back to
	     ;; the default value
	     (*name* (if url-param url-param *name*)))
    (render #'say-hi)))

(defpage say-hi
  (:h1 (format nil "Hi ~a!" *name*)))

And if we try it with localhost:8080/say-hi?name=smcn.io, we get:

say-hi with value

And removing the URL param, we get:

say-hi without value

Creating an API endpoint

To make things a little bit more interesting, we’re going to set up an API endpoint to do some small calculations.

We’ll implement the controller > handler flow again, but this time we don’t need to worry about a page as we’ll be returning some simple text.

;;;; calc.lisp

(defcontroller calc-controller "/calc"
  (:POST #'calc-handler))

(defhandler calc-handler
  "pong")

This is mostly the same code that we’ve been seeing all article. The only difference to note is that we’re handling a POST request now.

If we curl the endpoint, we should see a response:

calc response

Awesome!

We want to be able to send {"calculation": "(+ 1 2 3 4 5)"} to the endpoint and have it return 15.

Okay, yes, we’re going to eval a string. This is not production code and is merely here to show how to handle JSON. Don’t ever, ever, ever do this, ever.

The logic is simple:

  1. Convert the request to JSON
  2. Find the calculation
  3. Return the computed value

That’s easy enough to implement in LISP:

;;;; calc.lisp

(defhandler calc-handler
  (let* ((raw-request (hunchentoot:raw-post-data
		              ;; we want force-text as otherwise we're dealing
		              ;; with raw bytes
		              :force-text t))
	     (parsed-request (jonathan:parse raw-request))

		 ;; using :|| around the JSON key preserves text casing. LISP
		 ;; treats all symbols as uppercase by default.
	     (calculation (getf parsed-request :|calculation|)))

    (if calculation
	    (format nil "The answer is: ~a~%"
				;; don't let me see you using this or I'll tell your parents
	    	    (write-to-string (eval (read-from-string calculation))))
	    (format nil "please ensure that the calculation key exists~%"))))

And if we throw some data at it:

calc math

Defining some middleware

One issue with this is that the endpoint will accept any kind of request, regardless of content-type. calc-handler is adamant that it’s only going to deal with JSON and will break if you give it anything else. We could have a bunch of if statements littering the handlers whenever we come across this hard boundary, but it doesn’t look great.

Instead, we’re going to update defhandler to accept a list of functions that it’ll run before executing the body, with the idea that if a function doesn’t like what it sees, it can abort the request. This way the handler logic is focused on handling requests and nothing else.

We can use a simple catch and throw to end execution when required.

;;;; framework.lisp

(defmacro defhandler (name middleware &body body)
  `(defun ,name (*request*)
     (let ((middleware (list ,@middleware)))
       ;; Loop over the middleware list and call the function with the
       ;; incoming request. If any function throws :RETURN then break
       ;; out of the catch block and don't execute the BODY.
       (catch :return
	     (dolist (hook middleware)
	       (funcall hook *request*))
	     ,@body))))

And with that, let’s make things explicit and clean by creating a defmiddleware and a fail macro.

;;;; framework.lisp

(defmacro defmiddleware (name &body body)
  `(defun ,name (*request*)
       ,@body))

(defmacro fail (&body body)
  `(throw :return (progn ,@body)))

We can use the above the create a middleware which verifies that the incoming request is application/json.

;;;; middleware.lisp

(defmiddleware json-request-p
  (unless (string= (string-downcase (hunchentoot:header-in* "Content-Type"))
		           "application/json")

    (fail
	  ;; set the reponse code to 415: Unsupported Media Type
	  (setf (hunchentoot:return-code*) hunchentoot:+http-unsupported-media-type+)
	  (format nil "please only send json~%"))))

Finally, we can update the calc-handler to use the new macro

;;;; calc.lisp

(defhandler calc-handler (#'json-request-p)
  ...)

And a quick test

calc content-type

It works!


Final thoughts

LISP is incredibly powerful. It has been described as a programmable programming language. This makes it incredibly well suited to many different tasks, including web development.

In 64 lines (including comments) and just 6 macros, we were able to create a small framework which let us spin up web pages in no time at all.

This article obviously avoided tasks such as security, performance, and scaling, but I hope that you can see just how easy it is to implement ideas using very basic logic and a little know how.


  1. “Pretty” for a backend developer ↩︎