Next Gen Web Apps with Scala, BlueEyes, and MongoDB

2012-02-06 (Last Modified: 2012-02-06)

Web application architecture is in the midst of a big paradigm shift. Since the inception of the web we’ve been treating the browser like a thin client. Apps just dump markup to the browser which is then rendered. Every interaction requires a request back to the server which then returns more logic-less markup to the browser. In this model our web applications are server applications. There are certainly advantages to this model - especially when the markup consumers don’t have the capabilities to do anything more (or have inconsistent capabilities).

Web browser modernization has paved the way to move from the old “browser as a thin client” model to more of a Client/Server model where the client-side of the application actually runs in the browser. There are a lot of names for this new model, Ajax being a step along the way, and perhaps HTML5 being the full realization of Client/Server web apps.

There are hundreds of new libraries, tools, programming languages, and services that support the creation of these next generation web (and mobile) apps. One thing that is certainly lacking is one cohesive, end-to-end solution. I have no doubt that we will be seeing those crop up soon, but in the mean time lets pick a few libraries and build a couple simple apps to get a better understanding of what this new model looks like.

I’ve decided to try this with a stack consisting of Scala for my back-end code, BlueEyes for handling data access and serialization with transport over HTTP, MongoDB for data storage, and JavaScript with jQuery for the client-side. I’m going to keep this very simple and not pull in anything else. In future articles I’ll expand on this foundation.

So lets dive into some code. All of the code is on GitHub. If you want to get a local copy just do:

git clone git://github.com/jamesward/helloblueeyes.git

This project uses the Scala Build Tool (SBT) for downloading dependencies, compiling, and packaging. Follow the SBT setup instructions if you want to be able to compile the code on your machine. If you want to use IntelliJ or Eclipse then run one of the following commands:

sbt gen-idea

sbt eclipse

Lets start with some BlueEyes basics. BlueEyes is a lightweight web framework built on Netty (a really high performance, NIO-based HTTP library). BlueEyes makes it easy to define a tree of HTTP request handling patterns. I chose BlueEyes because it’s really lightweight and utilizes some nice features of the Scala language to make the code very concise. It’s kinda like a DSL for HTTP request handling.

To get started with BlueEyes we first need to create an AppServer that provides a way to start the server and a place to plug services into. Here is the code for the AppServer:

package net.interdoodle.example

object AppServer extends EnvBlueEyesServer with HelloHtmlServices with HelloJsonServices with HelloStartupShutdownServices with HelloMongoServices

My AppServer is an object which means it’s a singleton (that makes sense because I can’t have more than one of these in a single instance). The AppServer extends EnvBlueEyesServer which is a wrapper around the base BlueEyesServer that provides a way to get the HTTP port via an environment variable (this is necessary for running on Heroku - which we will get to later). The AppServer composes in a bunch of Services that actually setup HTTP request pattern trees.

Starting with the simplest one, HelloHtmlServices we have:

trait HelloHtmlServices extends BlueEyesServiceBuilder with BijectionsChunkString {
  val helloHtml = service("helloHtml", "0.1") { context =>
      request {
        path("/") {
          produce(text/html) {
            get { request =>
              val content = 
                Hello, world!
              
              val response = HttpResponse(content = Some(content.buildString(true)))
              Future.sync(response)
          }
        }
      }
    }
  }
}

Using the BlueEyesServiceBuilder the HelloHtmlServices trait creates a service that consists of a big pattern match that goes something like this: For a request to path “/”, produce an HTML response when there is a HTTP GET request. Notice that the response is actually wrapped in a Future. This is an important part of the next generation web app architecture. Because the underlying HTTP server is NIO-based (non-blocking), the actual HTTP handling threads can be very efficiently managed so that a thread is only in use when actual IO is happening. BlueEyes supports this by using Future wrapped responses, allowing the HTTP thread to be unblocked by back-end processing.

If you want to try this out locally run the following to compile the app and create a Unix script that makes it easy to start the AppServer:

sbt stage

Now start the AppServer process:

target/start net.interdoodle.example.AppServer

Verify that HelloHtmlServices is working by opening the following URL in your browser:

http://localhost:8080/

Now lets walk through a few steps to deploy this application on the cloud with Heroku.

Step 1) Sign up for a Heroku account

Step 2) Install the Heroku Toolbelt

Step 3) Login to Heroku from the command line:

heroku login

Step 4) Create a new application on Heroku using the Cedar stack:

heroku create -s cedar

Step 5) Deploy the app on Heroku:

git push heroku master

Step 6) Verify that the app works in your browser:

heroku open

Great! We have an app built with Scala and BlueEyes running on the cloud. Now for the next piece of the architecture. The UI of the application will be written in JavaScript and will run on the client-side. But we need to have a way to transfer data between the client and the server. JSON has become the regular way to do this because it is parsed easily and quickly on the client. So a BlueEyes service will generate some JSON data that can then be consumed by a JavaScript application running on the client. But how do we get that JavaScript application running on the client? This is the next piece of the architecture.

Ideally the JSON services are served separately from the client-side of the application (where the client-side includes all HTML, JavaScript, CSS, images, and other static assets). It is also ideal to serve the client-side from a Content Delivery Network (CDN) so that it’s edge cached and loads super fast. Dynamic assets (the JSON services) can’t be edge cached so there is a logical (and functional) separation between the client-side and the server-side. But there is a problem with this approach.

The CDN and the JSON services hosts each have different DNS names and browsers don’t allow cross-origin requests (actually they don’t allow access to the responses). There are a few different ways to deal with this (JSONP, iframe hackery, etc) but the approach I’m taking is simple. The JSON services server will serve an entry point / thin shim web page. That shim then loads the rest of the client-side from the CDN and since the origin of the web page is the same as the JSON services, there is no need to make a cross-origin request. (Side note: There is a new way to do cross-origin requests cleanly and natively in the browser, but it’s not ubiquitously supported yet.)

Here is HelloJsonServices that illustrates these patterns:

trait HelloJsonServices extends BlueEyesServiceBuilder with BijectionsChunkJson with BijectionsChunkString {
  val helloJson:HttpService[ByteChunk] = service("helloJson", "0.1") { context:HttpServiceContext =>
    request {
      path("/json") {
        contentType(application/json) {
          get { request: HttpRequest[JValue] =>
            val jstring = JString("Hello World!")
            val jfield = JField("result", jstring)
            val jobject = JObject(jfield :: Nil)
            val response = HttpResponse[JValue](content = Some(jobject))
            Future.sync(response)
          }
        } ~
        produce(text/html) {
          get { request: HttpRequest[ByteChunk] =>
            val contentUrl = System.getenv("CONTENT_URL")

            val content = 
              
              
              
            
            Future.sync(HttpResponse[String](content = Some(content.buildString(true))))
          }
        }
      }
    }
  }
}

This service handles requests to “/json” and if the requested content type is JSON then a simple JSON object is returned. If the requested content is HTML then the shim web page is returned. Inside the shim web page an environment variable, CONTENT_URL, is used to specify the URLs to load the JavaScript for the application.

The first JavaScript is the minified jQuery library. This abstracts some browser differences, makes doing the JSON request and modifying the webpage very simple. Here is the main client application hello_json.js:

$(function() {
    $.ajax("/json", {
        contentType: "application/json",
        success: function(data) {
            $("body").append(data.result);
        }
    });
});

This application has a function handler for when the page has loaded. That function makes an ajax request to “/json” with the “application/json” content type. Then on a successful response it just appends the result into the web page.

To run this locally first set the CONTENT_URL environment variable to the CDN where I’ve put the JavaScript files:

export CONTENT_URL=http://cdn-helloblueeyes.interdoodle.net/

Now start (or restart) the server and open:

http://localhost:8080/json

To test the JSON service locally run:

curl --header "Content-Type:application/json" http://localhost:8080/json

You can also use a local static file server for testing, just run net.interdoodle.example.httpstaticfileserver.HttpStaticFileServer and set CONTENT_URL accordingly.

If you want to try this on Heroku then set CONTENT_URL for your application:

heroku config:add CONTENT_URL=http://cdn-helloblueeyes.interdoodle.net/

This will set the environment variable and restart the application. Then navigate to the your Heroku application’s “/json” URL in your browser. Now both the server-side and client-side are cloudified!

Side Note: I’m using Amazon CloudFront as the CDN for this example. But any CDN would work fine. There are various ways to upload files to Amazon CloudFront (actually they get uploaded to Amazon S3) but one thing I’m looking into is a SBT plugin that will do this.

Now lets add some simple data access into the mix. I’ve chosen MongoDB because it has native support for JSON and we can take advantage of Hammersmith - a non-blocking MongoDB driver. In this example I’m not using Hammersmith since support for it hasn’t been officially added to BlueEyes (but at least there is the opportunity to easily switch to Hammersmith in the future).

Here is the code for HelloMongoServices:

trait HelloMongoServices extends BlueEyesServiceBuilder with MongoQueryBuilder with BijectionsChunkJson with BijectionsChunkString {

  val helloMongo = service("helloMongo", "0.1") {
    logging { log => context =>
      startup {
        // use MONGOLAB_URI in form: mongodb://username:password@host:port/database
        val mongolabUri = Properties.envOrElse("MONGOLAB_URI", "mongodb://127.0.0.1:27017/hello")
        val mongoURI = new MongoURI(mongolabUri)

        HelloConfig(new EnvMongo(mongoURI, context.config.configMap("mongo"))).future
      } ->
      request { helloConfig: HelloConfig =>
        path("/mongo") {
          contentType(application/json) {
            get { request: HttpRequest[JValue] =>
              helloConfig.database(selectAll.from("bars")) map { records =>
                HttpResponse[JValue](content = Some(JArray(records.toList)))
              }
            }
          } ~
          contentType(application/json) {
            post { request: HttpRequest[JValue] =>
              request.content map { jv: JValue =>
                helloConfig.database(insert(jv --> classOf[JObject]).into("bars"))
                Future.sync(HttpResponse[JValue](content = request.content))
              } getOrElse {
                Future.sync(HttpResponse[JValue](status = HttpStatus(BadRequest)))
              }
            }
          } ~
          produce(text/html) {
            get { request: HttpRequest[ByteChunk] =>
              val contentUrl = System.getenv("CONTENT_URL")

              val content = 
                              
                              
                              
                            
              Future.sync(HttpResponse[String](content = Some(content.buildString(true))))
            }
          }
        }
      } ->
      shutdown { helloConfig: HelloConfig =>
        Future.sync(())
      }
    }
  }
}

This service has a startup hook that creates a MongoDB connection based on the MONGOLAB_URI environment variable or a default value. Then there are three request handlers for the “/mongo” path. The first handles JSON HTTP GET requests by fetching some data from MongoDB and returning it. A JSON HTTP POST handler inserts the JSON that was sent into MongoDB. Then there is a HTML HTTP GET request handler that returns the web page shim that loads the JavaScript from the CDN.

Here is the client-side of the application hello_mongo.js:

$(function() {

    $("body").append('

<h4>
  Bars:
</h4>');
    $("body").append('

<ul id="bars">
  
</ul>');
    $("body").append('

<input id="bar" />');
    $("body").append('<button id="submit">GO!</button>');


    $("#submit").click(addbar);
    $("#bar").keyup(function(key) {
        if (key.which == 13) {
            addbar();
        }
    });

    loadbars();

});

function loadbars() {
    $.ajax("/mongo", {
        contentType: "application/json",
        success: function(data) {
            $("#bars").children().remove();
            $.each(data, function(index, item) {
                $("#bars").append("

<li>
  " + item.name + "
</li>");
            });
        }
    });
}

function addbar() {
    $.ajax({
        url: "/mongo",
        type: 'post',
        dataType: 'json',
        success: loadbars,
        data: '{"name": "' + $("#bar").val() + '"}',
        contentType: 'application/json'
    });
}

This JavaScript application starts by adding some elements to the page for displaying and entering “bars”. The loadbars() function makes a ajax GET request to the JSON service and then appends the result into the element on the page with the id of bars. The addbar() method makes a POST request to the JSON service, passing serialized JSON containing the name of the “bar” to the JSON service, then on success it calls loadbars().

If you want to run this locally then you will need a MongoDB server setup and running locally. Make sure CONTENT_URL is set correctly and if your MongoDB settings differ from the default, then set the MONGOLAB_URI environment variable. Start the AppServer process and then open:

http://localhost:8080/mongo

You should be able to add new “bars” and see them listed.

To run on Heroku you will need to add the MongoLab add-on. When the MongoLab add-on is added the MONGOLAB_URI environment variable on your Heroku application will be automatically set with the connection information for the newly provisioned MongoDB system. To add the MongoLab add-on just run:

heroku addons:add mongolab

Now open your Heroku app’s “/mongo” URL in your browser. Wahoo! You have a client/server browser app running on the cloud! Hopefully that has helped you to get a glimpse of where web application architecture is going. This is just part of the picture and I have a few other articles planned that will cover some of the other pieces. So keep watching here and let me know what you think.

Acknowledgments: Mike Slinn, a friend of mine that does Scala and Akka consulting, helped me understand BlueEyes and helped write some of the code for this project. Mike is also the author of the recently released book Composable Futures with Akka 2.0 with Java and Scala Code Examples.