Docker is a lightweight virtualization technology for Linux that promises to revolutionize the deployment and management of distributed applications. Rather than requiring a complete operating system, like a traditional virtual machine, Docker is built on top of Linux containers, a feature of the Linux kernel, that allows light-weight Docker containers to share a common kernel while isolating applications and their dependencies. There’s a very good Docker SlideShare presentation here that explains the philosophy behind Docker using the analogy of standardized shipping containers. Interesting that the standard shipping container has done more to create our global economy than all the free-trade treaties and international agreements put together. A Docker image is built from a script, called a ‘Dockerfile’. Each Dockerfile starts by declaring a parent image. This is very cool, because it means that you can build up your infrastructure from a layer of images, starting with general, platform images and then layering successively more application specific images on top. I’m going to demonstrate this by first building an image that provides a Mono development environment, and then creating a simple ‘Hello World’ console application image that runs on top of it. Because the Dockerfiles are simple text files, you can keep them under source control and version your environment and dependencies alongside the actual source code of your software. This is a game changer for the deployment and management of distributed systems. Imagine developing an upgrade to your software that includes new versions of its dependencies, including pieces that we’ve traditionally considered the realm of the environment, and not something that you would normally put in your source repository, like the Mono version that the software runs on for example. You can script all these changes in your Dockerfile, test the new container on your local machine, then simply move the image to test and then production. The possibilities for vastly simplified deployment workflows are obvious. Docker brings concerns that were previously the responsibility of an organization’s operations department and makes them a first class part of the software development lifecycle. Now your infrastructure can be maintained as source code, built as part of your CI cycle and continuously deployed, just like the software that runs inside it. Docker also provides docker index, an online repository of docker images. Anyone can create an image and add it to the index and there are already images for almost any piece of infrastructure you can imagine. Say you want to use RabbitMQ, all you have to do is grab a handy RabbitMQ images such as https://index.docker.io/u/tutum/rabbitmq/ and run it like this: docker run -d -p 5672:5672 -p 55672:55672 tutum/rabbitmq The –p flag maps ports between the image and the host. Let’s look at an example. I’m going to show you how to create a docker image for the Mono development environment and have it built and hosted on the docker index. Then I’m going to build a local docker image for a simple ‘hello world’ console application that I can run on my Ubuntu box. First we need to create a Docker file for our Mono environment. I’m going to use the Mono debian packages from directhex. These are maintained by the official Debian/Ubuntu Mono team and are the recommended way of installing the latest Mono versions on Ubuntu. Here’s the Dockerfile: #DOCKER-VERSION 0.9.1 # #VERSION 0.1 # # monoxide mono-devel package on Ubuntu 13.10 FROM ubuntu:13.10 MAINTAINER Mike Hadlow RUN sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q software-properties-common RUN sudo add-apt-repository ppa:directhex/monoxide -y RUN sudo apt-get update RUN sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q mono-devel Notice the first line (after the comments) that reads, ‘FROM ubuntu:13.10’. This specifies the parent image for this Dockerfile. This is the official docker Ubuntu image from the index. When I build this Dockerfile, that image will be automatically downloaded and used as the starting point for my image. But I don’t want to build this image locally. Docker provide a build server linked to the docker index. All you have to do is create a public GitHub repository containing your dockerfile, then link the repository to your profile on docker index. You can read the documentation for the details. The GitHub repository for my Mono image is at https://github.com/mikehadlow/ubuntu-monoxide-mono-devel. Notice how the Docker file is in the root of the repository. That’s the default location, but you can have multiple files in sub-directories if you want to support many images from a single repository. Now any time I push a change of my Dockerfile to GitHub, the docker build system will automatically build the image and update the docker index. You can see image listed here:https://index.docker.io/u/mikehadlow/ubuntu-monoxide-mono-devel/ I can now grab my image and run it interactively like this: $ sudo docker pull mikehadlow/ubuntu-monoxide-mono-devel Pulling repository mikehadlow/ubuntu-monoxide-mono-devel f259e029fcdd: Download complete 511136ea3c5a: Download complete 1c7f181e78b9: Download complete 9f676bd305a4: Download complete ce647670fde1: Download complete d6c54574173f: Download complete 6bcad8583de3: Download complete e82d34a742ff: Download complete $ sudo docker run -i mikehadlow/ubuntu-monoxide-mono-devel /bin/bash mono --version Mono JIT compiler version 3.2.8 (Debian 3.2.8+dfsg-1~pre1) Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com TLS: __thread SIGSEGV: altstack Notifications: epoll Architecture: amd64 Disabled: none Misc: softdebug LLVM: supported, not enabled. GC: sgen exit Next let’s create a new local Dockerfile that compiles a simple ‘hello world’ program, and then runs it when we run the image. You can follow along with these steps. All you need is a Ubuntu machine with Docker installed. First here’s our ‘hello world’, save this code in a file named hello.cs: using System; namespace Mike.MonoTest { public class Program { public static void Main() { Console.WriteLine("Hello World"); } } } Next we’ll create our Dockerfile. Copy this code into a file called ‘Dockerfile’: #DOCKER-VERSION 0.9.1 FROM mikehadlow/ubuntu-monoxide-mono-devel ADD . /src RUN mcs /src/hello.cs CMD ["mono", "/src/hello.exe"] Once again, notice the ‘FROM’ line. This time we’re telling Docker to start with our mono image. The next line ‘ADD . /src’, tells Docker to copy the contents of the current directory (the one containing our Dockerfile) into a root directory named ‘src’ in the container. Now our hello.cs file is at /src/hello.cs in the container, so we can compile it with the mono C# compiler, mcs, which is the line ‘RUN mcs /src/hello.cs’. Now we will have the executable, hello.exe, in the src directory. The line ‘CMD [“mono”, “/src/hello.exe”]’ tells Docker what we want to happen when the container is run: just execute our hello.exe program. As an aside, this exercise highlights some questions around what best practice should be with Docker. We could have done this in several different ways. Should we build our software independently of the Docker build in some CI environment, or does it make sense to do it this way, with the Docker build as a step in our CI process? Do we want to rebuild our container for every commit to our software, or do we want the running container to pull the latest from our build output? Initially I’m quite attracted to the idea of building the image as part of the CI but I expect that we’ll have to wait a while for best practice to evolve. Anyway, for now let’s manually build our image: $ sudo docker build -t hello . Uploading context 1.684 MB Uploading context Step 0 : FROM mikehadlow/ubuntu-monoxide-mono-devel ---> f259e029fcdd Step 1 : ADD . /src ---> 6075dee41003 Step 2 : RUN mcs /src/hello.cs ---> Running in 60a3582ab6a3 ---> 0e102c1e4f26 Step 3 : CMD ["mono", "/src/hello.exe"] ---> Running in 3f75e540219a ---> 1150949428b2 Successfully built 1150949428b2 Removing intermediate container 88d2d28f12ab Removing intermediate container 60a3582ab6a3 Removing intermediate container 3f75e540219a You can see Docker executing each build step in turn and storing the intermediate result until the final image is created. Because we used the tag (-t) option and named our image ‘hello’, we can see it when we list all the docker images: $ sudo docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE hello latest 1150949428b2 10 seconds ago 396.4 MB mikehadlow/ubuntu-monoxide-mono-devel latest f259e029fcdd 24 hours ago 394.7 MB ubuntu 13.10 9f676bd305a4 8 weeks ago 178 MB ubuntu saucy 9f676bd305a4 8 weeks ago 178 MB ... Now let’s run our image. The first time we do this Docker will create a container and run it. Each subsequent run will reuse that container: $ sudo docker run hello Hello World And that’s it. Imagine that instead of our little hello.exe, this image contained our web application, or maybe a service in some distributed software. In order to deploy it, we’d simply ask Docker to run it on any server we like; development, test, production, or on many servers in a web farm. This is an incredibly powerful way of doing consistent repeatable deployments. To reiterate, I think Docker is a game changer for large server side software. It’s one of the most exciting developments to have emerged this year and definitely worth your time to check out.

It’s a common pattern for command line tools to have multiple subcommands that run off of a single executable. For example, git fetch origin and git commit --amend both use the same executable /usr/bin/git to run. Each subcommand has its own set of required and optional parameters. This pattern is fairly easy to implement in your own Python command-line utilities using argparse. Here is a script that pretends to be git and provides the above two commands and arguments. #!/usr/bin/env python import argparse import sys class FakeGit(object): def __init__(self): parser = argparse.ArgumentParser( description='Pretends to be git', usage='''git [] The most commonly used git commands are: commit Record changes to the repository fetch Download objects and refs from another repository ''') parser.add_argument('command', help='Subcommand to run') # parse_args defaults to [1:] for args, but you need to # exclude the rest of the args too, or validation will fail args = parser.parse_args(sys.argv[1:2]) if not hasattr(self, args.command): print 'Unrecognized command' parser.print_help() exit(1) # use dispatch pattern to invoke method with same name getattr(self, args.command)() def commit(self): parser = argparse.ArgumentParser( description='Record changes to the repository') # prefixing the argument with -- means it's optional parser.add_argument('--amend', action='store_true') # now that we're inside a subcommand, ignore the first # TWO argvs, ie the command (git) and the subcommand (commit) args = parser.parse_args(sys.argv[2:]) print 'Running git commit, amend=%s' % args.amend def fetch(self): parser = argparse.ArgumentParser( description='Download objects and refs from another repository') # NOT prefixing the argument with -- means it's not optional parser.add_argument('repository') args = parser.parse_args(sys.argv[2:]) print 'Running git fetch, repository=%s' % args.repository if __name__ == '__main__': FakeGit() The argparse library gives you all kinds of great stuff. You can run ./git.py --help and get the following: usage: git [] The most commonly used git commands are: commit Record changes to the repository fetch Download objects and refs from another repository Pretends to be git positional arguments: command Subcommand to run optional arguments: -h, --help show this help message and exit You can get help on a particular subcommand with ./git.py commit --help. usage: git.py [-h] [--amend] Record changes to the repository optional arguments: -h, --help show this help message and exit --amend Want bash completion on your awesome new command line utlity? Try argcomplete, a drop in bash completion for Python + argparse.

I’ve just started looking at Docker. It’s a cool new technology that has the potential to make the management and deployment of distributed applications a great deal easier. I’d very much recommend checking it out. I’m especially interested in using it to deploy Mono applications because it promises to remove the hassle of deploying and maintaining the mono runtime on a multitude of Linux servers. I’ve been playing around creating new images and containers and debugging my Dockerfile, and I’ve wound up with lots of temporary containers and images. It’s really tedious repeatedly running ‘docker rm’ and ‘docker rmi’, so I’ve knocked up a couple of bash commands to bulk delete images and containers. Delete all containers: sudo docker ps -a -q | xargs -n 1 -I {} sudo docker rm {} Delete all un-tagged (or intermediate) images: sudo docker rmi $( sudo docker images | grep '' | tr -s ' ' | cut -d ' ' -f 3)

Every GitHub repository comes with its own wiki. This is a great place to put the documentation for your project. What isn’t clear from the wiki documentation is how to add images to your wiki. Here’s my step-by-step guide. I’m going to add a logo to the main page of my WikiDemo repository’s wiki: https://github.com/mikehadlow/WikiDemo/wiki/Main-Page First clone the wiki. You grab the clone URL from the button at the top of the wiki page. $ git clone git@github.com:mikehadlow/WikiDemo.wiki.git Cloning into 'WikiDemo.wiki'... Enter passphrase for key '/home/mike.hadlow/.ssh/id_rsa': remote: Counting objects: 6, done. remote: Compressing objects: 100% (3/3), done. remote: Total 6 (delta 0), reused 0 (delta 0) Receiving objects: 100% (6/6), done. Create a new directory called ‘images’ (it doesn’t matter what you call it, this is just a convention I use): $ mkdir images Then copy your picture(s) into the images directory (I’ve copied my logo_design.png file to my images directory). $ ls -l -rwxr-xr-x 1 mike.hadlow Domain Users 12971 Sep 5 2013 logo_design.png Commit your changes and push back to GitHub: $ git add -A $ git status # On branch master # Changes to be committed: # (use "git reset HEAD ..." to unstage) # # new file: images/logo_design.png # $ git commit -m "Added logo_design.png" [master 23a1b4a] Added logo_design.png 1 files changed, 0 insertions(+), 0 deletions(-) create mode 100755 images/logo_design.png $ git push Enter passphrase for key '/home/mike.hadlow/.ssh/id_rsa': Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (4/4), 9.05 KiB, done. Total 4 (delta 0), reused 0 (delta 0) To git@github.com:mikehadlow/WikiDemo.wiki.git 333a516..23a1b4a master -> master Now we can put a link to our image in ‘Main Page’: Save and there’s your image for all to see:

We all know we need to use connection pools where ever we connect to a database. All of the modern drivers using JDBC type 4 support it. In this post we will have look at an overview ofconnection pooling in spring applications and how to deal with same context in a non JEE enviorements (like tests). Most examples of connecting to database in spring is done using DriverManagerDataSource. If you don't read the documentation properly then you are going to miss a very important point. NOTE: This class is not an actual connection pool; it does not actually pool Connections. It just serves as simple replacement for a full-blown connection pool, implementing the same standard interface, but creating new Connections on every call. Useful for test or standalone environments outside of a J2EE container, either as a DataSource bean in a corresponding ApplicationContext or in conjunction with a simple JNDI environment. Pool-assuming Connection.close() calls will simply close the Connection, so any DataSource-aware persistence code should work. Yes, by default the spring applications does not use pooled connections. There are two ways to implement the connection pooling. Depending on who is managing the pool. If you are running in a JEE environment, then it is prefered use the container for it. In a non-JEE setup there are libraries which will help the application to manage the connection pools. Lets discuss them in bit detail below. 1. Server (Container) managed connection pool (Using JNDI) When the application connects to the database server, establishing the physical actual connection takes much more than the execution of the scripts. Connection pooling is a technique that was pioneered by database vendors to allow multiple clients to share a cached set of connection objects that provide access to a database resource. The JavaWorld article gives a good overview about this. In a J2EE container, it is recommended to use a JNDI DataSource provided by the container. Such a DataSource can be exposed as a DataSource bean in a Spring ApplicationContext via JndiObjectFactoryBean, for seamless switching to and from a local DataSource bean like this class. The below articles helped me in setting up the data source in JBoss AS. 1. DebaJava Post 2. JBoss Installation Guide 3. JBoss Wiki Next step is to use these connections created by the server from the application. As mentioned in the documentation you can use the JndiObjectFactoryBean for this. It is as simple as below If you want to write any tests using springs "SpringJUnit4ClassRunner" it can't load the context becuase the JNDI resource will not be available. For tests, you can then either set up a mock JNDI environment through Spring's SimpleNamingContextBuilder, or switch the bean definition to a local DataSource (which is simpler and thus recommended). As I was looking for a good solutions to this problem (I did not want a separate context for tests) this SO answer helped me. It sort of uses the various tips given in the Javadoc to good effect. The issue with the above solution is the repetition of code to create the JNDI connections. I have solved it using a customized runner SpringWithJNDIRunner. This class adds the JNDI capabilities to the SpringJUnit4ClassRunner. It reads the data source from "test-datasource.xml" file in the class path and binds it to the JNDI resource with name "java:/my-ds". After the execution of this code the JNDI resource is available for the spring container to consume. import javax.naming.NamingException; import org.junit.runners.model.InitializationError; import org.springframework.context.ApplicationContext; import org.springframework.context.support.ClassPathXmlApplicationContext; import org.springframework.mock.jndi.SimpleNamingContextBuilder; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; /** * This class adds the JNDI capabilities to the SpringJUnit4ClassRunner. * @author mkadicha * */ public class SpringWithJNDIRunner extends SpringJUnit4ClassRunner { public static boolean isJNDIactive; /** * JNDI is activated with this constructor. * * @param klass * @throws InitializationError * @throws NamingException * @throws IllegalStateException */ public SpringWithJNDIRunner(Class klass) throws InitializationError, IllegalStateException, NamingException { super(klass); synchronized (SpringWithJNDIRunner.class) { if (!isJNDIactive) { ApplicationContext applicationContext = new ClassPathXmlApplicationContext( "test-datasource.xml"); SimpleNamingContextBuilder builder = new SimpleNamingContextBuilder(); builder.bind("java:/my-ds", applicationContext.getBean("dataSource")); builder.activate(); isJNDIactive = true; } } } } To use this runner you just need to use the annotation @RunWith(SpringWithJNDIRunner.class) in your test. This class extends SpringJUnit4ClassRunner beacuse a there can only be one class in the @RunWith annotation. The JNDI is created only once is a test cycle. This class provides a clean solution to the problem. 2. Application managed connection pool If you need a "real" connection pool outside of a J2EE container, consider Apache's Jakarta Commons DBCP or C3P0. Commons DBCP's BasicDataSource and C3P0's ComboPooledDataSource are full connection pool beans, supporting the same basic properties as this class plus specific settings (such as minimal/maximal pool size etc). Below user guides can help you configure this. 1. Spring Docs 2. C3P0 Userguide 3. DBCP Userguide The below articles speaks about the general guidelines and best practices in configuring the connection pools. 1. SO question on Spring JDBC Connection pools 2. Connection pool max size in MS SQL Server 2008 3. How to decide the max number of connections 4. Monitoring the number of active connections in SQL Server 2008 Note:- All the text in italics are copied from the spring documentation of the DriverManagerDataSource.

this is another experiment with longer posts. previously, i used the time series example as the bed on which to test some ideas regarding feature design, to explain how we work and in general work out the rough patches along the way. i should probably note that these posts are purely fiction at this point. we have no plans to include a time series feature in ravendb at this time. i am trying to work out some thoughts in the open and get your feedback. at any rate, yesterday we had a request for cassandra style counters at the mailing list. and as long as i am doing feature design series, i thought that i could talk about how i would go about implementing this. again, consider this fiction, i have no plans of implementing this at this time. the essence of what we want is to be able to… count stuff. efficiently, in a distributed manner, with optional support for cross data center replication. very roughly, the idea is to have “sub counters”, unique for every node in the system. whenever you increment the value, we log this to our own sub counter, and then replicate it out. whenever you read it, we just sum all the data we have from all the sub counters. let us outline the various parts of the solution in the same order as the one i used for time series. storage a counter is just a named 64 bits signed integer. a counter name can be any string up to 128 printable characters. the external interface of the storage would look like this: 1: public struct counterincrement 2: { 3: public string name; 4: public long change; 5: } 6: 7: public struct counter 8: { 9: public string name; 10: public string source; 11: public long value; 12: } 13: 14: public interface icounterstorage 15: { 16: void localincrementbatch(counterincrement[] batch); 17: 18: counter[] read(string name); 19: 20: void replicatedupdates(counter[] updates); 21: } as you can see, this gives us very simple interface for the storage. we can either change the data locally (which modify our own storage) or we can get an update from a replica about its changes. there really isn’t much more to it, to be fair. the localincrementbatch() increment a local value, and read() will return all the values for a counter. there is a little bit of trickery involved in how exactly one would store the counter values. for now, i think we’ll store each counter as two step values. we’ll have a tree of multi tree values that will carry each value from each source. that means that a counter will take roughly 4kb or so. this is easy to work with and nicely fit the model voron uses internally. note that we’ll outline additional requirement for storage (searching for counter by prefix, iterating over counters, addresses of other servers, stats, etc) below. i’m not showing them here because they aren’t the major issue yet. over the wire skipping out on any optimizations that might be required, we will expose the following endpoints: get /counters/read?id=users/1/visits&users/1/posts <—will return json response with all the relevant values (already summed up). { “users/1/visits”: 43, “users/1/posts”: 3 } get /counters/read?id=users/1/visits&users/1/1/posts&raw=true <—will return json response with all the relevant values, per source. { “users/1/visits”: {“rvn1”: 21, “rvn2”: 22 } , “users/1/posts”: { “rvn1”: 2, “rvn3”: 1 } } post /counters/increment <– allows to increment counters. the request is a json array of the counter name and the change. for a real system, you’ll probably need a lot more stuff, metrics, stats, etc. but this is the high level design, so this would be enough. note that we are skipping the high performance stream based writes we outlined for time series. we’ll probably won’t need them, so that doesn’t matter, but they are an option if we need them. system behavior this is where it is really not interesting, there is very little behavior here, actually. we only have to read the data from the storage, sum it up, and send it to the user. hardly what i’ll call business logic. client api the client api will probably look something like this: 1: counters.increment("users/1/posts"); 2: counters.increment("users/1/visits", 4); 3: 4: using(var batch = counters.batch()) 5: { 6: batch.increment("users/1/posts"); 7: batch.increment("users/1/visits",5); 8: batch.submit(); 9: } note that we’re offering both batch and single api. we’ll likely also want to offer a fire & forget style, which will be able to offer even better performance (because they could do batching across more than a single thread), but that is out of scope for now. for simplicity sake, we are going to have the client just a container for all of endpoints that it knows about. the container would be responsible for… updating the client visible topology, selecting the best server to use at any given point, etc. user interface there isn’t much to it. just show a list of counter values in a list. allow to search by prefix, allow to dive into a particular counter and read its raw values, but that is about it. oh, and allow to delete a counter. deleting data honestly, i really hate deletes. they are very expensive to handle properly the moment you have more than a single node. in this case, there is an inherent race condition between a delete going out and another node getting an increment. and then there is the issue of what happens if you had a node down when you did the delete, etc. this just sucks. deletion are handled normally, (with the race condition caveat, obviously), and i’ll discuss how we replicate them in a bit. high availability / scale out by definition, we actually don’t want to have storage replication here. either log shipping or consensus based. we actually do want to have different values, because we are going to be modifying things independently on many servers. that means that we need to do replication at the database level. and that leads to some interesting questions. again, the hard part here is the deletes. actually, the really hard part is what we are going to do with the new server problem. the new server problem dictates how we are going to bring a new server into the cluster. if we could fix the size of the cluster, that would make things a lot easier. however, we are actually interested in being able to dynamically grow the cluster size. therefor, there are only two real ways to do it: add a new empty node to the cluster, and have it be filled from all the other servers. add a new node by backing up an existing node, and restoring as a new node. ravendb, for example, follows the first option. but it means that in needs to track a lot more information. the second option is actually a lot simpler, because we don’t need to care about keeping around old data. however, this means that the process of bringing up a new server would now be: update all nodes in the cluster with the new node address (node isn’t up yet, replication to it will fail and be queued). backup an existing node and restore at the new node. start the new node. the order of steps is quite important. and it would be easy to get it wrong. also, on large systems, backup & restore can take a long time. operationally speaking, i would much rather just be able to do something like, bring a new node into the cluster in “silent” mode. that is, it would get information from all the other nodes, and i can “flip the switch” and make it visible to clients at any point in time. that is how you do it with ravendb, and it is an incredibly powerful system, when used properly. that means that for all intents and purposes, we don’t do real deletes. what we’ll actually do is replace the counter value with delete marker. this turns deletes into a much simple “just another write”. it has the sad implication of not free disk space on deletes, but deletes tend to be rare, and it is usually fine to add a “purge” admin option that can be run on as needed basis. but that brings us to an interesting issue, how do we actually handle replication. the topology map to simplify things, we are going to go with one way replication from a node to another. that allows complex topologies like master-master, cluster-cluster, replication chain, etc. but in the end, this is all about a single node replication to another. the first question to ask is, are we going to replicate just our local changes, or are we going to have to replicate external changes as well? the problem with replicating external changes is that you may have the following topology: now, server a got a value and sent it to server b. server b then forwarded it to server c. however, at that point, we also have a the value from server a replicated directly to server c. which value is it supposed to pick? and what about a scenario where you have more complex topology? in general, because in this type of system, we can have any node accept writes, and we actually desire this to be the case , we don’t want this behavior. we want to only replicate local data, not all the data. of course, that leads to an annoying question, what happens if we have a 3 node cluster, and one node fails catastrophically. we can bring a new node in, and the other two nodes will be able to fill in their values via replication, but what about the node that is down? the data isn’t gone, it is still right there in the other two nodes, but we need a way to pull it out. therefor, i think that the best option would be to say that nodes only replicate their local state, except in the case of a new node. a new node will be told the address of an existing node in the cluster, at which point it will: register itself in all the nodes in the cluster (discoverable from the existing node). this assumes a standard two way replication link between all servers, if this isn’t the case, the operators would have the responsibility to setup the actual replication semantics on their own. new node now starts getting updates from all the nodes in the cluster. it keeps them in a log for now, not doing anything yet. ask that node for a complete update of all of its current state. when it has all the complete state of the existing node, it replays all of the remembered logs that it didn’t have a chance to apply yet. then it announces that it is in a valid state to start accepting client connections. note that this process is likely to be very sensitive to high data volumes. that is why you’ll usually want to select a backup node to read from, and that decision is an ops decision. you’ll also want to be able to report extensively on the current status of the node, since this can take a while, and ops will be watching this very closely. server name a node requires a unique name. we can use guids, but those aren’t readable, so we can use machine name + port, but those can change. ideally, we can require the user to set us up with a unique name. that is important for readability and for being able to alter see all the values we have in all the nodes. it is important that names are never repeated, so we’ll probably have a guid there anyway, just to be on the safe side. actual replication semantics since we have the new server problem down to an automated process, we can choose the drastically simpler model of just having an internal queue per each replication destination. whenever we make a change, we also make a note of that in the queue for that destination, then we start an async replication process to that server, sending all of our updates there. it is always safe to overwrite data using replication, because we are overwriting our own data, never anyone else. and… that is about it, actually. there are probably a lot of details that i am missing / would discover if we were to actually implement this. but i think that this is a pretty good idea about what this feature is about.

Gradle is a build tool on the JVM platform that’s been gaining prominence over the last few years. Gradle’s reach doesn’t stop with JVM languages though. The most recent releases, 1.10 and 1.11, have improved support for compiling native code (C / C++ / Objective C) and introduced support for the .NET platform. For Android projects, Google have made Gradle the de facto build tool. It’s getting a name for its flexibility, so when the opportunity came to build a single page Javascript webapp we decided to put the latest addition to the polyglot build tool arsenal through its paces. In this post I’m going to look at why we chose Gradle over a native JS build tool, provide a quick tour of the JS/CSS plugins, look at the embedded Jetty webserver plugin, discuss testing your code in CI, and a variety of other tips and gotchyas found in our experience during the project. Why not {{JavaScript_build_tool_flavour_of_the_month}? The momentum around JavaScript development is growing rapidly. So rapidly in fact that it feels like the book 1984 where one day your nation is fighting a war with one country, then all of a sudden wakes up, and it’s announced it has pledged an allegiance to a former enemy; your former ally becoming the new enemy. Twitter and the blogosphere recently became alive with talk about Gulp, a new JS build tool. This blog post ‘And just like that, Grunt and RequireJS are out – its all about Gulp and Browsify now’ and this Twitter post felt like the shifting of alliances in 1984. This is something you’ll have to factor into your JS build tool decision making. It also means however there is still room for contenders in this space. Having the same build tool for both client side and server side builds helps developers understand the project better. Our team had Groovy and Gradle exposure already and whilst we were happy to try a JS build tool, it would be nice to consider the same tools for both aspects of the project. A tool that works with your existing infrastructure is also important. The straw that broke the camel’s back was our client’s firewall. Firewalls are part and parcel of something we have to work around as consultants. Sometimes you win, sometimes you lose. Simply put, our client’s firewall made npm cry as the https proxy had some confusing message about SSL certificate algorithms. After trying to work around this for a while, I noticed we had a perfectly fine build tool that already worked fine with Java and our proxy. The question was asked: Are there any JavaScript plugins for Gradle? Reinventing the wheel I heard at a JavaScript BOF at JavaOne someone remark that ‘JavaScript had been rapidly climbing the tree of software development, reinventing tools that had already existed on established platforms, then falling off the tree taking each branch along with it on the way down.’ Although tongue-in-cheek, it did capture the frustrations of tools already built that were largely ignored, or unknown by the JavaScript community. From what I’ve seen so far, the other popular JVM build tool, Maven, has a large number of plugins to aid in the preparation of JavaScript files. Most of these use the Rhino JavaScript Engine that comes with Java in order to execute these utilities on the JVM platform where the build and testing is performed. The eco-system of running JavaScript utilities out of Rhino with ScriptEngineManager is well established. Additionally there are a lot of other tools written in Java to validate and compress JavaScript files – the Google Closure Compiler is one that is used by the Gradle JS and CSS plugin. It appears to have been relatively straight forward for plugin devs to repackage all these tools as Gradle plugins that kick off the same scripts from within. Where there isn’t a Java port or Jar, Gradle plugins or tasks can always launch JS tools from the command line. Kicking the tires So just what can Gradle with the available JS plugins do? Here’s a whirlwind tour of the available features. The Gradle JS and CSS plugins The Gradle JS plugin is a third party Gradle plugin written by Eric Wendelin. It can do file combination, minification, Gzip, JSHint and more. And although we didn’t need it for this project, you’ll be pleased to know it has require.js support as well. The Gradle CSS Plugin also written by Eric Wendelin, again supports combining and minification plus supports less compilation and csslint. To add these plugins to your build and associated tasks, all you have to do is add this to your build.gradle apply plugin: 'js' apply plugin: 'css' // define the dependencies gradle buildscript will use to build // the app (not the app itself) buildscript { repositories { mavenLocal() mavenCentral() } dependencies { classpath 'com.eriwen:gradle-css-plugin:1.8.0' classpath 'com.eriwen:gradle-js-plugin:1.9.0' } } You can then configure the tasks the plugin provides, combineJs, minifyJs, gzipJs as follows: combineJs { // pull together the source from string & file lists // eg. def core = ["$webAppDirName/init.js",...] source = core + application + devmode + bigquery + javascript.source.externalLibs.js.files + dfpFilters + chartdef + uxFilters // show the resolved files when gradle is run with -d source.each{ logger.debug ("$it") } dest = file("${buildDir}/all.js") } minifyJs { source = combineJs dest = file("${buildDir}/all.min.js") sourceMap = file("${buildDir}/all.sourcemap.json") closure { warningLevel = 'QUIET' compilerOptions.defineReplacements = ['MY_DBUG_FLAG':false] } } gzipJs { source = minifyJs.dest dest = file("${buildDir}/all.min.js.gz") } combineJs has a source property which takes a collection. We split our app into groups of JavaScript files and combine them with a list of files from a Gradle javascript sourceset called externLibs that references bootstrap and jquery in a separate dir in our project. The combine is done in the order you specify when specified in an ArrayList. For Gradle aficionados, this is why I’m not using sourcesets for our files – see here. The other reason is that all our client JS was in the same folder so lists were the most concise way to declare these file groups. You can see we can also debug the list to make sure we’ve pulled in the files we think we’ve asked for by using standard groovy code to iterate through the source collection, making use of the logger that is injected into the build file by Gradle. One of the neatest things about Gradle is that each task has a set of Input files or dirs, plus output files. Gradle can do some very intelligent introspection into whether or not a file has been modified and what dependent tasks need to be rerun. Spending a little effort to specify these properly means that your build won’t have to do unnecessary stuff repeatedly. The minifyJs task takes advantage of this. We can then declare the input for its source property as the dest outputfrom the previous task. Here we set source to combineJs and gradle figures out we mean the output of that task (we could have also said combineJs.dest). The cool thing about this is that Gradle will be lazy. If it notices that the output on the combineJs hasn’t changed, thus the input on minifyJs hasn’t changed, it will skip running that task. BTW, you can always force Gradle to run each task with the --rerun-tasks command line parameter. To help things along we can define the dependency chain on these tasks tasks.minifyJs.dependsOn tasks.combineJs tasks.gzipJs.dependsOn tasks.minifyJs and run gzipJs each time being confident it will only run the earlier tasks if their task inputs have changed. I’m not sure why the dependency chain isn’t configured by default in the JS plugin – perhaps it could be a reasonable enhancement for a future release? I mentioned that minifyJs uses the Google Closure Compiler under the hood and so I wanted to show our use of configuring it with closure config block, in particular the compilerOptions.defineReplacements option which wasn’t documented in the JS plugins docs. In our JS code we define a constant to indicate some developer mode functionality like so: // config.js /** * Flag to indicate console and extra logging throughout the app * @define {boolean} allow the value of this bool to be * overwritten at closure compiler / minification time * @const * @type {boolean} */ var MY_DBUG_FLAG = true; Then elsewhere throughout our code base, we have code like this which modifies the DOM and adds some helpful info about the applications state to the UI. if (MY_DBUG_FLAG) { $('.dashboard').append( 'Here's some dev info you wouldn't normally see' ); } defineReplacements allows the Google Closure compiler to redefine the MY_DBUG_FLAG constant at build time to false, and then through its optimisation, remove the if (MY_DBUG_FLAG) blocks from the resultant all.min.js entirely (since we’ve forced MY_DBUG_FLAG to false). There are plenty of other configuration options for the compiler but it’s not immediately apparent where these are set. Take a look at CompilerOptions.java in the Google Closure compiler source to see the properties it takes. CSS The CSS plugin adds combineCss, minifyCss, and gzipCss tasks. Here’s the config for the plugin & how we configured it. String[] cssDirs = ["src/main/webapp/$bootstrapPath/css", "src/main/webapp/libs/bootstrap-typeahead/0.9.3/css", "src/main/webapp/libs/bootstrap-datepicker/1.2.0/css", "src/main/webapp/$chosenPath", "src/main/webapp/$ajaxLoaderPath", "src/main/webapp/styles"] css.source { dev { css { srcDirs cssDirs include "*.css" include "*.min.css" exclude ".#*.css" // VCS creates these annoying files } } } Here’s another example of using the Gradle sourcesets, we have a list of our dependent CSS libs + our own CSS, using srcDirs to pull together these directories and let the source set definition define what files to include and exclude. Then pass it off to a combineCss source property. combineCss { source = css.source.dev.css.files dest = "${buildDir}/all.css" } combineCss << { source.each { logger.info "Combining $it"} } A small change made in this example is to add a doLast (aliased <<) closure on the combineCss task to do the non-essential logging of the files found after the task has been configured and actioned. I think it helps keep the build file a bit cleaner. The other thing you probably have noticed is the interleaving of properties such as ${buildDir}, $bootstrapPath, $webAppDirName. This is just Groovy syntax to refer to variables. Gradle does some binding magic to bind those variables to properties, either defined in the plugins, or within whatever you put in your gradle.properties file that lives in the same directory as build.gradle. # Path to Javascript files jsSrcDir=src/main/webapp/js # Path to json files used to describe UI controls jsonPath=src/main/webapp/json # Local Deployment wwwPath=/var/www openBrowserPostLocalDeploy=true # Library Paths jqueryPath=libs/jquery/2.0.3 ajaxLoaderPath=libs/ajaxloader/1.0 chosenPath=libs/chosen/1.0.0 bootstrapPath=libs/bootstrap/3.0.2 Just for completeness here are the minify and gzipCss tasks but now you are down with Gradle and Js, the below should be self-explanatory. minifyCss { source = combineCss dest = "${buildDir}/all.min.css" yuicompressor { // Optional lineBreakPos = -1 } } gzipCss { source = minifyCss dest = "${buildDir}/all.min.css.gz" } Deploy against the machine So we got to generating some minified, gzipped files. Great. Now we need to see our changes with an apache or nginx install. It’s a relatively short task to install through a couple of sudo apt-get commands on Ubuntu, or re-enabling web-sharing on OS X (google it) but admittedly a few hoops need to be jumped through. You also need to set some file permissions so you can let your build tool write to Apache’s /var/www dir. Once the formalities are out of the way though we can automate this deploy. Like any decent build tool, Gradle itself comes with tasks to copy, archive and manipulate files. Gradle copy tasks take a copy configuration closure which let you define properties on which file patterns you wish to include, exclude and which dirs (much like the source sets shown above). The copy tasks are also smart enough to know what’s already been copied and only copied changed files which can save a bit of time when it comes to deploying. Here are tasks for copying the all.min.js and css (plus gzipped versions and sourcemaps) task deployCombinedJsLocally(dependsOn: 'gzipJs', type:Copy) { from files(minifyJs.dest, minifyJs.sourceMap, gzipJs.dest) into wwwPath } task deployCombinedCssLocally(dependsOn: 'gzipCss', type: Copy) { doFirst { logger.quiet("Copying ${files(minifyCss.dest, gzipCss.dest)} ... to ${wwwPath}/css") } from files(minifyCss.dest, gzipCss.dest) into file("$wwwPath/css") } For the css task, I wanted to show an example of logging the files that would be copied but this time using Gradle’s default quiet log level so it would be printed out without needing to specify this flag. Also this logging happens within a doFirst closure block which will happen during the evaluation stage of the gradle build before the task executes. It would also be nice to also have a task to copy my un-minified files to the server: task deployJsLocally(type: Copy) { from files(javascript.source.pcr.js.files) into "$wwwPath/js" } task deployCssLocally(type: Copy) { from css.source.dev.css.files into "$wwwPath/css" } Finally we’d have a deploy task that referenced these and some copy tasks to pull in images, html and other non-css / js artefacts. task deployLocalMinified( description: 'Copy webapp artifacts to local apache', dependsOn: ['copyBootstrapFonts', 'copyJson', 'copyImages', 'addDevBootLibs', 'copyAllHtml', 'deployCombinedJsLocally', 'deployCombinedCssLocally'] ) {} An embedded web server Using a local Apache instance was all good and well, but as I mentioned it did require a developer to sort out their own installation. Gradle comes with a Jetty plugin out of the box and so we utilised this to provide a webserver which served our single page webapp. There was no need to provide a WEB-INF/web.xml, the default Jetty configuration is already configured to look in your webAppDir(src/main/webapp) and serve resources from that. Because it’s running in Jetty and out of the source folder as well, we could change the code out of our IDEs and have it update on the next reload – much better than having a grunt task to keep your source folders and apache in sync. apply plugin: 'jetty' // PROPERTIES (Common to jettyRun/RunWar/Stop tasks) // stop a running jetty stopPort = jettyStopPort as Integer stopKey = jettyStopKey jettyRun.doFirst { logger.quiet('Starting Jetty...') } jettyRun { scanIntervalSeconds = 5 contextPath = '/my-web-app' } IntelliJ has a neat feature that comes from its Webstorm IDE. If you install the LiveEdit plugin and the corresponding Chrome or Firefox extension, your code will update in the IDE whilst you are editing it. This saves a lot of time. You also get access to the debugging tools in IntelliJ as opposed to Chrome Dev Tools. The only tricky thing about setting this up was telling the Debug task in IntelliJ that http://localhost:8080/my-web-app/ was equal to /src/main/webapp so that breakpoints would work. Now any developer can simply check out our project from source control and start running the app by executing gradle jettyRun.It’s an important win – hence I’ve highlighted it. Multiple build files At this stage our build only looks after 3 concerns (CSS build, JS build and deployment) but our build.gradle file is starting to become a bit long. Gradle supports multiple build files so we can organise each concern into a separate build.gradle file to avoid clutter. Each buildscript has its own scope, applying its own plugins. That means that variables, methods and tasks that you or your plugins add are only in the scope of each child file. There isn’t a chance of other plugins polluting the namespace and causing issues you have to spend a lot of time to troubleshoot. They still have access to the project variable added by gradle to get access to configurations in other files if need be. We can define dependencies between our JS and CSS tasks. Our main build.gradle is now very short, with some apply commands like so: apply from: 'gradle/build-css.gradle' apply from: 'gradle/build-js.gradle' apply from: 'gradle/build-webserver.gradle' The buildscript‘s repository and dependency sections that define where to get the 3rd party CSS and JS plugins from can be moved into the relevant build-css and build-js.gradle files. However it’s not quite perfect. I did encounter a classpath issue with one of the JS plugins’ task and simply redefined the dependency in both the main projects build.gradle and the JS file. Overall though, keeping the concerns separate is much nicer for maintainability. Tests We used Jasmine with the JSTestDriver Jasmine plugin for this project. There was a Gradle Jasmine plugin but it’s no longer maintained and doesn’t work with the later versions of Gradle. That said, IntelliJ came to the rescue here and has a built-in Jasmine runner that automatically listens for source and test changes and reruns tests. For both our CI server and for kicking off tests from the command line, the gradle-js-plugin author wrote a task to kick off JSTD with Firefox in a headless browser. We adapted this for our build as follows: task jstd(type: Exec, description: 'runs JS tests through JsTestDriver') { commandLine = ['java', "-jar", "${configurations.testRuntime.files.find()}", '--config', "${projectDir}/jsTestDriverWithJasmine.jstd", '--port','9876', '--tests', 'all', '--testOutput', buildDir, '--browser', 'firefox'] // Then add --verbose flag to commandLine based on // the gradle log level. See later. } There are other JS testing frameworks supported. If you use buster, the Gradle plugin for buster.js keeps a warm VM running in the background checking for changes and re-running your tests. There is a grunt plugin, so you could just run whatever JavaScript tools you want. But as you learnt above, Grunt isn’t the new hotness anymore, so it may have limited appeal. ;-) For us, Grunt still relied on NPM so we left it alone. Quick start and CI friendly The other neat thing that Gradle has is a wrapper script that pulls down the version of Gradle you used to write your Gradle build script with. This is not only good for developers but for any build agents in your CI environment. You don’t need to have Gradle installed, you just clone the project and do gradlew jettyRun and it will go find Gradle, download and install it if it’s not there and then run gradle jettyRun as if you had it right from the beginning. It’s Groovy and it’s groovy If you are already a Javascript dev then much of the Groovy GDK will be easy to relate to. Groovy is dynamically typed as well, but the optional typing will be a welcome facility if you want to take it up. The tooling for Gradle and Groovy out of IntelliJ Community Edition (free) is quite neat. Netbeans (free) has recently made big strides in HTML5 and Javascript support and has strong Groovy support too. This may be a good opportunity to pick up another tool as well that will make your Javascript development easier too. Write your own tasks to do things in the build It’s easy to write your own Gradle tasks. We wrote our own task to modify the base HTML page of our single page webapp to load in different entry point scripts for development and production. We could then wire this task into the build, declare it as a dependency for producing the HTML for our war file or production like jetty deployment that referenced our combined minified gzipped CSS and JS files, or as a dependency to add in the head.js script that called all the individual dev scripts and css. enum DevMode { DEV, PROD, ORIGINAL }; task addProdBootLibs(type: WebAppBootLibTask, description: 'Modify the html of a single page app to include the dev or prod scripts and css') { html = file(webAppDirName + '/index.html') devMode = DevMode.PROD } task addDevBootLibs(type: WebAppBootLibTask, description: 'Modify the html of a single page app to include the dev or prod scripts and css') { html = file(webAppDirName + '/index.html') devMode = DevMode.DEV } task revertBootLibs(type: WebAppBootLibTask, description: 'Revert html to use a placeholder message to remind devs to run the correct script') { html = file(webAppDirName + '/index.html') devMode = DevMode.ORIGINAL } class WebAppBootLibTask extends DefaultTask { final def tag = "" final def endTag = "" @InputFile File html @Input DevMode devMode @OutputFile @Optional File outputHtml def devModeToInclude = [(DevMode.ORIGINAL): ORIGINAL_INCLUDE, (DevMode.DEV): DEV_INCLUDE, (DevMode.PROD): PROD_INCLUDE] final static def ORIGINAL_INCLUDE = """ Libraries Required Run the addDev/ProdBootLibs gradle task to add the correct libraries for the target environment to the project. """ final static def DEV_INCLUDE = """ """ final static def PROD_INCLUDE = """ """ @TaskAction def void applyScriptAndStylesheetTags() { if (html.exists()) { def htmlReplacement = html.getText().replaceAll( "$tag\\p{all}*$endTag", "$tag${devModeToInclude[devMode]}$endTag") logger.info(htmlReplacement) if (outputHtml == null) outputHtml = html outputHtml.setText(htmlReplacement, 'UTF-8') } } } Open your webapp after deploy Because it was Java, we could use the JDK’s java.awt.Desktop to open links to the webapp after a deploy was kicked off locally. import static java.awt.Desktop.* deployLocalMinified << { // open the webapp in your browser desktop.browse "http://localhost/report.html".toURI() } SCP, SSH Because Gradle comes installed with its own ant, it can run any Ant task. We used the ant SSH plugin to SCP our packed distributions to our build box and another task to execute an SSH script remotely to deploy. The deep dark Gradle cookie jar It can get easy to spoil yourself with all of Gradle’s features. The simple stuff is simple but it did take me a few reads of the manual to get my head around it. You can pick out the first 10 chapters and get the basics which should be enough to keep you out of trouble for a while. However venture a little bit out of the norm and then you have to start really thinking about reading a few more chapters. This isn’t a bad thing. The tool in itself is quite broad and the Gradleware team have done a good job of breaking the chapters up to focus on just functionality that you need. But you do get to a point where you start realising that you need to understand more of the Gradle build lifecycle, how external properties work, the different lifecycles of tasks. And I won’t sugarcoat it, since JavaScript is not an official plugin from Gradleware (yet) and not maintained full-time it still has a few quirks both in documentation and the occasional bug. The community around it is good, but you just have to be prepared to roll your sleeves up. The Open Source culture is alive and well both in the Gradle and Groovy communities. When getting stuck it’s pretty easy to clone a plugins git repo to understand how a plugin works. The issues page for the CSS/JS plugins do list a few workarounds too so it’s worth looking there. However, I appreciate that this is not for everyone, especially if you are a Javascript dev with no Java/Groovy experience. Learning this used up more time than I initially planned for Gradle but I do hope JavaScript devs can benefit from this blog to take it up. Also, the experience has been great to learn more about Gradle anyway and thus helped make me aware of things we can apply in our Java builds too. Other lessons learnt So I listed a whole bunch of positives earlier, here were some of the things that irked myself and my colleagues as we picked up the tool. Tasks. When to use << and the task keyword prefix. Some plugins add default tasks. In order to configure them you just go lesscss { source = css.dev.css.files // css sourceset dest = ”$buildDir/out.css” } But the properties are executed every time the script is run for any task. So say I wanted to log what my sourceset had evaluated to, my instinct would be to stick a logging message lesscss { source = css.dev.css.files// css sourceset dest=”$buildDir/out.css” logger.quiet(“Compiling these files $source into $dest”) } If you go run another task, you’ll see that this closure gets evaluated regardless of whether you asked it to or not. gradle –q combineCss Compiling these files [‘src/main/javascript/styles/foo.css’,’src/main/javascript/styles/bar.css’] into /home/username/projects/gradle-js-blog/build … :combineCss task executed successfully. ‘Compiling these files…’? What, thats from lesscss???? Why am I seeing another build task output? What you need is to define the task with doFirst or doLast (shorthand <<) in order to tell Gradle that block isn’t to run during build evaluation. It turns out to be is that lesscss isn’t a task at all. This is called a convention mapping and we are configuring properties in the build evaluation step of Gradle rather than steps to run as part of running the task. It’s a bit confusing since this is also a Groovy bit of code. You can do manipulation on those properties at build evaluation time which is neat but the difference isn’t immediately apparent. Getting the log level flag passed into Gradle Gradle comes with three logging flags. --quiet, --info and --debug. By default Gradle will only show if a task passed, failed or was skipped. Much unlike Maven. Thus, you’ll only use info or debug if you want to see the output of the steps. It would be nice for JS tools to mimic the log level that was passed into the Gradle build. The project object that’s available in every buildscript exposes its logger however calling logger.level returned null. A workaround described on the Gradle forums indicated that using the logger.isXXXXXXEnabled methods worked, so we wrote a task to query this and apply the correct loglevel to the task. task someTaskIdLikeToReuseTheLogLevelPassedToGradle(type: Exec) { // setup commandLine array with path to tool and other flags String runnerMode = getRunnerMode() if (runnerMode) commandLine += ['--verbose', '--runnerMode', runnerMode ] } private String getRunnerMode() { def runnerMode = null if (logger.isDebugEnabled()) { runnerMode = 'DEBUG' } else if (logger.isInfoEnabled()) { runnerMode = 'INFO' } else if (logger.isQuietEnabled()) { runnerMode = 'QUIET' } runnerMode } More to come from the official gradle guys There is more JS support coming. There is a large thread on the public gradle mailing list discussing putting more JS in. Already there are undocumented plugins for Rhino and Coffeescript (you can find these in the Javadoc for Gradle but missing from the users guide and DSL which indicate that they are in progress). File watching is also on the road map so hopefully more plugins can have the watch/build/reload cycle that the buster.js and grunt plugins afford. Haven’t yet covered Artefact resolution from a public repository like Github or Maven Central for JS libs is still a popular topic on the forums. The gradle-js-plugin covers the former with experimental support, and the Gradleware team are working on the later. Because of the problems I had getting npm to work over our proxy I stayed away from this anyway. However require.js support in the gradle-js-plugin out of the box which will get you some of the way there. Build Successful Projects with JavaScript only web front ends are ubiquitous nowadays. Being able to use the same tools to build your front and back end infrastructure allows your team to have a better sense of the project as a whole. There are a number of things that make Gradle attractive – an out of the box web platform to develop with, smart tasks that are more aware of their dependencies, the ability to integrate with the JDK and existing Java tools and the ability to fall back to calling JS command line tools – all provide a huge amount of flexibility. As a Java and Groovy dev, using a familiar tool in Gradle to build for a completely different platform was both a learning and rewarding experience. It did come with its frustrations particularly as you step from pre-configured conventional Gradle build tasks to more specific ones but I think we became better build masters for it. Although I haven’t touched on it in this blog, being able to integrate with a Java web container also availed us to some quick wins when it came to implementing security for our single page webapp. We could rely on the Java Servlet 3 spec to make light work of the authentication and authorisation aspects of our app. In addition, we can package our JavaScript app up as a war and give it to our clients deployment team who just think they are deploying a Java app. Since that’s the infrastructure they maintain and are used to then that works out well for everyone. There is no stopping the JavaScript build tool train, at least for a while. For JavaScript devs looking at their next build tool replacement, Gradle is a worthy choice for any platform. Sure you have to learn Groovy, but it’s not a bad language and JS devs shouldn’t have much trouble adjusting. You may find that if you try Gradle, Groovy and a Java based IDEs and then decide it’s not for you, at least you’ve got some exposure to new tools you didn’t expect to have in your tool box as well as an awareness of how other JS build tools could behave. And that, in my mind at least, is a nice prize for any software practitioner to come away with. This blog was written with Gradle 1.10

a short "how-to" based on an issue one of my work mates recently faced when trying to automate the creation of an msi package on jenkins. normally, visual studio solutions can be build on jenkins by using the appropriate msbuild plugin . apparently though, for visual studio setup projects, msbuild cannot be used and one has to switch to using visual studio itself to execute the build. so the first approach was to use devenv.exe as follows devenv.exe visualstudiosolution.sln /build "release" while this works, the problem is that it is an "async call", meaning that the compilation goes on in the background while the console from which the build is executed, immediately returns. obviously this isn't suited for being used on jenkins. searching around for a while, it turned out that you have to use devenv.com instead of devenv.exe : "c:\program files (x86)\microsoft visual studio 10.0\common7\ide\devenv.com"visualstudiosolution.sln /build "release" once you got that, integrating everything into jenkins is quite straightforward: (obviously you may also simply set an environment variable pointing to devenv.com on your build server rather than indicating the entire path)

Java EE in general and Context and Dependency Injection has been part of the Vaadin ecosystem since ages. Recently, Spring Vaadin is a joint effort of the Vaadin and the Spring teams to bring the Spring framework into the Vaadin ecosystem, lead by Petter Holmström for Vaadin and Josh Long for Pivotal. Integration is based on the Spring Boot project - and its sub-modules, that aims to ease creating new Spring web projects. This article assumes the reader is familiar enough with Spring Boot. If not the case, please take some time to get to understand basic notions about the library. Note that at the time of this writing, there's no release for Spring Vaadin. You'll need to clone the project and build it yourself. The first step is to create the UI. In order to display usage of Spring's Dependency Injection, it should use a service dependency. Let's injection the UI through Constructor Injection to favor immutability. The only addition to a standard UI is to annotate it with org.vaadin.spring.@VaadinUI. @VaadinUI public class VaadinSpringExampleUi extends UI { private HelloService helloService; public VaadinSpringExampleUi(HelloService helloService) { this.helloService = helloService; } @Override protected void init(VaadinRequest vaadinRequest) { String hello = helloService.sayHello(); setContent(new Label(hello)); } } The second step is standard Spring Java configuration. Let's create two configuration classes, one for the main context and the other for the web one. Two thing of note: The method instantiating the previous UI has to be annotated with org.vaadin.spring.@UIScope in addition to standard Spring org.springframework.context.annotation.@Bean to bind the bean lifecycle to the new scope provided by the Spring Vaadin library. At the time of this writing, a RequestContextListener bean must be provided. In order to be compliant with future versions of the library, it's a good practice to annotate the instantiating method with @ConditionalOnMissingBean(RequestContextListener.class). @Configuration public class MainConfig { @Bean public HelloService helloService() { return new HelloService(); } } @Configuration public class WebConfig extends MainConfig { @Bean @ConditionalOnMissingBean(RequestContextListener.class) public RequestContextListener requestContextListener() { return new RequestContextListener(); } @Bean @UIScope public VaadinSpringExampleUi exampleUi() { return new VaadinSpringExampleUi(helloService()); } } The final step is to create a dedicated WebApplicationInitializer. Spring Boot already offers a concrete implementation, we just need to reference our previous configuration classes as well as those provided by Spring Vaadin, namely VaadinAutoConfiguration and VaadinConfiguration. public class ApplicationInitializer extends SpringBootServletInitializer { @Override protected SpringApplicationBuilder configure(SpringApplicationBuilder application) { return application.showBanner(false) .sources(MainConfig.class) .sources(VaadinAutoConfiguration.class, VaadinConfiguration.class) .sources(WebConfig.class); } } At this point, we demonstrated a working Spring Vaadin sample application. Code for this article can be browsed and forked on Github.

Here is a short snippet of Ansible playbook that installs R and any required packages to any nodes of the cluster: - name: Making sure R is installed apt: pkg=r-base state=installed - name: adding a few R packages command: /usr/bin/Rscript --slave --no-save --no-restore-history -e "if (! ('{{item}' %in% installed.packages()[,'Package'])) install.packages(pkgs={{item}, repos=c('http://www.freestatistics.org/cran/'))" with_items: - rjson - rPython - plyr - psych - reshape2 You should replace the repos with one chosen from the list of Cran mirrors. Note that the command above installs each package only if it is not already present, but messes up the “changed” status of Ansible’s PLAY RECAP by incorrectly reporting a change per R package at every run. Find more big data technical posts on my blog.

A while back, I wrote an article showing how to Live Migrate Your VMs in One Line of Powershell between non-clustered Windows Server 2012 Hyper-V hosts using Shared Nothing Live Migration. Since then, I’ve been asked a few times for how this type of parallel Live Migration would be performed for highly available virtual machines between Hyper-V hosts within a cluster. In this article, we’ll walk through the steps of doing exactly that … via Windows PowerShell on Windows Server 2012 or 2012 R2 or our FREE Hyper-V Server 2012 R2 bare-metal, enterprise-grade hypervisor in a clustered configuration. Wait! Do I need PowerShell to Live Migrate multiple VMs within a Cluster? Well, actually … No. You could certainly use the Failover Cluster Manager GUI tool to select multiple highly available virtual machines, right-click and select Move | Live Migration … Failover Cluster Manager – Performing Multi-VM Live Migration But, you may wish to script this process for other reasons … perhaps to efficiently drain all VM’s from a host as part of a maintenance script that will be performing other tasks. Can I use the same PowerShell cmdlets for Live Migrating within a Cluster? Well, actually … No again. When VMs are made highly available resources within a cluster, they’re managed as cluster group resources instead of being standalone VM resources. As a result, we have a different set of Cluster-aware PowerShell cmdlets that we use when managing these cluster groups. To perform a scripted multi-VM Live Migration, we’ll be leveraging three of these cmdlets: Get-ClusterNode, Get-ClusterGroup and Move-ClusterVirtualMachineRole Now, let’s see that one line of PowerShell! Before getting to the point of actually performing the multi-VM Live Migration in a single PowerShell command line, we first need to setup a few variables to handle the "what" and "where" of moving these VMs. First, let’s specify the name of the cluster with which we’ll be working. We’ll store it in a $clusterName variable. $clusterName = read-host -Prompt "Cluster name" Next, we’ll need to select the cluster node to which we’ll be Live Migrating the VMs. Lets use the Get-ClusterNode and Out-GridView cmdlets together to prompt for the cluster node and store the value in a $targetClusterNode variable. $targetClusterNode = Get-ClusterNode -Cluster $clusterName | Out-GridView -Title "Select Target Cluster Node" ` -OutputMode Single And then, we’ll need to create a list of all the VMs currently running in the cluster. We can use the Get-ClusterGroup cmdlet to retrieve this list. Below, we have an example where we are combining this cmdlet with a Where-Object cmdlet to return only the virtual machine cluster groups that are running on any node except the selected target cluster node. After all, it really doesn’t make any sense to Live Migrate a VM to the same node on which it’s currently running! $haVMs = Get-ClusterGroup -Cluster $clusterName | Where-Object {($_.GroupType -eq "VirtualMachine") ` -and ($_.OwnerNode -ne $targetClusterNode.Name)} We’ve stored the resulting list of VMs in a $haVMs variable. Ready to Live Migrate! OK … Now we have all of our variables defined for the cluster, the target cluster node and the list of VMs from which to choose. Here’s our single line of PowerShell to do the magic … $haVMs | Out-GridView -Title "Select VMs to Move" –PassThru | Move-ClusterVirtualMachineRole -MigrationType Live ` -Node $targetClusterNode.Name -Wait 0 Proceed with care: Keep in mind that your target cluster node will need to have sufficient available resources to run the VM's that you select for Live Migration. Of course, it's best to initially test tasks like this in your lab environment first. Here’s what is happening in this single PowerShell command line: We’re passing the list of VMs stored in the $haVMs variable to the Out-GridView cmdlet. Out-GridView prompts for which VMs to Live Migrate and then passes the selected VMs down the PowerShell object pipeline to the Move-ClusterVirtualMachineRole cmdlet. This cmdlet initiates the Live Migration for each selected VM, and because it’s using a –Wait 0 parameter, it initiates each Live Migration one-after-another without waiting for the prior task to finish. As a result, all of the selected VMs will Live Migrate in parallel, up to the maximum number of concurrent Live Migrations that you’ve configured on these cluster nodes. The VMs selected beyond this maximum will simply queue up and wait their turn. Unlike some competing hypervisors, Hyper-V doesn't impose an artificial hard-coded limit on how many VMs for you can Live Migrate concurrently. Instead, it's up to you to set the maximum to a sensible value based on your hardware and network capacity. Do you have your own PowerShell automation ideas for Hyper-V? Feel free to share your ideas in the Comments section below. See you in the Clouds! - Keith

What is It Query by example is an alternative querying technique supported by the main JPA vendors but not by the JPA specification itself. QBE returns a result set depending on the properties that were set on an instance of the queried class. So if I create an Address entity and fill in the city field then the query will select all the Address entities having the same city field as the given Address entity. The typical use case of QBE is evaluating a search form where the user can fill in any search fields and gets the results based on the given search fields. In this case QBE can reduce code size significantly. When to Use · Using many fields of an entity in a query · User selects which fields of an Entity to use in a query · We are refactoring the entities frequently and don’t want to worry about breaking the queries that rely on them Limitations · QBE is not available in JPA 1.0 or 2.0 · Version properties, identifiers and associations are ignored · The query object should be annotated with @Entity Test Data I used the following entities to test the QBE feature of Hibernate: · Address (long id, String city, String street, String countryISO2Code, AddressType addressType) · AddressType (Integer type, String description) Imports The examples will refer to the following classes: import org.hibernate.Criteria; import org.hibernate.Session; import org.hibernate.criterion.Example; import org.hibernate.criterion.Restrictions; import org.junit.Test; import java.util.List; Utility Methods I also made two utility methods to present a list of the two entity types: private void listAddresses(List addresses) { for (Address address : addresses) { System.out.println(address.getId() + ", " + address.getCountryISO2Code() + ", " + address.getCity() + ", " + address.getStreet() + ", " + address.getAddressType().getType() + ", " + address.getAddressType().getDescription()); } } private void listAddressTypes(List addressTypes) { for (AddressType addressType : addressTypes) { System.out.println(addressType.getType() + ", " + addressType.getDescription()); } } Example 1: Equals This example code returns the Address entities matching the given CountryISO2Code and City. Method: @Test public void testEquals() throws Exception { Session session = (Session) entityManager.getDelegate(); Address address = new Address(); address.setCountryISO2Code("US"); address.setCity("CHICAGO"); Example addressExample = Example.create(address); Criteria criteria = session.createCriteria(Address.class).add(addressExample); listAddresses(criteria.list()); } Result: 75, US, CHICAGO, Los Angeles Way2, 6, Customer 170, US, CHICAGO, Jackson Blvd 33a, 4, Delivery 63, US, CHICAGO, Main Avenue 1, 5, Bill to 37, US, CHICAGO, Jackson Blvd 33a, 4, Delivery 36, US, CHICAGO, Jackson Blvd 33a, 4, Delivery Example 2: Id Limitation This example presents that id fields in the query object are ignored. Method: @Test public void testIdLimitation() throws Exception { Session session = (Session) entityManager.getDelegate(); Address address = new Address(); address.setCountryISO2Code("US"); address.setCity("CHICAGO"); address.setId(100); // setting id is ignored Example addressExample = Example.create(address); Criteria criteria = session.createCriteria(Address.class).add(addressExample); listAddresses(criteria.list()); } Result: 75, US, CHICAGO, Los Angeles Way2, 6, Customer 170, US, CHICAGO, Jackson Blvd 33a, 4, Delivery 63, US, CHICAGO, Main Avenue 1, 5, Bill to 37, US, CHICAGO, Jackson Blvd 33a, 4, Delivery 36, US, CHICAGO, Jackson Blvd 33a, 4, Delivery Example 3: Association Limitation Associations of the query object are ignored, too. Method: @Test public void testAssociationLimitation() throws Exception { Session session = (Session) entityManager.getDelegate(); Address address = new Address(); address.setCountryISO2Code("US"); address.setCity("CHICAGO"); AddressType addressType = new AddressType(); addressType.setType(5); address.setAddressType(addressType); // setting an association is ignored Example addressExample = Example.create(address); Criteria criteria = session.createCriteria(Address.class).add(addressExample); listAddresses(criteria.list()); } Result: 75, US, CHICAGO, Los Angeles Way2, 6, Customer 170, US, CHICAGO, Jackson Blvd 33a, 4, Delivery 63, US, CHICAGO, Main Avenue 1, 5, Bill to 37, US, CHICAGO, Jackson Blvd 33a, 4, Delivery 36, US, CHICAGO, Jackson Blvd 33a, 4, Delivery Example 4: Like QBE supports like in the query object if we enable it with Example.enableLike(). Method: @Test public void testLike() throws Exception { Session session = (Session) entityManager.getDelegate(); Address address = new Address(); address.setCountryISO2Code("US"); address.setCity("AT%"); Example addressExample = Example.create(address).enableLike(); Criteria criteria = session.createCriteria(Address.class).add(addressExample); listAddresses(criteria.list()); } Result: 83, US, ATLANTA, null, 6, Customer 184, US, ATLANTA, null, 1, Shipper 25, US, ATLANTA, null, 1, Shipper Example 5: ExcludeProperty We can exclude a property with Example.excludeProperty(String propertyName). Method: @Test public void testExcludeProperty() throws Exception { Session session = (Session) entityManager.getDelegate(); Address address = new Address(); address.setCountryISO2Code("US"); address.setCity("AT%"); Example addressExample = Example.create(address).enableLike() .excludeProperty("countryISO2Code"); // countryISO2Code is a property of Address Criteria criteria = session.createCriteria(Address.class).add(addressExample); listAddresses(criteria.list()); } Result: 154, GR, ATHENS, BETA ALPHA Street 5, 2, Consignee 83, US, ATLANTA, null, 6, Customer 25, US, ATLANTA, null, 1, Shipper 184, US, ATLANTA, null, 1, Shipper Example 6: IgnoreCase Case-insensitive search is supported by Example.ignoreCase(). Method: @Test public void testIgnoreCase() throws Exception { Session session = (Session) entityManager.getDelegate(); AddressType addressType = new AddressType(); addressType.setDescription("customer"); Example addressTypeExample = Example.create(addressType).ignoreCase(); Criteria criteria = session.createCriteria(AddressType.class) .add(addressTypeExample); listAddressTypes(criteria.list()); } Result: 6, Customer Example 7: ExcludeZeroes We can ignore 0 values of the query object by Example.excludeZeroes(). Method: @Test public void testExcludeZeroes() throws Exception { Session session = (Session) entityManager.getDelegate(); AddressType addressType = new AddressType(); addressType.setType(0); addressType.setDescription("Customer"); Example addressTypeExample = Example.create(addressType) .excludeZeroes(); Criteria criteria = session.createCriteria(AddressType.class) .add(addressTypeExample); listAddressTypes(criteria.list()); } Result: 6, Customer Example 8: Combining with Criteria QBE can be combined with criteria query. In this example we add further restriction to the query object using criteria query. Method: @Test public void testCombiningWithCriteria() throws Exception { Session session = (Session) entityManager.getDelegate(); AddressType addressType = new AddressType(); addressType.setDescription("Customer"); Example addressTypeExample = Example.create(addressType); Criteria criteria = session .createCriteria(AddressType.class).add(addressTypeExample) .add(Restrictions.eq("type", 6)); listAddressTypes(criteria.list()); } Result: 6, Customer Example 9: Association With criteria query we can filter both sides of an association, using two query objects. Method: @Test public void testAssociation() throws Exception { Session session = (Session) entityManager.getDelegate(); Address address = new Address(); address.setCountryISO2Code("US"); AddressType addressType = new AddressType(); addressType.setType(6); Example addressExample = Example.create(address); Example addressTypeExample = Example.create(addressType); Criteria criteria = session.createCriteria(Address.class).add(addressExample) .createCriteria("addressType").add(addressTypeExample); // addressType is a property of Address listAddresses(criteria.list()); } Result: 84, US, BOSTON, null, 6, Customer 83, US, ATLANTA, null, 6, Customer 82, US, SAN FRANCISCO, null, 6, Customer 75, US, CHICAGO, Los Angeles Way2, 6, Customer EclipseLink EclipseLink QBE uses QueryByExamplePolicy, ReadObjectQuery and JpaHelper: QueryByExamplePolicy qbePolicy =newQueryByExamplePolicy(); qbePolicy.excludeDefaultPrimitiveValues(); Address address =newAddress(); address.setCity("CHICAGO"); ReadObjectQuery roq =newReadObjectQuery(address, qbePolicy); Query query =JpaHelper.createQuery(roq, entityManager); OpenJPA OpenJPA uses OpenJPAQueryBuilder: CriteriaQuery cq = openJPAQueryBuilder.createQuery(Address.class); Address address =newAddress(); address.setCity("CHICAGO"); cq.where(openJPAQueryBuilder.qbe(cq.from(Address.class), address); References Hibernate: · Srinivas Guruzu and Gary Mak: Hibernate Recipes: A Problem-Solution Approach (Apress) · http://docs.jboss.org/hibernate/core/3.3/reference/en/html/querycriteria.html#querycriteria-examples · http://www.java2s.com/Code/Java/Hibernate/CriteriaQBEQueryByExampleCriteria.htm · http://www.dzone.com/snippets/hibernate-query-example · http://gal-levinsky.blogspot.de/2012/01/qbe-pattern.html Hibernate associations: · http://stackoverflow.com/questions/9309884/query-by-example-on-associations · http://stackoverflow.com/questions/8236596/hibernate-query-by-example-equivalent-of-association-criteria-query JPA: · http://stackoverflow.com/questions/2880209/jpa-findbyexample EclipseLink: · http://www.coderanch.com/t/486528/ORM/databases/findByExample-JPA-book OpenJPA: · http://www.ibm.com/developerworks/java/library/j-typesafejpa/#N10C18

If you ever need to persuade management why it might be better to deploy a larger change in multiple stages and push it to customers gradually, read on. A deployment of many changes is risky. We want therefore to deploy them in a way which minimizes the risk of harm to our customers and our companies. The deployment can be done either in an all-at-once (also known as big-bang) way or a gradual way. We will argue here for the more gradual (“stepwise”) approach. Big-bang or stepwise deployment? A big-bang deployment seems to be the natural thing to do: the full solution is developed and tested and then replaces the current system at once. However, it has two crucial flaws. First, it assumes that most defects can be discovered by testing. However, due to differences in test/prod environments, unknown dependencies, and the sheer scale of a typical larger system there always will be problems that are not discovered until production deployment or even until the application runs for a while in production (whichapplies even to airplanes). The more parts have been changed, the more of these production defects will happen at the same time. A gradual deployment makes it possible to discover and handle them one by one. Second, the more complex the deployment, the higher chance of human error(s), i.e. the deployment itself is a likely source of serious defects. Some of the drawbacks of a big-bang deployment in more detail: Complexity: A big-bang deployment requires coordination of many people and “moving parts” that depend on each other, providing a huge opportunity for human mistake (i.e. there will be mistakes). Lot of time: Such a deployment requires lot of time (typically also more than planed/expected) and thus lot of downtime when users cannot use the system. Hard troubleshooting: With a network of inter-dependent parts that changed all at the same time, while perhaps also changing the infrastructure (i.e. connections between them), it is extremely hard to pinpoint the source of defects, thus considerably increasing the time to detect and correct defects while also increasing the risk of people stepping on the toes of each other and “panic fixes” that either cause more problems than they remove or are not good enough (as the rollback that sped upKnight’s downfall). Rollback is likely either impossible or equally time-consuming and risky as the deployment itself, thus increasing the impact of defects and inviting even more human errors. Impact: Deploying everything to all users at the same time means that everybody will be impacted by a potential defect/error/mistake. Long freeze: All needs to be tested together after all development is finished, which requires a lot of time while the code is frozen and no more fixes and changes can get into production for weeks. Risk mitigation The goal of a good deployment plan is to mitigate the risk of the deployment and get it to an acceptable level. There are two aspects to risk: the probability of a defect and the impact of the defect. The following table shows how the possible measures affect them: Defect probability reduction Defect impact reduction testing stepwise deployment gradual migration of users to the new version (f.ex. 1 in 1000 or particular subsets) rollback mechanism => these also lead to much lower time to detect and fix defects Practices for stepwise deployment Enable stepwise deployment: Use parallel change and other Continuous Delivery techniques to make it possible to deploy updated components independently from each other and to switch on/off new features and to switch what versions of the components they depend on are currently used. (Parallel change – keeping the old and new code and being able to use one or the other – is crucial here. Also notice that parallel change applies also to data – you will need to evolve your data schema gradually and keep both old and new one at the same time in a period of time.) Enable rollback. The previous measure – stepwise deployment – makes it also easy(ier) to roll-back the changes by switching to a previous version of a dependency or by switching back to the old code. Migrate users gradually to the new version, i.e. expose the new version only to a small subset of the users initially and increase that subset until everybody uses it. This can be done f.ex. by deploying to only a subset of servers and sending a random/particular subset of users to the new servers but there are also ways if you have only a single machine. (See f.ex. my post Webapp Blue-Green Deployment Without Breaking Sessions/With Fallback With HAProxy.) Monitoring – make sure you are able to monitor flow of users through the system and detect any anomalies and errors early, long before angry calls from the business. Tools such as Logstash, Google Analytics (with custom events from JavaScript), client-side error logging via one of existing services or a custom solution are invaluable. About these ads

This morning an interesting topic was posted to the Apache ServiceMix user forum, asking the question: To ServiceMix or not ServiceMix. In my mind the short answer is: NO Guillaume Nodet one of the key architects and long time committer on Apache ServiceMix already had his mind set 3 years ago when he wrong this blog post - Thoughts about ServiceMix. What has happened on the ServiceMix project was that the ServiceMix kernel was pulled out of ServiceMix into its own project - Apache Karaf. That happened in spring 2009, which Guillaume also blogged about. So is all that bad? No its IMHO all great. In fact having the kernel as a separate project, and Camel and CXF as the integration and WS/RS frameworks, would allow the ServiceMix team to focus on building the ESB that truly had value-add. But that did not happen. ServiceMix did not create a cross product security model, web console, audit and trace tooling, clustering, governance, service registry, and much more that people were looking for in an ESB (or related to a SOA suite). There were only small pieces of it, but never really baked well into the project. That said its not too late. I think the ServiceMix project is dying, but if a lot of people in the community step up, and contribute and work on these things, then it can bring value to some users. But I seriously doubt this will happen. PS: 6 years ago I was working as a consultant and looked at the next integration platform for a major Danish organization, and we looked at ServiceMix back then and dismissed it due its JBI nature, and the new OSGi based architecture was only just started. And frankly it has taken a long long time to mature Apache Karaf / Felix / Aries and the other pieces in OSGi to what they are today to offer a stable and sound platform for users to build their integration applications. That was not the case 4-6 years ago. Okay No to ServiceMix - what are my options then? So what should use you instead of ServiceMix? Well in my mind you have at least these two options. 1) Use Apache Karaf and add the pieces you need, such as Camel, CXF, ActiveMQ and build your own ESB. These individual projects have regular releases, and you can upgrade as you need. The ServiceMix project only has the JBI components in additional, that you should NOT use. Only legacy users that got on the old ServiceMix 3.x wagon may need to use this in a graceful upgrade from JBI to Karaf based containers. 2) Take a look at fabric8. IMHO fabric8 is all that value-add the ServiceMix project did not create, and a lot more. James Strachan, just blogged today about some of his thoughts on fabric8, JBoss Fuse, and Karaf. I encourage you to take a read. For example he talks about how fabric becomes poly container, so you have a much wider choice of which containers/JVM to run your integration applications. OSGi is no longer a requirement. (IMHO that is very very existing and potentially a changer). I encourage you to check out fabric8 web-site, and also read the overview and motivation sections of the documentation. And then check out some of the videos. After the upcoming JBoss Fuse 6.1 release, the Fuse team at Red Hat will have more time and focus to bring the documentation at fabric8 up to date covering all the functionality we have (there is a lot more), and as well bring out a 1.0 community released using pure community releases. This gives end users a 100% free to use out of the box release. And users looking for a commercial release can then use JBoss Fuse. Best of both worlds. Summary Okay back to the question - to ServiceMix or not. Then NO. Innovation happens outside ServiceMix, and also more and more outside Apache. If you have thoughts then you can share those in comments to this blog, or better yet, get involved in the discussion forum at the ServiceMix user forum. PPS: The thoughts on this blog is mine alone, and are not any official words from my employer.

This article was originally written by Jeff Morris In the introduction to this series, I discussed some of the motivation for rewriting .NET SDK, the goals, objectives and the major features of the upcoming 2.0 release, and we examined the high-level architecture (10,000 feet view) of a Couchbase Server Client SDK. In this post we will go over the design and development of one of the core configuration components of a Couchbase SDK: Server Configuration. Introduction A Couchbase SDK client requires configuration from two sources: the Client Configuration, which defines the IP of the cluster to connect to, number of connections to use and other important information regarding how the client will interact with the cluster, and the Server Configuration, which defines the current state of the cluster (e.g. number of nodes, buckets that are available, etc.), thus driving the internal state of a client (Cluster Map) This post will only discuss the Server Configuration aspects and will largely revolve around implementing several well-defined interfaces or contracts. HTTP Streaming Configuration Currently, most clients use a “bootstrapping” technique via client configuration and a “Streaming Configuration” exposed by the Couchbase REST API. This is supported by versions of Couchbase from 2.2 and back. The usual approach is as follows: Within the “uris” element of a Client Configuration (semantics very per client), a URL is defined for which to start the bootstrapping process: http://[SERVER]:8091/pools The response is then parsed and the a request is made to get the buckets configuration: http://[SERVER]:8091/pools/default?uuid=[UUID] This response is parsed and another request is made to get streaming URL from: http://[SERVER]:8091/pools/default/buckets?v=[VERSION]&uuid=[UUID] Finally, the streaming URL connection is made which is long-lived and raises events in the client with respect to changes in the cluster: http://[SERVER]:8091/pools/default/bucketsStreaming/default?bucket_uuid=[UUID] The client will then change its internal state to match that of the current server configuration. There are some problems with this approach, among others: The “streaming URL” is resource intensive to create and maintain (mainly memory) on the server-side During a rebalance or failover situation, the cluster configuration may change many, many times. Each time this happens the client must tear down all of its resources (socket connections, VBucket mappings) and build its state up again and again, which can leads to reduced throughput, latency, higher than expected memory and CPU usage, and so on and so forth… Operations that are in-flight may be terminated and then re-tried on a new config state – it’s as if the “carpet has been pulled out from underneath them”. Responding to NOT_MY_VBUCKET responses are handled in-efficiently by simple trying the next node in the list – there is no information to help the client in which node to re-direct the operation to. A New Model for Configuration Management: CCCP While the streaming HTTP “bootstrapping” approach has worked reasonably well for most clients, the downsides have begun to outweigh the plusses, thus a new model for updating client configuration has been defined is available starting with the 2.5 version of the Couchbase Server: Client Cluster Configuration Publication or “CCCP”. CCCP introduces a new operation to be used before or after authentication to request configuration as well as a mechanism for returning configuration information when a NOT_MY_VBUCKET response is returned for a failed operation. In this case CCCP supporting SDK, the client will react by using the configuration to update itself before resending the operation. Note that a NOT_MY_VBUCKET is the standard response that is returned by the cluster when the cluster itself has changed (during a rebalance or failover scenario for example) and the client has not yet “synched” up and is using a stale configuration, resulting in an invalid key mapping. Whereas the “bootstrapping” approach is somewhat of a “pull” type operation, CCCP is either “push” or “pull” depending upon whether the request was initiated by the client (via an explicit CMD_GET_CLUSTER_CONFIG operation) or by the server itself (via a NOT_MY_VBUCKET response to an operation). We will go over CCCP in more detail in a later post. File Based Configuration One other semi-supported configuration option exists: file based configuration. File based configuration is primarily useful for testing and development and we will provide an implementation in the test projects to remove some of the dependencies that are difficult to replicate and or cause false positives when running the test suite. Structural Architecture View Internally the Server Configuration component of the client is a provider based model, in which multiple implementations of a configuration provider can be configured in the client and then a strategy can be chosen to determine which provider should be used. The default is a simple linear, fallback approach where the first configured provider is used and then if it fails the next provider in sequence will take its place. Here is a diagram showing the main actor objects and the relationships with some of other key objects within the client which will be discussed in subsequent posts: A description of each follows: ConfigurationProvider: a source which shall yield a new ConfigInfo. It’s the responsibility of the provider to provide the mechanism for fetching the configuration from its source. ConfigurationInformation: the configuration info contains a list of possible nodes and the VBucket map informing clients about which servers within said nodes a given key should be forwarded to. ConfigurationManager: bridge between the client and the providers and the strategy taken to determine which provider to use and what retry logic to apply. A more detailed document of this architecture can be found here. Please note that this, like all development, is an evolutionary process, so expect some changes and revisions over time. Conclusion and Next Steps This post discussed the history (HTTP Streaming) and the future (CCCP) of Couchbase SDK Server Configuration Management. In the next post we will go into detail the implementation of the HTTP Streaming configuration provider which is required for clients targeting pre-2.5 versions of the Couchbase Server.

At the end of last year I was playing around with running scheduled tasks to monitor a Neo4j cluster and one of the problems I ran into was that the monitoring would sometimes exit. I eventually realised that this was because a RuntimeException was being thrown inside the Runnable method and I wasn’t handling it. The following code demonstrates the problem: import java.util.ArrayList; import java.util.List; import java.util.concurrent.*; public class RunnableBlog { public static void main(String[] args) throws ExecutionException, InterruptedException { ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(new Runnable() { @Override public void run() { System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis()); throw new RuntimeException("game over"); } }, 0, 1000, TimeUnit.MILLISECONDS).get(); System.out.println("exit"); executor.shutdown(); } } If we run that code we’ll see the RuntimeException but the executor won’t exit because the thread died without informing it: Exception in thread "main" pool-1-thread-1 -> 1391212558074 java.util.concurrent.ExecutionException: java.lang.RuntimeException: game over at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at RunnableBlog.main(RunnableBlog.java:11) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120) Caused by: java.lang.RuntimeException: game over at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) At the time I ended up adding a try catch block and printing the exception like so: public class RunnableBlog { public static void main(String[] args) throws ExecutionException, InterruptedException { ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis()); throw new RuntimeException("game over"); } catch (RuntimeException e) { e.printStackTrace(); } } }, 0, 1000, TimeUnit.MILLISECONDS).get(); System.out.println("exit"); executor.shutdown(); } } This allows the exception to be recognised and as far as I can tell means that the thread executing the Runnable doesn’t die. java.lang.RuntimeException: game over pool-1-thread-1 -> 1391212651955 at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) pool-1-thread-1 -> 1391212652956 java.lang.RuntimeException: game over at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) pool-1-thread-1 -> 1391212653955 java.lang.RuntimeException: game over at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) This worked well and allowed me to keep monitoring the cluster. However, I recently started reading ‘Java Concurrency in Practice‘ (only 6 years after I bought it!) and realised that this might not be the proper way of handling the RuntimeException. public class RunnableBlog { public static void main(String[] args) throws ExecutionException, InterruptedException { ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis()); throw new RuntimeException("game over"); } catch (RuntimeException e) { Thread t = Thread.currentThread(); t.getUncaughtExceptionHandler().uncaughtException(t, e); } } }, 0, 1000, TimeUnit.MILLISECONDS).get(); System.out.println("exit"); executor.shutdown(); } } I don’t see much difference between the two approaches so it’d be great if someone could explain to me why this approach is better than my previous one of catching the exception and printing the stack trace.

Learn how to set up a four node Hadoop cluster using AWS EC2, PuTTy(gen), and WinSCP.

i’ve been pushing this branching model for something like 14 years now. it’s nice to see facebook say a little more about their trunk based development . of course they’re not doing it because they read anything i wrote, as the practice isn’t mine, it’s been hanging around in the industry for many years, but always as bridesmaid so to speak. if not trunk, what? mainline? mainline as popularized by clearcase is what we’re trying to kill. at least historically. it’s very different to trunk based development, and even having vastly improved merge tools doesn’t make it better – you still risk regressions, and huge nerves around ordering of releases. clearcase’s best-practices also foisted a ‘many repos’ (vobs) on teams using it, and that courted the whole conway’s law prophesy. i mentioned conway’s law before in scaling trunk based development and it concerns undue self-importance of teams around arbitrary separations. multiple small repos for a dvcs ? there is a great statement by a reddit user in the programming section of reddit, in conjunction with the facebook announcement: comment ref or all comments this redditor is right, there’s a lack of atomicity around a many-repos design, that stymies bisect. it could be that git subtrees (not submodules) are a way of getting that back (thanks @chris_stevenson on a back channel). there’s also a real problem moving code easily between repos (with history) though @offbytwo (back channel again) points out that subtrees carefully used can help do that. trunk at google vs facebook tuesday’s announcement was from facebook, and to give some balance, there’s deeper info on google’s trunk design in: google’s scaled trunk based development . subsetting the trunk for checkouts tl;dr: different google have many thousands of buildable and deployable things, which have very different release schedules. facebook don’t as they substantially have the php web-app, and apps for ios and android in different repos. well at least the main php web-app is in the mercurial trunk they talked about on tuesday. i’m not sure how the ios and android apps are managed, but at least the android one is outside the main trunk. google subset their trunk. i posted about that on monday . in that article i pointed out that the checkout can grow (or shrink) depending on the nature of the change being undertaken. it’s very different to a multiple-small-repos design. facebook don’t subset their trunk on checkout, as they do not need to; the head revisions of everything in that trunk are not big enough for a c: drive or ide to buckle. there’s also no compile stage for php , for regular development work. maximized sharing of code tl;dr: the same code is shared using globbed directories within the source tree. it’s shared as source files, in situ, rather than classes in a jar (or equivalent). refactoring tl;dr: the same developers take on refactorings where appropriate. sure it means a bigger atomic commit, but knowing all the affected source is in front of you as you do the refactoring is comforting. at least, knowing that if intellij (or eclipse, etc) completes the refactoring there’s a very strong possibility that the build will stay green, and that you’re only going to have a slight impact on other people’s working copy, and only if they are concurrently editing the same files. bigger refactoring probably still require a warning email. super tooling of the build phase tl;dr: the same google have what amounts to a super-computer doing the compilation for them (all languages that are compiled). all developers and all ci daemons leverage it. and by effective super-computer, i mean previous-compiled bits and pieces are pulled out of an internal cloud-map-thing for source permutations that have been compiled before. the distributed hashmap is possibly lru centric rather that everything forever. facebook don’t have that big hashmap of recently compiled bits and pieces, but they do have hiphop in the toolchain (originally a php to c++ compiler) which is interesting because at face value php is an interpreted language and ‘compile’ makes no sense. hiphop was created to reduce the server footprint and requirements for production deployments, while still being 100% functionally identical to the interpreted php app. it’s also faster in production. more recently hiphop became a virtual machine. it continues to be incrementally improved. like google, facebook can measure cost-benefit of continued work on it (prod rack space & prod electricity vs developer salaries). source-control weapons of choice tl;dr: different google use perforce for their trunk (with additional tooling), and many (but not all) developers use git on their local workstation to gain local-branching with an inhouse developed bridge for interop with perforce. facebook uses mercurial with additional tooling for the central server/repo. it’s unclear whether developers, by habit, exist with the mercurial client software or use git which can interop with mercurial backends. both google and facebook do trunk based development of course. branches & merge pain tl;dr: the same they don’t have merge pain, because as a rule developers are not merging to/from branches. at least up to the central repo’s server they are not. on workstations, developers may be merging to/from local branches, and rebasing when the push something that’s “done” back to the central repo. release engineers might cherry-pick defect fixes from time to time, but regular developers are not merging (you should not count to-working-copy merges) eating own dog-food tl;dr: mostly different all staff at facebook use a not-live-yet version of the web-app for all of their communication, documentation, management etc. if there’s a bug everyone feels it – though selenium2 functional tests and zillions of unit-tests guard against that happening too often. google has too many different apps for the team making each to be said to be a daily user of it. for example the adsense developer may use a dog-food version of gmail, but they are making adsense, so are hardly hurting themselves as they are not minute by minute using the interface as part of their regular existence at google. code review tl;dr: same both google and facebook insist on code reviews before the commit is accepted into the remote repo’s trunk for all others to use. there’s no mechanism of code review that’s more efficient or effective. google back in 2009 were pivoting incoming changes to the trunk around the code-review process managed by mondrian. i wrote about that in “continuous review #1” in december . i think they are unchanged in that respect: developers actively push their commit after a code review has been completed. facebook have just flipped to mercurial (from subversion). in the article linked to at the top of the page, facebook have not mentioned “pull request” or “patch queue”, or indeed “code review”. the article was mostly about speed, robustness and scale. i suspect they are sitting within the semantics of mercurials patch-queue processing though, although assigning a bot to it rather than a human. update: simon stewart pinged me and reminded me that they use (and made) phabricator. he spoke about it in a mobile@scale presentation, and that video is here . in the video he says the review is queue based now, but that they experimenting with landing the change sets into the master now. the video is from november, and was for the android + ios platforms, but it is likely to be used today for the main trunk for the php web-app. automated testing tl;dr: same heavy reliance on unit tests (not necessarily made in a tdd style). later in an build pipeline, selenium2 tests (for web-apps at least) kick in to guard the functional quality of deployed app. manual qa tl;dr: mostly the same both companies have progressively moved way from manual qa and dedicated testing professionals, towards developers testing their own stuff at discrete moments (note the dog-food item above too). prod release frequency tl;dr: it varies. facebook for the main web app, are twice a day presently (at least on weekdays). i published info on that at the start of last year. google have many apps with different release schedules, and some are “many times a day”, while others are “planned releases every few weeks”. many are in between. prod db deployment tl;dr: mostly the same database (or equivalent) table shapes (or equivalent) are designed to be forwards/backwards compatible as far as possible. pull requests as part of workflow tl;dr: same etsy, github, and other high throughput organizations are trunking by some definition, but using pull-requests to merge in things being done. it has different obligations if done, but google and facebook are not doing this in their trunks – they both essentially push (after review). refer the ‘code review’ section above. common code ownership tl;dr: the same you can commit to any part of the source tree, provided it passed a fair code review. notional owners of directories within the source tree take a boy-scout pledge to do their best with unsolicited incoming change-lists. there are strong permissions in the google perforce implementation, but the pledge means that contributions are not often rejected if the merit is there. build is ever broken tl;dr: the same almost never. directionality of merge for prod bug fixes tl;dr: the same trunk receives the defect fix, it gets cherry picked to the release branch. the release branch might have been made from a tag, if it didn’t exist before. binary dependencies tl;dr: the same checked into source-control without version suffixing (harmonized versions across all apps). e.g. – log4j.jar rather than log4j-1.2.8.jar.

One of the great things about git is how fast it is. You can create a new branch, or switch to another branch, almost as fast as you can type the command. This tends to lower the impedance of branching. As a result, many individuals and teams will naturally converge on a process where they create many, many branches. If you’re like me, you may have 30 branches at any given time. This can make viewing all the branches unwieldy. Once I week or so, I would go on a branch deletion spree by manually copying and pasting multiple branch names into a git branch -D statement. The basic use case is that you want to delete any branches that are already merged into master. Here is a python script that automated just that. from subprocess import check_output import sys def get_merged_branches(): ''' a list of merged branches, not couting the current branch or master ''' raw_results = check_output('git branch --merged upstream/master', shell=True) return [b.strip() for b in raw_results.split('\n') if b.strip() and not b.startswith('*') and b.strip() != 'master'] def delete_branch(branch): return check_output('git branch -D %s' % branch, shell=True).strip() if __name__ == '__main__': dry_run = '--confirm' not in sys.argv for branch in get_merged_branches(): if dry_run: print branch else: print delete_branch(branch) if dry_run: print '*****************************************************************' print 'Did not actually delete anything yet, pass in --confirm to delete' print '*****************************************************************' To print the branches that would be deleted, just execute python delete_merged_branches.py. To actually delete the branches, execute python delete_merged_branches.py --confirm.

I'm passionate about front end optimization and have been for years. My original inspiration was Steve Souders and his Even Faster Web Sites talk at OSCON 2008. Since then, I've optimized this blog, made it even faster with a new design, doubled the speed of several apps for clients and showed how to make AppFuse faster. As part of my Devoxx 2013 presentation, I showed how to do page speed optimization in a Java webapp. I developed a couple AngularJS apps last year. To concat and minify their stylesheets and scripts, I used mechanisms that already existed in the projects. On one project, it was Ant and its concat task. On the other, it was part of a Grails application, so I used the resources and yui-minify-resources plugins. The Angular project I'm working on now will be published on a web server, as well as bundled in an iOS native app. Therefore, I turned to Grunt to do the optimization this time. I found it to be quite simple, once I figured out how to make it work with Angular. Based on my findings, I submitted a pull request to add Grunt to angular-seed. Below are the steps I used to add Grunt to my Angular project. Install Grunt's command line interface with "sudo npm install -g grunt-cli". Edit package.json to include a version number (e.g. "version": "1.0.0"). Add Grunt plugins in package.json to do concat/minify/asset versioning: "grunt": "~0.4.1", "grunt-contrib-concat": "~0.3.0", "grunt-contrib-uglify": "~0.2.7", "grunt-contrib-cssmin": "~0.7.0", "grunt-usemin": "~2.0.2", "grunt-contrib-copy": "~0.5.0", "grunt-rev": "~0.1.0", "grunt-contrib-clean": "~0.5.0" Create a Gruntfile.js that runs all the plugins. module.exports = function (grunt) { grunt.initConfig({ pkg: grunt.file.readJSON('package.json'), clean: ["dist", '.tmp'], copy: { main: { expand: true, cwd: 'app/', src: ['**', '!js/**', '!lib/**', '!**/*.css'], dest: 'dist/' }, shims: { expand: true, cwd: 'app/lib/webshim/shims', src: ['**'], dest: 'dist/js/shims' } }, rev: { files: { src: ['dist/**/*.{js,css}', '!dist/js/shims/**'] } }, useminPrepare: { html: 'app/index.html' }, usemin: { html: ['dist/index.html'] }, uglify: { options: { report: 'min', mangle: false } } }); grunt.loadNpmTasks('grunt-contrib-clean'); grunt.loadNpmTasks('grunt-contrib-copy'); grunt.loadNpmTasks('grunt-contrib-concat'); grunt.loadNpmTasks('grunt-contrib-cssmin'); grunt.loadNpmTasks('grunt-contrib-uglify'); grunt.loadNpmTasks('grunt-rev'); grunt.loadNpmTasks('grunt-usemin'); // Tell Grunt what to do when we type "grunt" into the terminal grunt.registerTask('default', [ 'copy', 'useminPrepare', 'concat', 'uglify', 'cssmin', 'rev', 'usemin' ]); }; Add comments to app/index.html so usemin knows what files to process. The comments are the important part, your files will likely be different. ... A couple of things to note: 1) the copy task copies the "shims" directory from Webshims lib because it loads files dynamically and 2) setting "mangle: false" on the uglify task is necessary for Angular's dependency injection to work. I tried to use grunt-ngmin with uglify and had no luck. After making these changes, I'm able to run "grunt" and get an optimized version of my app in the "dist" folder of my project. For development, I continue to run the app from my "app" folder, so I don't currently have a need for watching and processing assets on-the-fly. That could change if I start using LESS or CoffeeScript. The results speak for themselves: from 27 requests to 5 on initial load, and only 3 requests for less than 2K after that. YSlow Page Speed No optimization 75 27 HTTP requests / 464K 55/100 Apache optimization (gzip and expires headers) 89 initial load: 26 requests / 166K primed cache: 4 requests / 40K 88/100 Apache + concat/minified/versioned files 98 initial load: 5 requests / 136K primed cache: 3 requests / 1.4K 93/100