I’ve just started looking at Docker. It’s a cool new technology that has the potential to make the management and deployment of distributed applications a great deal easier. I’d very much recommend checking it out. I’m especially interested in using it to deploy Mono applications because it promises to remove the hassle of deploying and maintaining the mono runtime on a multitude of Linux servers. I’ve been playing around creating new images and containers and debugging my Dockerfile, and I’ve wound up with lots of temporary containers and images. It’s really tedious repeatedly running ‘docker rm’ and ‘docker rmi’, so I’ve knocked up a couple of bash commands to bulk delete images and containers. Delete all containers: sudo docker ps -a -q | xargs -n 1 -I {} sudo docker rm {} Delete all un-tagged (or intermediate) images: sudo docker rmi $( sudo docker images | grep '' | tr -s ' ' | cut -d ' ' -f 3)

There is actually nothing very special about writing a Spring-boot web application purely using Scala, it just works! In this blog entry, I will slowly transform a Java based Spring-boot application completely to Scala - the Java based sample is available at this github location - https://github.com/bijukunjummen/spring-boot-mvc-test To start with, I had the option of going with either a maven based build or gradle based build - I opted to go with a gradle based build as gradle has a greatscala plugin, so for scala support the only changes to a build.gradle build script is the following: ... apply plugin: 'scala' ... jar { baseName = 'spring-boot-scala-web' version = '0.1.0' } dependencies { ... compile 'org.scala-lang:scala-library:2.10.2' ... } Essentially adding in the scala plugin and specifying the version of the scala-library. Now, I have one entity, a Hotel class, it transforms to the following with Scala: package mvctest.domain .... @Entity class Hotel { @Id @GeneratedValue @BeanProperty var id: Long = _ @BeanProperty var name: String = _ @BeanProperty var address: String = _ @BeanProperty var zip: String = _ } Every property is annotated with @BeanProperty annotation to instruct scala to generate the Java bean based getter and setter on the variables. With the entity in place a Spring-data repository for CRUD operations on this entity transforms from: import mvctest.domain.Hotel; import org.springframework.data.repository.CrudRepository; public interface HotelRepository extends CrudRepository { } to the following in Scala: import org.springframework.data.repository.CrudRepository import mvctest.domain.Hotel import java.lang.Long trait HotelRepository extends CrudRepository[Hotel, Long] And the Scala based controller which uses this repository to list the Hotels - vi... import org.springframework.web.bind.annotation.RequestMapping import org.springframework.stereotype.Controller import mvctest.service.HotelRepository import org.springframework.beans.factory.annotation.Autowired import org.springframework.ui.Model @Controller @RequestMapping(Array("/hotels")) class HotelController @Autowired() (private val hotelRepository: HotelRepository) { @RequestMapping(Array("/list")) def list(model: Model) = { val hotels = hotelRepository.findAll() model.addAttribute("hotels", hotels) "hotels/list" } } Here the constructor autowiring of the HotelRepository just works!. Do note the slightly awkward way of specifying the @Autowired annotation for constructor based injection. Finally, Spring-boot based application requires a main class to bootstrap the entire application, where this bootstrap class looks like this with Java: @Configuration @EnableAutoConfiguration @ComponentScan public class SampleWebApplication { public static void main(String[] args) { SpringApplication.run(SampleWebApplication.class, args); } } In scala, though I needed to provide two classes, one to specify the annotation and other to bootstrap the application, there may be better way to do this(blame it on my lack of Scala depth!) - package mvctest import org.springframework.context.annotation.Configuration import org.springframework.boot.autoconfigure.EnableAutoConfiguration import org.springframework.context.annotation.ComponentScan import org.springframework.boot.SpringApplication @Configuration @EnableAutoConfiguration @ComponentScan class SampleConfig object SampleWebApplication extends App { SpringApplication.run(classOf[SampleConfig]); } and that's it, with this set-up the entire application just works, the application can be started up with the following: ./gradlew build && java -jar build/libs/spring-boot-scala-web-0.1.0.jar and the sample endpoint listing the hotels accessed at this url: http://localhost:8080/hotels/list I have the entire git project available at this github location: https://github.com/bijukunjummen/spring-boot-scala-web In conclusion, Scala can be considered a first class citizen for a Spring-boot based application and there is no special configuration required to get a Scala based Spring-boot application to work. It just works!

this is another experiment with longer posts. previously, i used the time series example as the bed on which to test some ideas regarding feature design, to explain how we work and in general work out the rough patches along the way. i should probably note that these posts are purely fiction at this point. we have no plans to include a time series feature in ravendb at this time. i am trying to work out some thoughts in the open and get your feedback. at any rate, yesterday we had a request for cassandra style counters at the mailing list. and as long as i am doing feature design series, i thought that i could talk about how i would go about implementing this. again, consider this fiction, i have no plans of implementing this at this time. the essence of what we want is to be able to… count stuff. efficiently, in a distributed manner, with optional support for cross data center replication. very roughly, the idea is to have “sub counters”, unique for every node in the system. whenever you increment the value, we log this to our own sub counter, and then replicate it out. whenever you read it, we just sum all the data we have from all the sub counters. let us outline the various parts of the solution in the same order as the one i used for time series. storage a counter is just a named 64 bits signed integer. a counter name can be any string up to 128 printable characters. the external interface of the storage would look like this: 1: public struct counterincrement 2: { 3: public string name; 4: public long change; 5: } 6: 7: public struct counter 8: { 9: public string name; 10: public string source; 11: public long value; 12: } 13: 14: public interface icounterstorage 15: { 16: void localincrementbatch(counterincrement[] batch); 17: 18: counter[] read(string name); 19: 20: void replicatedupdates(counter[] updates); 21: } as you can see, this gives us very simple interface for the storage. we can either change the data locally (which modify our own storage) or we can get an update from a replica about its changes. there really isn’t much more to it, to be fair. the localincrementbatch() increment a local value, and read() will return all the values for a counter. there is a little bit of trickery involved in how exactly one would store the counter values. for now, i think we’ll store each counter as two step values. we’ll have a tree of multi tree values that will carry each value from each source. that means that a counter will take roughly 4kb or so. this is easy to work with and nicely fit the model voron uses internally. note that we’ll outline additional requirement for storage (searching for counter by prefix, iterating over counters, addresses of other servers, stats, etc) below. i’m not showing them here because they aren’t the major issue yet. over the wire skipping out on any optimizations that might be required, we will expose the following endpoints: get /counters/read?id=users/1/visits&users/1/posts <—will return json response with all the relevant values (already summed up). { “users/1/visits”: 43, “users/1/posts”: 3 } get /counters/read?id=users/1/visits&users/1/1/posts&raw=true <—will return json response with all the relevant values, per source. { “users/1/visits”: {“rvn1”: 21, “rvn2”: 22 } , “users/1/posts”: { “rvn1”: 2, “rvn3”: 1 } } post /counters/increment <– allows to increment counters. the request is a json array of the counter name and the change. for a real system, you’ll probably need a lot more stuff, metrics, stats, etc. but this is the high level design, so this would be enough. note that we are skipping the high performance stream based writes we outlined for time series. we’ll probably won’t need them, so that doesn’t matter, but they are an option if we need them. system behavior this is where it is really not interesting, there is very little behavior here, actually. we only have to read the data from the storage, sum it up, and send it to the user. hardly what i’ll call business logic. client api the client api will probably look something like this: 1: counters.increment("users/1/posts"); 2: counters.increment("users/1/visits", 4); 3: 4: using(var batch = counters.batch()) 5: { 6: batch.increment("users/1/posts"); 7: batch.increment("users/1/visits",5); 8: batch.submit(); 9: } note that we’re offering both batch and single api. we’ll likely also want to offer a fire & forget style, which will be able to offer even better performance (because they could do batching across more than a single thread), but that is out of scope for now. for simplicity sake, we are going to have the client just a container for all of endpoints that it knows about. the container would be responsible for… updating the client visible topology, selecting the best server to use at any given point, etc. user interface there isn’t much to it. just show a list of counter values in a list. allow to search by prefix, allow to dive into a particular counter and read its raw values, but that is about it. oh, and allow to delete a counter. deleting data honestly, i really hate deletes. they are very expensive to handle properly the moment you have more than a single node. in this case, there is an inherent race condition between a delete going out and another node getting an increment. and then there is the issue of what happens if you had a node down when you did the delete, etc. this just sucks. deletion are handled normally, (with the race condition caveat, obviously), and i’ll discuss how we replicate them in a bit. high availability / scale out by definition, we actually don’t want to have storage replication here. either log shipping or consensus based. we actually do want to have different values, because we are going to be modifying things independently on many servers. that means that we need to do replication at the database level. and that leads to some interesting questions. again, the hard part here is the deletes. actually, the really hard part is what we are going to do with the new server problem. the new server problem dictates how we are going to bring a new server into the cluster. if we could fix the size of the cluster, that would make things a lot easier. however, we are actually interested in being able to dynamically grow the cluster size. therefor, there are only two real ways to do it: add a new empty node to the cluster, and have it be filled from all the other servers. add a new node by backing up an existing node, and restoring as a new node. ravendb, for example, follows the first option. but it means that in needs to track a lot more information. the second option is actually a lot simpler, because we don’t need to care about keeping around old data. however, this means that the process of bringing up a new server would now be: update all nodes in the cluster with the new node address (node isn’t up yet, replication to it will fail and be queued). backup an existing node and restore at the new node. start the new node. the order of steps is quite important. and it would be easy to get it wrong. also, on large systems, backup & restore can take a long time. operationally speaking, i would much rather just be able to do something like, bring a new node into the cluster in “silent” mode. that is, it would get information from all the other nodes, and i can “flip the switch” and make it visible to clients at any point in time. that is how you do it with ravendb, and it is an incredibly powerful system, when used properly. that means that for all intents and purposes, we don’t do real deletes. what we’ll actually do is replace the counter value with delete marker. this turns deletes into a much simple “just another write”. it has the sad implication of not free disk space on deletes, but deletes tend to be rare, and it is usually fine to add a “purge” admin option that can be run on as needed basis. but that brings us to an interesting issue, how do we actually handle replication. the topology map to simplify things, we are going to go with one way replication from a node to another. that allows complex topologies like master-master, cluster-cluster, replication chain, etc. but in the end, this is all about a single node replication to another. the first question to ask is, are we going to replicate just our local changes, or are we going to have to replicate external changes as well? the problem with replicating external changes is that you may have the following topology: now, server a got a value and sent it to server b. server b then forwarded it to server c. however, at that point, we also have a the value from server a replicated directly to server c. which value is it supposed to pick? and what about a scenario where you have more complex topology? in general, because in this type of system, we can have any node accept writes, and we actually desire this to be the case , we don’t want this behavior. we want to only replicate local data, not all the data. of course, that leads to an annoying question, what happens if we have a 3 node cluster, and one node fails catastrophically. we can bring a new node in, and the other two nodes will be able to fill in their values via replication, but what about the node that is down? the data isn’t gone, it is still right there in the other two nodes, but we need a way to pull it out. therefor, i think that the best option would be to say that nodes only replicate their local state, except in the case of a new node. a new node will be told the address of an existing node in the cluster, at which point it will: register itself in all the nodes in the cluster (discoverable from the existing node). this assumes a standard two way replication link between all servers, if this isn’t the case, the operators would have the responsibility to setup the actual replication semantics on their own. new node now starts getting updates from all the nodes in the cluster. it keeps them in a log for now, not doing anything yet. ask that node for a complete update of all of its current state. when it has all the complete state of the existing node, it replays all of the remembered logs that it didn’t have a chance to apply yet. then it announces that it is in a valid state to start accepting client connections. note that this process is likely to be very sensitive to high data volumes. that is why you’ll usually want to select a backup node to read from, and that decision is an ops decision. you’ll also want to be able to report extensively on the current status of the node, since this can take a while, and ops will be watching this very closely. server name a node requires a unique name. we can use guids, but those aren’t readable, so we can use machine name + port, but those can change. ideally, we can require the user to set us up with a unique name. that is important for readability and for being able to alter see all the values we have in all the nodes. it is important that names are never repeated, so we’ll probably have a guid there anyway, just to be on the safe side. actual replication semantics since we have the new server problem down to an automated process, we can choose the drastically simpler model of just having an internal queue per each replication destination. whenever we make a change, we also make a note of that in the queue for that destination, then we start an async replication process to that server, sending all of our updates there. it is always safe to overwrite data using replication, because we are overwriting our own data, never anyone else. and… that is about it, actually. there are probably a lot of details that i am missing / would discover if we were to actually implement this. but i think that this is a pretty good idea about what this feature is about.

In my previous post, you have seen how we can start a WebLogic admin and multiple managed servers. One downside with that instruction is that those processes will start in foreground and the STDOUT are printed on terminal. If you intended to run these severs as background services, you might want to try the WebLogic node manager wlscontrol.sh tool. I will show you how you can get Node Manager started here. The easiest way is still to create the domain directory with the admin server running temporary and then create all your servers through the /console application as described in last post. Once you have these created, then you may shut down all these processes and start it with Node Manager. 1. cd $WL_HOME/server/bin && startNodeManager.sh & 3. $WL_HOME/common/bin/wlscontrol.sh -d mydomain -r $HOME/domains/mydomain -c -f startWebLogic.sh -s myserver START 4. $WL_HOME/common/bin/wlscontrol.sh -d mydomain -r $HOME/domains/mydomain -c -f startManagedWebLogic.sh -s appserver1 START The first step above is to start and run your Node Manager. It is recommended you run this as full daemon service so even OS reboot can restart itself. But for this demo purpose, you can just run it and send to background. Using the Node Manager we can then start the admin in step 2, and then to start the managed server on step 3. The NodeManager can start not only just the WebLogic server for you, but it can also monitor them and automatically restart them if they were terminated for any reasons. If you want to shutdown the server manually, you may use this command using Node Manager as well: $WL_HOME/common/bin/wlscontrol.sh -d mydomain -s appserver1 KILL The Node Manager can also be used to start servers remotely through SSH on multiple machines. Using this tool effectively can help managing your servers across your network. You may read more details here: http://docs.oracle.com/cd/E23943_01/web.1111/e13740/toc.htm TIPS1: If there is problem when starting server, you may wnat to look into the log files. One log file is the/servers//logs/.out of the server you trying to start. Or you can look into the Node Manager log itself at $WL_HOME/common/nodemanager/nodemanager.log TIPS2: You add startup JVM arguments to each server starting with Node Manager. You need to create a file under /servers//data/nodemanager/startup.properties and add this key value pair:Arguments = -Dmyapp=/foo/bar TIPS3: If you want to explore Windows version of NodeManager, you may want to start NodeManager without native library to save yourself some trouble. Try adding NativeVersionEnabled=false to$WL_HOME/common/nodemanager/nodemanager.properties file.

As well as being key value store, Redis offers a publish subscribe messaging implementation. This post will describe a simple scenario, using Spring Data Redis, of adding a message domain object to a repository via a REST call, publishing that message to a channel, subscribers to that channel receiving that message who as a result set any long polling deferred results with the message. The two key classes in the Redis publish subscribe mechanism are the RedisTemplate class and the RedisMessageListenerContainer class. The RedisTemplate contains the JedisConnectionFactory which holds the Redis connection details and as well as the methods to manipulate the key value stores, there’s a publish method calledconvertAndSend. This method takes two arguments. The first being the channel name of where the messages need to be published to and the second being the object to be sent. In this example, the publishing of the message is done after the Message is persisted via an aspect. @Aspect @Component public class MessageAspect extends AbstractRedisAspect { private static final Logger LOGGER = LoggerFactory .getLogger(MessageAspect.class); @Value("${messaging.redis.channel.messages}") private String channelName; @After("execution(* com.city81.redisPubSub.repository.MessageDao.save(..))") public void interceptMessage(JoinPoint joinPoint) { Message message = (Message) joinPoint.getArgs()[0]; // this publishes the message this.redisTemplate.convertAndSend(channelName, message); } } The RedisMessageListenerContainer, as well as holding the JedisConnectionFactory, holds a map of message listeners where the key is a message listener instance and the value the channel. The message listener instance references a class which implements the onMessage method of theMessageListener interface. When a message is published, those subscribers who are listening to that channel will then receive the published message via the onMessage method. The published message contains the serialised object that was sent in the body of the Redis Message and needs to be deserialised and cast to the original object. public void onMessage( org.springframework.data.redis.connection.Message redisMessage, byte[] pattern) { Message message = (Message) SerializationUtils.deserialize(redisMessage.getBody()); // set the deferred results for the user for (DeferredResult deferredResult : this.messageDeferredResultList) { deferredResult.setResult(message); } } The DeferredResult list is populated by calls to the REST service's getNewMessage method. This will in turn, in the MessageManager, create a DeferredResult object, add it to the list and return the object to the client. public DeferredResult getNewMessage() throws Exception { final DeferredResult deferredResult = new DeferredResult(deferredResultTimeout); deferredResult.onCompletion(new Runnable() { public void run() { messageDeferredResultList.remove(deferredResult); } }); deferredResult.onTimeout(new Runnable() { public void run() { messageDeferredResultList.remove(deferredResult); } }); messageDeferredResultList.add(deferredResult); return deferredResult; } The GitHub repo for this example contains two simple HTML pages, one which starts a long poll request and another which adds a message. These will call the below REST web service. @Controller @RequestMapping("/messages") public class MessageAPIController { @Inject private MessageManager messageManager; // // ADD A MESSAGE // @RequestMapping(value = "/add", method = RequestMethod.POST, produces = "application/json") @ResponseBody public Message addMessage( @RequestParam(required = true) String text) throws Exception { return messageManager.addMessage(text); } // // LONG POLLING // @RequestMapping(value = "/watch", method = RequestMethod.GET, produces = "application/json") @ResponseBody public DeferredResult getNewMessage() throws Exception { return messageManager.getNewMessage(); } } A further enhancement to the above to ensure messages aren't missed in between long polling requests would be to store the messages in Redis in a sorted set with the score being the message's creation timestamp. The Redis publish mechanism could then be used to tell the subscriber that there are new messages in Redis and it could then retrieve them based on the time of the last request, and return a collection of messages back to the client in the DeferredResult object.

In the previous post we used avro-tools commands to serialize and deserialize data. In this post we post we will use Avro Java API for achieving the same. We will use same sample data and schema from our previous post. The java code for serializing and deserializing data without generating the code for schema is given below: package com.rishav.avro; import java.io.File; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; import java.util.Iterator; import java.util.LinkedHashMap; import org.apache.avro.Schema; import org.apache.avro.file.DataFileReader; import org.apache.avro.file.DataFileWriter; import org.apache.avro.generic.GenericData; import org.apache.avro.generic.GenericDatumReader; import org.apache.avro.generic.GenericDatumWriter; import org.apache.avro.generic.GenericRecord; import org.apache.avro.io.BinaryDecoder; import org.apache.avro.io.DatumReader; import org.apache.avro.io.DatumWriter; import org.codehaus.jackson.JsonFactory; import org.codehaus.jackson.JsonParseException; import org.codehaus.jackson.JsonProcessingException; import org.codehaus.jackson.map.ObjectMapper; import org.json.simple.JSONObject; public class AvroExampleWithoutCodeGeneration { public void serialize() throws JsonParseException, JsonProcessingException, IOException { InputStream in = new FileInputStream("resources/StudentActivity.json"); // create a schema Schema schema = new Schema.Parser().parse(new File("resources/StudentActivity.avsc")); // create a record to hold json GenericRecord AvroRec = new GenericData.Record(schema); // create a record to hold course_details GenericRecord CourseRec = new GenericData.Record(schema.getField("course_details").schema()); // this file will have AVro output data File AvroFile = new File("resources/StudentActivity.avro"); // Create a writer to serialize the record DatumWriter datumWriter = new GenericDatumWriter(schema); DataFileWriter dataFileWriter = new DataFileWriter(datumWriter); dataFileWriter.create(schema, AvroFile); // iterate over JSONs present in input file and write to Avro output file for (Iterator it = new ObjectMapper().readValues( new JsonFactory().createJsonParser(in), JSONObject.class); it.hasNext();) { JSONObject JsonRec = (JSONObject) it.next(); AvroRec.put("id", JsonRec.get("id")); AvroRec.put("student_id", JsonRec.get("student_id")); AvroRec.put("university_id", JsonRec.get("university_id")); LinkedHashMap CourseDetails = (LinkedHashMap) JsonRec.get("course_details"); CourseRec.put("course_id", CourseDetails.get("course_id")); CourseRec.put("enroll_date", CourseDetails.get("enroll_date")); CourseRec.put("verb", CourseDetails.get("verb")); CourseRec.put("result_score", CourseDetails.get("result_score")); AvroRec.put("course_details", CourseRec); dataFileWriter.append(AvroRec); } // end of for loop in.close(); dataFileWriter.close(); } // end of serialize method public void deserialize () throws IOException { // create a schema Schema schema = new Schema.Parser().parse(new File("resources/StudentActivity.avsc")); // create a record using schema GenericRecord AvroRec = new GenericData.Record(schema); File AvroFile = new File("resources/StudentActivity.avro"); DatumReader datumReader = new GenericDatumReader(schema); DataFileReader dataFileReader = new DataFileReader(AvroFile, datumReader); System.out.println("Deserialized data is :"); while (dataFileReader.hasNext()) { AvroRec = dataFileReader.next(AvroRec); System.out.println(AvroRec); } } public static void main(String[] args) throws JsonParseException, JsonProcessingException, IOException { AvroExampleWithoutCodeGeneration AvroEx = new AvroExampleWithoutCodeGeneration(); AvroEx.serialize(); AvroEx.deserialize(); } } For generating the schema java code from Avro json schema we can use avro-tools jar. The command for same is given below: java -jar avro-tools-1.7.5.jar compile schema StudentActivity.avsc Output path can be source folder for the project or we can add the generated java class files to Eclipse IDE manually. The java code for serializing and deserializing data with generating the code for schema is similar to above code except that in previous code we were assiging values to a GenericRecord and in this one we are assigning values to the generated Avro object: package com.rishav.avro; import java.io.File; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; import java.util.Iterator; import java.util.LinkedHashMap; import org.apache.avro.Schema; import org.apache.avro.file.DataFileReader; import org.apache.avro.file.DataFileWriter; import org.apache.avro.generic.GenericData; import org.apache.avro.generic.GenericDatumReader; import org.apache.avro.generic.GenericDatumWriter; import org.apache.avro.generic.GenericRecord; import org.apache.avro.io.DatumReader; import org.apache.avro.io.DatumWriter; import org.codehaus.jackson.JsonFactory; import org.codehaus.jackson.JsonParseException; import org.codehaus.jackson.JsonProcessingException; import org.codehaus.jackson.map.ObjectMapper; import org.json.simple.JSONObject; public class AvroExampleWithCodeGeneration { public void serialize() throws JsonParseException, JsonProcessingException, IOException { InputStream in = new FileInputStream("resources/StudentActivity.json"); // create a schema Schema schema = new Schema.Parser().parse(new File("resources/StudentActivity.avsc")); // create an object to hold json record StudentActivity sa = new StudentActivity(); // create an object to hold course_details Activity a = new Activity(); // this file will have AVro output data File AvroFile = new File("resources/StudentActivity.avro"); // Create a writer to serialize the record DatumWriter datumWriter = new GenericDatumWriter(schema); DataFileWriter dataFileWriter = new DataFileWriter(datumWriter); dataFileWriter.create(schema, AvroFile); // iterate over JSONs present in input file and write to Avro output file for (Iterator it = new ObjectMapper().readValues( new JsonFactory().createJsonParser(in), JSONObject.class); it.hasNext();) { JSONObject JsonRec = (JSONObject) it.next(); sa.setId((CharSequence) JsonRec.get("id")); sa.setStudentId((Integer) JsonRec.get("student_id")); sa.setUniversityId((Integer) JsonRec.get("university_id")); LinkedHashMap CourseDetails = (LinkedHashMap) JsonRec.get("course_details"); a.setCourseId((Integer) CourseDetails.get("course_id")); a.setEnrollDate((CharSequence) CourseDetails.get("enroll_date")); a.setVerb((CharSequence) CourseDetails.get("verb")); a.setResultScore((Double) CourseDetails.get("result_score")); sa.setCourseDetails(a); dataFileWriter.append(sa); } // end of for loop in.close(); dataFileWriter.close(); } // end of serialize method public void deserialize () throws IOException { // create a schema Schema schema = new Schema.Parser().parse(new File("resources/StudentActivity.avsc")); // create a record using schema GenericRecord AvroRec = new GenericData.Record(schema); File AvroFile = new File("resources/StudentActivity.avro"); DatumReader datumReader = new GenericDatumReader(schema); DataFileReader dataFileReader = new DataFileReader(AvroFile, datumReader); System.out.println("Deserialized data is :"); while (dataFileReader.hasNext()) { AvroRec = dataFileReader.next(AvroRec); System.out.println(AvroRec); } } public static void main(String[] args) throws JsonParseException, JsonProcessingException, IOException { AvroExampleWithoutCodeGeneration AvroEx = new AvroExampleWithoutCodeGeneration(); AvroEx.serialize(); AvroEx.deserialize(); } } In next post we will see how Avro deals with schema evolution.

Vert.x is a lightweight, high performance application platform for the JVM Modern web applications and the rise of mobile clients redefined what is expected from a web server. Node.js was the first technology that recognized the paradigm shift and offered a solution. The application platform Vert.x takes some of the innovations from Node.js and makes them available on the JVM, combining fresh ideas with one of the most sophisticated and fastest runtime environments available. Vert.x comes with a set of exciting features that make it interesting for anybody developing web applications. Non-blocking, event driven runtime Vert.x provides a non-blocking, event-driven runtime. If a server has to do a task that requires waiting for a response (e.g. requesting data from a database) there are two possibilities how this can be implemented: blocking and non-blocking. The traditional approach is a synchronous or blocking call. The program flow pauses and waits for the answer to return. To be able to handle more than one request in parallel, the server would execute each request in a different thread. The advantage is a relatively simple programming model, but the downside is a significant amount of overhead if the number of threads becomes large. The second solution is a non-blocking call. Instead of waiting for the answer, the caller continues execution, but provides a callback that will be executed once data arrives. This approach requires a (slightly) more complex programming model, but has a lot less overhead. In general a non-blocking approach results in much better performance when a large number of requests need to be served in parallel. Simple to use concurrency and scalability A Vert.x application consists of loosely coupled components, which can be rearranged to match increasing performance requirements Vert.x applications are written using an Actor-like concurrency model. An application consists of several components, the so-called Verticles, which run independently. A Verticle runs single-threaded and communicates with other Verticles by exchanging messages on the global event-bus. Because they do not share state, Verticles can run in parallel. The result is an easy to use approach for writing multi-threaded applications.You can create several Verticles which are responsible for the same task and the runtime will distribute the workload among them, which means you can take full advantage of all CPU cores without much effort. Verticles can also be distributed between several machines. This will be transparent to the application code. The Verticles use the same mechanisms to communicate as if they would run on the same machine. This makes it extremely easy to scale your application. Vert.x supports the most popular languages on the JVM. Support for Scala and Clojure is on the way. Polyglot Unlike many other application platforms, Vert.x is polyglot. Applications can be written in several languages. It is even possible to use different languages in the same application. At this point Java, Python, Groovy, Ruby, and JavaScript can be used and support for Scala and Clojure is on the way. Conclusion Vert.x is a relatively young platform and subsequently the ecosystem is not as rich as that of the more established platforms. Nevertheless for the most common tasks, there are extensions available.The advantages of Vert.x are astonishing. Its non-blocking, event-driven nature is extremely well-suited for modern web applications. Vert.x makes it easy to write concurrent applications that scale effortless from a single low-end machine to a cluster with several high-end servers. Add the fact that you can use most popular languages for the JVM and you have a web developers dream come true.

Java EE in general and Context and Dependency Injection has been part of the Vaadin ecosystem since ages. Recently, Spring Vaadin is a joint effort of the Vaadin and the Spring teams to bring the Spring framework into the Vaadin ecosystem, lead by Petter Holmström for Vaadin and Josh Long for Pivotal. Integration is based on the Spring Boot project - and its sub-modules, that aims to ease creating new Spring web projects. This article assumes the reader is familiar enough with Spring Boot. If not the case, please take some time to get to understand basic notions about the library. Note that at the time of this writing, there's no release for Spring Vaadin. You'll need to clone the project and build it yourself. The first step is to create the UI. In order to display usage of Spring's Dependency Injection, it should use a service dependency. Let's injection the UI through Constructor Injection to favor immutability. The only addition to a standard UI is to annotate it with org.vaadin.spring.@VaadinUI. @VaadinUI public class VaadinSpringExampleUi extends UI { private HelloService helloService; public VaadinSpringExampleUi(HelloService helloService) { this.helloService = helloService; } @Override protected void init(VaadinRequest vaadinRequest) { String hello = helloService.sayHello(); setContent(new Label(hello)); } } The second step is standard Spring Java configuration. Let's create two configuration classes, one for the main context and the other for the web one. Two thing of note: The method instantiating the previous UI has to be annotated with org.vaadin.spring.@UIScope in addition to standard Spring org.springframework.context.annotation.@Bean to bind the bean lifecycle to the new scope provided by the Spring Vaadin library. At the time of this writing, a RequestContextListener bean must be provided. In order to be compliant with future versions of the library, it's a good practice to annotate the instantiating method with @ConditionalOnMissingBean(RequestContextListener.class). @Configuration public class MainConfig { @Bean public HelloService helloService() { return new HelloService(); } } @Configuration public class WebConfig extends MainConfig { @Bean @ConditionalOnMissingBean(RequestContextListener.class) public RequestContextListener requestContextListener() { return new RequestContextListener(); } @Bean @UIScope public VaadinSpringExampleUi exampleUi() { return new VaadinSpringExampleUi(helloService()); } } The final step is to create a dedicated WebApplicationInitializer. Spring Boot already offers a concrete implementation, we just need to reference our previous configuration classes as well as those provided by Spring Vaadin, namely VaadinAutoConfiguration and VaadinConfiguration. public class ApplicationInitializer extends SpringBootServletInitializer { @Override protected SpringApplicationBuilder configure(SpringApplicationBuilder application) { return application.showBanner(false) .sources(MainConfig.class) .sources(VaadinAutoConfiguration.class, VaadinConfiguration.class) .sources(WebConfig.class); } } At this point, we demonstrated a working Spring Vaadin sample application. Code for this article can be browsed and forked on Github.

We all know what XML is right? Just in case not, no problem here is what it is all about. 5 Now, what the computer really needs is the number five and some context around it. In XML you (human and computer) can see how it represents context to five. Now lets say instead you have a business XML document like FPML 32.00 150000 1.00 EUR 405000 2001-07-17Z NONE EUR 2.70 ISDA2002 ISDA2002Equity TODO GBEN Party A Party B That is a lot of extra unnecessary data points. Now lets look at this using Apache Avro. With Avro, the context and the values are separated. This means the schema/structure of what the information is does not get stored or streamed over and over and over and over (and over) again. The Avro schema is hashed. So the data structure only holds the value and the computer understands the fingerprint (the hash) of the schema and can retrieve the schema using the fingerprint. 0x d7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592 This type of implementation is pretty typical in the data space. When you do this you can reduce your data between 20%-80%. When I tell folks this they immediately ask, “why such a large gap of unknowns”. The answer is because not every XML is created the same. But that is the problem because you are duplicating the information the computer needs to understand the data. XML is nice for humans to read, sure … but that is not optimized for the computer. Here is a converter we are working on https://github.com/stealthly/xml-avro to help get folks off of XML and onto lower cost, open source systems. This allows you to keep parts of your systems (specifically the domain business code) using the XML and not having to be changed (risk mitigation) but store and stream the data with less overhead (optimize budget).

In one of my previous posts I explained how we can convert json data to avro data and vice versa using avro tools command line option. Today I was trying to see what options we have for converting csv data to avro format, as of now we don't have any avro tool option to accomplish this . Now, we can either write our own java program (MapReduce program or a simple java program) or we can use various SerDe's available with Hive to do this quickly and without writing any code :) To convert csv data to Avro data using Hive we need to follow the steps below: Create a Hive table stored as textfile and specify your csv delimiter also. Load csv file to above table using "load data" command. Create another Hive table using AvroSerDe. Insert data from former table to new Avro Hive table using "insert overwrite" command. To demonstrate this I will use use the data below (student.csv): 0,38,91 0,65,28 0,78,16 1,34,96 1,78,14 1,11,43 Now execute below queries in Hive: --1. Create a Hive table stored as textfile USE test; CREATE TABLE csv_table ( student_id INT, subject_id INT, marks INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE; --2. Load csv_table with student.csv data LOAD DATA LOCAL INPATH "/path/to/student.csv" OVERWRITE INTO TABLE test.csv_table; --3. Create another Hive table using AvroSerDe CREATE TABLE avro_table ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' TBLPROPERTIES ( 'avro.schema.literal'='{ "namespace": "com.rishav.avro", "name": "student_marks", "type": "record", "fields": [ { "name":"student_id","type":"int"}, { "name":"subject_id","type":"int"}, { "name":"marks","type":"int"}] }'); --4. Load avro_table with data from csv_table INSERT OVERWRITE TABLE avro_table SELECT student_id, subject_id, marks FROM csv_table; Now you can get data in Avro format from Hive warehouse folder. To dump this file to local file system use below command: hadoop fs -cat /path/to/warehouse/test.db/avro_table/* > student.avro If you want to get json data from this avro file you can use avro tools command: java -jar avro-tools-1.7.5.jar tojson student.avro > student.json So we can easily convert csv to avro and csv to json also by just writing 4 HQLs.

A while back, I wrote an article showing how to Live Migrate Your VMs in One Line of Powershell between non-clustered Windows Server 2012 Hyper-V hosts using Shared Nothing Live Migration. Since then, I’ve been asked a few times for how this type of parallel Live Migration would be performed for highly available virtual machines between Hyper-V hosts within a cluster. In this article, we’ll walk through the steps of doing exactly that … via Windows PowerShell on Windows Server 2012 or 2012 R2 or our FREE Hyper-V Server 2012 R2 bare-metal, enterprise-grade hypervisor in a clustered configuration. Wait! Do I need PowerShell to Live Migrate multiple VMs within a Cluster? Well, actually … No. You could certainly use the Failover Cluster Manager GUI tool to select multiple highly available virtual machines, right-click and select Move | Live Migration … Failover Cluster Manager – Performing Multi-VM Live Migration But, you may wish to script this process for other reasons … perhaps to efficiently drain all VM’s from a host as part of a maintenance script that will be performing other tasks. Can I use the same PowerShell cmdlets for Live Migrating within a Cluster? Well, actually … No again. When VMs are made highly available resources within a cluster, they’re managed as cluster group resources instead of being standalone VM resources. As a result, we have a different set of Cluster-aware PowerShell cmdlets that we use when managing these cluster groups. To perform a scripted multi-VM Live Migration, we’ll be leveraging three of these cmdlets: Get-ClusterNode, Get-ClusterGroup and Move-ClusterVirtualMachineRole Now, let’s see that one line of PowerShell! Before getting to the point of actually performing the multi-VM Live Migration in a single PowerShell command line, we first need to setup a few variables to handle the "what" and "where" of moving these VMs. First, let’s specify the name of the cluster with which we’ll be working. We’ll store it in a $clusterName variable. $clusterName = read-host -Prompt "Cluster name" Next, we’ll need to select the cluster node to which we’ll be Live Migrating the VMs. Lets use the Get-ClusterNode and Out-GridView cmdlets together to prompt for the cluster node and store the value in a $targetClusterNode variable. $targetClusterNode = Get-ClusterNode -Cluster $clusterName | Out-GridView -Title "Select Target Cluster Node" ` -OutputMode Single And then, we’ll need to create a list of all the VMs currently running in the cluster. We can use the Get-ClusterGroup cmdlet to retrieve this list. Below, we have an example where we are combining this cmdlet with a Where-Object cmdlet to return only the virtual machine cluster groups that are running on any node except the selected target cluster node. After all, it really doesn’t make any sense to Live Migrate a VM to the same node on which it’s currently running! $haVMs = Get-ClusterGroup -Cluster $clusterName | Where-Object {($_.GroupType -eq "VirtualMachine") ` -and ($_.OwnerNode -ne $targetClusterNode.Name)} We’ve stored the resulting list of VMs in a $haVMs variable. Ready to Live Migrate! OK … Now we have all of our variables defined for the cluster, the target cluster node and the list of VMs from which to choose. Here’s our single line of PowerShell to do the magic … $haVMs | Out-GridView -Title "Select VMs to Move" –PassThru | Move-ClusterVirtualMachineRole -MigrationType Live ` -Node $targetClusterNode.Name -Wait 0 Proceed with care: Keep in mind that your target cluster node will need to have sufficient available resources to run the VM's that you select for Live Migration. Of course, it's best to initially test tasks like this in your lab environment first. Here’s what is happening in this single PowerShell command line: We’re passing the list of VMs stored in the $haVMs variable to the Out-GridView cmdlet. Out-GridView prompts for which VMs to Live Migrate and then passes the selected VMs down the PowerShell object pipeline to the Move-ClusterVirtualMachineRole cmdlet. This cmdlet initiates the Live Migration for each selected VM, and because it’s using a –Wait 0 parameter, it initiates each Live Migration one-after-another without waiting for the prior task to finish. As a result, all of the selected VMs will Live Migrate in parallel, up to the maximum number of concurrent Live Migrations that you’ve configured on these cluster nodes. The VMs selected beyond this maximum will simply queue up and wait their turn. Unlike some competing hypervisors, Hyper-V doesn't impose an artificial hard-coded limit on how many VMs for you can Live Migrate concurrently. Instead, it's up to you to set the maximum to a sensible value based on your hardware and network capacity. Do you have your own PowerShell automation ideas for Hyper-V? Feel free to share your ideas in the Comments section below. See you in the Clouds! - Keith

JDave, Concordion, Easyb, JBehave, Cucumber are all compared here briefly for your convenience.

This morning an interesting topic was posted to the Apache ServiceMix user forum, asking the question: To ServiceMix or not ServiceMix. In my mind the short answer is: NO Guillaume Nodet one of the key architects and long time committer on Apache ServiceMix already had his mind set 3 years ago when he wrong this blog post - Thoughts about ServiceMix. What has happened on the ServiceMix project was that the ServiceMix kernel was pulled out of ServiceMix into its own project - Apache Karaf. That happened in spring 2009, which Guillaume also blogged about. So is all that bad? No its IMHO all great. In fact having the kernel as a separate project, and Camel and CXF as the integration and WS/RS frameworks, would allow the ServiceMix team to focus on building the ESB that truly had value-add. But that did not happen. ServiceMix did not create a cross product security model, web console, audit and trace tooling, clustering, governance, service registry, and much more that people were looking for in an ESB (or related to a SOA suite). There were only small pieces of it, but never really baked well into the project. That said its not too late. I think the ServiceMix project is dying, but if a lot of people in the community step up, and contribute and work on these things, then it can bring value to some users. But I seriously doubt this will happen. PS: 6 years ago I was working as a consultant and looked at the next integration platform for a major Danish organization, and we looked at ServiceMix back then and dismissed it due its JBI nature, and the new OSGi based architecture was only just started. And frankly it has taken a long long time to mature Apache Karaf / Felix / Aries and the other pieces in OSGi to what they are today to offer a stable and sound platform for users to build their integration applications. That was not the case 4-6 years ago. Okay No to ServiceMix - what are my options then? So what should use you instead of ServiceMix? Well in my mind you have at least these two options. 1) Use Apache Karaf and add the pieces you need, such as Camel, CXF, ActiveMQ and build your own ESB. These individual projects have regular releases, and you can upgrade as you need. The ServiceMix project only has the JBI components in additional, that you should NOT use. Only legacy users that got on the old ServiceMix 3.x wagon may need to use this in a graceful upgrade from JBI to Karaf based containers. 2) Take a look at fabric8. IMHO fabric8 is all that value-add the ServiceMix project did not create, and a lot more. James Strachan, just blogged today about some of his thoughts on fabric8, JBoss Fuse, and Karaf. I encourage you to take a read. For example he talks about how fabric becomes poly container, so you have a much wider choice of which containers/JVM to run your integration applications. OSGi is no longer a requirement. (IMHO that is very very existing and potentially a changer). I encourage you to check out fabric8 web-site, and also read the overview and motivation sections of the documentation. And then check out some of the videos. After the upcoming JBoss Fuse 6.1 release, the Fuse team at Red Hat will have more time and focus to bring the documentation at fabric8 up to date covering all the functionality we have (there is a lot more), and as well bring out a 1.0 community released using pure community releases. This gives end users a 100% free to use out of the box release. And users looking for a commercial release can then use JBoss Fuse. Best of both worlds. Summary Okay back to the question - to ServiceMix or not. Then NO. Innovation happens outside ServiceMix, and also more and more outside Apache. If you have thoughts then you can share those in comments to this blog, or better yet, get involved in the discussion forum at the ServiceMix user forum. PPS: The thoughts on this blog is mine alone, and are not any official words from my employer.

This article was originally written by Jeff Morris In the introduction to this series, I discussed some of the motivation for rewriting .NET SDK, the goals, objectives and the major features of the upcoming 2.0 release, and we examined the high-level architecture (10,000 feet view) of a Couchbase Server Client SDK. In this post we will go over the design and development of one of the core configuration components of a Couchbase SDK: Server Configuration. Introduction A Couchbase SDK client requires configuration from two sources: the Client Configuration, which defines the IP of the cluster to connect to, number of connections to use and other important information regarding how the client will interact with the cluster, and the Server Configuration, which defines the current state of the cluster (e.g. number of nodes, buckets that are available, etc.), thus driving the internal state of a client (Cluster Map) This post will only discuss the Server Configuration aspects and will largely revolve around implementing several well-defined interfaces or contracts. HTTP Streaming Configuration Currently, most clients use a “bootstrapping” technique via client configuration and a “Streaming Configuration” exposed by the Couchbase REST API. This is supported by versions of Couchbase from 2.2 and back. The usual approach is as follows: Within the “uris” element of a Client Configuration (semantics very per client), a URL is defined for which to start the bootstrapping process: http://[SERVER]:8091/pools The response is then parsed and the a request is made to get the buckets configuration: http://[SERVER]:8091/pools/default?uuid=[UUID] This response is parsed and another request is made to get streaming URL from: http://[SERVER]:8091/pools/default/buckets?v=[VERSION]&uuid=[UUID] Finally, the streaming URL connection is made which is long-lived and raises events in the client with respect to changes in the cluster: http://[SERVER]:8091/pools/default/bucketsStreaming/default?bucket_uuid=[UUID] The client will then change its internal state to match that of the current server configuration. There are some problems with this approach, among others: The “streaming URL” is resource intensive to create and maintain (mainly memory) on the server-side During a rebalance or failover situation, the cluster configuration may change many, many times. Each time this happens the client must tear down all of its resources (socket connections, VBucket mappings) and build its state up again and again, which can leads to reduced throughput, latency, higher than expected memory and CPU usage, and so on and so forth… Operations that are in-flight may be terminated and then re-tried on a new config state – it’s as if the “carpet has been pulled out from underneath them”. Responding to NOT_MY_VBUCKET responses are handled in-efficiently by simple trying the next node in the list – there is no information to help the client in which node to re-direct the operation to. A New Model for Configuration Management: CCCP While the streaming HTTP “bootstrapping” approach has worked reasonably well for most clients, the downsides have begun to outweigh the plusses, thus a new model for updating client configuration has been defined is available starting with the 2.5 version of the Couchbase Server: Client Cluster Configuration Publication or “CCCP”. CCCP introduces a new operation to be used before or after authentication to request configuration as well as a mechanism for returning configuration information when a NOT_MY_VBUCKET response is returned for a failed operation. In this case CCCP supporting SDK, the client will react by using the configuration to update itself before resending the operation. Note that a NOT_MY_VBUCKET is the standard response that is returned by the cluster when the cluster itself has changed (during a rebalance or failover scenario for example) and the client has not yet “synched” up and is using a stale configuration, resulting in an invalid key mapping. Whereas the “bootstrapping” approach is somewhat of a “pull” type operation, CCCP is either “push” or “pull” depending upon whether the request was initiated by the client (via an explicit CMD_GET_CLUSTER_CONFIG operation) or by the server itself (via a NOT_MY_VBUCKET response to an operation). We will go over CCCP in more detail in a later post. File Based Configuration One other semi-supported configuration option exists: file based configuration. File based configuration is primarily useful for testing and development and we will provide an implementation in the test projects to remove some of the dependencies that are difficult to replicate and or cause false positives when running the test suite. Structural Architecture View Internally the Server Configuration component of the client is a provider based model, in which multiple implementations of a configuration provider can be configured in the client and then a strategy can be chosen to determine which provider should be used. The default is a simple linear, fallback approach where the first configured provider is used and then if it fails the next provider in sequence will take its place. Here is a diagram showing the main actor objects and the relationships with some of other key objects within the client which will be discussed in subsequent posts: A description of each follows: ConfigurationProvider: a source which shall yield a new ConfigInfo. It’s the responsibility of the provider to provide the mechanism for fetching the configuration from its source. ConfigurationInformation: the configuration info contains a list of possible nodes and the VBucket map informing clients about which servers within said nodes a given key should be forwarded to. ConfigurationManager: bridge between the client and the providers and the strategy taken to determine which provider to use and what retry logic to apply. A more detailed document of this architecture can be found here. Please note that this, like all development, is an evolutionary process, so expect some changes and revisions over time. Conclusion and Next Steps This post discussed the history (HTTP Streaming) and the future (CCCP) of Couchbase SDK Server Configuration Management. In the next post we will go into detail the implementation of the HTTP Streaming configuration provider which is required for clients targeting pre-2.5 versions of the Couchbase Server.

At the end of last year I was playing around with running scheduled tasks to monitor a Neo4j cluster and one of the problems I ran into was that the monitoring would sometimes exit. I eventually realised that this was because a RuntimeException was being thrown inside the Runnable method and I wasn’t handling it. The following code demonstrates the problem: import java.util.ArrayList; import java.util.List; import java.util.concurrent.*; public class RunnableBlog { public static void main(String[] args) throws ExecutionException, InterruptedException { ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(new Runnable() { @Override public void run() { System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis()); throw new RuntimeException("game over"); } }, 0, 1000, TimeUnit.MILLISECONDS).get(); System.out.println("exit"); executor.shutdown(); } } If we run that code we’ll see the RuntimeException but the executor won’t exit because the thread died without informing it: Exception in thread "main" pool-1-thread-1 -> 1391212558074 java.util.concurrent.ExecutionException: java.lang.RuntimeException: game over at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at RunnableBlog.main(RunnableBlog.java:11) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120) Caused by: java.lang.RuntimeException: game over at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) At the time I ended up adding a try catch block and printing the exception like so: public class RunnableBlog { public static void main(String[] args) throws ExecutionException, InterruptedException { ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis()); throw new RuntimeException("game over"); } catch (RuntimeException e) { e.printStackTrace(); } } }, 0, 1000, TimeUnit.MILLISECONDS).get(); System.out.println("exit"); executor.shutdown(); } } This allows the exception to be recognised and as far as I can tell means that the thread executing the Runnable doesn’t die. java.lang.RuntimeException: game over pool-1-thread-1 -> 1391212651955 at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) pool-1-thread-1 -> 1391212652956 java.lang.RuntimeException: game over at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) pool-1-thread-1 -> 1391212653955 java.lang.RuntimeException: game over at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) This worked well and allowed me to keep monitoring the cluster. However, I recently started reading ‘Java Concurrency in Practice‘ (only 6 years after I bought it!) and realised that this might not be the proper way of handling the RuntimeException. public class RunnableBlog { public static void main(String[] args) throws ExecutionException, InterruptedException { ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis()); throw new RuntimeException("game over"); } catch (RuntimeException e) { Thread t = Thread.currentThread(); t.getUncaughtExceptionHandler().uncaughtException(t, e); } } }, 0, 1000, TimeUnit.MILLISECONDS).get(); System.out.println("exit"); executor.shutdown(); } } I don’t see much difference between the two approaches so it’d be great if someone could explain to me why this approach is better than my previous one of catching the exception and printing the stack trace.

Learn how to set up a four node Hadoop cluster using AWS EC2, PuTTy(gen), and WinSCP.

Get started with Spring IDE and the Spring Tool Suite – a set of plugins to simplify the development of Spring-based applications in Eclipse.

A typical Maven and Spring web application has a fair amount of XML and verbosity to it. Add in Jersey and Spring Security and you can have hundreds of lines of XML before you even start to write your Java code. As part of a recent project, I was tasked with upgrading a webapp like this to use Spring 4 and Spring Boot. I also figured I'd try to minimize the XML. This is my story on how I upgraded to Spring 4, Jersey 2, Java 8 and Spring Boot 0.5.0 M6. When I started, the app was using Spring 3.2.5, Spring Security 3.1.4 and Jersey 1.18. The pom.xml had four Jersey dependencies, three Spring dependencies and three Spring Security dependencies, along with a number of exclusions for "jersey-spring". Upgrading to Spring 4 Upgrading to Spring 4 was easy, I changed the version property to 4.0.0.RC2 and added the new Spring bill of materials to my pom.xml. I also add the Spring milestone repo since Spring 4 won't be released to Maven central until tomorrow. org.springframework spring-framework-bom ${spring.framework.version} pom import spring-milestones http://repo.spring.io/milestone true Next, I removed all the references to ${spring.framework.version} in dependencies since it'd be controlled by Maven's dependency management feature. org.springframework spring-web - ${spring.framework.version} I also changed to use Maven 3's wildcard syntax to exclude multiple dependencies. com.sun.jersey.contribs jersey-spring org.springframework - spring - - - org.springframework - spring-core - - - org.springframework - spring-web - - - org.springframework - spring-beans - - - org.springframework - spring-context + * I confirmed the upgrade worked by running "mvn dependency:tree | grep spring", followed by "mvn jetty:run" and viewing the app in my browser. Upgrading to Jersey 2 The next item I tackled was upgrading to Jersey 2.4.1. I changed the version number in my pom.xml, then added the Jersey BOM. org.glassfish.jersey jersey-bom ${jersey.version} pom import You might ask "why Jersey?" if we already have Spring MVC and its REST support? You might also ask why not Play or Grails instead of a Java + Spring stack? For this particular project, I recommended technology options, and these were certainly among them. However, the team chose differently and I support their decision. The project is creating an iOS app, as well as a responsive HTML5 mobile/desktop app. We figured we had enough risk with new technologies on the front-end that we should play it a bit safer on the backend. To make the backend work a bit sexier, we've decided to allow Spring 4, Java 8 and possibly some reactive principles. Next, I changed from the old com.sun.jersey dependencies to org.glassfish.jersey and removed jersey-spring. org.glassfish.jersey.containers jersey-container-servlet org.glassfish.jersey.media jersey-media-json-jackson The last thing I needed to do was change the servlet-class and param-name in web.xml: jersey-servlet org.glassfish.jersey.servlet.ServletContainer jersey.config.server.provider.packages com.raibledesigns.boot.service 1 Requiring Java 8 Requiring Java 8 to compile was easy enough. I added the maven-compiler-plugin to enforce a minimum version. maven-compiler-plugin 3.1 1.8 1.8 I downloaded the latest Java 8 SDK and installed it. Then I set my JAVA_HOME to use it. export JAVA_HOME=`/usr/libexec/java_home -v 1.8` Integrating Spring Boot I learned about Spring Boot a few weeks ago at Devoxx. Josh Long gave me a 3-minute demo at the speaker's dinner and showed me enough to peak my interest. To integrate it into my project, I started with the Quick Start. I added the boot-parent, dependencies for web, security and actuator (logging, metrics, etc.) and the Maven plugin. I removed all the Spring and Spring Security dependencies. org.springframework.boot spring-boot-starter-parent 0.5.0.M6 ... spring-milestones http://repo.spring.io/milestone ... org.springframework.boot spring-boot-starter-web org.springframework.boot spring-boot-starter-security org.springframework.boot spring-boot-starter-actuator ... org.springframework.boot spring-boot-maven-plugin Upon restarting my app, I got an error about spring-security.xml using a 3.1 XSD. I fixed it by changing to 3.2. Next, I wanted to eliminate web.xml. First of all, I created an ApplicationInitializer so the WAR could be started from the command line. package com.raibledesigns.boot.config; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.boot.builder.SpringApplicationBuilder; import org.springframework.boot.web.SpringBootServletInitializer; import org.springframework.context.annotation.ComponentScan; import org.springframework.context.annotation.Configuration; @Configuration @EnableAutoConfiguration @ComponentScan public class ApplicationInitializer extends SpringBootServletInitializer { @Override protected SpringApplicationBuilder configure(SpringApplicationBuilder application) { return application.sources(ApplicationInitializer.class); } public static void main(String[] args) { SpringApplication.run(ApplicationInitializer.class, args); } } However, after adding this, I received the following error on startup: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor': Invocation of init method failed; nested exception is java.lang.AbstractMethodError: org.hibernate.validator.internal.engine.ConfigurationImpl .getDefaultParameterNameProvider()Ljavax/validation/ParameterNameProvider; Adding hibernate-validator as a dependency solved this problem: org.hibernate hibernate-validator To configure Spring Security without web.xml and spring-security.xml, I created WebSecurityConfig.java: package com.raibledesigns.boot.config; import org.springframework.context.annotation.Configuration; import org.springframework.core.Ordered; import org.springframework.core.annotation.Order; import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder; import org.springframework.security.config.annotation.web.builders.HttpSecurity; import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity; import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter; @Configuration @EnableWebSecurity @Order(Ordered.LOWEST_PRECEDENCE - 6) public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http.authorizeRequests() .antMatchers("/", "/home").permitAll() .antMatchers("/v1.0/**").hasRole("USER") .anyRequest().authenticated(); http.httpBasic().realmName("My API"); } @Override protected void configure(AuthenticationManagerBuilder authManagerBuilder) throws Exception { authManagerBuilder.inMemoryAuthentication() .withUser("test").password("test123").roles("USER"); } } To configure Jersey without web.xml, I created a JerseyConfig class: package com.raibledesigns.boot.config; import org.glassfish.jersey.filter.LoggingFilter; import org.glassfish.jersey.jackson.JacksonFeature; import org.glassfish.jersey.server.ResourceConfig; import org.glassfish.jersey.server.ServerProperties; import javax.ws.rs.ApplicationPath; @ApplicationPath("/v1.0") public class JerseyConfig extends ResourceConfig { public JerseyConfig() { packages("com.raibledesigns.boot.service"); property(ServerProperties.BV_SEND_ERROR_IN_RESPONSE, true); property(ServerProperties.JSON_PROCESSING_FEATURE_DISABLE, false); property(ServerProperties.MOXY_JSON_FEATURE_DISABLE, true); property(ServerProperties.WADL_FEATURE_DISABLE, true); register(LoggingFilter.class); register(JacksonFeature.class); } } Finally, I created MvcConfig.java to set the welcome page. package com.raibledesigns.boot.config; import org.springframework.context.annotation.Configuration; import org.springframework.web.servlet.config.annotation.ViewControllerRegistry; import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter; @Configuration public class MvcConfig extends WebMvcConfigurerAdapter { @Override public void addViewControllers(ViewControllerRegistry registry) { registry.addViewController("/").setViewName("index"); } } To cleanup, I deleted src/main/webapp/WEB-INF and created src/main/resources/logback.xml: Since Boot doesn't support JSP out-of-the-box, I renamed my index.jsp file to index.html and changed the URL in it to point to "/v1.0/hello". I was pleased to see that everything worked nicely. I learned shortly after that I could remove the Spring BOM since Spring Boot uses a property to control its Spring version. The only issue I found is when started the app with "mvn package && java -jar target/app.war", it failed to initialize Jersey. I tried adding a @Bean for the servlet: @Bean public ServletRegistrationBean jerseyServlet() { ServletRegistrationBean registration = new ServletRegistrationBean(new ServletContainer(), "/v1.0/*"); registration.addInitParameter(ServletProperties.JAXRS_APPLICATION_CLASS, JerseyConfig.class.getName()); return registration; } Unfortunately, when running it using "java -jar", I get the following error: org.glassfish.hk2.api.MultiException: A MultiException has 1 exceptions. They are: 1. org.glassfish.jersey.server.internal.scanning.ResourceFinderException: java.io.FileNotFoundException: /.../target/app.war!/WEB-INF/classes (No such file or directory) at org.jvnet.hk2.internal.Utilities.justCreate(Utilities.java:869) at org.jvnet.hk2.internal.ServiceLocatorImpl.create(ServiceLocatorImpl.java:814) at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:906) at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:898) at org.glassfish.jersey.server.ApplicationHandler.createApplication(ApplicationHandler.java:300) at org.glassfish.jersey.server.ApplicationHandler.(ApplicationHandler.java:279) at org.glassfish.jersey.servlet.WebComponent.(WebComponent.java:302) This seems strange since there is a WEB-INF/classes in my WAR. Regardless, this is not a Boot problem per se, but more of a Jersey issue. From one of the Boot developers: The whole idea with Boot is that servlets are just a transport - they are a means to an end, and hopefully not the only one - the "container" is Spring, not the servlet container. We probably could add some form of support for SCI but only by hacking the containers since the spec really doesn't allow for much control of their lifecycle. It hasn't been a priority so far. Summary I hope this article is useful to see how you to upgrade your Java webapps to use Spring 4 and Spring Boot. I've created a boot-makeover project on GitHub with all the code mentioned. You can also view the commits for each step.

when you build applications based on mule ee (enterprise edition) and you are using maven to build your projects, you will notice you have dependencies to libraries that are not available in the public maven repos. to add these libraries to your local maven repo the mule distribution comes with a script ‘populate_m2_repo’ which is described here how to use it. now that is okay if you are the only developer and you are running your continuous integration on your local machine. in my case we are using artifactory as our company maven repository and also our build server is using it as the maven repo. so what i wanted was not to populate my local repository but the artifactory instance with all mule libraries. to do so i did two things: first make sure that maven is authorised to add libraries to artifactory. you can do this by adding the following to your settings.xml: artifactory admin password second step is to modify the original ‘populate_m2_repo.groovy’ script. replace the following line: mvn(["install:install-file", "-dgroupid=${project.groupid}", "-dartifactid=${project.artifactid}", "-dversion=${version}", "-dpackaging=pom", "-dfile=${localpom.canonicalpath}"]) with mvn(["deploy:deploy-file", "-dgroupid=${project.groupid}", "-dartifactid=${project.artifactid}", "-dversion=${version}", "-dpackaging=pom", "-dfile=${localpom.canonicalpath}", "-drepositoryid=arti", "-durl=http://localhost:8080/artifactory/libs-release-local" ]) and do the same for the line: def args = ["install:install-file", "-dgroupid=${pomprops.groupid}", "-dartifactid=${pomprops.artifactid}", "-dversion=${pomprops.version}", "-dpackaging=jar", "-dfile=${f.canonicalpath}", "-dpomfile=${localpom.canonicalpath}"] by replacing it with: def args = ["deploy:deploy-file", "-dgroupid=${pomprops.groupid}", "-dartifactid=${pomprops.artifactid}", "-dversion=${pomprops.version}", "-dpackaging=jar", "-dfile=${f.canonicalpath}", "-dpomfile=${localpom.canonicalpath}", "-drepositoryid=arti", "-durl=http://localhost:8080/artifactory/libs-release-local" ] now you can run the script with: ./populate_m2_repo bla as you can see it doesn’t really matter what you supply as m2_repo_home here because the libraries are uploaded to artifactory anyway. if you want you can replace the hardcoded url for artifactory in the script with the supplied parameter but in my case this solution was sufficient

there are a lot of design patterns, architectures and design concepts that can be applied to technical aspects of implementing a product. for exapmle, we have mvc architecture that isolates the view, controller and model of the application. we have a factory pattern that defines how to create objects and so on. yet, these all limit themselves to the technical aspects of the product, while there exists no definite pre-defined architecture or patterns to design and build "business functions". the plug and play architecture i want to describe in this article is a "functional architecture", that defines a pattern to design the business functionality to get the most out of it. modular architecture is a design technique where functionality of a program is separated into independent, interchangeable modules, such that each module contains everything necessary to execute one aspect of the desired functionality. typically, in modular architecture, separation is done based on technical aspects. for eg., a module is created for database interaction, another is created for logging and so on. the advantage of a modular architecture is that you can easily replace or add components without affecting the rest of the application. the modules clearly define the interfaces that are used to interact with the module and these are tied at compile time to other modules. what if we could extend this architecture to "business functionality" rather than just limiting it to technical aspects of a product. we already design "business functionality" in a modular manner. but, we do not consciously look at it as modules with clear-cut interfaces for inputs and outputs. defining the plug and play architecture the plug and play architecture extends the techniques of modular architecture to business functions. for eg., a module can be created to encompass all functions related to order, while another can be created to encompass all functions related to quote. i will call these independent modules implementing business functions as flows. an application, in this architecture, is defined as a "collection of flows working together". the same advantage of modular architecture applies here, i.e., you can replace, add or remove flows without affecting the rest of the application. an added advantage to extending modules to business functions, is that the flows can be combined in different paths to get different business functionality. further, extending this definition to be able to add, remove or replace modules at "runtime" rather than tying them together at compile time gives us the flexibility to customize the application at runtime without "in-built flags and if clauses". the beauty of this extension of the architecture is that "it gives us the flexibility to create a set of loosely coupled flows that can be bound together at runtime as opposed to coding one tightly integrated application that is rigid and not malleable". to further see how this architecture works, let us consider a set of loosely coupled flows coded as per the diagram: there are 5 flows coded and deployed. each flow exposes an output and accepts a set of inputs. for eg., the "product listing" flow encapsulates the following business functions create products manually by admin import products by admin provide services to view, search and list products expose an output, "product" data that can be linked to other flows another eg., is the cart flow. this encapsulates the following business functions create a new cart add items to the cart check out the cart expose an output, "cart" data that can be linked to other flows accept an input, "cart item" that can be linked to the output from other flows we can connect the output "product" data from "product listing" flow to the "cart item" input of the cart flow. this is not tightly coupled at compile time and is left as an open gate or an input for the cart flow. at runtime the gate is closed or left open as required by the customer. from the diagram we see that we have 5 gates that can be closed. leading us to create different applications at runtime by closing the correct gates. for eg., we can just enable "product listing" for a customer which provides just the basic features of product listing. gates 4 and 6 can be closed to create an application with the features create, list products add products to a cart send an enquiry for the products added to the cart in another variation instead of gates 4 and 6, gate 7 can be closed. the application now has the features create, list products send an enquiry for single products by plugging the "inputs" of flows with different "outputs" from other flows, varied applications can be created. this is called a plug and play architecture. smart is implemented in this architecture to provide a highly flexible container to build saas products. need for plug and play architecture when a product is exposed as a multi-tenanted saas product, it cannot be created with "one size fits all" tenet. to break this tenet we need a flexible, customizable runtime environment where features can be varied based on a customer requirement without affecting other customers serviced by the same application. the plug and play architecture provides this flexibility to the saas.