In database design classes in Computer Science, we learn that normalization is a good thing. And it certainly is a good thing, for databases. In the case of APIs, it is a different story. If a client must do multiple GETs to obtain the data it needs, or multiple PUTs or POSTs to send up data, just because your database happens to be normalized, then something is wrong. One of the functions of an API Server is to de-normalize your data so that clients are spared from making extra REST API calls, with all of the overhead which goes with that. Mugunth Kumar explains this very well in this excellent presentation, using Twitter as an example. When you do a GET on a tweet, it not only returns you the Tweet itself, but also other information (e.g. description of the Twitter user who sent the tweet). This saves the API client (often a mobile app) from making another request for that data. Effectively, the API Server has gathered up that data, which may come from different database tables, and de-normalized it for the response. You can try it out yourself here, by looking at the JSON which comes back from this Twitter API GET the most recent Tweet from my timeline. Many Vordel customers are using the API Server to gather together the data which is returned to the API clients, often taking this data from multiple sources (not only databases, but also message queues and even from other APIs). This data is then amalgamated into single JSON or XML structures. It often then cached at the API Server, in this structure. In this way, clients are spared from doing multiple calls, and instead (like the Twitter API example above) get the data they need in one request, or can PUT or POST up data in one action, rather than piecemeal. De-normalization is key to this process, and is one of the great benefits of an API Server.

Spock is a BBD testing framework that allows for easy BDD tests to be written. The framework is an extension upon JUnit which allows for easy IDE integration and using existing JUnit functionality. Spock tests are written in Groovy and can be used for writing a wide range of tests from small unit tests to full application integration tests. Without going into too much detail on how to write Spock based tests (see below for a few excellent links), lets go through how we can use the framework to build integration tests for testing a RESTful API. Our first RESTful API Test package com.wolfware.integration import groovyx.net.http.RESTClient import spock.lang.* import spock.lang.Specification import com.movideo.spock.extension.APIVersion import com.movideo.spock.extension.EnvironmentEndPoint @APIVersion(minimimApiVersion="1.0.0.0") class GetAuthenticationToken extends Specification { @EnvironmentEndPoint protected def environmentHost def "Get authentication token XML from API for valid account"() { given: "a valid account" def authenticationTokenRequestParams = ['key':"AAABBBCCC123", 'user':"myauthemail@bla.com"] and: "a client to get the authentication token XML" def client = new RESTClient(environmentHost) when: "we attempt to retrieve authentication token XML" def resp = client.get(path : "/authenticate", query : authenticationTokenRequestParams) then: "we should get a valid authentication token XML response" assert resp.data.token.isEmpty() == false // lots more asserts } } As you can see, apart from the @APIVersion and @EnvironmentEndPoint annotations (these are Spock extensions as explained later), the spec is a fairly simple Spock test. This specification has a feature that, as the name suggests, gets a authentication token in XML format and validates it. Lets look at each step: Given The url parameters required to get a authentication token from the RESTful service When using the Groovy RestClient to call the RESTful service for the authentication token details Then We can assert all the details of the response. The thing I really like about Spock is the readability of the tests. From the name being a descriptive sentence rather than some short hand with _ throughout to make a valid method name to being able to easily see where setup of the test is done and then the expectations and assertions. Trying to test any environment RESTful service I've found that when trying to write integration tests, there has either been: Hard coded environment details and the code branched for each environment making it near impossible to keep code in sync as merge hell becomes the norm. Config files that define the environment are used to define environment details, again checked into each branch for each environment. Trying to follow the principles of continuous delivery, it would be great to be able to use the same code base to test against any environment. This is where Spock Extensions come into play to help us out. Spock Extensions In short Spock allows us to extend it to perform other functionality during the test life-cycle (a great post on extensions can be read on this excellent blog post). I've developed two extensions which help to make the idea of running the same test suite across different environments easier. The @EnvironmentEndPoint Extension The aim of this Spock extension is to have a placeholder variable in code that at run-time, can be defined with the environment host of the RESTful services that we want to test. package com.movideo.runtime.extension.custom import org.apache.commons.logging.Log import org.apache.commons.logging.LogFactory import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension import org.spockframework.runtime.extension.AbstractMethodInterceptor import org.spockframework.runtime.extension.IMethodInvocation import org.spockframework.runtime.model.FieldInfo import org.spockframework.runtime.model.SpecInfo /** * Spock Environment Annotation Extension */ class EnvironmentEndPointExtension extends AbstractAnnotationDrivenExtension { private static final Log LOG = LogFactory.getLog(getClass()); private static def config = new ConfigSlurper().parse(new File('src/test/resources/SpockConfig.groovy').toURL()) /** * env environment variable * * Defaults to {@code LOCAL_END_POINT} */ private static final String envString = System.getProperties().getProperty("env", config.envHost); static { LOG.info("Environment End Point [" + envString + "]") } /** * {@inheritDoc} */ @Override void visitFieldAnnotation(EnvironmentEndPoint annotation, FieldInfo field) { def interceptor = new EnvironmentInterceptor(field, envString) interceptor.install(field.parent.getTopSpec()) } } /** * * Environment Intercepter * */ class EnvironmentInterceptor extends AbstractMethodInterceptor { private final FieldInfo field private final String envString EnvironmentInterceptor(FieldInfo field, String envString) { this.field = field this.envString = envString } private void injectEnvironmentHost(target) { field.writeValue(target, envString) } @Override void interceptSetupMethod(IMethodInvocation invocation) { injectEnvironmentHost(invocation.target) invocation.proceed() } @Override void install(SpecInfo spec) { spec.setupMethod.addInterceptor this } } The EnvironmentEndPointExtension class defines the following: config: is a ConfigSlurper that parses a config file 'SpockConfig.groovy' that is used to define the default environment host (envHost) envString: gets the value of 'env' from all System Properties (these include run-time properties) and defaults to config.envHost With the environment host able to be accessed by Spock, now we need to inject this into the placeholder variable for Spock tests to access. An interceptor is created which is used to inject(field.writeValue method) the value of the environment host into the placeholder variable. This placeholder is the one that the @EnvironmentEndPoint is annotating. When the test is run, the interceptor sets the placeholder variable and the test can then use this value as the host for the RestClient object. When running the Spock tests either the default value from the config file will be used or the JVM argument -Denv=? can be used. This makes running the same test code base against any environment so much easier. A note on Gradle builds. By default, Gradle will not pass through JVM arguments through to forked processes such as running tests. The code snippet below shows how to achieve this: /* * Required to pass all system properties to Test tasks. * Not default for Gradle to pass system properties through to forked processes. */ tasks.withType(Test) { def config = new ConfigSlurper().parse(new File('src/test/resources/SpockConfig.groovy').toURL()) systemProperty 'env', System.getProperty('env', config.envHost) } This allows all tasks that are a type of 'Test' to have some custom code run. In this case, we are defining the 'SpockConfig.groovy' config file and then setting 'systemPropery' within Gradle Test tasks to 'env' and either getting the value from the passed in JVM argument or from the config file. With this code in the build.gradle, we're able to run all tests via a Gradle test build, which will produce lovely test reports (in Gradle HTML and JUnit XML). The @APIVersion Extension Another integration testing problem I've found is that if we try and develop our tests first (or at least during the process of developing a feature or bug fix) that running the same tests against an environment that doesn't yet have the new code base (but we are using the same test code base everywhere), we'll have failing tests that aren't really failures as the new code isn't there yet. To help solve this problem, I've developed the @APIVersion extension to help with this issue. As newly developed code should be deployed with a new version, we can use this version to compare to a minimum version that a test can be run against. package com.movideo.runtime.extension.custom import groovyx.net.http.RESTClient import java.lang.annotation.Annotation import java.util.regex.Pattern import org.apache.commons.logging.Log import org.apache.commons.logging.LogFactory import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension import org.spockframework.runtime.model.FeatureInfo import org.spockframework.runtime.model.SpecInfo /** * API Version Extension * */ class APIVersionExtension extends AbstractAnnotationDrivenExtension { /** * Logger */ private static final Log LOG = LogFactory.getLog(getClass()); /** * */ private static def config = new ConfigSlurper().parse(new File('src/test/resources/SpockConfig.groovy').toURL()) /** * env environment variable * * Defaults to {@code LOCAL_END_POINT} */ private static final String envString = System.getProperties().getProperty("env", config.envHost); /** * Version REGX pattern */ private static final def VERSION_PATTERN = Pattern.compile(".", Pattern.LITERAL); /** * Max version length */ private static final def MAX_VERSION_LENGTH = 4; /** * Current API Version */ private static final def CURRENT_API_VERSION = getDeployedAPIVersion(); /** * {@inheritDoc} */ @Override void visitFeatureAnnotation(APIVersion annotation, FeatureInfo feature) { if(!isApiVersionGreaterThanMinApiVersion(annotation, feature.name)) { feature.setSkipped(true) } } /** * {@inheritDoc} */ @Override public void visitSpecAnnotation(APIVersion annotation, SpecInfo spec) { if(!isApiVersionGreaterThanMinApiVersion(annotation, spec.name)) { spec.setSkipped(true) } } /** * Get the current deployed API version * * Performs a HTTP request to the current deployed API version. Parses the returned data and get the {@code version} node data. * @return current deployed API version */ private static String getDeployedAPIVersion() { def apiVersion = null try { def client = new RESTClient(envString) def resp = client.get(path : config.versionServiceUri) apiVersion = resp.data.version LOG.info("Current deployed API version [" + apiVersion + "]"); } catch (ex) { APIVersionError apiVersionError = new APIVersionError("Error occurred attempting to get current deployed API version from %s", envString + config.versionServiceUri); apiVersionError.setStackTrace(ex.stackTrace); throw apiVersionError; } return apiVersion } * @param annotation * @param infoName * @return */ private boolean isApiVersionGreaterThanMinApiVersion(APIVersion annotation, String infoName) { def isApiVersionGreaterThanMinApiVersion = true def minApiVersionRequired = annotation.minimimApiVersion(); // normalise both version id's def apiVersionNormalised = normaliseVersion(CURRENT_API_VERSION); def minApiVersionRequiredNormalised = normaliseVersion(minApiVersionRequired); // compare version id's int cmp = apiVersionNormalised.compareTo(minApiVersionRequiredNormalised); // if the comparison is less than 0, min API version is greater than the deployed API version if(cmp < 0) { LOG.info("min api version [" + minApiVersionRequired + "] greater than api version [" + CURRENT_API_VERSION + "], skipping [" + infoName + "]") isApiVersionGreaterThanMinApiVersion = false } return isApiVersionGreaterThanMinApiVersion } * @param version * @return */ private String normaliseVersion(String version) { String[] split = VERSION_PATTERN.split(version); StringBuilder sb = new StringBuilder(); for (String s : split) { sb.append(String.format("%" + MAX_VERSION_LENGTH + 's', s)); } return sb.toString(); } } The @APIVersion extension defines the same environment config as the @EnvironmentEndPoint extension does so that the environment can be injected and used purely for accessing the API version endpoint without the need for @EnvironmentEndPoint. The RESTful API version endpoint is required to be setup and publicly available. The @APIVersion extension will call this service to get details about the version of RESTful API. The version response data should be as follows: Media API 1.51.1 The @APIVersion extension will look for the version data to define what the current deployed version of the RESTful API is. Once the version of the RESTful API is known, the extension then checks the minimum API version required. Example @APIVersion(minimimApiVersion="1.0.0.0") The extension then uses this value to compare against the response data version and if the required version is greater than that of the deployed RESTful API services, then the test is skipped. This extension annotation can be placed on Specification's or Feature's allowing whole Specs to have a minimum version and / or Features to have their own minimum version. This extension has made writing integration tests with Spock even more portable and allows for a 'build once' set of tests that can be run against any environment, with some small changes to allow getting the API version. The SpockConfig.groovy file Here is an example of the SpockConfig.groovy config file used to configure defaults for both @EnvironmentEndPoint and @APIVersion extensions. versionServiceUri="/public/serviceInformation" envHost="http://api.preview.movideo.com" The 'versionServiceUri' is required for @APIVersion extension as the URI for the RESTful API version The 'envHost' is required for both @APIVersion and @EnvironmentEndPoint extensions as the host of the RESTful API Go and start testing Hopefully these Spock extensions might help your Spock integration tests. The framework is really easy and fun to use to build essential tests for the whole test stack. Checkout my GitHub projects for the code for both extensions. Hope this post has been helpful and hopefully I'll post something sooner for my next post. References and really helpful links Spock Homepage Annotation Driven Extensions With Spock

Integration Testing is an often overlooked area in enterprise development. This is primarily due to the associated complexities in setting up the necessary infrastructure for an integration test. For applications backed by databases, it’s fairly complicated and time-consuming to setup databases for integration tests, and also to clean those up once test is complete (ex. data files, schemas etc.), to ensure repeatability of tests. While there have been many tools (ex. DBUnit) and mechanisms (ex. rollback after test) to assist in this, the inherent complexity and issues have been there always. But if you are working with MongoDB, there’s a cool and easy way to do your unit tests, with almost the simplicity of writing a unit test with mocks. With ‘EmbedMongo’, we can easily setup an embedded MongoDB instance for testing, with in-built clean up support once tests are complete. In this article, we will walkthrough an example where EmbedMongo is used with JUnit for integration testing a Repository Implementation. Here’s the technology stack that we will be using. MongoDB 2.2.0 EmbedMongo 1.26 Spring Data – Mongo 1.0.3 Spring Framework 3.1 The Maven POM for the above setup looks like this. 4.0.0 com.yohanliyanage.blog.mongoit mongo-it 1.0 org.springframework.data spring-data-mongodb 1.0.3.RELEASE compile junit junit 4.10 test org.springframework spring-context 3.1.3.RELEASE compile de.flapdoodle.embed de.flapdoodle.embed.mongo 1.26 test Or if you prefer Gradle (by the way, Gradle is an awesome build tool which you should check out if you haven’t done so already). apply plugin: 'java' apply plugin: 'eclipse' sourceCompatibility = 1.6 group = "com.yohanliyanage.blog.mongoit" version = '1.0' ext.springVersion = '3.1.3.RELEASE' ext.junitVersion = '4.10' ext.springMongoVersion = '1.0.3.RELEASE' ext.embedMongoVersion = '1.26' repositories { mavenCentral() maven { url 'http://repo.springsource.org/release' } } dependencies { compile "org.springframework:spring-context:${springVersion}" compile "org.springframework.data:spring-data-mongodb:${springMongoVersion}" testCompile "junit:junit:${junitVersion}" testCompile "de.flapdoodle.embed:de.flapdoodle.embed.mongo:${embedMongoVersion}" } To begin with, here’s the document that we will be storing in Mongo. package com.yohanliyanage.blog.mongoit.model; import org.springframework.data.mongodb.core.index.Indexed; import org.springframework.data.mongodb.core.mapping.Document; /** * A Sample Document. * * @author Yohan Liyanage * */ @Document public class Sample { @Indexed private String key; private String value; public Sample(String key, String value) { super(); this.key = key; this.value = value; } public String getKey() { return key; } public void setKey(String key) { this.key = key; } public String getValue() { return value; } public void setValue(String value) { this.value = value; } } To assist with storing and managing this document, let’s write up a simple Repository implementation. The Repository Interface is as follows. package com.yohanliyanage.blog.mongoit.repository; import java.util.List; import com.yohanliyanage.blog.mongoit.model.Sample; /** * Sample Repository API. * * @author Yohan Liyanage * */ public interface SampleRepository { /** * Persists the given Sample. * @param sample */ void save(Sample sample); /** * Returns the list of samples with given key. * @param sample * @return */ List findByKey(String key); } And the implementation… package com.yohanliyanage.blog.mongoit.repository; import java.util.List; import static org.springframework.data.mongodb.core.query.Query.query; import static org.springframework.data.mongodb.core.query.Criteria.*; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.data.mongodb.core.MongoOperations; import org.springframework.stereotype.Repository; import com.yohanliyanage.blog.mongoit.model.Sample; /** * Sample Repository MongoDB Implementation. * * @author Yohan Liyanage * */ @Repository public class SampleRepositoryMongoImpl implements SampleRepository { @Autowired private MongoOperations mongoOps; /** * {@inheritDoc} */ public void save(Sample sample) { mongoOps.save(sample); } /** * {@inheritDoc} */ public List findByKey(String key) { return mongoOps.find(query(where("key").is(key)), Sample.class); } /** * Sets the MongoOps implementation. * * @param mongoOps the mongoOps to set */ public void setMongoOps(MongoOperations mongoOps) { this.mongoOps = mongoOps; } } To wire this up, we need a Spring Bean Configuration. Note that we do not need this for testing. But for the sake of completion, I have included this. The XML configuration is as follows. And now we are ready to write the Integration Test for our Repository Implementation using Embed Mongo. Ideally, the integration tests should be placed in a separate source directory, just like we place our unit tests (ex. src/test/java => src/integration-test/java). However, neither Maven nor Gradle supports this out of the box (yet – v1.2. For Gradle, there’s an on going discussion for this facility). Nevertheless, both Maven and Gradle are flexible, so you can configure the POM / build.gradle to handle this. However, to keep this discussion simple and focused, I will be placing the Integration Tests in the ‘src/test/java’, but I do not recommend this for a real application. Let’s start writing up the Integration Test. First, let’s begin with a simple JUnit based Test for the methods. package com.yohanliyanage.blog.mongoit.repository; import static org.junit.Assert.fail; import org.junit.After; import org.junit.Before; import org.junit.Test; /** * Integration Test for {@link SampleRepositoryMongoImpl}. * * @author Yohan Liyanage */ public class SampleRepositoryMongoImplIntegrationTest { private SampleRepositoryMongoImpl repoImpl; @Before public void setUp() throws Exception { repoImpl = new SampleRepositoryMongoImpl(); } @After public void tearDown() throws Exception { } @Test public void testSave() { fail("Not yet implemented"); } @Test public void testFindByKey() { fail("Not yet implemented"); } } When this JUnit Test Case initializes, we need to fire up EmbedMongo to start an embedded Mongo server. Also, when the Test Case ends, we need to cleanup the DB. The below code snippet does this. package com.yohanliyanage.blog.mongoit.repository; import static org.junit.Assert.fail; import java.io.IOException; import org.junit.*; import org.springframework.data.mongodb.core.MongoTemplate; import com.mongodb.Mongo; import com.yohanliyanage.blog.mongoit.model.Sample; import de.flapdoodle.embed.mongo.MongodExecutable; import de.flapdoodle.embed.mongo.MongodProcess; import de.flapdoodle.embed.mongo.MongodStarter; import de.flapdoodle.embed.mongo.config.MongodConfig; import de.flapdoodle.embed.mongo.config.RuntimeConfig; import de.flapdoodle.embed.mongo.distribution.Version; import de.flapdoodle.embed.process.extract.UserTempNaming; /** * Integration Test for {@link SampleRepositoryMongoImpl}. * * @author Yohan Liyanage */ public class SampleRepositoryMongoImplIntegrationTest { private static final String LOCALHOST = "127.0.0.1"; private static final String DB_NAME = "itest"; private static final int MONGO_TEST_PORT = 27028; private SampleRepositoryMongoImpl repoImpl; private static MongodProcess mongoProcess; private static Mongo mongo; private MongoTemplate template; @BeforeClass public static void initializeDB() throws IOException { RuntimeConfig config = new RuntimeConfig(); config.setExecutableNaming(new UserTempNaming()); MongodStarter starter = MongodStarter.getInstance(config); MongodExecutable mongoExecutable = starter.prepare(new MongodConfig(Version.V2_2_0, MONGO_TEST_PORT, false)); mongoProcess = mongoExecutable.start(); mongo = new Mongo(LOCALHOST, MONGO_TEST_PORT); mongo.getDB(DB_NAME); } @AfterClass public static void shutdownDB() throws InterruptedException { mongo.close(); mongoProcess.stop(); } @Before public void setUp() throws Exception { repoImpl = new SampleRepositoryMongoImpl(); template = new MongoTemplate(mongo, DB_NAME); repoImpl.setMongoOps(template); } @After public void tearDown() throws Exception { template.dropCollection(Sample.class); } @Test public void testSave() { fail("Not yet implemented"); } @Test public void testFindByKey() { fail("Not yet implemented"); } } The initializeDB() method is annotated with @BeforeClass to start this before test case beings. This method fires up an embedded MongoDB instance which is bound to the given port, and exposes a Mongo object which is set to use the given database. Internally, EmbedMongo creates the necessary data files in temporary directories. When this method executes for the first time, EmbedMongo will download the necessary Mongo implementation (denoted by Version.V2_2_0 in above code) if it does not exist already. This is a nice facility specially when it comes to Continuous Integration servers. You don’t have to manually setup Mongo in each of the CI servers. That’s one less external dependency for the tests. In the shutdownDB() method, which is annotated with @AfterClass, we stop the EmbedMongo process. This triggers the necessary cleanups in EmbedMongo to remove the temporary data files, restoring the state to where it was before Test Case was executed. We have now updated setUp() method to build a Spring MongoTemplate object which is backed by the Mongo instance exposed by EmbedMongo, and to setup our RepoImpl with that template. The tearDown() method is updated to drop the ‘Sample’ collection to ensure that each of our test methods start with a clean state. Now it’s just a matter of writing the actual test methods. Let’s start with the save method test. @Test public void testSave() { Sample sample = new Sample("TEST", "2"); repoImpl.save(sample); int samplesInCollection = template.findAll(Sample.class).size(); assertEquals("Only 1 Sample should exist collection, but there are " + samplesInCollection, 1, samplesInCollection); } We create a Sample object, pass it to repoImpl.save(), and assert to make sure that there’s only one Sample in the Sample collection. Simple, straight-forward stuff. And here’s the test method for findByKey method. @Test public void testFindByKey() { // Setup Test Data List samples = Arrays.asList( new Sample("TEST", "1"), new Sample("TEST", "25"), new Sample("TEST2", "66"), new Sample("TEST2", "99")); for (Sample sample : samples) { template.save(sample); } // Execute Test List matches = repoImpl.findByKey("TEST"); // Note: Since our test data (populateDummies) have only 2 // records with key "TEST", this should be 2 assertEquals("Expected only two samples with key TEST, but there are " + matches.size(), 2, matches.size()); } Initially, we setup the data by adding a set of Sample objects into the data store. It’s important that we directly use template.save() here, because repoImpl.save() is a method under-test. We are not testing that here, so we use the underlying “trusted” template.save() during data setup. This is a basic concept in Unit / Integration testing. Then we execute the method under test ‘findByKey’, and assert to ensure that only two Samples matched our query. Likewise, we can continue to write more tests for each of the repository methods, including negative tests. And here’s the final Integration Test file. package com.yohanliyanage.blog.mongoit.repository; import static org.junit.Assert.*; import java.io.IOException; import java.util.Arrays; import java.util.List; import org.junit.*; import org.springframework.data.mongodb.core.MongoTemplate; import com.mongodb.Mongo; import com.yohanliyanage.blog.mongoit.model.Sample; import de.flapdoodle.embed.mongo.MongodExecutable; import de.flapdoodle.embed.mongo.MongodProcess; import de.flapdoodle.embed.mongo.MongodStarter; import de.flapdoodle.embed.mongo.config.MongodConfig; import de.flapdoodle.embed.mongo.config.RuntimeConfig; import de.flapdoodle.embed.mongo.distribution.Version; import de.flapdoodle.embed.process.extract.UserTempNaming; /** * Integration Test for {@link SampleRepositoryMongoImpl}. * * @author Yohan Liyanage */ public class SampleRepositoryMongoImplIntegrationTest { private static final String LOCALHOST = "127.0.0.1"; private static final String DB_NAME = "itest"; private static final int MONGO_TEST_PORT = 27028; private SampleRepositoryMongoImpl repoImpl; private static MongodProcess mongoProcess; private static Mongo mongo; private MongoTemplate template; @BeforeClass public static void initializeDB() throws IOException { RuntimeConfig config = new RuntimeConfig(); config.setExecutableNaming(new UserTempNaming()); MongodStarter starter = MongodStarter.getInstance(config); MongodExecutable mongoExecutable = starter.prepare(new MongodConfig(Version.V2_2_0, MONGO_TEST_PORT, false)); mongoProcess = mongoExecutable.start(); mongo = new Mongo(LOCALHOST, MONGO_TEST_PORT); mongo.getDB(DB_NAME); } @AfterClass public static void shutdownDB() throws InterruptedException { mongo.close(); mongoProcess.stop(); } @Before public void setUp() throws Exception { repoImpl = new SampleRepositoryMongoImpl(); template = new MongoTemplate(mongo, DB_NAME); repoImpl.setMongoOps(template); } @After public void tearDown() throws Exception { template.dropCollection(Sample.class); } @Test public void testSave() { Sample sample = new Sample("TEST", "2"); repoImpl.save(sample); int samplesInCollection = template.findAll(Sample.class).size(); assertEquals("Only 1 Sample should exist in collection, but there are " + samplesInCollection, 1, samplesInCollection); } @Test public void testFindByKey() { // Setup Test Data List samples = Arrays.asList( new Sample("TEST", "1"), new Sample("TEST", "25"), new Sample("TEST2", "66"), new Sample("TEST2", "99")); for (Sample sample : samples) { template.save(sample); } // Execute Test List matches = repoImpl.findByKey("TEST"); // Note: Since our test data (populateDummies) have only 2 // records with key "TEST", this should be 2 assertEquals("Expected only two samples with key TEST, but there are " + matches.size(), 2, matches.size()); } } On a side note, one of the key concerns with Integration Tests is the execution time. We all want to keep our test execution times as low as possible, ideally a couple of seconds to make sure that we can run all the tests during CI, with minimal build and verification times. However, since Integration Tests rely on underlying infrastructure, usually Integration Tests take time to run. But with EmbedMongo, this is not the case. In my machine, above test suite runs in 1.8 seconds, and each test method takes only .166 seconds max. See the screenshot below. I have uploaded the code for above project into GitHub. You can download / clone it from here: https://github.com/yohanliyanage/blog-mongo-integration-tests. For more information regarding EmbedMongo, refer to their site at GitHub https://github.com/flapdoodle-oss/embedmongo.flapdoodle.de.

for people in hurry, refer the steps and the demo . introduction control bus pattern is a enterprise integration pattern is used to control distributed systems in spring integration . in this blog, i will show you how a control bus can control your application or a component to start or stop listening to jms message . in this example, we are using jms queue to start and stop the jms inbound-channel-adapter , we can also do this with jdbc inbound-channel-adapter and control this thru an external application. the other way to do the same is by using mbean as in this example . in this use case, there is a spring integration flow. this spring integration flow can be controlled by sending start / stop message to inbound-channel-adapter from a activemq jms queue. details control bus with spring integration control bus spring integration jms to start implementing this use case, we write the junit test 1st. if you notice once the inboundadapter is started the message is received from the adapteroutchannel. once the inboundadapter is stopped no message is received. this is demonstrated as below, @test public void democontrolbus() { assertnull(adapteroutputchanel.receive(1000)); controlchannel.send(new genericmessage("@inboundadapter.start()")); assertnotnull(adapteroutputchanel.receive(1000)); controlchannel.send(new genericmessage("@inboundadapter.stop()")); assertnull(adapteroutputchanel.receive(1000)); } the test configuration looks as below, if you run the “mvn test” the tests work. in the main configuration, we will be configuring actual queues and jms inbound-channel-adapter as below, now when you start the component as “run on server” in sts ide and post a message on myqueue, you can see the subscribers received the messages on the console. you can issue “@inboundadapter.stop()” on the controlbusqueue, it will stop the inbound-channel-adapter, it will also throw java.lang.interruptedexception, it looks like a false alarm. to test if the inbound-channel-adapter is stopped, post a message on to myqueue, the component will not process the message. now issue “@inboundadapter.start()” on the controlbusqueue, it will process the earlier message and start listening for new messages. conclusion if you notice in this blog, we can control the component to listen to message using control bus. the other way to do the same is by using mbean as in this example .

JAX-RS is a framework designed to help you write RESTful applications both on the client and server side. With Java EE 7 is being slated to be released next year, 2013, JAX-RS is one of the specifications getting a deep revision. JAX-RS 2.0 is currently in the Public Draft phase at the JCP, so now is a good time to discuss some of the key new features so that you can start playing with your favorite JAX-RS implementation and give some valuable feedback the expert group needs to finalize the specification. Key features in 2.0: Client API Server-side Asynchronous HTTP Filters and Interceptors This article gives a brief overview of each of these features. Client Framework One huge thing missing from JAX-RS 1.0 was a client API. While it was easy to write a portable JAX-RS service, each JAX-RS implementation defined their own proprietary API. JAX-RS 2.0 fills in this gap with a fluent, low-level, request building API. Here's a simple example: Client client = ClientFactory.newClient(); WebTarget target = client.target("http://example.com/shop"); Form form = new Form().param("customer", "Bill") .param("product", "IPhone 5") .param("CC", "4444 4444 4444 4444"); Response response = target.request().post(Entity.form(form)); assert response.getStatus() == 200; Order order = response.readEntity(Order.class); Let's dissect this example code. The Client interface manages and configures HTTP connections. It is also a factory for WebTargets. WebTargets represent a specific URI. You build and execute requests from a WebTarget instance. Response is the same class defined in JAX-RS 1.0, but it has been expanded to support the client side. The example client, allocates an instance of a Client, creates a WebTarget, then posts form data to the URI represented by the WebTarget. The Response object is tested to see if the status is 200, then the application class Order is extracted from the response using the readEntity() method. The MessageBodyReader and MessageBodyWriter content handler interfaces defined in JAX-RS 1.0 are reused on the client side. When the readEntity() method is invoked in the example code, a MessageBodyReader is matched with the response's Content-Type and the Java type (Order) passed in as a parameter to the readEntity() method. If you are optimistic that your service will return a successful response, there are some nice helper methods that allow you to get the Java object directly without having to interact with and write additional code around a Response object. Customer cust = client.target("http://example.com/customers") .queryParam("name", "Bill Burke") .request().get(Customer.class); In this example we target a URI and specify an additional query parameter we want appended to the request URI. The get() method has an additional parameter of the Java type we want to unmarshal the HTTP response to. If the HTTP response code is something other than 200, OK, JAX-RS picks an exception that maps to the error code from a defined exception hierarchy in the JAX-RS client API. Asynchronous Client API The JAX-RS 2.0 client framework also supports an asynchronous API and a callback API. This allows you to execute HTTP requests in the background and either poll for a response or receive a callback when the request finishes. Future future = client.target("http://e.com/customers") .queryParam("name", "Bill Burke") .request() .async() .get(Customer.class); try { Customer cust = future.get(1, TimeUnit.MINUTES); } catch (TimeoutException ex) { System.err.println("timeout"); } The Future interface is a JDK interface that has been around since JDK 5.0. The above code executes an HTTP request in the background, then blocks for one minute while it waits for a response. You could also use the Future to poll to see if the request is finished or not. Here's an example of using a callback interface. InvocationCallback callback = new InvocationCallback { public void completed(Response res) { System.out.println("Request success!"); } public void failed(ClientException e) { System.out.println("Request failed!");n } }; client.target("http://example.com/customers") .queryParam("name", "Bill Burke") .request() .async() .get(callback); In this example, we instantiate an implementation of the InvocationCallback interface. We invoke a GET request in the background and register this callback instance with the request. The callback interface will output a message on whether the request executed successfully or not. Those are the main features of the client API. I suggest browsing the specification and Javadoc to learn more. Server-Side Asynchronous HTTP On the server-side, JAX-RS 2.0 provides support for asynchronous HTTP. Asynchronous HTTP is generally used to implement long-polling interfaces or server-side push. JAX-RS 2.0 support for Asynchronous HTTP is annotation driven and is very analogous with how the Servlet 3.0 specification handles asynchronous HTTP support through the AsyncContext interface. Here's an example of writing a crude chat program. @Path("/listener") public class ChatListener { List listeners = ...some global list...; @GET public void listen(@Suspended AsyncResponse res) { list.add(res); } } For those of you who have used the Servlet 3.0 asynchronous interfaces, the above code may look familiar to you. An AsyncResponse is injected into the JAX-RS resource method via the @Suspended annotation. This act disassociates the calling thread to the HTTP socket connection. The example code takes the AsyncResponse instance and adds it to a application-defined global List object. When the JAX-RS method returns, the JAX-RS runtime will do no response processing. A different thread will handle response processing. @Path("/speaker") public class ChatSpeaker { List listeners = ...some global list...; @POST @Consumes("text/plain") public void speak(String speech) { for (AsyncResponse res : listeners) { res.resume(Response.ok(speech, "text/plain").build());n } } } When a client posts text to this ChatSpeaker interface, the speak() method loops through the list of registered AsyncResponses and sends back an 200, OK response with the posted text. Those are the main features of the asynchronous HTTP interface, check out the Javadocs for a deeper detail. Filters and Entity Interceptors JAX-RS 2.0 has an interceptor API that allows framework developers to intercept request and response processing. This powerful API allows framework developers to transparently add orthogonal concerns like authentication, caching, and encoding without polluting application code. Prior to JAX-RS 2.0 many JAX-RS providers like Resteasy, Jersey, and Apache CXF wrote their own proprietary interceptor frameworks to deliver various features in their implementations. So, while JAX-RS 2.0 filters and interceptors can be a bit complex to understand please note that it is very use-case driven based on real-world examples. I wrote a blog on JAX-RS interceptor requirements awhile back to help guide the JAX-RS 2.0 JSR Expert Group on defining such an API. The blog is a bit dated, but hopefully you can get the gist of why we did what we did. JAX-RS 2.0 has two different concepts for interceptions: Filters and Entity Interceptors. Filters are mainly used to modify or process incoming and outgoing request headers or response headers. They execute before and after request and response processing. Entity Interceptors are concerned with marshaling and unmarshalling of HTTP message bodies. They wrap around the execution of MessageBodyReader and MessageBodyWriter instances. Server Side Filters On the server-side you have two different types of filters. ContainerRequestFilters run before your JAX-RS resource method is invoked. ContainerResponseFilters run after your JAX-RS resource method is invoked. As an added caveat, ContainerRequestFilters come in two flavors: pre-match and post-matching. Pre-matching ContainerRequestFilters are designated with the @PreMatching annotation and will execute before the JAX-RS resource method is matched with the incoming HTTP request. Pre-matching filters often are used to modify request attributes to change how it matches to a specific resource. For example, some firewalls do not allow PUT and/or DELETE invocations. To circumvent this limitation many applications tunnel the HTTP method through the HTTP header X-Http-Method-Override. A pre-matching ContainerRequestFilter could implement this behavior. @Provider public class HttpOverride implements ContainerRequestFilter { public void filter(ContainerRequestContext ctx) { String method = ctx.getHeaderString("X-Http-Method-Override"); if (method != null) ctx.setMethod(method); } } Post matching ContainerRequestFilters execute after the Java resource method has been matched. These filters can implement a range of features for example, annotation driven security protocols. After the resource class method is executed, JAX-RS will run all ContainerResponseFilters. These filters allow you to modify the outgoing response before it is marshalled and sent to the client. One example here is a filter that automatically sets a Cache-Control header. @Provider public class CacheControlFilter implements ContainerResponseFilter { public void filter(ContainerRequestContext req, ContainerResponseContext res) { if (req.getMethod().equals("GET")) { req.getHeaders().add("Cache-Control", cacheControl); } } } Client Side Filters On the client side you also have two types of filters: ClientRequestFilter and ClientResponseFilter. ClientRequestFilters run before your HTTP request is sent over the wire to the server. ClientResponseFilters run after a response is received from the server, but before the response body is unmarshalled. A good example of client request and response filters working together is a client-side cache that supports conditional GETs. The ClientRequestFilter would be responsible for setting the If-None-Match or If-Modified-Since headers if the requested URI is already cached. Here's what that code might look like. @Provider public class ConditionalGetFilter implements ClientRequestFilter { public void filter(ClientRequestContext req) { if (req.getMethod().equals("GET")) { CacheEntry entry = cache.getEntry(req.getURI()); if (entry != null) { req.getHeaders().putSngle("If-Modified-Since", entry.getLastModified()); } } } } The ClientResponseFilter would be responsible for either buffering and caching the response, or, if a 302, Not Modified response was sent back, to edit the Response object to change its status to 200, set the appropriate headers and buffer to the currently cached entry. This code would be a bit more complicated, so for brevity, we're not going to illustrate it within this article. Reader and Writer Interceptors While filters modify request or response headers, interceptors deal with message bodies. Interceptors are executed in the same call stack as their corresponding reader or writer. ReaderInterceptors wrap around the execution of MessageBodyReaders. WriterInterceptors wrap around the execution of MessageBodyWriters. They can be used to implement a specific content-encoding. They can be used to generate digital signatures or to post or pre-process a Java object model before or after it is marshalled. Here's an example of a GZIP encoding WriterInterceptor. @Provider public class GZIPEndoer implements WriterInterceptor { public void aroundWriteTo(WriterInterceptorContext ctx) throws IOException, WebApplicationException { GZIPOutputStream os = new GZIPOutputStream(ctx.getOutputStream()); try { ctx.setOutputStream(os); return ctx.proceed(); } finally { os.finish(); } } } Resource Method Filters and Interceptors Sometimes you want a filter or interceptor to only run for a specific resource method. You can do this in two different ways: register an implementation of DynamicFeature or use the @NameBinding annotation. The DynamicFeature interface is executed at deployment time for each resource method. You just use the Configurable interface to register the filters and interceptors you want for the specific resource method. @Provider public class ServerCachingFeature implements DynamicFeature { public void configure(ResourceInfo resourceInfo, Configurable configurable) { if (resourceInfo.getMethod().isAnnotationPresent(GET.class)) { configurable.register(ServerCacheFilter.class); } } } On the other hande, @NameBinding works a lot like CDI interceptors. You annotate a custom annotation with @NameBinding and then apply that custom annotation to your filter and resource method @NameBinding public @interface DoIt {} @DoIt public class MyFilter implements ContainerRequestFilter {...} @Path public class MyResource { @GET @DoIt public String get() {...} Wrapping Up Well, those are the main features of JAX-RS 2.0. There's also a bunch of minor features here and there, but youll have to explore them yourselves. If you want to testdrive JAX-RS 2.0 (and hopefully also give feedback to the expert group), Red Hat's Resteasy 3.0 and Oracle's Jersey project have implementations you can download and use. Useful Links Below are some useful links. I've also included links to some features in Resteasy that make use of filters and interceptors. This code might give you a more in-depth look into what you can do with this new JAX-RS 2.0 feature. JAX-RS 2.0 Public Draft Specification Resteasy 3.0 Download Jersey Resteasy 3.0 client cache implementation code (to see how filters interceptors work on client side) Doseta digital signature headers (good use case or interceptors) File suffix content negotiation implementation (server-side filter example) Other server-side examples (cache-control annotations, gzip encoding, role-based security)

Following up on the previous article where we highlighted the top 20 features of Code Completion, I’d like to talk about the top Refactoring features that help make IntelliJ IDEA an extremely useful development tool. IntelliJ IDEA was the first Java IDE to implement the extensive set of refactorings worked out and recommended by Martin Fowler, driving other IDEs to offer this feature. Nowadays it’s hard to imagine an IDE that doesn’t provide at least a basic set of refactorings. However, it’s never about the number of refactorings you can use, but rather about how confident you feel using them. That’s why IntelliJ IDEA has always focused on refactoring productivity and refactoring safety. It’s great to improve your code quickly, but you’ve got to make sure your changes are safe to the project as a whole. In this article I give an overview of the most important refactoring features that not everyone knows and uses, which make IntelliJ IDEA really shine. Out-of-the-box support for languages and frameworks The first and perhaps the most impressive aspect of refactorings in IntelliJ IDEA is its all-encompassing support for languages and frameworks. It not only recognizes many languages, expressions and dialects (even nested inside each other), but also their relationships within the project. You can safely call refactorings for any statement at the caret, and IntelliJ IDEA will take care of applying the corresponding changes to every piece of code related to the change. This includes SQL expressions; database table definitions; Spring expressions and annotations and configurations; JSF expressions; hibernate mappings; and more. For instance, you call the Rename refactoring on a class within a JPA statement. IntelliJ IDEA recognizes that you need to rename a JPA entity class, and applies changes to the class and every JPA or other expression in the project — in mere seconds. Undo Another aspect that changes the user experince significantly is how safe and easy you can undo any change resulting from even a complicated refactoring, with just one click. Don’t be afraid to apply changes, because you can always roll them back! Find and replace code duplicates Another thing that makes some developers think IntelliJ IDEA understands their code as well as they do (or better), is detection of code duplicates. This feature is available as a separate refactoring, which you can call on any project scope, and as a part of any other refactoring, such as introduce constant, variable, method, etc. Just apply the refactoring and IntelliJ IDEA willmake appropriate changes to your code to remove duplicates. Try it just once—and you’ll wonder how you’ve lived without it all along. Rename and name patterns recognition What could be simpler than the Rename refactoring, you ask? Well, IntelliJ IDEA offers incredible additional support for this refactoring. When you use it, the IDE offers to apply the corresponding changes to getters and setters, variables, constants, test classes and methods, implementation classes, etc. This can be a huge time-saver and a lot of help in keeping your code clean Type migration Another useful feature you will rarely find in other IDEs is type migration. Have you ever used some type for a long time and then decided to change it? I’m sure you have. IntelliJ IDEA takes care of automatically applying changes to method return types, local variables, parameters and other data-flow-dependent type entries across the entire project. You can even switch between arrays and collections, and the IDE will make all the changes for you. Invert boolean If we can automate type migration, why not do the same with semantics? Exactly. For example, IntelliJ IDEA can correctly invert all usages of a boolean member or variable. Safe delete As I hinted earlier, the real benefits of refactoring are always in the details. IntelliJ IDEA tries to keep things simple for you, but there’s a lot intelligence lurking behind every feature. Even with simple deletion, it ensures that not a single line of code gets broken. String fragments Yet another time saver not found in other IDEs. IntelliJ IDEA can even extract a part of a string expression. Just select the fragment you need, and the IDE will take care of the rest. Other productivity-boosting features Many other refactorings in IntelliJ IDEA also include productivity-boosting features. For instance, you can easily change the type of extracted variable (or parameter) via ⇧⇥, just in-place, as well as replace all occurrences or declare it final. If you extract a field, the IDE will prompt you to choose where you want to initialize it. If you do it within a test, it will suggest that you initialize it in a setUp method. Inline to anonymous Everyone is used to inlining methods. However, not everyone knows that IntelliJ IDEA also provides inline refactoring for constructors. This is especially useful for such classes as Thread or Runnable. After you call it, all usages will be inlined into anonymous classes. Clone class The Clone class refactoring is yet another example of how something so simple can still save your time. As most other refactorings, it is available from a usage and helps you create a copy of any class you need. Encapsulate fields This feature is quite simple and is present in most IDEs. It helps you encapsulate fields with one click. IntelliJ IDEA goes a bit further: it can do it for a whole class at once. Consistent behavior Most Java refactorings in IntelliJ IDEA are also available from non-Java files where references to Java classes exist. Since it comes with out-of-the-box support for many custom frameworks, it offers the same shortcuts and consistent behavior for all refactorings. Framework specific refactorings In addition to Java refactorings, IntelliJ IDEA offers refactorings specific to custom frameworks, such as Spring, Java EE, Android, etc. For example, you can easily morph any component of an Android application into another type, right from the designer. Framework-specific refactorings are a wide-ranging topic that’s probably out of the scope of this article. I hope to cover it later, as well as refactorings specific to other languages, such as Scala, Groovy, JavaScript, CSS, and XML. Additional refactorings The total number of refactorings available in IntelliJ IDEA is quite high. There are about 35 Java only refactorings, plus a large number of refactorings specific to other frameworks and languages. Whichever definition of refactoring you use, it’s got more of them than any other Java IDE. Here’s a list of just the unique ones Make static Inline super class Replace inheritance with delegation Extract method object Remove middleman Wrap return value Move instance method Convert to instance method Replace temp with query Refactor this If you cannot recall the shortcut for a particular refactoring, or if you don’t feel like using the mouse, IntelliJ IDEA offers Refactor this action available via ⌘⇧⌥T. It shows you the list of refactorings applicable at the current context. Structural replace The last feature for today is Structural replace available via ⌘⇧M. This is a very powerful tool, but also the least obvious. Thank to its advanced code analysis, IntelliJ IDEA knows pretty much everything about your code. This makes possible Structural replace, which lets you use language-specific tokens in lookup and replace expressions. For example, we have a library with a new version where a static method was replaced with a singleton. To update our code, we can use the following structural replace expressions: com.ij.j2ee.MakeUtil.$MethodCall$($Params$) for lookup and com.ij.j2ee.MakeUtil.getInstance().$MethodCall$($Params$) for replace. IntelliJ IDEA will find, resolve and replaces all usages correctly, regardless of how the class was imported. Structural replace can be rather complicated at first, but once you learn how to use it, it can save you a lot of time. Summary I hope this article helps you to discover the powerful refactoring functionality hidden in IntelliJ IDEA. The more you know about your IDE, the more time it can save you every day, and the more productive you become. Go ahead and get the most out of your IntelliJ IDEA!

The Internet doesn't lack expositions on REST architecture, RESTful services, and their implementation in Java. But, here is another one. Why? Because I couldn't find something concise enough to point readers of the eValhalla blog series. What is REST? The acronym stands for Representational State Transfer. It refers to an architectural style (or pattern) thought up by one of the main authors of the HTTP protocol. Don't try to infer what the phrase "representational state transfer" could possibly mean. It sounds like there's some transfer of state that's going on between systems, but that's a bit of a stretch. Mostly, there's transfer of resources between clients and servers. The clients initiate requests and get back responses. The responses are resources in some standard media type such as XML or JSON or HTML. But, and that's a crucial aspect of the paradigm, the interaction itself is stateless. That's a major architectural departure from the classic client-server model of the 90s. Unlike classic client-server, there's no notion of a client session here. REST is offered not as a procotol, but as an architectural paradigm. However, in reality we are pretty much talking about HTTP of which REST is an abstraction. The core aspects of the architecture are (1) resource identifiers (i.e. URIs); (2) different possible representations of resources, or internet media types (e.g. application/json); (3) CRUD operations support for resources like the HTTP methods GET, PUT, POST and DEL. Resources are in principle decoupled from their identifiers. That means the environment can deliver a cached version or it can load balance somehow to fulfill the request. In practice, we all know URIs are actually addresses that resolve to particular domains so there's at least that level of coupling. In addition, resources are decoupled from their representation. A server may be asked to return HTML or XML or something else. There's content negotiation going on where the server may offer the desired representation or not. The CRUD operations have constraints on their semantics that may or may not appear obvious to you. The GET, PUT and DEL operations require that a resource be identified while POST is supposed to create a new resource. The GET operation must not have side-effects. So all other things being equal, one should be able to invoke GET many times and get back the same result. PUT updates a resource, DEL removes it and therefore they both have side-effects just like POST. On the other hand, just like GET, PUT may be repeated multiple times always to the same effect. In practice, those semantics are roughly followed. The main exception is the POST method which is frequently used to send data to the server for some processing, but without necessarily expecting it to create a new resource. Implementing RESTful services revolves around implementing those CRUD operations for various resources. This can be done in Java with the help of a Java standard API called JAX-RS. REST in Java = JAX-RS = JSR 311 In the Java world, when it comes to REST, we have the wonderful JAX-RS. And I'm not being sarcastic! This is one of those technologies that the Java Community Process actually got right, unlike so many other screw ups. The API is defined as JSR 311 and it is at version 1.1, with work on version 2.0 under way. The beauty of JAX-RS is that it is almost entirely driven by annotations. This means you can turn almost any class into a RESTful service. You can simply turn a POJO into a REST endpoint by annotating it with JSR 311 annotations. Such an annotated POJOs is called a resource class in JAX-RS terms. Some of the JAX-RS annotations are at the class level, some at the method level and others at the method parameter level. Some are available both at class and method levels. Ultimately the annotations combine to make a given Java method into a RESTful endpoint accessible at an HTTP-based URL. The annotations must specify the following elements: The relative path of the Java method - this is accomplished with @Path annotation. What the HTTP verb is, i.e. what CRUD operation is being performed - this is done by specifying one of @GET, @PUT, @POST or @DELETE annotations. The media type accepted (i.e. the representation format) - @Consumes annotation. The media type returned - @Produces annotation. The two last ones are optional. If omitted, then all media types are assumed possible. Let's look at a simple example and take it apart: import javax.ws.rs.*; @Path("/mail") @Produces("application/json") public class EmailService { @POST @Path("/new") public String sendEmail(@FormParam("subject") String subject, @FormParam("to") String to, @FormParam("body") String body) { return "new email sent"; } @GET @Path("/new") public String getUnread() { return "[]"; } @DELETE @Path("/{id}") public String deleteEmail(@PathParam("id") int emailid) { return "delete " + id; } @GET @Path("/export") @Produces("text/html") public String exportHtml(@QueryParam("searchString") @DefaultValue("") String search) { return "..."; } } The class define a RESTful interface for a hypothetical HTTP-based email service. The top-level path mail is relative to the root application path. The root application path is associated with the JAX-RS javax.ws.rs.core.Application that you extend to plugin into the runtime environment. Then we've declared with the @Produces annotation that all methods in that service produce JSON. This is just a class-default that one can override for individual methods like we've done in the exportHtml method. The sendMail method defines a typical HTTP post where the content is sent as an HTML form. The intent here would be to post to http://myserver.com/mail/new a form for a new email that should be sent out. As you can see, the API allows you to bind each separate form field to a method parameter. Note also that you have a different method for the exact same path. If you do an HTTP get at /mail/new, the Java method annotated with @GET will be called instead. Presumably the semantics of get /mail/new would be to obtain the list of unread emails. Next, note how the path of the deleteEmail method is parametarized by an integer id of the email to delete. The curly braces indicate that "id" is actually a parameter. The value of that parameter is bound to the whatever is annotated with @PathParam("id"). Thus if we do an HTTP delete at http://myserver.com/mail/453 we would be calling the deleteEmail method with argument emailid=453. Finally, the exportHtml method demonstrates how we can get a handle on query parameters. When you annotate a parameter with @QueryParam("x") the value is taken from the HTTP query parameter named x. The @DefaultValue annotation provides a default in case that query parameter is missing. So, calling http://myserver.org/mail/export?searchString=RESTful will call the exportHtml method with a parameter search="RESTful". To expose this service, first we need to write an implementation of javax.ws.rs.core.Application. That's just a few lines: public class MyRestApp extends javax.ws.rs.core.Application { public Set>Class> getClasses() { HashSet S = new HashSet(); S.add(EmailService.class); return S; } } How this gets plugged into your server depends on your JAX-RS implementation. Before we leave the API, I should mentioned that there's more to it. You do have access to a Request and Response objects. You have annotations to access other contextual information and metadata like HTTP headers, cookies etc. And you can provide custom serialization and deserialization between media types and Java objects. RESTful vs Web Services Web services (SOAP, WSDL) were heavily promoted in the past decade, but they didn't become as ubiquitous as their fans had hoped. Blame XML. Blame the rigidity of the XML Schema strong typing. Blame the tremendous overhead, the complexity of deploying and managing a web service. Or, blame the frequent compatibility nightmares between implementations. Reasons are not hard to find and the end result is that RESTful services are much easier to develop and use. But there is a flip side! The simplicity of RESTful services means that one has less guidance in how to map application logic to a REST API. One of the issues is that instead of the programmatic types we have in programming languages, we have the Java primitives and media types. Fortunately, JAX-RS allows to implement whatever conversions we want between actual Java method arguments and what gets sent on the wire. The other issue is the limited set of operations that a REST service can offer. While with web services, you define the operation and its semantics just as in a general purpose programming language, with RESTful you're stuck with get, put, post and delete. So, free from the type mismatch nightmare, but tied into only 4 possible operations. This is not as bad as it seems if you view those operations as abstract, meta operations. The key point when designing RESTful services, whether you are exposing existing application logic or creating a new one, is to think in terms of data resources. That's not so hard since most of what common business applications do is manipulate data. First, because every single thing is identified as a resource, one must come up with an appropriate naming schema. Because URIs are hierarchical, it is easy to devise a nested structure like /productcategory/productname/version/partno. Second, one must decide what kinds of representations are to be supported, both in output and input. For a modern AJAX webpp, we'd mostly use JSON. I would recommend JSON over XML even in a B2B setting where servers talk to each other. Finally, one must categorize business operation as one of GET, PUT, POST and DELETE. This is probably a bit less intuitive, but it's just a matter of getting used to. For example, instead of thinking about a "Checkout Shopping Cart" operation, think about POSTing a new order. Instead of thinking about a "Login User" operation think about GETing an authentication token. In general, every business operation manipulates some data in some way. Therefore, every business operation can fit into this crude CRUD model. Clearly, most read-only operations should be a GET. However, sometimes you have to send a large chunk of data to the server in which case you should use POST. For example you could post some very time consuming query that require a lot of text to specify. Then the resource you are creating is for example the query result. Another way to decide if you should POST or no is if you have a unique resource identifier. If not, then use POST. Obviously, operations that cause some data to be removed should be a DELETE. The operations that "store" data are PUT and again POST. Deciding between those two is easy: use PUT whenever you are modifying an existing resource for which you have an identifier. Otherwise, use POST. Implementations & Resources There are several implementations to choose from. Since, I haven't tried them all, I can't offer specific comments. Most of them used to require a servlet containers. The Restlet framework by Jerome Louvel never did, and that's why I liked it. Its documentation leaves to be desired and if you look at its code, it's over-architected to a comical degree, but then what ambitious Java framework isn't. Another newcomer that is strictly about REST and seems lightweight is Wink, an Apache incubated project. I haven't tried it, but it looks promising. And of course, one should not forget the reference implementation Jersey. Jersey has the advantage of being the most up-to-date with the spec at any given time. Originally it was dependent on Tomcat. Nowadays, it seems it can run standalone so it's on par with Restlet which I mentioned first because that's what I have mostly used. Here are some further reading resources, may their representational state be transferred to your brain and properly encoded from HTML/PDF to a compact and efficient neural net: The Wikipedia article on REST is not in very good shape, but still a starting point if you want to dig deeper into the conceptual framework. Refcard from Dzone.com: http://refcardz.dzone.com/refcardz/rest-foundations-restful#refcard-download-social-buttons-display Wink's User Guide seems well written. Since it's an implementation of JAX-RS, it's a good documentation of that technology. https://dzone.com/articles/putting-java-rest: A fairly good show-and-tell introduction to the JAX-RS API, with a link in there to a more in-depth description of REST concepts by the same author. Worth the read. http://jcp.org/en/jsr/detail?id=311: The official JSR 311 page. Download the specification and API Javadocs from there. http://jsr311.java.net/nonav/javadoc/index.html: Online access of JSR 311 Javadocs. If you know of something better, something nice, please post it in a comment and I'll include in this list. PS: I'm curious if people start new projects with Servlets, JSP/JSF these days? I would be curious as to what the rationale would be to pick those over AJAX + RESTful services communication via JSON. As I said above, this entry is intended to help readers of the eValhalla blogs series which chronicles the development of the eValhalla project following precisely the AJAX+REST model.

A short blog about a topic I was discussing last week with a customer: testing SOAP Web Services. If you follow my blog you would know by now that I’m not a fan of unit testing in MOCK environments. Not because I don’t like it or I have religious believes that don’t allow me to use JUnit and Mockito. It’s just because with the work I do (mostly Java EE using application servers) my code runs in a managed environment (i.e. containers) and when I start mocking all the container’s services, it becomes cumbersome and useless. Few months ago I wrote a post about integration testing with Arquillian. But you don’t always need Arquillian to test inside a container because today, most of the containers are light and run in-memory. Think of an in-memory database. An in-memory web container. An in-memory EJB container. So first, let’s write a SOAP Web Service. I’m using the one I use on my book : a SOAP Web Service that validates a credit card. If you look at the code, there is nothing special about it (the credit card validation algorithm is a dummy one: even numbers are valid, odd are invalid). Let’s start with the interface: import javax.jws.WebService; @WebService public interface Validator { public boolean validate(CreditCard creditCard); } Then the SOAP Web Service implementation: @WebService(endpointInterface = "org.agoncal.book.javaee7.chapter21.Validator") public class CardValidator implements Validator { public boolean validate(CreditCard creditCard) { Character lastDigit = creditCard.getNumber().charAt(creditCard.getNumber().length() - 1); return Integer.parseInt(lastDigit.toString()) % 2 != 0; } } In this unit test I instantiate the CardValidator class and invoke the validate method. This is acceptable, but what if your SOAP Web Serivce uses Handlers ? What if it overrides mapping with the webservice.xml deployment descriptor ? Uses the WebServiceContext ? In short, what if your SOAP Web Service uses containers’ services ? Unit testing becomes useless. So let’s test your SOAP Web Service inside the container and write an the integration test. For that we can use an in-memory web container. And I’m not just talking about a GlassFish, JBoss or Tomcat, but something as simple as the web container that come with the SUN’s JDK. Sun’s implementation of Java SE 6 includes a light-weight HTTP server API and implementation : com.sun.net.httpserver. Note that this default HTTP server is in a com.sun package. So this might not be portable depending on the version of your JDK. Instead of using the default HTTP server it is also possible to plug other implementations as long as they provide a Service Provider Implementation (SPI) for example Jetty’s J2se6HttpServerSPI. So this is how an integration test using an in memory web container can look like: public class CardValidatorIT { @Test public void shouldCheckCreditCardValidity() throws MalformedURLException { // Publishes the SOAP Web Service Endpoint endpoint = Endpoint.publish("http://localhost:8080/cardValidator", new CardValidator()); assertTrue(endpoint.isPublished()); assertEquals("http://schemas.xmlsoap.org/wsdl/soap/http", endpoint.getBinding().getBindingID()); // Data to access the web service URL wsdlDocumentLocation = new URL("http://localhost:8080/cardValidator?wsdl"); String namespaceURI = "http://chapter21.javaee7.book.agoncal.org/"; String servicePart = "CardValidatorService"; String portName = "CardValidatorPort"; QName serviceQN = new QName(namespaceURI, servicePart); QName portQN = new QName(namespaceURI, portName); // Creates a service instance Service service = Service.create(wsdlDocumentLocation, serviceQN); Validator cardValidator = service.getPort(portQN, Validator.class); // Invokes the web service CreditCard creditCard = new CreditCard("12341234", "10/10", 1234, "VISA"); assertFalse("Credit card should be valid", cardValidator.validate(creditCard)); creditCard.setNumber("12341233"); assertTrue("Credit card should not be valid", cardValidator.validate(creditCard)); // Unpublishes the SOAP Web Service endpoint.stop(); assertFalse(endpoint.isPublished()); } } The Endpoint.publish() method uses by default the light-weight HTTP server implementation that is included in Sun’s Java SE 6. It publishes the SOAP Web Service and starts listening on URL http://localhost:8080/cardValidator. You can even go to http://localhost:8080/cardValidator?wsdl to see the generated WSDL. The integration test looks for the WSDL document, creates a service using the WSDL information, gets the port to the SOAP Web Service and then invokes the validate method. The method Endpoint.stop() stops the publishin of the service and shutsdown the in-memory web server. Again, you should be careful as this integration test uses the default HTTP server which is in a com.sun package and therefore not portable.

If you've read some of my other articles on this blog you probably know I'm a fan of HTML5. With HTML5 we get all this interesting functionality, directly in the browser, in a way that, eventually, is standard across browsers. One of the new HTML5 APIs that is slowly moving through the standardization process is the Web Audio API. With this API, currently only supported in Chrome, we get access to all kinds of interesting audio components you can use to create, modify and visualize sounds (such as the following spectrogram). So why do I start with visualizations? It looks nice, that's one reason, but not the important one. This API provides a number of more complex components, whose behavior is much easier to explain when you can see what happens. With a filter you can instantly see whether some frequencies are filtered, instead of trying to listen to the resulting audio for thse changes. There are many interesting examples that use this API. The problem is, though, that getting started with this API and with digital signal processing (DSP) usually isn't explained. In this article I'll walk you through a couple of steps that shows how to do the following: Create a signal volume meter Visualize the frequencies using a spectrum analyzer And show a time based spectrogram We start with the basic setup that we can use as the basis for the components we'll create. Setting up the basic If we want to experiment with sound, we need some sound source. We could use the microphone (as we'll do later in this series), but to keep it simple, for now we'll just use an mp3 as our input. To get this working using web audio we have to take the following steps: Load the data Read it in a buffer node and play the sound Load the data With the web audio we can use different types of audio sources. We've got a MediaElementAudioSourceNode that can be used to use the audio provided by a media element. There's also a MediaStreamAudioSourceNode. With this audio source node we can use the microphone as input (see my previous article on sound recognition). Finally there is the AudioBufferSourceNode. With this node we can load the data from an existing audio file (e.g mp3) and use that as input. For this example we'll use this last approach. // create the audio context (chrome only for now) var context = new webkitAudioContext(); var audioBuffer; var sourceNode; // load the sound setupAudioNodes(); loadSound("wagner-short.ogg"); function setupAudioNodes() { // create a buffer source node sourceNode = context.createBufferSource(); // and connect to destination sourceNode.connect(context.destination); } // load the specified sound function loadSound(url) { var request = new XMLHttpRequest(); request.open('GET', url, true); request.responseType = 'arraybuffer'; // When loaded decode the data request.onload = function() { // decode the data context.decodeAudioData(request.response, function(buffer) { // when the audio is decoded play the sound playSound(buffer); }, onError); } request.send(); } function playSound(buffer) { sourceNode.buffer = buffer; sourceNode.noteOn(0); } // log if an error occurs function onError(e) { console.log(e); } In this example you can see a couple of functions. The setupAudioNodes function creates a BufferSource audio node and connects it to the destination. The loadSound function shows how you can load an audio file. The buffer which is passed into the playSound function contains decoded audio that can be used by the web audio API. In this example I use an .ogg file, for a complete overview of the formats supported look at: https://sites.google.com/a/chromium.org/dev/audio-video Play the sound To play this audio file, all we have to do is turn the source node on, this is done in the playSound function: function playSound(buffer) { sourceNode.buffer = buffer; sourceNode.noteOn(0); } You can test this out at the following page: Example 1: Loading and playing a sound with Web Audio API. When you open that page, you'll hear some music. Nothing to spectacular for now, but nevertheless an easy way to load audio that'll use for the rest of this article. The first item on our list was the volume meter. Create a volume meter One of the basic scenario's, and often one of the first steps someone new to this API tries to create, is a simple signal volume meter (or an UV meter). I expected this to be a standard component in this API, where I could just read off the signal strength as a property. But, no such node exists. But not to worry, with the components that are available, it's pretty easy (not straightforward, but easy nevertheless) to get an indication of the signal strength of your audio file. Int this section we'll create the following simple volume meter: As you can see this is a simple volume meter where we measure the signal strength for the left and the right audio channel. This is drawn on the canvas, but you could have also used divs or svg to visualize this. Lets start with a single volume meter, instead of one for each channel. For this we need to do the following: Create an analyzer node: With this node we get realtime information about the data that is processed. This data we use to determine the signal strength Create a javascript node: We use this node as a timer to update the volume meters with new information Connect everything together Analyser node With the analyser node we can perform real-time frequency and time domain analysis. From the specification: a node which is able to provide real-time frequency and time-domain analysis information. The audio stream will be passed un-processed from input to output. I won't go into the mathematical details behind this node, since there are many articles out there that explain how this works (a good one is the chapter on fourier transformation from here). What you should now about this node is that it splits up the signal in frequency buckets and we get the amplitude (the signal strenght) for each set of frequencies (the bucket). The best way to understand this, is to skip a bit ahead in this article and look at the frequency distribution we'll create later on. This image plots the result from the analyser node. The frequencies increase from left to right, and the height of the bar shows the strength of that specific frequency bucket. More on this later on in the article. For now we don't want to see the strength of the separate frequency buckets, but the strength of the total signal. For this we'll just add all the strenghts from each bucket and divide it by the number of buckets. First we need to create an analyzer node // setup a analyzer analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.3; analyser.fftSize = 1024; This creates an analyzer node whose result will be used to create the volume meter. We use a smoothingTimeConstant to make the meter less jittery. With this variable we use input from a longer time period to calculate the amplitudes, this results in a more smooth meter. The fftSize determine how many buckets we get containing frequency information. If we have a fftSize of 1024 we get 512 buckets (more info on this in the book on DPS and fourier transformations). When this node receives a stream of data, it analyzes this stream and provides us with information about the frequencies in that signal and their strengths. We now need a timer to update the meter at regular intervals. We could use the standard javascript setInterval function, but since we're looking at the Web Audio API lets use one of its nodes. The JavaScriptNode. The javascript node With the javascriptnode we can process the raw audio data directly from javascript. We can use this to write our own analyzers or complex components. We're not going to do that, though. When creating the javascript node, you can specify the interval at which it is called. We'll use that feature to update the meter at regulat intervals. Creating a javascript node is very easy. // setup a javascript node javascriptNode = context.createJavaScriptNode(2048, 1, 1); This will create a javascriptnode that is called whenever the 2048 frames have been sampled. Since our data is sampled at 44.1k, this function will be called approximately 21 times a second. Now what happens when this function is called: // when the javascript node is called // we use information from the analyzer node // to draw the volume javascriptNode.onaudioprocess = function() { // get the average, bincount is fftsize / 2 var array = new Uint8Array(analyser.frequencyBinCount); analyser.getByteFrequencyData(array); var average = getAverageVolume(array) // clear the current state ctx.clearRect(0, 0, 60, 130); // set the fill style ctx.fillStyle=gradient; // create the meters ctx.fillRect(0,130-average,25,130); } function getAverageVolume(array) { var values = 0; var average; var length = array.length; // get all the frequency amplitudes for (var i = 0; i < length; i++) { values += array[i]; } average = values / length; return average; } In these two functions we calculate the average and draw the meter directly on the canvas (using a gradient so we have nice colors). Now all we have to do is connect the output from the audiosource to the analyser, the analyser to the javasource node (and if we want audio to hear, we also need to connect something to the destionation). Connect everything together Connecting everything together is easy: function setupAudioNodes() { // setup a javascript node javascriptNode = context.createJavaScriptNode(2048, 1, 1); // connect to destination, else it isn't called javascriptNode.connect(context.destination); // setup a analyzer analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.3; analyser.fftSize = 1024; // create a buffer source node sourceNode = context.createBufferSource(); // connect the source to the analyser sourceNode.connect(analyser); // we use the javascript node to draw at a specific interval. analyser.connect(javascriptNode); // and connect to destination, if you want audio sourceNode.connect(context.destination); } And that's it. This will draw a single volume meter, for the complete signal. Now what do we do when we want to have a volume meter for each channel. For this we use a ChannelSplitter. Let's dive right into the code to connect everything: function setupAudioNodes() { // setup a javascript node javascriptNode = context.createJavaScriptNode(2048, 1, 1); // connect to destination, else it isn't called javascriptNode.connect(context.destination); // setup a analyzer analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.3; analyser.fftSize = 1024; analyser2 = context.createAnalyser(); analyser2.smoothingTimeConstant = 0.0; analyser2.fftSize = 1024; // create a buffer source node sourceNode = context.createBufferSource(); splitter = context.createChannelSplitter(); // connect the source to the analyser and the splitter sourceNode.connect(splitter); // connect one of the outputs from the splitter to // the analyser splitter.connect(analyser,0,0); splitter.connect(analyser2,1,0); // we use the javascript node to draw at a // specific interval. analyser.connect(javascriptNode); // and connect to destination sourceNode.connect(context.destination); } As you can see we don't really change much. We introduce a new node, the splitter node. This node splits the sound into a left and a right channel. These channels can be processed separately. With this layout the following happens: The audiosource creates a signal based on the buffered audio. This signal is sent to the splitter, who splits the signal into a left and right stream. Each of these two streams is processed by their own realtime analyser. From the javascript node, we now get the information from both analysers and plot both meters I've shown step 1 through 3, let's quickly move on the step 4. For this we simply add the following to the onaudioprocess node: javascriptNode.onaudioprocess = function() { // get the average for the first channel var array = new Uint8Array(analyser.frequencyBinCount); analyser.getByteFrequencyData(array); var average = getAverageVolume(array); // get the average for the second channel var array2 = new Uint8Array(analyser2.frequencyBinCount); analyser2.getByteFrequencyData(array2); var average2 = getAverageVolume(array2); // clear the current state ctx.clearRect(0, 0, 60, 130); // set the fill style ctx.fillStyle=gradient; // create the meters ctx.fillRect(0,130-average,25,130); ctx.fillRect(30,130-average2,25,130); } And now we've got two signal meters, one for each channel. Example 2: Visualize the signal strength with a volume meter. Or view the result on youtube: Now lets see how we can get the view of the frequencies I showed earlier. Create a frequency spectrum With all the work we already did in the previous section, creating a frequency spectrum overview is now very easy. We're going to aim for this: We set up the nodes just like we did in the first example: function setupAudioNodes() { // setup a javascript node javascriptNode = context.createJavaScriptNode(2048, 1, 1); // connect to destination, else it isn't called javascriptNode.connect(context.destination); // setup a analyzer analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.3; analyser.fftSize = 512; // create a buffer source node sourceNode = context.createBufferSource(); sourceNode.connect(analyser); analyser.connect(javascriptNode); // sourceNode.connect(context.destination); } So this time we don't split the channels and we set the fftSize to 512. This means we get 256 bars that represent our frequency. We now just need to alter the onaudioprocess method and the gradient we use: var gradient = ctx.createLinearGradient(0,0,0,300); gradient.addColorStop(1,'#000000'); gradient.addColorStop(0.75,'#ff0000'); gradient.addColorStop(0.25,'#ffff00'); gradient.addColorStop(0,'#ffffff'); // when the javascript node is called // we use information from the analyzer node // to draw the volume javascriptNode.onaudioprocess = function() { // get the average for the first channel var array = new Uint8Array(analyser.frequencyBinCount); analyser.getByteFrequencyData(array); // clear the current state ctx.clearRect(0, 0, 1000, 325); // set the fill style ctx.fillStyle=gradient; drawSpectrum(array); } function drawSpectrum(array) { for ( var i = 0; i < (array.length); i++ ){ var value = array[i]; ctx.fillRect(i*5,325-value,3,325); } }; In the drawSpectrum function we iterate over the array, and draw a vertical bar based on the value. That's it. For a live example, click on the following link: Example 3: Visualize the frequency spectrum. Or view it on youtube: And then the final one. The spectrogram. Time based spectrogram When you run the previous demo you see the strength of the various frequency buckets in real time. While this is a nice visualization, it doesn't allow you to analyze information over a period of time. If you want to do that you can create a spectrogram. With a spectrogram we plot a single line for each measurement. The y-axis represents the frequency, the x-asis the time and the color of a pixel the strength of that frequency. It can be used to analyze the received audio, and also creates nice looking images. The good thing, is that to output this data we don't have to change much from what we've already got in place. The only function that'll change is the onaudioprocess node and we'll create a slightly different analyser. analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0; analyser.fftSize = 1024; The enalyser we create here has an fftSize of 1024, this means we get 512 frequency buckets with strengths. So we can draw a spectrogram that has a height of 512 pixels. Also note that the smoothingTimeConstant is set to 0. This means we don't use any of the previous results in the analysis. We want to show the real information, not provide a smooth volume meter or frequency spectrum analysis. The easiest way to draw a spectrogram is by just start drawing the line at the left, and for each new set of frequencies increase the x-coordinate by one. The problem is that this will quickly fill up our canvas, and we'll only be able to see the first half a minute of the audio. To fix this, we need some creative canvas copying. The complete code for drawing the spectrogram is shown here: // create a temp canvas we use for copying and scrolling var tempCanvas = document.createElement("canvas"), tempCtx = tempCanvas.getContext("2d"); tempCanvas.width=800; tempCanvas.height=512; // used for color distribution var hot = new chroma.ColorScale({ colors:['#000000', '#ff0000', '#ffff00', '#ffffff'], positions:[0, .25, .75, 1], mode:'rgb', limits:[0, 300] }); ... // when the javascript node is called // we use information from the analyzer node // to draw the volume javascriptNode.onaudioprocess = function () { // get the average for the first channel var array = new Uint8Array(analyser.frequencyBinCount); analyser.getByteFrequencyData(array); // draw the spectrogram if (sourceNode.playbackState == sourceNode.PLAYING_STATE) { drawSpectrogram(array); } } function drawSpectrogram(array) { // copy the current canvas onto the temp canvas var canvas = document.getElementById("canvas"); tempCtx.drawImage(canvas, 0, 0, 800, 512); // iterate over the elements from the array for (var i = 0; i < array.length; i++) { // draw each pixel with the specific color var value = array[i]; ctx.fillStyle = hot.getColor(value).hex(); // draw the line at the right side of the canvas ctx.fillRect(800 - 1, 512 - i, 1, 1); } // set translate on the canvas ctx.translate(-1, 0); // draw the copied image ctx.drawImage(tempCanvas, 0, 0, 800, 512, 0, 0, 800, 512); // reset the transformation matrix ctx.setTransform(1, 0, 0, 1, 0, 0); } To draw the spectrogram we do the following: We copy what is currently drawn to a hidden canvas Next we draw a line of the current values at the far right of the canvas We set the translate on the canvas to -1 We copy the copied information back to the original canvas (that is now drawn 1 pixel to the left) And reset the transformation matrix See a running example here: Example 4: Create a spectrogram Or view it here: Last thing I'd like to mention regarding the code is the chroma.js library I used for the colors. If you ever need to draw something color or gradient related (e.g maps, strengths, levels) you can easily create color scales with this library. Two final pointers, I know I'll get questions about: Volume could be represented as a magnitude, just didn't want to complicate matters for this. The spectogram doesn't use logarithmic scales. Once again, didn't want to complicate things

ASP.NET Web API is a great tool to build an API with. Or as my buddy Kristof Rennen (and the French) always say: “it makes you ‘api”. One of the things I like a lot is the fact that you can do very powerful things that you know and love from the ASP.NET MVC stack, like, for example, using filter attributes. Action filters, result filters and… authorization filters. Say you wanted to protect your API and make use of the controller’s User property to return user-specific information. You probably will add an [Authorize] attribute (to ensure the user is authenticated) to either the entire API controller or to one of its action methods, like this: [Authorize] public class SuperSecretController : ApiController { public string Get() { return string.Format("Hello, {0}", User.Identity.Name); } } Great! But how will your application know who’s calling? Forms authentication doesn’t really make sense for a lot of API’s. Configuring IIS and switching to Windows authentication or basic authentication may be an option. But not every ASP.NET Web API will live in IIS, right? And maybe you want to use some other form of authentication for your API, for example one that uses a custom HTTP header containing an API key? Let’s see how you can do that… Our API authentication? An API key API keys may make sense for your API. They provide an easy means of authenticating your API consumers based on a simple token that is passed around in a custom header. OAuth2 may make sense as well, but even that one boils down to a custom Authorization header at the HTTP level. (hint: the approach outlined in this post can be used for OAuth2 tokens as well) Let’s build our API and require every API consumer to pass in a custom header, named “X-ApiKey”. Calls to our API will look like this: GET http://localhost:60573/api/v1/SuperSecret HTTP/1.1 Host: localhost:60573 X-ApiKey: 12345 In our SuperSecretController above, we want to make sure that we’re working with a traditional IPrincipal which we can query for username, roles and possibly even claims if needed. How do we get that identity there? Translating the API key using a DelegatingHandler The title already gives you a pointer. We want to add a plugin into ASP.NET Web API’s pipeline which replaces the current thread’s IPrincipal with one that is mapped from the incoming API key. That plugin will come in the form of a DelegatingHandler, a class that’s plugged in really early in the ASP.NET Web API pipeline. I’m not going to elaborate on what DelegatingHandler does and where it fits, there’s a perfect post on that to be found here. Our handler, which I’ll call AuthorizationHeaderHandler will be inheriting ASP.NET Web API’s DelegatingHandler. The method we’re interested in is SendAsync, which will be called on every request into our API. public class AuthorizationHeaderHandler : DelegatingHandler { protected override Task SendAsync( HttpRequestMessage request, CancellationToken cancellationToken) { // ... } } This method offers access to the HttpRequestMessage, which contains everything you’ll probably be needing such as… HTTP headers! Let’s read out our X-ApiKey header, convert it to a ClaimsIdentity (so we can add additional claims if needed) and assign it to the current thread: public class AuthorizationHeaderHandler : DelegatingHandler { protected override Task SendAsync( HttpRequestMessage request, CancellationToken cancellationToken) { IEnumerable apiKeyHeaderValues = null; if (request.Headers.TryGetValues("X-ApiKey", out apiKeyHeaderValues)) { var apiKeyHeaderValue = apiKeyHeaderValues.First(); // ... your authentication logic here ... var username = (apiKeyHeaderValue == "12345" ? "Maarten" : "OtherUser"); var usernameClaim = new Claim(ClaimTypes.Name, username); var identity = new ClaimsIdentity(new[] {usernameClaim}, "ApiKey"); var principal = new ClaimsPrincipal(identity); Thread.CurrentPrincipal = principal; } return base.SendAsync(request, cancellationToken); } } Easy, no? The only thing left to do is registering this handler in the pipeline during your application’s start: GlobalConfiguration.Configuration.MessageHandlers.Add(new AuthorizationHeaderHandler()); From now on, any request coming in with the X-ApiKey header will be translated into an IPrincipal which you can easily use throughout your web API. Enjoy! PS: if you’re looking into OAuth2, I’ve used a similar approach in “ASP.NET Web API OAuth2 delegation with Windows Azure Access Control Service” to handle OAuth2 tokens.

The content of this article was originally written by Tae Jin Gu on the Cubrid blog. When there is an obstacle, or when a Java based Web application is running much slower than expected, we need to use thread dumps. If thread dumps feel like very complicated to you, this article may help you very much. Here I will explain what threads are in Java, their types, how they are created, how to manage them, how you can dump threads from a running application, and finally how you can analyze them and determine the bottleneck or blocking threads. This article is a result of long experience in Java application debugging. Java and Thread A web server uses tens to hundreds of threads to process a large number of concurrent users. If two or more threads utilize the same resources, a contention between the threads is inevitable, and sometimes deadlock occurs. Thread contention is a status in which one thread is waiting for a lock, held by another thread, to be lifted. Different threads frequently access shared resources on a web application. For example, to record a log, the thread trying to record the log must obtain a lock and access the shared resources. Deadlock is a special type of thread contention, in which two or more threads are waiting for the other threads to complete their tasks in order to complete their own tasks. Different issues can arise from thread contention. To analyze such issues, you need to use the thread dump. A thread dump will give you the information on the exact status of each thread. Background Information for Java Threads Thread Synchronization A thread can be processed with other threads at the same time. In order to ensure compatibility when multiple threads are trying to use shared resources, one thread at a time should be allowed to access the shared resources by using thread synchronization. Thread synchronization on Java can be done using monitor. Every Java object has a single monitor. The monitor can be owned by only one thread. For a thread to own a monitor that is owned by a different thread, it needs to wait in the wait queue until the other thread releases its monitor. Thread Status In order to analyze a thread dump, you need to know the status of threads. The statuses of threads are stated on java.lang.Thread.State. Figure 1: Thread Status. NEW: The thread is created but has not been processed yet. RUNNABLE: The thread is occupying the CPU and processing a task. (It may be in WAITING status due to the OS's resource distribution.) BLOCKED: The thread is waiting for a different thread to release its lock in order to get the monitor lock. WAITING: The thread is waiting by using a wait, join or park method. TIMED_WAITING: The thread is waiting by using a sleep, wait, join or park method. (The difference from WAITING is that the maximum waiting time is specified by the method parameter, and WAITING can be relieved by time as well as external changes.) Thread Types Java threads can be divided into two: daemon threads; and non-daemon threads. Daemon threads stop working when there are no other non-daemon threads. Even if you do not create any threads, the Java application will create several threads by default. Most of them are daemon threads, mainly for processing tasks such as garbage collection or JMX. A thread running the 'static void main(String[] args)’ method is created as a non-daemon thread, and when this thread stops working, all other daemon threads will stop as well. (The thread running this main method is called the VM thread in HotSpot VM.) Getting a Thread Dump We will introduce the three most commonly used methods. Note that there are many other ways to get a thread dump. A thread dump can only show the thread status at the time of measurement, so in order to see the change in thread status, it is recommended to extract them from 5 to 10 times with 5-second intervals. Getting a Thread Dump Using jstack In JDK 1.6 and higher, it is possible to get a thread dump on MS Windows using jstack. Use PID via jps to check the PID of the currently running Java application process. [user@linux ~]$ jps -v 25780 RemoteTestRunner -Dfile.encoding=UTF-8 25590 sub.rmi.registry.RegistryImpl 2999 -Dapplication.home=/home1/user/java/jdk.1.6.0_24 -Xms8m 26300 sun.tools.jps.Jps -mlvV -Dapplication.home=/home1/user/java/jdk.1.6.0_24 -Xms8m Use the extracted PID as the parameter of jstack to obtain a thread dump. [user@linux ~]$ jstack -f 5824 A Thread Dump Using jVisualVM Generate a thread dump by using a program such as jVisualVM. Figure 2: A Thread Dump Using visualvm. The task on the left indicates the list of currently running processes. Click on the process for which you want the information, and select the thread tab to check the thread information in real time. Click the Thread Dump button on the top right corner to get the thread dump file. Generating in a Linux Terminal Obtain the process pid by using ps -ef command to check the pid of the currently running Java process. [user@linux ~]$ ps - ef | grep java user 2477 1 0 Dec23 ? 00:10:45 ... user 25780 25361 0 15:02 pts/3 00:00:02 ./jstatd -J -Djava.security.policy=jstatd.all.policy -p 2999 user 26335 25361 0 15:49 pts/3 00:00:00 grep java Use the extracted pid as the parameter of kill –SIGQUIT(3) to obtain a thread dump. Thread Information from the Thread Dump File "pool-1-thread-13" prio=6 tid=0x000000000729a000 nid=0x2fb4 runnable [0x0000000007f0f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) - locked <0x0000000780b7e688> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:167) at java.io.BufferedReader.fill(BufferedReader.java:136) at java.io.BufferedReader.readLine(BufferedReader.java:299) - locked <0x0000000780b7e688> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(BufferedReader.java:362) ) Thread name: When using Java.lang.Thread class to generate a thread, the thread will be named Thread-(Number), whereas when using java.util.concurrent.ThreadFactory class, it will be named pool-(number)-thread-(number). Priority: Represents the priority of the threads. Thread ID: Represents the unique ID for the threads. (Some useful information, including the CPU usage or memory usage of the thread, can be obtained by using thread ID.) Thread status: Represents the status of the threads. Thread callstack: Represents the call stack information of the threads. Thread Dump Patterns by Type When Unable to Obtain a Lock (BLOCKED) This is when the overall performance of the application slows down because a thread is occupying the lock and prevents other threads from obtaining it. In the following example, BLOCKED_TEST pool-1-thread-1 thread is running with <0x0000000780a000b0> lock, while BLOCKED_TEST pool-1-thread-2 and BLOCKED_TEST pool-1-thread-3 threads are waiting to obtain <0x0000000780a000b0> lock. Figure 3: A thread blocking other threads. "BLOCKED_TEST pool-1-thread-1" prio=6 tid=0x0000000006904800 nid=0x28f4 runnable [0x000000000785f000] java.lang.Thread.State: RUNNABLE at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:282) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) - locked <0x0000000780a31778> (a java.io.BufferedOutputStream) at java.io.PrintStream.write(PrintStream.java:432) - locked <0x0000000780a04118> (a java.io.PrintStream) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272) at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85) - locked <0x0000000780a040c0> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168) at java.io.PrintStream.newLine(PrintStream.java:496) - locked <0x0000000780a04118> (a java.io.PrintStream) at java.io.PrintStream.println(PrintStream.java:687) - locked <0x0000000780a04118> (a java.io.PrintStream) at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:44) - locked <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState) at com.nbp.theplatform.threaddump.ThreadBlockedState$1.run(ThreadBlockedState.java:7) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000780a31758> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "BLOCKED_TEST pool-1-thread-2" prio=6 tid=0x0000000007673800 nid=0x260c waiting for monitor entry [0x0000000008abf000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:43) - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState) at com.nbp.theplatform.threaddump.ThreadBlockedState$2.run(ThreadBlockedState.java:26) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000780b0c6a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "BLOCKED_TEST pool-1-thread-3" prio=6 tid=0x00000000074f5800 nid=0x1994 waiting for monitor entry [0x0000000008bbf000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:42) - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState) at com.nbp.theplatform.threaddump.ThreadBlockedState$3.run(ThreadBlockedState.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000780b0e1b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) When in Deadlock Status This is when thread A needs to obtain thread B's lock to continue its task, while thread B needs to obtain thread A's lock to continue its task. In the thread dump, you can see that DEADLOCK_TEST-1 thread has 0x00000007d58f5e48 lock, and is trying to obtain 0x00000007d58f5e60 lock. You can also see that DEADLOCK_TEST-2 thread has 0x00000007d58f5e60 lock, and is trying to obtain 0x00000007d58f5e78 lock. Also, DEADLOCK_TEST-3 thread has 0x00000007d58f5e78 lock, and is trying to obtain 0x00000007d58f5e48 lock. As you can see, each thread is waiting to obtain another thread's lock, and this status will not change until one thread discards its lock. Figure 4: Threads in a Deadlock status. "DEADLOCK_TEST-1" daemon prio=6 tid=0x000000000690f800 nid=0x1820 waiting for monitor entry [0x000000000805f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197) - waiting to lock <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182) - locked <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers: - None "DEADLOCK_TEST-2" daemon prio=6 tid=0x0000000006858800 nid=0x17b8 waiting for monitor entry [0x000000000815f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197) - waiting to lock <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182) - locked <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers: - None "DEADLOCK_TEST-3" daemon prio=6 tid=0x0000000006859000 nid=0x25dc waiting for monitor entry [0x000000000825f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197) - waiting to lock <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182) - locked <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers: - None When Continuously Waiting to Receive Messages from a Remote Server The thread appears to be normal, since its state keeps showing as RUNNABLE. However, when you align the thread dumps chronologically, you can see that socketReadThread thread is waiting infinitely to read the socket. Figure 5: Continuous Waiting Status. "socketReadThread" prio=6 tid=0x0000000006a0d800 nid=0x1b40 runnable [0x00000000089ef000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) - locked <0x00000007d78a2230> (a java.io.InputStreamReader) at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:107) - locked <0x00000007d78a2230> (a java.io.InputStreamReader) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:93) at java.io.InputStreamReader.read(InputStreamReader.java:151) at com.nbp.theplatform.threaddump.ThreadSocketReadState$1.run(ThreadSocketReadState.java:27) at java.lang.Thread.run(Thread.java:662) When Waiting The thread is maintaining WAIT status. In the thread dump, IoWaitThread thread keeps waiting to receive a message from LinkedBlockingQueue. If there continues to be no message for LinkedBlockingQueue, then the thread status will not change. Figure 6: Waiting status. "IoWaitThread" prio=6 tid=0x0000000007334800 nid=0x2b3c waiting on condition [0x000000000893f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007d5c45850> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:440) at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:629) at com.nbp.theplatform.threaddump.ThreadIoWaitState$IoWaitHandler2.run(ThreadIoWaitState.java:89) at java.lang.Thread.run(Thread.java:662) When Thread Resources Cannot be Organized Normally Unnecessary threads will pile up when thread resources cannot be organized normally. If this occurs, it is recommended to monitor the thread organization process or check the conditions for thread termination. Figure 7: Unorganized Threads. How to Solve Problems by Using Thread Dump Example 1: When the CPU Usage is Abnormally High 1. Extract the thread that has the highest CPU usage. [user@linux ~]$ ps -mo pid.lwp.stime.time.cpu -C java PID LWP STIME TIME %CPU 10029 - Dec07 00:02:02 99.5 - 10039 Dec07 00:00:00 0.1 - 10040 Dec07 00:00:00 95.5 From the application, find out which thread is using the CPU the most. Acquire the Light Weight Process (LWP) that uses the CPU the most and convert its unique number (10039) into a hexadecimal number (0x2737). 2. After acquiring the thread dump, check the thread's action. Extract the thread dump of an application with a PID of 10029, then find the thread with an nid of 0x2737. "NioProcessor-2" prio=10 tid=0x0a8d2800 nid=0x2737 runnable [0x49aa5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x74c52678> (a sun.nio.ch.Util$1) - locked <0x74c52668> (a java.util.Collections$UnmodifiableSet) - locked <0x74c501b0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at external.org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65) at external.org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:708) at external.org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Extract thread dumps several times every hour, and check the status change of the threads to determine the problem. Example 2: When the Processing Performance is Abnormally Slow After acquiring thread dumps several times, find the list of threads with BLOCKED status. " DB-Processor-13" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f000] java.lang.Thread.State: BLOCKED (on object monitor) at beans.ConnectionPool.getConnection(ConnectionPool.java:102) - waiting to lock <0xe0375410> (a beans.ConnectionPool) at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111) at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43) "DB-Processor-14" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f020] java.lang.Thread.State: BLOCKED (on object monitor) at beans.ConnectionPool.getConnection(ConnectionPool.java:102) - waiting to lock <0xe0375410> (a beans.ConnectionPool) at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111) at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43) " DB-Processor-3" daemon prio=5 tid=0x00928248 nid=0x8b waiting for monitor entry [0x000000000825d080] java.lang.Thread.State: RUNNABLE at oracle.jdbc.driver.OracleConnection.isClosed(OracleConnection.java:570) - waiting to lock <0xe03ba2e0> (a oracle.jdbc.driver.OracleConnection) at beans.ConnectionPool.getConnection(ConnectionPool.java:112) - locked <0xe0386580> (a java.util.Vector) - locked <0xe0375410> (a beans.ConnectionPool) at beans.cus.Cue_1700c.GetNationList(Cue_1700c.java:66) at org.apache.jsp.cue_1700c_jsp._jspService(cue_1700c_jsp.java:120) Acquire the list of threads with BLOCKED status after getting the thread dumps several times. If the threads are BLOCKED, extract the threads related to the lock that the threads are trying to obtain. Through the thread dump, you can confirm that the thread status stays BLOCKED because <0xe0375410> lock could not be obtained. This problem can be solved by analyzing stack trace from the thread currently holding the lock. There are two reasons why the above pattern frequently appears in applications using DBMS. The first reason is inadequate configurations. Despite the fact that the threads are still working, they cannot show their best performance because the configurations for DBCP and the like are not adequate. If you extract thread dumps multiple times and compare them, you will often see that some of the threads that were BLOCKED previously are in a different state. The second reason is the abnormal connection. When the connection with DBMS stays abnormal, the threads wait until the time is out. In this case, even after extracting the thread dumps several times and comparing them, you will see that the threads related to DBMS are still in a BLOCKED state. By adequately changing the values, such as the timeout value, you can shorten the time in which the problem occurs. Coding for Easy Thread Dump Naming Threads When a thread is created using java.lang.Thread object, the thread will be named Thread-(Number). When a thread is created using java.util.concurrent.DefaultThreadFactory object, the thread will be named pool-(Number)-thread-(Number). When analyzing tens to thousands of threads for an application, if all the threads still have their default names, analyzing them becomes very difficult, because it is difficult to distinguish the threads to be analyzed. Therefore, you are recommended to develop the habit of naming the threads whenever a new thread is created. When you create a thread using java.lang.Thread, you can give the thread a custom name by using the creator parameter. public Thread(Runnable target, String name); public Thread(ThreadGroup group, String name); public Thread(ThreadGroup group, Runnable target, String name); public Thread(ThreadGroup group, Runnable target, String name, long stackSize); When you create a thread using java.util.concurrent.ThreadFactory, you can name it by generating your own ThreadFactory. If you do not need special functionalities, then you can use MyThreadFactory as described below: import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ThreadFactory; import java.util.concurrent.atomic.AtomicInteger; public class MyThreadFactory implements ThreadFactory { private static final ConcurrentHashMap POOL_NUMBER = new ConcurrentHashMap(); private final ThreadGroup group; private final AtomicInteger threadNumber = new AtomicInteger(1); private final String namePrefix; public MyThreadFactory(String threadPoolName) { if (threadPoolName == null) { throw new NullPointerException("threadPoolName"); } POOL_NUMBER.putIfAbsent(threadPoolName, new AtomicInteger()); SecurityManager securityManager = System.getSecurityManager(); group = (securityManager != null) ? securityManager.getThreadGroup() : Thread.currentThread().getThreadGroup(); AtomicInteger poolCount = POOL_NUMBER.get(threadPoolName); if (poolCount == null) { namePrefix = threadPoolName + " pool-00-thread-"; } else { namePrefix = threadPoolName + " pool-" + poolCount.getAndIncrement() + "-thread-"; } } public Thread newThread(Runnable runnable) { Thread thread = new Thread(group, runnable, namePrefix + threadNumber.getAndIncrement(), 0); if (thread.isDaemon()) { thread.setDaemon(false); } if (thread.getPriority() != Thread.NORM_PRIORITY) { thread.setPriority(Thread.NORM_PRIORITY); } return thread; } } Obtaining More Detailed Information by Using MBean You can obtain ThreadInfo objects using MBean. You can also obtain more information that would be difficult to acquire via thread dumps, by using ThreadInfo. ThreadMXBean mxBean = ManagementFactory.getThreadMXBean(); long[] threadIds = mxBean.getAllThreadIds(); ThreadInfo[] threadInfos = mxBean.getThreadInfo(threadIds); for (ThreadInfo threadInfo : threadInfos) { System.out.println( threadInfo.getThreadName()); System.out.println( threadInfo.getBlockedCount()); System.out.println( threadInfo.getBlockedTime()); System.out.println( threadInfo.getWaitedCount()); System.out.println( threadInfo.getWaitedTime()); } You can acquire the amount of time that the threads WAITed or were BLOCKED by using the method in ThreadInfo, and by using this you can also obtain the list of threads that have been inactive for an abnormally long period of time. In Conclusion In this article I was concerned that for developers with a lot of experience in multi-thread programming, this material may be common knowledge, whereas for less experienced developers, I felt that I was skipping straight to thread dumps, without providing enough background information about the thread activities. This was because of my lack of knowledge, as I was not able to explain the thread activities in a clear yet concise manner. I sincerely hope that this article will prove helpful for many developers.

SOAP headers can be added to a Web service request in different ways, if you use Apache CXF. The way I prefer is the one I’ve mentioned here – as it doesn’t require changes to wsdl or method signatures and it’s much faster as it doesn’t break streaming and the memory overhead is less. The headers in the list are streamed at the appropriate time to the wire according to the databinding object found in the Header object. About SOAP headers, Like any good messaging protocol, SOAP defines the concept of a message header. It is an optional part of any SOAP message. If it exists, the header contains application-specific information (like authentication, payment, etc) about the SOAP message i.e. information about the message, or about the context in which the message is sent, or basically whatever the creator of the message thought was a good idea to put there instead of the actual body of the message. If the Header element is present, it must be the first child element of the Envelope element. /** * @author Singaram Subramanian * */ /* Create a ClientProxyFactoryBean reference and assign it an instance of JaxWsProxyFactoryBean, a factory for creating JAX-WS proxies. This class provides access to the internal properties used to set-up proxies. Using it provides more control than the standard JAX-WS APIs. */ ClientProxyFactoryBean factory = new JaxWsProxyFactoryBean(); factory.setServiceClass(singz.ws.cxf.sample.SampleServiceInterface.class); // Set the web service endpoint URL here factory.setAddress("http://xxx.xxx.com/services/SampleService/v1"); SampleServiceInterface serviceClient = (SampleServiceInterface) factory.create(); // Get the underlying Client object from the proxy object of service interface Client proxy = ClientProxy.getClient(serviceClient); // Creating SOAP headers to the web service request // Create a list for holding all SOAP headers List headersList = new ArrayList(); Header testSoapHeader1 = new Header(new QName("uri:singz.ws.sample", "soapheader1"), "SOAP Header Message 1", new JAXBDataBinding(String.class)); Header testSoapHeader2 = new Header(new QName("uri:singz.ws.sample", "soapheader2"), "SOAP Header Message 2", new JAXBDataBinding(String.class)); headersList.add(testSoapHeader1); headersList.add(testSoapHeader2); // Add SOAP headers to the web service request proxy.getRequestContext().put(Header.HEADER_LIST, headersList); More on this @ http://cxf.apache.org/faq.html#FAQ-HowcanIaddsoapheaderstotherequest%2Fresponse%3F

For my own interest I started exploring Spring REST Services with GWT. It took some time to figure it out and then I came accross RestyGWT. With the help of RestyGWT, I managed to integrate GWT with Spring REST Services. My idea was to keep the GWT way of creating the service and serviceAsync and yet make a REST call. Below are the steps which will help achieve above. My preffered development environment is Eclipse, so as a prerequisite you must have Eclipse with Maven support installed. Lets Begin, Create a Maven project, goto File-->New-->Other. In the wizard type "Maven". Select Maven Project and click on Next. In the "Select project name and location" page of the wizard, make sure that "Create a simple project (skip archetype selection)" option is checked, hit "Next" to continue with default values. In the "Enter group id for the artifact" page of the wizard, enter values for group id and artifactid. Select the packaging as "war", hit "Finish" to exit the wizard and to create your project. Modify the POM file to add required dependencies as below 4.0.0 com.sagar.restgwt RestGWT 0.0.1-SNAPSHOT war com.google.gwt gwt-servlet ${gwt.version} runtime com.google.gwt gwt-user ${gwt.version} provided org.fusesource.restygwt restygwt 1.2 javax.ws.rs jsr311-api 1.1 provided org.codehaus.jackson jackson-mapper-asl 1.4.1 org.springframework spring-core ${org.springframework.version} org.springframework spring-web ${org.springframework.version} org.springframework spring-webmvc ${org.springframework.version} . 2.5.0-rc1 1.6 3.1.1.RELEASE UTF-8 ${project.build.directory}\${project.build.finalName} restgwt ${webappDirectory}/WEB-INF/classes org.codehaus.mojo gwt-maven-plugin 2.5.0-rc1 compile test generateAsync ${webappDirectory} org.apache.maven.plugins maven-compiler-plugin ${java-version} ${java-version} true "Update Project Configuration" by Right clicking on your project-->Maven. The below steps assumes that your are aware of the GWT project structure. Create your GWT module. This can be done by installing GWT plugin for Eclipse. Once the GWT module is ready update your .gwt.xml with the below given content. In the client package create the service to make REST Call. With this approach we dont have to create the ServiceAsync interface. We will be creating our service interface by extending the "RestService", provided by RestyGWT. Code Snippet: InfoService.Java [ A service interface to make REST call. ] @Path("/service") public interface InfoService extends RestService { public static class Util { private static InfoService instance; public static InfoService getService() { if (instance == null) { instance = GWT.create(InfoService.class); } Resource resource = new Resource(GWT.getModuleBaseURL() + "service"); ((RestServiceProxy) instance).setResource(resource); return instance; } } @GET @Path("/loadInfo") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.APPLICATION_JSON) public void getInfo(MethodCallback callback); } OrderConfirmation.java [ Model which will be returned as a response. ] public class OrderConfirmation { public String message; public Long ready_time; /** * Example of how to create an instance of a JsonEncoderDecoder for a data * transfer object. */ public interface OrderConfirmationJED extends JsonEncoderDecoder { } @Override public String toString() { if (GWT.isClient()) { OrderConfirmationJED jed = GWT.create(OrderConfirmationJED.class); return jed.encode(this).toString(); } return super.toString(); } } RestGWT.java [ GWT module entrypoint to see things running. ] public class RestGWT implements EntryPoint { public void onModuleLoad() { Button button = new Button("Click Me"); button.addClickHandler(new ClickHandler() { @Override public void onClick(ClickEvent event) { InfoService.Util.getService().getInfo(new MethodCallback() { @Override public void onSuccess(Method method, OrderConfirmation response) { RootPanel.get().add(new Label(response.toString())); } @Override public void onFailure(Method method, Throwable exception) { GWT.log("Error"); } }); } }); RootPanel.get().add(button); } } Create the Spring managed controller as below, this should be presnt in the "server" package as per GWT project structure. Also you can notice that we don't have to implement our service interface. RestGWTController.java [ Spring managed controller. ] @Controller public class RestGWTController { @RequestMapping(value = "/loadInfo", method = RequestMethod.GET, headers = "Accept=application/json") public @ResponseBody OrderConfirmation handleRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { GreetingServiceEndpoint endpoint = greetingService.getGreetingServiceEndpointPort(); OrderConfirmation confirmation = new OrderConfirmation(); confirmation.message = endpoint.sayHello(); confirmation.ready_time = System.currentTimeMillis() + 1000 * 60 * 30; return confirmation; } } Settings required for Spring src/main/webapp/WEB-INF/web.xml Rest GWT This is web-project for RestGWT contextConfigLocation /WEB-INF/applicationContext.xml org.springframework.web.context.ContextLoaderListener Spring MVC Dispatcher Servlet org.springframework.web.servlet.DispatcherServlet contextConfigLocation /WEB-INF/classes/action-servlet.xml 1 Spring MVC Dispatcher Servlet /restgwt/service/* RestGWT.html src/main/webapp/WEB-INF/applicationCotext.xml src/main/webapp/WEB-INF/classes/action-servlet.xml Now all configuration is completed. To build the application right click on your project → Run As → Maven Install. This will create the war file in your project's target/restgwt folder. To test this approach we are going to deploy our web application to an Apache Tomcat 7 server. To launch the application point your browser to the following address http://localhost:8080/restgwt/ Enjoy Coding Nilabh

this simple cheat sheet contains all the key goals, principals and design patterns that you should be aware of when designing soa services and contains helpful links to places where you can find more in-depth information on each topic. when i was studying for my soa certified architect exams, i kept notes on all the best bits from the course material. after 9 months and several hundred hours of study, i found that there were certain key pieces of information that i kept referring back to time and time again, such as… how do you define service-orientation? what are the goals and strategic benefits of having a service-oriented business? what are the design principals you should apply to soa service design & soa governance? what are the characteristics of soa based businesses – how can you recognise one? what are the most useful soa design patterns and how are they grouped? i thought it might be useful to bring all this information together into one place, so in collaboration with soagrowers we have published a free pdf cheat sheet on soa service design which you can print out and keep close to hand so it’s there whenever you need it. it’s not meant to be an exhaustive guide – it’s just a set of place-holders to remind you of the topics that may be of relevance to you when designing services. however, it should prove useful to any service architect or developer who’s interested in service design or anyone who is going through the same certification programme as i did – even if you just use it as a check-list or aide-mémoire . none of it is particularly technology specific. the same set of goals, principals and patterns can be applied equally to soap based web services , restful services or any other kind of distributed components – that’s the beauty of service-orientation, it’s vendor and technology neutral. in the sheet i’ve also highlighted something that often get’s overlooked when technologists have the lead on soa implementations:- soa has some very attractive and unique business benefits that can only be fully realised when you apply the design paradigm correctly. for my money, it’s this outcome oriented viewpoint (the business case if you like) that really differentiates soa from other tactics like eai/esb, but all too often this message gets lost in the melee . we hope you find it useful. to get your copy of the soa service design cheat sheet, just click on the image below. if you like it please share it (there are handy share buttons on the page below). click on the image to download the pdf get involved. did you find this useful? is there something you think could be added or removed? did you notice how esb is just a small fraction of the bigger picture? let me know your thoughts in the comments below. ————————————————————————— updated: 18/09/2012. i’ve now added a small section on contract first service design, just because it so fundamentally underpins many of the most important goals, principals and patterns used to deliver successful soa. for more information on contract first, see spring-ws’s excellent whitepaper . contract-first isn’t just a soap thing by the way. ‘contract’ in a soa design context means operations, data types, policies and anything else to do with the service’s public facia. so although rest has an implicit contract with predetermined operations (get, put, post, etc.) it still has data type’s and flexible url’s that convey some meaning. therefore, if you want to make a rest architecture more interoperable and less brittle for clients, it helps to plan these datatypes and url’s in advance if you can so they become better standardised and therefore more reusable.

For a given WSDL, there are several different ways to generate Java web service code (CXF, Axis2, etc..). And depending on certain settings within the WSDL file and settings used by the relevant build tool, there are different ways of exposing those services described in the WSDL. This post will briefly document the generating of Java code for a WSDL using Maven and the jaxws wsimport plugin. It will also show the difference in the services exposed when using wrapped and non-wrapped bindings. Below is an extract from a pom.xml to generate the Java code: org.codehaus.mojo jaxws-maven-plugin 1.10 wsimport City81SOAPService.wsdl ${basedir}/src/wsdl generate-sources generate-sources ......... ${project.build.directory}/generated-sources/jaxws-wsimport true true true ${basedir}/src/jax-ws-catalog.xml For the below WSDL file, the wsimport plugin will generate the following classes: com\city81\soap\Balance.java com\city81\soap\City81SOAP.java com\city81\soap\City81SOAPImplService.java com\city81\soap\CreateCustomer.java com\city81\soap\CreateCustomerResponse.java com\city81\soap\CreateCustomerResponseType.java com\city81\soap\CreateStatus.java com\city81\soap\ObjectFactory.java com\city81\soap\package-info.java For the above settings, the generated City81SOAP class will be as below: @WebService(name = "City81SOAP", targetNamespace = "http://soap.city81.com/") @SOAPBinding(parameterStyle = SOAPBinding.ParameterStyle.BARE) @XmlSeeAlso({ ObjectFactory.class }) public interface City81SOAP { @WebMethod(action = "http://soap.city81.com/createCustomer") @WebResult(name = "createCustomerResponse", targetNamespace = "http://soap.city81.com/", partName = "params") public CreateCustomerResponse createCustomer(@WebParam(name = "createCustomer", targetNamespace = "http://soap.city81.com/", partName = "params") CreateCustomer params); } The binding style as can be seen from the @SOAPBinding annotation at the head of the class is BARE ie non-wrapped. The method's args and return parameters are in each case represented as a single Java object. CreateCustomer and CreateCustomerResponse. This has happened because in the pom.xml file, there is a bindingDirectory tag which points to a folder containing a binding.xml file. This file, shown below, has an enableWrapperStyle tag and the boolean value of false. false If the boolean was true, or if there was no bindingDirectory tag in the pom.xml file, then the default SOAP binding style would be used ie WRAPPED. This would then result in the below generated City81SOAP class: @WebService(name = "City81SOAP", targetNamespace = "http://soap.city81.com/") @XmlSeeAlso({ ObjectFactory.class }) public interface City81SOAP { @WebMethod(action = "http://soap.city81.com/createCustomer") @RequestWrapper(localName = "createCustomer", targetNamespace = "http://soap.city81.com/", className = "com.city81.soap.CreateCustomer") @ResponseWrapper(localName = "createCustomerResponse", targetNamespace = "http://soap.city81.com/", className = "com.city81.soap.CreateCustomerResponse") public void createCustomer( @WebParam(name = "surname", targetNamespace = "") String surname, @WebParam(name = "firstName", targetNamespace = "") String firstName, @WebParam(name = "balance", targetNamespace = "") Balance balance, @WebParam(name = "customerId", targetNamespace = "", mode = WebParam.Mode.OUT) Holder customerId, @WebParam(name = "status", targetNamespace = "", mode = WebParam.Mode.OUT) Holder status); } The method's args are now individual Java objects and the return parameters are each represented as Holder objects with a WebParam.Mode.OUT value denoting they are return objects. This means that return objects are set as opposed to actually being returned in the method's signature. Another way to specify bindings other than using the binding.xml file is to embed the enableWrapperStyle as a child of the portType but if a WSDL is from a third party, then having to change it every time a new version of the WSDL is released is open to errors. false ... Back to the generated interfaces, and these of course need to be implemented. For an interface with a binding type of BARE, the implemented class would look like below: @WebService(targetNamespace = "http://soap.city81.com/", name = "City81SOAP", portName = "City81SOAPImplPort", serviceName = "City81SOAPImplService") @SOAPBinding(style = SOAPBinding.Style.DOCUMENT, use = SOAPBinding.Use.LITERAL, parameterStyle = SOAPBinding.ParameterStyle.BARE) public class City81SOAPImpl implements City81SOAP { @Override public CreateCustomerResponse createCustomer(CreateCustomer createCustomer) { CreateCustomerResponse createCustomerResponse = new CreateCustomerResponse(); ..... return createCustomerResponse; } } In the case of WRAPPED binding style, the SOAPBinding annotation would include parameterStyle = SOAPBinding.ParameterStyle.WRAPPED and the createCustomer method would be as below: public void createCustomer( String surname, String firstName, Balance balance, Holder customerId, Holder status) { customerId= new Holder("1"); status = new Holder(CreateStatus.CREATE_PENDING); } This post shows that there are different ways to ultimately achieve the same result.

People tell me A and B, They tell me how I have to see, Things that I have seen already clear, So they push me then from side to side (I Want Out - Helloween) Developing an application that uses RESTful web API may imply developing server and client side. Writing integration tests for the server side can be as easy as using Arquillian to start up server and REST-assured to test that the services works as expected. The problem is how to test the client side. In this post, we are going to see how to test the client side apart from using mocks. As a brief description, to test the client side, what we need is a local server which can return recorded JSON responses. The rest-client-driver is a library which simulates a RESTful service. You can set expectations on the HTTP requests you want to receive during a test. So it is exactly what we need for our java client side. Note that this project is really helpful to write tests when we are developing RESTful web clients for connecting to services developed by third parties like Flickr Rest API, Jira Rest API, Github ... First thing to do is adding rest-client-driver dependency: com.github.rest-driver rest-client-driver 1.1.27 test Next step we are going to create a very simple Jersey application which simply invokes a get method to required URI. public class GithubClient { private static final int HTTP_STATUS_CODE_OK = 200; private String githubBaseUri; public GithubClient(String githubBaseUri) { this.githubBaseUri = githubBaseUri; } public String invokeGetMethod(String resourceName) { Client client = Client.create(); WebResource webResource = client.resource(githubBaseUri+resourceName); ClientResponse response = webResource.type("application/json") .accept("application/json").get(ClientResponse.class); int statusCode = response.getStatus(); if(statusCode != HTTP_STATUS_CODE_OK) { throw new IllegalStateException("Error code "+statusCode); } return response.getEntity(String.class); } } And now we want to test that invokeGetMethod really gets the required resource. Let's suppose that this method in production code will be responsible of getting all issues name from a project registered on github. Now we can start to write the test: @Rule public ClientDriverRule driver = new ClientDriverRule(); @Test public void issues_from_project_should_be_retrieved() { driver.addExpectation( onRequestTo("/repos/lordofthejars/nosqlunit/issues"). withMethod(Method.GET), giveResponse(GET_RESPONSE)); GithubClient githubClient = new GithubClient(driver.getBaseUrl()); String issues = githubClient.invokeGetMethod("/repos/lordofthejars/nosqlunit/issues"); assertThat(issues, is(GET_RESPONSE)); } We use ClientDriverRule @Rule annotation to add the client-driver to a test. And then using methods provided by RestClientDriver class, expectations are recorded. See how we are setting the base URL using driver.getBaseUrl() With rest-client-driver we can also record http status response using giveEmptyResponse method: @Test(expected=IllegalStateException.class) public void http_errors_should_throw_an_exception() { driver.addExpectation( onRequestTo("/repos/lordofthejars/nosqlunit/issues") .withMethod(Method.GET), giveEmptyResponse().withStatus(401)); GithubClient githubClient = new GithubClient(driver.getBaseUrl()); githubClient.invokeGetMethod("/repos/lordofthejars/nosqlunit/issues"); } And obviously we can record a put action: Note that in this example, we are setting that our request should contain given message body to response a 204 status code. This is a very simple example, but keep in mind that also works with libraries like gson or jackson. Also rest-driver project comes with a module that can be used to assert server responses (like REST-assured project) but this topic will be addressed into another post. I wish you have found this post useful. We keep learning, Alex.

when writing testable code, your first port of call is often to abstract any dependencies and make them easy to mock. this is the same for any of your codebase that talks to ftp servers. testing the way your code behaves under real world conditions makes integration tests important regardless of abstraction, though. here’s a simple trick to test ftp code in the wild. a recent project of mine has involved writing code that talks to ftp servers with the goal of adding additional continuous integration automation to a project. although all of my main methods are easily abstracted and injectable, my project still needs to actually talk to ftp servers at the end of the day, and i need to test that these very methods do the right thing when they are met with different conditions; be they bad credentials, lack of read/write permissions etc. the challenge integration tests can be brittle at the best of times, so ensuring that they are repeatable and can be setup and torn down can often be almost as much of a challenge as writing your actual code itself. an ftp server is usually a static service that is installed on a server. you might think that running one and ensuring it stays up and doesn’t get hacked just so that all your integration tests work is a necessary evil, but there is an easier way. run local. run often. i was running an ftp server on my build server just so that it was “always around” for my tests until i stumbled across an interesting project over on github to do just this . the approach i'm about to show you doesn’t need you to go to the effort of running a dedicated server at all. all you need to do is add a single executable to your unit test project and wrap your unit test in a using statement. the ftp server executable is a single file ftp server called ftpdmin which offers a read/write ftp server that can be fired up from the command line with a minimum feature set and only a few command line parameters to make it all tick. by implementing idisposable the helper class that wraps around this command line exe allows you to take advantage of the using() pattern to take care of your executable’s lifetime and have it die when your code is done testing. steps to make it happen download ftpdmin from here . add the exe to the root of your test project (you can put this anywhere, but you’ll have to update the helper class below). now add the exe to your project (i.e “view all items” in your test project’s solution explorer, and add the exe). set the exe to “copy always” in it’s solution properties. add the following code to a helper class in your test project: public class ftptestserver: idisposable { private readonly process ftpprocess; public ftptestserver(string rootdirectory, int port = 21, bool allowuploads = true) { var psinfo = new processstartinfo { filename = appdomain.currentdomain.basedirectory + "\\ftpdmin.exe", arguments = string.format("-p {0} -ha 127.0.0.1 \"{1}\" {2}", port, rootdirectory, allowuploads ? string.empty : "-g"), windowstyle = processwindowstyle.hidden }; ftpprocess = process.start(psinfo); } public void dispose() { if (ftpprocess.hasexited) return; ftpprocess.kill(); ftpprocess.waitforexit(); } } now you can enjoy being able to write really clean integration testing code that starts and ftp server every time you run your tests and then tear it down when your test is done. an example integration test showing connecting to “127.0.0.1”: [testmethod] public void ftpcode_upload_canconnect() { try { // fire up a new ftp server instance using (new ftptestserver(rootdirectory: "./")) { // code that talks to an ftp server on 127.0.0.1 } } catch (webexception e) { assert.fail("failed to connect to our ftp server"); } } how awesome is that? the power of using ftpdmin is that it can be told to deny write permissions to simulate bad user permissions as well: [testmethod] public void ftpcode_upload_throwswebexception() { try { // fire up a new ftp server instance using (new ftptestserver(rootdirectory: "./", allowuploads: false)) { // code that talks to an ftp server on 127.0.0.1 } } catch (webexception e) { assert.fail("our code failed to upload a file because of invalid permissions"); } } all in all, the above has been a complete lifesaver when it comes to making my integration test projects portable – if a new developer joins my project, they instantly get access to my ftp test harness just by pulling down my project’s source code.

Date and time handling in Java is a somewhat tricky part when you are new to the language. Time can be accessed via the static method System.currentTimeMillis() which returns the current time in milliseconds from January 1st 1970. If you prefer to work with Objects instead you can use java.util.Date, a class whose methods are mostly deprecated in recent versions of Java. To work with time offsets, say add one month to a date, there is java.util.GregorianCalendar. All in all, those methods described here are not very convenient to work with. Java 7 and below are lacking a good date and time API. The Joda Time library is a common drop-in if you need to work with date/time. With JSR 310 (Java Specification Request) this is about to change. JSR 310 adds a new date, time and calendar API to Java 8. The ThreeTen project provides a reference implementation to this new API and can already be utilized in current Java projects (I however recommend not to do this for production). As the README states: The API is currently considered usable and accurate, yet incomplete and subject to change. If you use this API you must be able to handle incompatible changes in later versions. Building ThreeTen Building the ThreeTen project is relatively easy. It requires both Git and Ant to be installed on your system. git clone git://github.com/ThreeTen/threeten.git cd threeten ant This will first fetch the most recent version of ThreeTen and then start the build process using ant. Note that building the library also requires either OpenJDK 1.6 or Oracle JDK 1.6. JSR 310 The new API specifies a number of new classes which are divided into the categories of continuous and human time. Continuous time is based on Unix time and is represented as a single incrementing number. Class Description Instant A point in time in nanoseconds from January 1st 1970 Duration An amount of time measured in nanoseconds Human time is based on fields that we use in our daily lifes such as day, hour, minute and second. It is represented by a group of classes, some of which we will discuss in this article. Class Description LocalDate a date, without time of day, offset or zone LocalTime the time of day, without date, offset or zone LocalDateTime the date and time, without offset or zone OffsetDate a date with an offset such as +02:00, without time of day or zone OffsetTime the time of day with an offset such as +02:00, without date or zone OffsetDateTime the date and time with an offset such as +02:00, without a zone ZonedDateTime the date and time with a time zone and offset YearMonth a year and month MonthDay month and day Year/MonthOfDay/DayOfWeek/... classes for the important fields DateTimeFields stores a map of field-value pairs which may be invalid Calendrical access to the low-level API Period a descriptive amount of time, such as "2 months and 3 days" In addition to the above classes three support classes have been implemented. The Clock class wraps the current time and date, ZoneOffset is a time offset from UTC and ZoneId defines a time zone such as 'Australia/Brisbane'. Using the API Getting the current time The current time is represented by the Clock class. The class is abstract, so you can not create instances of it. The systemUTC() static method will return the current time based on your system clock and set to UTC. import javax.time.Clock; Clock clock = Clock.systemUTC(); To use the default time zone on your system there also is systemDefaultZone(). Clock clock = Clock.systemDefaultZone(); The millis() method can then be used to access the current time in milliseconds from January 1st, 1970. This shows, that the Clock class and all subclasses are wrapped around System.currentTimeMillis(). Clock clock = Clock.systemDefaultZone(); long time = clock.millis(); Working with time zones To work with time zones you need to import the ZoneId class. The class provides a method to get the default system time zone: import javax.time.ZoneId; import javax.time.Clock; ZoneId zone = ZoneId.systemDefault(); Clock clock = Clock.system(zone); As seen above, the ZoneId can then be used to get an instance of a Clock with that time zone. Other time zones can be accessed by their name, e.g.: ZoneId zone = ZoneId.of("Europe/Berlin"); Clock clock = Clock.system(zone); Getting human date and time Working with a time represented in a single long variable is not what we wanted. We want to work with objects that represent human readable time. The LocalDate, LocalTime and LocalDateTime classes do just that. import javax.time.LocalDate; // The now() method returns the current DateTime LocalDate date = LocalDate.now(); System.out.printf("%s-%s-%s", date.getYear(), date.getMonthValue(), date.getDayOfMonth() ); Using LocalDate to print the current date Doing calculations with times and dates One of the most important functionalities of JSR-310 is that you can do calculations with dates and times. The API makes it very easy to do that. import javax.time.LocalTime; import javax.time.Period; import static javax.time.calendrical.LocalPeriodUnit.HOURS; Period p = Period.of(5, HOURS); LocalTime time = LocalTime.now(); LocalTime newTime; newTime = time.plus(5, HOURS); // or newTime = time.plusHours(5); // or newTime = time.plus(p); Three ways of adding 5 hours to the current time Each class that represents human time implements the AdjustableDateTime interface. The interface requires the plus and the minus method that take a value and a PeriodUnit as argument. Conclusion This article gave a (very) brief introduction into the new date and time API that will ship with Java 8. The API seems to be very consistent and well thought through and provides many ways to interact with dates and times. Upon release of Java 8 the API will be moved from the javax.time package over to java.time, so there will be no conflict if you start using the current implementation.

First of all we should define EIPs and why we should use them. As the name implies, these are tested solutions for specific design problems encountered during many years in the development of IT systems. And what is all the more important is that they are technology-agnostic which means it does not matter what programming language or operating system you use. Patterns are divided into seven sections: 1. Messaging Systems, 2. Messaging Channels, 3. Message Constructions, 4. Message Routing, 5. Message Transformation, 6. Messaging endpoints, 7. System management. The purpose of this article is to encourage you to use the patterns so I will discuss briefly only one or two such patterns from each of above sections. If you want to explore then further, visit http://www.eaipatterns.com/ or read Gregor Hohpe’s book mentioned in the introduction of this series. Message Channel (from Messaging Systems) A message channel is a logical channel which is used to connect the applications. One application writes messages to the channel and the other one (or others) reads that message from the channel. Message queue and message topic are examples of message channels. Message Translator (from Messaging Systems) Message translator transforms messages in one format to another. For example one application sends a message in XML format, but the other accepts only JSON messages so one of the parties (or mediator) has to transform XML data to JSON. This is probably the most widely used integration pattern. Publish-Subscribe Channel (from Messaging Channels) This type of channel broadcasts an event or notification to all subscribed receivers. This is in contrast with a point-to-point channel . Each subscriber receive the message once and next copy of this message is deleted from channel. The most common implementation of this patter is messaging topic. Dead Letter Channel (from Messaging Channels) The Dead Letter Channel describe scenario, what to do if the messaging system determines that it cannot deliver a message to the specified recipient. This may be caused for example by connection problems or other exception like overflowed memory or disc space. Usually, before sending the message to the Dead Letter Channel, multiple attempts to redeliver message are taken. Correlation Identifier (from Message Construction) Correlation Identifier gives the possibility to match request and reply message when asynchronous messaging system is used. This is usually accomplished in the following way: Producer: Generate unique correlation identifier. Producer: Send message with attached generated correlation identifier. Consumer: Process messages and send reply with attached correlation identifier given in request message. Producer: Correlate request and reply message based on correlation identifier. Content-Based Router (from Message Routing) Content-Based Router examines message contents and route messages based on data contained in the message. Content Enricher (from Message Transformation) Content Enricher as the name suggests enrich message with missing information. Usually external data source like database or web service is used. Event-Driven Consumer (from Messaging Endpoints) Event-Driver Consumer enables you to provide a action that is called automatically by the messaging channel or transport layer. It is asynchronous type of pattern because receiver does not have a running thread until a callback thread delivers a message. Polling Consumer (from Messaging Endpoints) Polling Consumer is used when we want receiver to poll for a message, process it and next poll for another. What is very important is that this pattern is synchronous because it blocks thread until a message is received. This is in contrast with a event-driven consumer. An example of using this pattern is file polling. Wire Tap (from System Management) Wire Tap copy a message and route it to a separate channel, while the original message is forwarded to the destination channel. Usually Wire Tap is used to inspect message or for analysis purposes.

If you have ever used a WordPress premium theme then you would of seen the custom theme options page that is available. The theme options page that is found under the appearance menu which allows the admin of the WordPress site to change some of the settings on the theme. Most premium themes will come with options to change the colors of fonts, backgrounds, change images or font types...anything that allows you to style the WordPress theme. Some of the most common fields to change are: Theme Options - To edit the theme logo, change the stylesheet, upload a new favicon, Add Google analytics code, enter your feedburner URL and add custom CSS. Styling Options - Change the background colour or change the background image. Fonts - Change the font on all your header tags or the main content text. Social - Providing you theme with your social media profiles will make it easier to link to them in parts of your theme or display your latest tweets. Option pages can also be used on plugins to change settings and to customize the plugin. Examples Of Theme Options Pages Here is what some of the theme options page from premium themes look like. How To Build A Theme Option Page When creating an option page there are a few things you need to setup. Add Menu - If you want to display the menu under the appearance menu or if you want to give the options page it's own menu. Add Sections - These are sections of settings you are adding to the options page. Register Settings - Settings are the different fields you are adding to the options page, they need to be registered with the settings API. Display Settings - The settings API will be used to call a function to display the setting. Validate Setting - When the user saves the settings field the input will need to be validated before stored in the options table. Feedback Messages - When the settings are saved you need to be able to feedback to the user if the settings were saved successfully or if there was an error during validation. To help us perform all these tasks there is a WordPress API called the Settings API. This API allows admin pages to handle setting forms semi-automatically. With the API you can define pages for the settings, sections for the settings and fields for the settings. This works by registering setting fields to be displayed within sections and page will display these sections. WordPress uses the Settings API by default on existing admin pages, this means that by using the Settings API you can add to existing pages by registering new settings. All validation must be performed by the developer of the settings pages but the Settings API will control the creation of the form and storing the values in the form in the options table. Add Menu To WordPress Admin When adding a menu to the WordPress admin screen you have loads of flexibility you have the option of adding brand new menu items or adding the menu as a sub menu. To add a top level menu just use the following function add_menu_page(). $page_title - The title used on the settings page. $menu_title - The title used on the menu. $capability - Only displays the menu if the user matches this capability. $menu_slug - The unique name of the menu slug. $function - This is the callback function to run to display the page. $icon_url - Display a icon just for the menu. $position - This allows you to choose when the menu item appears in the list. If you prefer to have the menu under the appearance parent menu you can use the following code snippet. Or you can use the function add_theme_page() which will add a sub-menu under the appearance menu. add_theme_page( $page_title, $menu_title, $capability, $menu_slug, $function); Registering The Settings To start off we need to register the settings group we are going to store the settings page values. This will use the Settings API to define the group of settings, we will then add the settings to a group. When you store the settings in this group they are stored in the wp_options database table so you can get these values out at a later date. The wp_options table is a key value pairing stored in the database. This is what you should use when storing long term data on your WordPress site. If you are storing a lot of data it's best practice to turn the data into an array and store it under one key, instead of storing all the values over multiple keys. This means that if you have a settings page to change the site logo, background color, font, font size etc, you won't have an option for each of these but you will group them into an option group. The reason you do this is to increase on database efficiency by not adding too many rows to the options database. To register settings on the Settings API you need to use the function register_setting(). The parameters you pass into this are: Option Group - The name of the group of settings you are going to store. This must match the group name used in the settings_field() function. Option name - The name of the option which will be saved, this is the key that is used in the options table. Sanitize Callback - This is the function that is used to validate the settings for this option group. Add Sections To Settings Once the settings are registered we can add section groups to the Settings API. This will allow us to organise the settings on the page, so that you can add styles to display these differently on the page. The benefit of adding sections on your Settings API is so that we can call the function do_settings_sections() as this will display all the settings under this one section. To create you own settings all you have to do is use the function add_settings_section(). The parameters you need to use on this function are: Id - String to use for the ID of the section. Title - The title to use on the section. Callback - This is the function that will display the settings on the page. Page - This is the page that is displaying the section, should match the menu slug of the page. Add Fields To The Sections The last important function we need to use to add settings to the page is the add_settings_field() function, this is used as part of the Settings API to define fields to a section. The function will need to know the page slug and the section Id before you can define the settings to use. All the settings which you setup here will be stored in the options table under the key used in the register_settings() function. To use this function you need to add the following parameters. ID - ID of the field Title - Title of the field. Callback - Function used to display the setting. This is very important as it is used to display the input field you want. Page - Page which is going to display the field should be the same as the menu slug on the section. Section - Section Id which the field will be added to. $args - Additional arguments which are passed to the callback function. Example Of Using The Settings API There is a lot of information to take in above to create this settings page so it can seem a bit complicated but once you get your head around the structure the Settings API uses it's actually quite easy to understand. The best way to understand how this all works is to show you with an example. Create A Theme Option Page With A Textbox Field In this example we will create a theme option page and add a textbox on the page to add additional text to the index.php. Just add the following to your functions.php file to create a theme options page. First we start off by creating the menu item under the appearance menu by using the add_theme_page() function on the admin_menu action. /** * Theme Option Page Example */ function pu_theme_menu() { add_theme_page( 'Theme Option', 'Theme Options', 'manage_options', 'pu_theme_options.php', 'pu_theme_page'); } add_action('admin_menu', 'pu_theme_menu'); As you can see above we set the callback function to the theme options page to be pu_theme_page so we need to create this function to display our page. Here we create a form to submit to the options.php so that we can save in the options table, we call settings_fields() to the get the settings in register_settings() and use the do_settings_sections() function to display our settings. /** * Callback function to the add_theme_page * Will display the theme options page */ function pu_theme_page() { ?> Custom Theme Options Created by Paulund. 'text', 'id' => 'pu_textbox', 'name' => 'pu_textbox', 'desc' => 'Example of textbox description', 'std' => '', 'label_for' => 'pu_textbox', 'class' => 'css_class' ); add_settings_field( 'example_textbox', 'Example Textbox', 'pu_display_setting', 'pu_theme_options.php', 'pu_text_section', $field_args ); } The callback function on creating sections can be used to add addition information that will appear above every section, on this example we are just leaving it blank. /** * Function to add extra text to display on each section */ function pu_display_section($section){ } The callback function on the add_settings_field() function is pu_display_setting, this is the function that is going to echo the display of any input's on the page. The parameter to this function is the $args value on the add_settings_field() we can use this to add things like id, name, default value etc. We want to get any existing values from the wp_option table to display any values which previously typed in by the user, do to this we get the values from the table by using the get_option() function. /** * Function to display the settings on the page * This is setup to be expandable by using a switch on the type variable. * In future you can add multiple types to be display from this function, * Such as checkboxes, select boxes, file upload boxes etc. */ function pu_display_setting($args) { extract( $args ); $option_name = 'pu_theme_options'; $options = get_option( $option_name ); switch ( $type ) { case 'text': $options[$id] = stripslashes($options[$id]); $options[$id] = esc_attr( $options[$id]); echo ""; echo ($desc != '') ? "$desc" : ""; break; } } Finally we can validate the values added to the form by creating the validation callback function pu_validate_settings. All this does at the moment is loop through the inputs passed to it and checks if it's a letter or a number. The return of this function is what will be added to the database. /** * Callback function to the register_settings function will pass through an input variable * You can then validate the values and the return variable will be the values stored in the database. */ function pu_validate_settings($input) { foreach($input as $k => $v) { $newinput[$k] = trim($v); // Check the input is a letter or a number if(!preg_match('/^[A-Z0-9 _]*$/i', $v)) { $newinput[$k] = ''; } } return $newinput; } If you copy all the snippets above into your functions.php file you will see this options form under the appearance menu. Using Theme Options Within Your Theme Now that you understand how to create a theme options page you need to be able to use this value in your theme so you can change the settings. All the settings are stored in the wp_options table with WordPress it's very easy to get these values out all you have to do is use the get_option() function. The option name is the name you put on the register_settings() function. So in our example above you will use this code. The $options variable will now store an array of the values from the theme options, which you can display the value of the textbox we put on the page by using this snippet. Conclusion That's the basics that you need to understand to use the Settings API, now you can take this information and create your own theme options page. Experiment with different input types you can add to the form, experiment with different validation methods you want to use. In future tutorials I will post how you can use some of the inbuilt WordPress third party applications to create a better user experience on your theme options panel. This will include things like color pickers, date pickers, jQuery UI features etc. As you can see we have created a settings option page in just over 100 lines of code, so it's not a hard thing to do but that are a few steps to it and the features can be expanded on. For this reason people have created theme option frameworks to allow you to easily create a theme option page with much higher level of complexity with the options. But like many other frameworks I always recommend you learn the basics before using a framework, this is why it's important to understand how the Settings API works before using or creating a settings page framework.