This article will describe the complete root cause analysis of a recent Java deadlock problem observed from a Weblogic 11g production system running on the IBM JVM 1.6.This case study will also demonstrate the importance of mastering Thread Dump analysis skills; including for the IBM JVM Thread Dump format. Environment specification Java EE server: Oracle Weblogic Server 11g & Spring 2.0 OS: AIX 5.3 Java VM: IBM JRE 1.6.0 Platform type: Portal & ordering application Monitoring and troubleshooting tools JVM Thread Dump (IBM JVM format) Compuware Server Vantage (Weblogic JMX monitoring & alerting) Problem overview A major stuck Threads problem was observed & reported from Compuware Server Vantage and affecting 2 of our Weblogic 11g production managed servers causing application impact and timeout conditions from our end users. Gathering and validation of facts As usual, a Java EE problem investigation requires gathering of technical and non-technical facts so we can either derived other facts and/or conclude on the root cause. Before applying a corrective measure, the facts below were verified in order to conclude on the root cause: · What is the client impact? MEDIUM (only 2 managed servers / JVM affected out of 16) · Recent change of the affected platform? Yes (new JMS related asynchronous component) · Any recent traffic increase to the affected platform? No · How does this problem manifest itself? A sudden increase of Threads was observed leading to rapid Thread depletion · Did a Weblogic managed server restart resolve the problem? Yes, but problem is returning after few hours (unpredictable & intermittent pattern) - Conclusion #1: The problem is related to an intermittent stuck Threads behaviour affecting only a few Weblogic managed servers at the time - Conclusion #2: Since problem is intermittent, a global root cause such as a non-responsive downstream system is not likely Thread Dump analysis – first pass The first thing to do when dealing with stuck Thread problems is to generate a JVM Thread Dump. This is a golden rule regardless of your environment specifications & problem context. A JVM Thread Dump snapshot provides you with crucial information about the active Threads and what type of processing / tasks they are performing at that time. Now back to our case study, an IBM JVM Thread Dump (javacore.xyz format) was generated which did reveal the following Java Thread deadlock condition below: 1LKDEADLOCK Deadlock detected !!! NULL --------------------- NULL 2LKDEADLOCKTHR Thread "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012CC08B00) 3LKDEADLOCKWTR is waiting for: 4LKDEADLOCKMON sys_mon_t:0x0000000126171DF8 infl_mon_t: 0x0000000126171E38: 4LKDEADLOCKOBJ weblogic/jms/frontend/FESession@0x07000000198048C0/0x07000000198048D8: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012E560500) 3LKDEADLOCKWTR which is waiting for: 4LKDEADLOCKMON sys_mon_t:0x000000012884CD60 infl_mon_t: 0x000000012884CDA0: 4LKDEADLOCKOBJ weblogic/jms/frontend/FEConnection@0x0700000019822F08/0x0700000019822F20: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012CC08B00) This deadlock situation can be translated as per below: - Weblogic Thread #8 is waiting to acquire an Object monitor lock owned by Weblogic Thread #10 - Weblogic Thread #10 is waiting to acquire an Object monitor lock owned by Weblogic Thread #8 Conclusion: both Weblogic Threads #8 & #10 are waiting on each other; forever! Now before going any deeper in this root cause analysis, let me provide you a high level overview on Java Thread deadlocks. Java Thread deadlock overview Most of you are probably familiar with Java Thread deadlock principles but did you really experience a true deadlock problem? From my experience, true Java deadlocks are rare and I have only seen ~5 occurrences over the last 10 years. The reason is that most stuck Threads related problems are due to Thread hanging conditions (waiting on remote IO call etc.) but not involved in a true deadlock condition with other Thread(s). A Java Thread deadlock is a situation for example where Thread A is waiting to acquire an Object monitor lock held by Thread B which is itself waiting to acquire an Object monitor lock held by Thread A. Both these Threads will wait for each other forever. This situation can be visualized as per below diagram: Thread deadlock is confirmed…now what can you do? Once the deadlock is confirmed (most JVM Thread Dump implementations will highlight it for you), the next step is to perform a deeper dive analysis by reviewing each Thread involved in the deadlock situation along with their current task & wait condition.Find below the partial Thread Stack Trace from our problem case for each Thread involved in the deadlock condition: ** Please note that the real application Java package name was renamed for confidentiality purposes ** Weblogic Thread #8 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" J9VMThread:0x000000012CC08B00, j9thread_t:0x00000001299E5100, java/lang/Thread:0x070000001D72EE00, state:B, prio=1 (native thread ID:0x111200F, native priority:0x1, native policy:UNKNOWN) Java callstack: at weblogic/jms/frontend/FEConnection.stop(FEConnection.java:671(Compiled Code)) at weblogic/jms/frontend/FEConnection.invoke(FEConnection.java:1685(Compiled Code)) at weblogic/messaging/dispatcher/Request.wrappedFiniteStateMachine(Request.java:961(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.syncRequest(DispatcherImpl.java:184(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.dispatchSync(DispatcherImpl.java:212(Compiled Code)) at weblogic/jms/dispatcher/DispatcherAdapter.dispatchSync(DispatcherAdapter.java:43(Compiled Code)) at weblogic/jms/client/JMSConnection.stop(JMSConnection.java:863(Compiled Code)) at weblogic/jms/client/WLConnectionImpl.stop(WLConnectionImpl.java:843) at org/springframework/jms/connection/SingleConnectionFactory.closeConnection(SingleConnectionFactory.java:342) at org/springframework/jms/connection/SingleConnectionFactory.resetConnection(SingleConnectionFactory.java:296) at org/app/JMSReceiver.receive() …………………………………………………………………… Weblogic Thread #10 "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" J9VMThread:0x000000012E560500, j9thread_t:0x000000012E35BCE0, java/lang/Thread:0x070000001ECA9200, state:B, prio=1 (native thread ID:0x4FA027, native priority:0x1, native policy:UNKNOWN) Java callstack: at weblogic/jms/frontend/FEConnection.getPeerVersion(FEConnection.java:1381(Compiled Code)) at weblogic/jms/frontend/FESession.setUpBackEndSession(FESession.java:755(Compiled Code)) at weblogic/jms/frontend/FESession.consumerCreate(FESession.java:1025(Compiled Code)) at weblogic/jms/frontend/FESession.invoke(FESession.java:2995(Compiled Code)) at weblogic/messaging/dispatcher/Request.wrappedFiniteStateMachine(Request.java:961(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.syncRequest(DispatcherImpl.java:184(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.dispatchSync(DispatcherImpl.java:212(Compiled Code)) at weblogic/jms/dispatcher/DispatcherAdapter.dispatchSync(DispatcherAdapter.java:43(Compiled Code)) at weblogic/jms/client/JMSSession.consumerCreate(JMSSession.java:2982(Compiled Code)) at weblogic/jms/client/JMSSession.setupConsumer(JMSSession.java:2749(Compiled Code)) at weblogic/jms/client/JMSSession.createConsumer(JMSSession.java:2691(Compiled Code)) at weblogic/jms/client/JMSSession.createReceiver(JMSSession.java:2596(Compiled Code)) at weblogic/jms/client/WLSessionImpl.createReceiver(WLSessionImpl.java:991(Compiled Code)) at org/springframework/jms/core/JmsTemplate102.createConsumer(JmsTemplate102.java:204(Compiled Code)) at org/springframework/jms/core/JmsTemplate.doReceive(JmsTemplate.java:676(Compiled Code)) at org/springframework/jms/core/JmsTemplate$10.doInJms(JmsTemplate.java:652(Compiled Code)) at org/springframework/jms/core/JmsTemplate.execute(JmsTemplate.java:412(Compiled Code)) at org/springframework/jms/core/JmsTemplate.receiveSelected(JmsTemplate.java:650(Compiled Code)) at org/springframework/jms/core/JmsTemplate.receiveSelected(JmsTemplate.java:641(Compiled Code)) at org/app/JMSReceiver.receive() …………………………………………………………… As you can see in the above Thread Strack Traces, such deadlock did originate from our application code which is using the Spring framework API for the JMS consumer implementation (very useful when not using MDB’s). The Stack Traces are quite interesting and revealing that both Threads are in a race condition against the same Weblogic JMS consumer session / connection and leading to a deadlock situation: - Weblogic Thread #8 is attempting to reset and close the current JMS connection - Weblogic Thread #10 is attempting to use the same JMS Connection / Session in order to create a new JMS consumer - Thread deadlock is triggered! Root cause: non Thread safe Spring JMS SingleConnectionFactory implementation A code review and a quick research from Spring JIRA bug database did reveal the following Thread safe defect below with a perfect correlation with the above analysis: # SingleConnectionFactory's resetConnection is causing deadlocks with underlying OracleAQ's JMS connection https://jira.springsource.org/browse/SPR-5987 A patch for Spring SingleConnectionFactory was released back in 2009 which did involve adding proper synchronized{} block in order to prevent Thread deadlock in the event of a JMS Connection reset operation: synchronized (connectionMonitor) { //if condition added to avoid possible deadlocks when trying to reset the target connection if (!started) { this.target.start(); started = true; } } Solution Our team is currently planning to integrate this Spring patch in to our production environment shortly. The initial tests performed in our test environment are positive. Conclusion I hope this case study has helped understand a real-life Java Thread deadlock problem and how proper Thread Dump analysis skills can allow you to quickly pinpoint the root cause of stuck Thread related problems at the code level. Please don’t hesitate to post any comment or question.

Continuing on the theme of temporary storage for transient messages used within Spring Integration flows, the claim-check model offers configurable storage for message payloads. The advantage in using this Enterprise Integration pattern, compared against header enrichment, is that objects don't have to be packed into the header using a Header Enrichment technique. They can be stored in a local Java Map, an IMDB, cache or anything else that be used to hold data. Several advantages using this approach are evident. Firstly, performance and efficiency. When using header enrichment, if message payloads need to be managed outside of the JVM that generates the enriched message header, the object will not be available unless it's serialised and transported around the distributed application. This could be costly in terms of performance and transport efficiency. The key factor here is the frequency of remote dispatch and the size of the header object. In specific circumstances the claim-check pattern may offer an advantage here, objects can be serialised and/or transformed into a storage specific format and stored internally in memory or externally in a data store. Secondly, accessibility. It's conceivable that message payloads undergoing claim-check processing may need to be accessed by third party applications that are unable to receive Spring Integration messages. The claim-check pattern allows this type of processing to take place. Thirdly, resiliency is offered. A data store can be chosen that guarantees persistence for messages in order that they can be recovered following failure. The following code details how the claim-check pattern can be used: The gateway used is specified as the following Java class: package com.l8mdv.sample; import org.springframework.integration.Message; import org.springframework.integration.annotation.Gateway; public interface ClaimCheckGateway { public static final String CLAIM_CHECK_ID = "CLAIM_CHECK_ID"; @Gateway (requestChannel = "claim-check-in-channel") public Message send(Message message); } Lastly, this can all be tested by using the following JUnit test case: package com.l8mdv.sample; import org.junit.Assert; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.integration.Message; import org.springframework.integration.support.MessageBuilder; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import static com.l8mdv.sample.ClaimCheckGateway.CLAIM_CHECK_ID; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration( locations = {"classpath:META-INF/spring/claim-check.xml"} ) public class ClaimCheckIntegrationTest { @Autowired ClaimCheckGateway claimCheckGateway; @Test public void locatePayloadInHeader() { String payload = "Sample test message."; Message message = MessageBuilder.withPayload(payload).build(); Message response = claimCheckGateway.send(message); Assert.assertTrue(response.getPayload().equals(payload)); Assert.assertTrue(response.getHeaders().get(CLAIM_CHECK_ID) != null); } }

Cross-site request forgery attacks (CSRF) are very common in web applications and can cause significant harm if allowed. If you have never heard of CSRF I recommend you check out OWASPs page about it. Luckily preventing CSRF attacks is quite simple, I’ll try to show you how they work and how we can defend from them in the least obtrusive way possible in Java based web apps. Imagine you are about to perform a money transfer in your bank’s secure web page, when you click on the transfer option a form page is loaded that allows you to choose the debit and credit accounts, and enter the amount of money to move. When you are satisfied with your options you press “submit” and send the form information to your bank’s web server, which in turns performs the transaction. Now add the following to the picture, a malicious website (which you think harmless of course) is open on another window/tab of your browser while you are innocently moving all your millions in your bank’s site. This evil site knows the bank’s web forms structure, and as you browse through it, it tries to post transactions withdrawing money from your accounts and depositing it on the evil overlord’s accounts, it can do it because you have an open and valid session with the banks site in the same browser! This is the basis for a CSRF attack. One simple and effective way to prevent it is to generate a random (i.e. unpredictable) string when the initial transfer form is loaded and send it to the browser. The browser then sends this piece of data along with the transfer options, and the server validates it before approving the transaction for processing. This way, malicious websites cannot post transactions even if they have access to a valid session in a browser. To implement this mechanism in Java I choose to use two filters, one to create the salt for each request, and another to validate it. Since the users request and subsequent POST or GETs that should be validated do not necessarily get executed in order, I decided to use a time based cache to store a list of valid salt strings. The first filter, used to generate a new salt for a request and store it in the cache can be coded as follows: package com.ricardozuasti.csrf; import com.google.common.cache.Cache; import com.google.common.cache.CacheBuilder; import com.google.common.cache.CacheLoader; import com.google.common.cache.LoadingCache; import java.io.IOException; import java.security.SecureRandom; import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; import javax.servlet.*; import javax.servlet.http.HttpServletRequest; import org.apache.commons.lang.RandomStringUtils; public class LoadSalt implements Filter { @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { // Assume its HTTP HttpServletRequest httpReq = (HttpServletRequest) request; // Check the user session for the salt cache, if none is present we create one Cache csrfPreventionSaltCache = (Cache) httpReq.getSession().getAttribute("csrfPreventionSaltCache"); if (csrfPreventionSaltCache == null){ csrfPreventionSaltCache = CacheBuilder.newBuilder() .maximumSize(5000) .expireAfterWrite(20, TimeUnit.MINUTES) .build(); httpReq.getSession().setAttribute("csrfPreventionSaltCache", csrfPreventionSaltCache); } // Generate the salt and store it in the users cache String salt = RandomStringUtils.random(20, 0, 0, true, true, null, new SecureRandom()); csrfPreventionSaltCache.put(salt, Boolean.TRUE); // Add the salt to the current request so it can be used // by the page rendered in this request httpReq.setAttribute("csrfPreventionSalt", salt); chain.doFilter(request, response); } @Override public void init(FilterConfig filterConfig) throws ServletException { } @Override public void destroy() { } } I used Guava CacheBuilder to create the salt cache since it has both a size limit and an expiration timeout per entry. To generate the actual salt I used Apache Commons RandomStringUtils, powered by Java 6 SecureRandom to ensure a strong generation seed. This filter should be used in all requests ending in a page that will link, post or call via AJAX a secured transaction, so in most cases it’s a good idea to map it to every request (maybe with the exception of static content such as images, CSS, etc.). It’s mapping in your web.xml should look similar to: ... loadSalt com.ricardozuasti.csrf.LoadSalt ... loadSalt * ... As I said, to validate the salt before executing secure transactions we can write another filter: package com.ricardozuasti.csrf; import com.google.common.cache.Cache; import java.io.IOException; import javax.servlet.*; import javax.servlet.http.HttpServletRequest; public class ValidateSalt implements Filter { @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { // Assume its HTTP HttpServletRequest httpReq = (HttpServletRequest) request; // Get the salt sent with the request String salt = (String) httpReq.getParameter("csrfPreventionSalt"); // Validate that the salt is in the cache Cache csrfPreventionSaltCache = (Cache) httpReq.getSession().getAttribute("csrfPreventionSaltCache"); if (csrfPreventionSaltCache != null && salt != null && csrfPreventionSaltCache.getIfPresent(salt) != null){ // If the salt is in the cache, we move on chain.doFilter(request, response); } else { // Otherwise we throw an exception aborting the request flow throw new ServletException("Potential CSRF detected!! Inform a scary sysadmin ASAP."); } } @Override public void init(FilterConfig filterConfig) throws ServletException { } @Override public void destroy() { } } You should configure this filter for every request that needs to be secure (i.e. retrieves or modifies sensitive information, move money, etc.), for example: ... validateSalt com.ricardozuasti.csrf.ValidateSalt ... validateSalt /transferMoneyServlet ... After configuring both servlets all your secured requests should fail :). To fix it you have to add, to each link and form post that ends in a secure URL, the csrfPreventionSalt parameter containing the value of the request parameter with the same name. For example, in an HTML form within a JSP page: ... ... ... Of course you can write a custom tag, a nice Javascript code or whatever you prefer to inject the new parameter in every needed link/form.

This article will provide you with a tutorial on how you can quickly pinpoint the Java Thread contributors to a high CPU problem on the Windows OS. Windows, like other OS such as Linux, Solaris & AIX allow you to monitor the CPU utilization at the process level but also for individual Thread executing a task within a process. For this tutorial, we created a simple Java program that will allow you to learn this technique in a step by step manner. Troubleshooting tools The following tools will be used below for this tutorial: - Windows Process Explorer (to pinpoint high CPU Thread contributors) - JVM Thread Dump (for Thread correlation and root cause analysis at code level) High CPU simulator Java program The simple program below is simply looping and creating new String objects. It will allow us to perform this CPU per Thread analysis. I recommend that you import it in an IDE of your choice e.g. Eclipse and run it from there. You should observe an increase of CPU on your Windows machine as soon as you execute it. package org.ph.javaee.tool.cpu; /** * HighCPUSimulator * @author Pierre-Hugues Charbonneau * http://javaeesupportpatterns.blogspot.com * */ public class HighCPUSimulator { private final static int NB_ITERATIONS = 500000000; // ~1 KB data footprint private final static String DATA_PREFIX = "datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata"; /** * @param args */ public static void main(String[] args) { System.out.println("HIGH CPU Simulator 1.0"); System.out.println("Author: Pierre-Hugues Charbonneau"); System.out.println("http://javaeesupportpatterns.blogspot.com/"); try { for (int i = 0; i < NB_ITERATIONS; i++) { // Perform some String manipulations to slowdown and expose looping process... String data = DATA_PREFIX + i; } } catch (Throwable any) { System.out.println("Unexpected Exception! " + any.getMessage() + " [" + any + "]"); } System.out.println("HighCPUSimulator done!"); } } Step #1 – Launch Process Explorer The Process Explorer tool visually shows the CPU usage dynamically. It is good for live analysis. If you need historical data on CPU per Thread then you can also use Windows perfmon with % Processor Time & Thread Id data counters. You can download Process Explorer from the link below: http://technet.microsoft.com/en-us/sysinternals/bb896653 In our example, you can see that the Eclipse javaw.exe process is now using ~25% of total CPU utilization following the execution of our sample program. Step #2 – Launch Process Explorer Threads view The next step is to display the Threads view of the javaw.exe process. Simply right click on the javaw.exe process and select Properties. The Threads view will be opened as per below snapshot: - The first column is the Thread Id (decimal format) - The second column is the CPU utilization % used by each Thread - The third column is also another counter indicating if Thread is running on the CPU In our example, we can see our primary culprit is Thread Id #5996 using ~ 25% of CPU. Step #3 – Generate a JVM Thread Dump At this point, Process Explorer will no longer be useful. The goal was to pinpoint one or multiple Java Threads consuming most of the Java process CPU utilization which is what we achieved. In order to go the next level in your analysis you will need to capture a JVM Thread Dump. This will allow you to correlate the Thread Id with the Thread Stack Trace so you can pinpoint that type of processing is consuming such high CPU. JVM Thread Dump generation can be done in a few manners. If you are using JRockit VM you can simply use the jrcmd tool as per below example: Once you have the Thread Dump data, simply search for the Thread Id and locate the Thread Stack Trace that you are interested in. For our example, the Thread “Main Thread” which was fired from Eclipse got exposed as the primary culprit which is exactly what we wanted to demonstrate. "Main Thread" id=1 idx=0x4 tid=5996 prio=5 alive, native_blocked at org/ph/javaee/tool/cpu/HighCPUSimulator.main (HighCPUSimulator.java:31) at jrockit/vm/RNI.c2java(IIIII)V(Native Method) -- end of trace Step #4 – Analyze the culprit Thread(s) Stack Trace and determine root cause At this point you should have everything that you need to move forward with the root cause analysis. You will need to review each Thread Stack Trace and determine what type of problem you are dealing with. That final step is typically where you will spend most of your time and problem can be simple such as infinite looping or complex such as garbage collection related problems. In our example, the Thread Dump did reveal the high CPU originates from our sample Java program around line 31. As expected, it did reveal the looping condition that we engineered on purpose for this tutorial. for (int i = 0; i < NB_ITERATIONS; i++) { // Perform some String manipulations to slowdown and expose looping process... String data = DATA_PREFIX + i; } I hope this tutorial has helped you understand how you can analyze and help pinpoint root cause of Java CPU problems on Windows OS. Please stay tuned for more updates, the next article will provide you with a Java CPU troubleshooting guide including how to tackle that last analysis step along with common problem patterns.

On the surface these methods do the same thing in Java; Thread.yield(), Thread.sleep(0), Object.wait(0,1) and LockSupport.parkNanos(1) They all wait a sort period of time, but how much that is varies a surprising amount and between platforms. Timing a short delay The following code times how long it takes to repeatedly call those methods. import java.util.concurrent.locks.LockSupport; public class Pausing { public static void main(String... args) throws InterruptedException { int repeat = 10000; for (int i = 0; i < 3; i++) { long time0 = System.nanoTime(); for (int j = 0; j < repeat; j++) Thread.yield(); long time1 = System.nanoTime(); for (int j = 0; j < repeat; j++) Thread.sleep(0); long time2 = System.nanoTime(); synchronized (Thread.class) { for (int j = 0; j < repeat/10; j++) Thread.class.wait(0, 1); } long time3 = System.nanoTime(); for (int j = 0; j < repeat/10; j++) LockSupport.parkNanos(1); long time4 = System.nanoTime(); System.out.printf("The average time to yield %.1f μs, sleep(0) %.1f μs, " + "wait(0,1) %.1f μs and LockSupport.parkNanos(1) %.1f μs%n", (time1 - time0) / repeat / 1e3, (time2 - time1) / repeat / 1e3, (time3 - time2) / (repeat/10) / 1e3, (time4 - time3) / (repeat/10) / 1e3); } } } On Windows 7 The average time to yield 0.3 μs, sleep(0) 0.6 μs, wait(0,1) 999.9 μs and LockSupport.parkNanos(1) 1000.0 μs The average time to yield 0.3 μs, sleep(0) 0.6 μs, wait(0,1) 999.5 μs and LockSupport.parkNanos(1) 1000.1 μs The average time to yield 0.2 μs, sleep(0) 0.5 μs, wait(0,1) 1000.0 μs and LockSupport.parkNanos(1) 1000.1 μs On RHEL 5.x The average time to yield 1.1 μs, sleep(0) 1.1 μs, wait(0,1) 2003.8 μs and LockSupport.parkNanos(1) 3.8 μs The average time to yield 1.1 μs, sleep(0) 1.1 μs, wait(0,1) 2004.8 μs and LockSupport.parkNanos(1) 3.4 μs The average time to yield 1.1 μs, sleep(0) 1.1 μs, wait(0,1) 2005.6 μs and LockSupport.parkNanos(1) 3.1 μs In summary If you want to wait for a short period of time, you can't assume that all these methods do the same thing, nor will be the same between platforms.

Here are some notes on adding Camel (from v2.3+) exception handling to a JavaDSL route. There are various approaches/options available. These notes cover the important distinctions between approaches... Default handling The default mode uses the DefaultErrorHandler strategy which simply propagates any exception back to the caller and ends the route immediately. This is rarely the desired behavior, at the very least, you should define a generic/global exception handler to log the errors and put them on a queue for further analysis (during development, testing, etc). onException(Exception) .to("log:GeneralError?level=ERROR") .to("activemq:queue:GeneralErrorQueue"); Try-catch-finally This approach mimics the Java for exception handling and is designed to be very readable and easy to implement. It inlines the try/catch/finally blocks directly in the route and is useful for route specific error handling. from("direct:start") .doTry() .process(new MyProcessor()) .doCatch(Exception.class) .to("mock:error"); .doFinally() .to("mock:end"); onException This approach defines the exception clause separately from the route. This makes the route and exception handling code more readable and reusable. Also, the exception handling will apply to any routes defined in its CamelContext. onException(Exception.class) .to("mock:error"); from("direct:start") .process(new MyProcessor()) .to("mock:end"); Handled/Continued These APIs provide valuable control over the flow. Adding handled(true) tells Camel to not propagate the error back to the caller (should almost always be used). The continued(true) tells Camel to resume the route where it left off (rarely used, but powerful). These can both be used to control the flow of the route in interesting ways, for example... from("direct:start") .process(new MyProcessor()) .to("mock:end"); //send the exception back to the client (rarely used, clients need a meaningful response) onException(ClientException.class) .handled(false) //default .log("error sent back to the client"); //send a readable error message back to the client and handle the error internally onException(HandledException.class) .handled(true) .setBody(constant("error")) .to("mock:error"); //ignore the exception and continue the route (can be dangerous, use wisely) onException(ContinuedException.class) .continued(true); Using a processor for more control If you need more control of the handler code, you can use an inline Processor to get a handle to the exception that was thrown and write your own handler code... onException(Exception.class) .handled(true) .process(new Processor() { public void process(Exchange exchange) throws Exception { Exception exception = (Exception) exchange.getProperty(Exchange.EXCEPTION_CAUGHT); //log, email, reroute, etc. } }); Summary Overall, the exception handling is very flexible and can meet almost any scenario you can come up with. For the sake of focusing on the basics, many advanced features haven't been covered here. For more details, see these pages on Camel's site... http://camel.apache.org/error-handling-in-camel.html http://camel.apache.org/try-catch-finally.html http://camel.apache.org/exception-clause.html

So I recently wrote a post about good old reliable Apache Commons StringUtils, which provoked a couple of comments, one of which was that Google Guava provides better mechanisms for joining and splitting Strings. I have to admit, this is a corner of Guava I've yet to explore. So thought I ought to take a closer look, and compare with StringUtils, and I have to admit I was surprised at what I found. Splitting strings eh? There can't be many different ways of doing this surely? Well Guava and StringUtils do take a sylisticly different approach. Lets start with the basic usage. // Apache StringUtils... String[] tokens1 = StringUtils.split("one,two,three",','); // Guava splitter... Iterable tokens2 = Splitter.on(',').split("one,two,three"); So, my first observation is that Splitter is more object orientated. You have to create a splitter object, which you then use to do the splitting. Whereas the StringUtils splitter methods uses a more functional style, with static methods. Here I much prefer Splitter. Need a reusable splitter that splits comma separated lists? A splitter that also trims leading and trailing white space, and ignores empty elements? Not a problem: Splitter niceCommaSplitter = Splitter.on(',') .omitEmptyString() .trimResults(); niceCommaSplitter.split("one,, two, three"); //"one","two","three" niceCommaSplitter.split(" four , five "); //"four","five" That looks really useful, any other differences? The other thing to notice is that Splitter returns an Iterable, whereas StringUtils.split returns a String array. Don't really see that making much of a difference, most of the time I just want to loop through the tokens in order anyway! I also didn't think it was a big deal, until I examined the performance of the two approaches. To do this I tried running the following code: final String numberList = "One,Two,Three,Four,Five,Six,Seven,Eight,Nine,Ten"; long start = System.currentTimeMillis(); for(int i=0; i<1000000; i++) { StringUtils.split(numberList , ','); } System.out.println(System.currentTimeMillis() - start); start = System.currentTimeMillis(); for(int i=0; i<1000000; i++) { Splitter.on(',').split(numberList ); } System.out.println(System.currentTimeMillis() - start); On my machine this output the following times: 594 31 Guava's Splitter is almost 10 times faster! Now this is a much bigger difference than I was expecting, Splitter is over 10 times faster than StringUtils. How can this be? Well, I suspect it's something to do with the return type. Splitter returns an Iterable, whereas StringUtils.split gives you an array of Strings! So Splitter doesn't actually need to create new String objects. It's also worth noting you can cache your Splitter object, which results in an even faster runtime. Blimey, end of argument? Guava's Splitter wins every time? Hold on a second. This isn't quite the full story. Notice we're not actually doing anything with the result of the Strings? Like I mentioned, it looks like the Splitter isn't actually creating any new Strings. I suspect it's actually deferring this to the Iterator object it returns. So can we test this? Sure thing. Here's some code to repeatedly check the lengths of the generated substrings: final String numberList = "One,Two,Three,Four,Five,Six,Seven,Eight,Nine,Ten"; long start = System.currentTimeMillis(); for(int i=0; i<1000000; i++) { final String[] numbers = StringUtils.split(numberList, ','); for(String number : numbers) { number.length(); } } System.out.println(System.currentTimeMillis() - start); Splitter splitter = Splitter.on(','); start = System.currentTimeMillis(); for(int i=0; i<1000000; i++) { Iterable numbers = splitter.split(numberList); for(String number : numbers) { number.length(); } } System.out.println(System.currentTimeMillis() - start); On my machine this outputs: 609 2048 Guava's Splitter is almost 4 times slower! Indeed, I was expecting them to be about the same, or maybe Guava slightly faster, so this is another surprising result. Looks like by returning an Iterable, Splitter is trading immediate gains, for longer term pain. There's also a moral here about making sure performance tests are actually testing something useful. In conclusion I think I'll still use Splitter most of the time. On small lists the difference in performance is going to be negligible, and Splitter just feels much nicer to use. Still I was surprised by the result, and if you're splitting lots of Strings and performance is an issue, it might be worth considering switching back to Commons StringUtils.

An old customer recently asked me if I had a solution for how to integrate between their existing JMS infrastructure on Websphere MQ with RabbitMQ. Although I know that RabbitMQ has the shovel plugin which can bridge between Rabbit instances I've yet not found a good plugin for JMS <-> AMQP forwarding. The first thing that came to my mind was to utilize a Spring Integration mediation as SI has excellent support for both JMS and Rabbit. Curious as I am I started a PoC and this is the result. It takes messages of a JMS queue and forwards to an AMQP exchange that is bound to a queue the consumer application is supposed to listen to. I used an external HornetQ instance in JBoss 6.1 as the JMS Provider, but I am 100% secure that the same setup would work for Websphere MQ as they both implement JMS. Be aware that I've done no performance tweaking or QoS setup yet as this is just a proof-of-concept. For a real setup you'd probably have to think about delivery guarantees versus performance and etc... The code will be available at a GitHub repository near you soon.. SpringContext in XML: org.jnp.interfaces.NamingContextFactory jnp://localhost:1099 org.jnp.interfaces:org.jboss.naming ConnectionFactory Maven POM: 4.0.0 org.rl si.jmstorabbit 0.0.1-SNAPSHOT jar si.jmstorabbit http://maven.apache.org UTF-8 2.2.5.Final 2.1.0.RELEASE springsource-release http://repository.springsource.com/maven/bundles/release false springsource-external http://repository.springsource.com/maven/bundles/external false org.springframework.integration spring-integration-core ${spring.integration.version} org.springframework.integration spring-integration-file ${spring.integration.version} org.springframework.integration spring-integration-amqp ${spring.integration.version} org.springframework.integration spring-integration-jms ${spring.integration.version} junit junit 3.8.1 test org.springframework spring-context 3.0.7.RELEASE jboss jnp-client 4.2.2.GA org.hornetq hornetq-core-client ${hornet.version} org.hornetq hornetq-jms-client ${hornet.version} org.hornetq hornetq-jms ${hornet.version} jboss jboss-common-client 3.2.3 org.jboss.netty netty 3.2.7.Final javax.jms jms 1.1

In this article we'll look at how you can connect from Scala to RabbitMQ so you can support the AMQP protocol from your applications. In this example I'll use the Play Framework 2.0 as container (for more info on this see my other article on this subject) to run the application in, since Play makes developing with Scala a lot easier. This article will also use Akka actors to send and receive the messages from RabbitMQ. What is AMQP First, a quick introduction into AMQP. AMQP stands for "Advanced Message Queueing Protocol" and is an open standard for messaging. The AMQP homepage states their vision as this: "To become the standard protocol for interoperability between all messaging middleware". AMQP defines a transport level protocol for exchanging messages that can be used to integrate applications from a number of different platform, languages and technologies. There are a number of tools implementing this protocol, but one that is getting more and more attention is RabbitMQ. RabbitMQ is an open source, erlang based message broker that uses AMQP. All application that can speak AMQP can connect to and make use of RabbitMQ. So in this article we'll show how you can connect from your Play2/Scala/Akka based application to RabbitMQ. In this article we'll show you how to do implement the two most common scenarios: Send / recieve: We'll configure one sender to send a message every couple of seconds, and use two listeners that will read the messages, in a round robin fashion, from the queue. Publish / subscribe: For this example we'll create pretty much the same scenario, but this time, the listeners will both get the message at the same time. I assume you've got an installation of RabbitMQ. If not follow the instructions from their site. Setup basic Play 2 / Scala project For this example I created a new Play 2 project. Doing this is very easy: jos@Joss-MacBook-Pro.local:~/Dev/play-2.0-RC2$ ./play new Play2AndRabbitMQ _ _ _ __ | | __ _ _ _| | | '_ \| |/ _' | || |_| | __/|_|\____|\__ (_) |_| |__/ play! 2.0-RC2, http://www.playframework.org The new application will be created in /Users/jos/Dev/play-2.0/PlayAndRabbitMQ What is the application name? > PlayAndRabbitMQ Which template do you want to use for this new application? 1 - Create a simple Scala application 2 - Create a simple Java application 3 - Create an empty project > 1 OK, application PlayAndRabbitMQ is created. Have fun! I am used to work from Eclipse with the scala-ide pluging, so I execute play eclipsify and import the project in Eclipse. The next step we need to do is set up the correct dependencies. Play uses sbt for this and allows you to configure your dependencies from the build.scala file in your project directory. The only dependency we'll add is the java client library from RabbitMQ. Even though Lift provides a scala based AMQP library, I find using the RabbitMQ one directly just as easy. After adding the dependency my build.scala looks like this: import sbt._ import Keys._ import PlayProject._ object ApplicationBuild extends Build { val appName = "PlayAndRabbitMQ" val appVersion = "1.0-SNAPSHOT" val appDependencies = Seq( "com.rabbitmq" % "amqp-client" % "2.8.1" ) val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings( ) } Add rabbitMQ configuration to the config file For our examples we can configure a couple of things. The queue where to send the message to, the exchange to use, and the host where RabbitMQ is running. In a real world scenario we would have more configuration options to set, but for this case we'll just have these three. Add the following to your application.conf so that we can reference it from our application. #rabbit-mq configuration rabbitmq.host=localhost rabbitmq.queue=queue1 rabbitmq.exchange=exchange1 We can now access these configuration files using the ConfigFactory. To allow easy access create the following object: object Config { val RABBITMQ_HOST = ConfigFactory.load().getString("rabbitmq.host"); val RABBITMQ_QUEUE = ConfigFactory.load().getString("rabbitmq.queue"); val RABBITMQ_EXCHANGEE = ConfigFactory.load().getString("rabbitmq.exchange"); } Initialize the connection to RabbitMQ We've got one more object to define before we'll look at how we can use RabbitMQ to send and receive messages. to work with RabbitMQ we require a connection. We can get a connection to a server by using a ConnectionFactory. Look at the javadocs for more information on how to configure the connection. object RabbitMQConnection { private val connection: Connection = null; /** * Return a connection if one doesn't exist. Else create * a new one */ def getConnection(): Connection = { connection match { case null => { val factory = new ConnectionFactory(); factory.setHost(Config.RABBITMQ_HOST); factory.newConnection(); } case _ => connection } } } Start the listeners when the application starts We need to do one more thing before we can look at the RabbitMQ code. We need to make sure our message listeners are registered on application startup and our senders start sending. Play 2 provides a GlobalSettings object for this which you can extend to execute code when your application starts. For our example we'll use the following object (remember, this needs to be stored in the default namespace: import play.api.mvc._ import play.api._ import rabbitmq.Sender object Global extends GlobalSettings { override def onStart(app: Application) { Sender.startSending } } We'll look at this Sender.startSending operation, which initializes all the senders and receivers in the following sections. Setup send and receive scenario Let's look at the Sender.startSending code that will setup a sender that sends a msg to a specific queue. For this we use the following piece of code: object Sender { def startSending = { // create the connection val connection = RabbitMQConnection.getConnection(); // create the channel we use to send val sendingChannel = connection.createChannel(); // make sure the queue exists we want to send to sendingChannel.queueDeclare(Config.RABBITMQ_QUEUE, false, false, false, null); Akka.system.scheduler.schedule(2 seconds, 1 seconds , Akka.system.actorOf(Props( new SendingActor(channel = sendingChannel, queue = Config.RABBITMQ_QUEUE))) , "MSG to Queue"); } } class SendingActor(channel: Channel, queue: String) extends Actor { def receive = { case some: String => { val msg = (some + " : " + System.currentTimeMillis()); channel.basicPublish("", queue, null, msg.getBytes()); Logger.info(msg); } case _ => {} } } In this code we take the following steps: Use the factory to retrieve a connection to RabbitMQ Create a channel on this connection to use in communicating with RabbitMQ Use the channel to create the queue (if it doesn't exist yet) Schedule Akka to send a message to an actor every second. This all should be pretty straightforward. The only (somewhat) complex part is the scheduling part. What this schedule operation does is this. We tell Akka to schedule a message to be sent to an actor. We want a 2 seconds delay before it is fired, and we want to repeat this job every second. The actor that should be used for this is the SendingActor you can also see in this listing. This actor needs access to a channel to send a message and this actor also needs to know where to send the message it receives to. This is the queue. So every second this Actor will receive a message, append a timestamp, and use the provided channel to send this message to the queue: channel.basicPublish("", queue, null, msg.getBytes());. Now that we send a message each second it would be nice to have listeners on this queue that can receive messages. For receiving messages we've also created an Actor that listens indefinitely on a specific queue. class ListeningActor(channel: Channel, queue: String, f: (String) => Any) extends Actor { // called on the initial run def receive = { case _ => startReceving } def startReceving = { val consumer = new QueueingConsumer(channel); channel.basicConsume(queue, true, consumer); while (true) { // wait for the message val delivery = consumer.nextDelivery(); val msg = new String(delivery.getBody()); // send the message to the provided callback function // and execute this in a subactor context.actorOf(Props(new Actor { def receive = { case some: String => f(some); } })) ! msg } } } This actor is a little bit more complex than the one we used for sending. When this actor receives a message (kind of message doesn't matter) it starts listening on the queue it was created with. It does this by creating a consumer using the supplied channel and tells the consumers to start listening on the specified queue. The consumer.nextDelivery() method will block until a message is waiting in the configured queue. Once a message is received, a new Actor is created to which the message is sent. This new actor passes the message on to the supplied method, where you can put your business logic. To use this listener we need to supply the following arguments: Channel: Allows access to RabbitMQ Queue: The queue to listen to for messages f: The function that we'll execute when a message is received. The final step for this first example is glueing everything together. We do this by adding a couple of method calls to the Sender.startSending method. def startSending = { ... val callback1 = (x: String) => Logger.info("Recieved on queue callback 1: " + x); setupListener(connection.createChannel(),Config.RABBITMQ_QUEUE, callback1); // create an actor that starts listening on the specified queue and passes the // received message to the provided callback val callback2 = (x: String) => Logger.info("Recieved on queue callback 2: " + x); // setup the listener that sends to a specific queue using the SendingActor setupListener(connection.createChannel(),Config.RABBITMQ_QUEUE, callback2); ... } private def setupListener(receivingChannel: Channel, queue: String, f: (String) => Any) { Akka.system.scheduler.scheduleOnce(2 seconds, Akka.system.actorOf(Props(new ListeningActor(receivingChannel, queue, f))), ""); } In this code you can see that we define a callback function, and use this callback function, together with the queue and the channel to create the ListeningActor. We use the scheduleOnce method to start this listener in a separate thread. Now with this code in place we can run the application (play run) open up localhost:9000 to start the application and we should see something like the following output. [info] play - Starting application default Akka system. [info] play - Application started (Dev) [info] application - MSG to Exchange : 1334324531424 [info] application - MSG to Queue : 1334324531424 [info] application - Recieved on queue callback 2: MSG to Queue : 1334324531424 [info] application - MSG to Exchange : 1334324532522 [info] application - MSG to Queue : 1334324532522 [info] application - Recieved on queue callback 1: MSG to Queue : 1334324532522 [info] application - MSG to Exchange : 1334324533622 [info] application - MSG to Queue : 1334324533622 [info] application - Recieved on queue callback 2: MSG to Queue : 1334324533622 [info] application - MSG to Exchange : 1334324534722 [info] application - MSG to Queue : 1334324534722 [info] application - Recieved on queue callback 1: MSG to Queue : 1334324534722 [info] application - MSG to Exchange : 1334324535822 [info] application - MSG to Queue : 1334324535822 [info] application - Recieved on queue callback 2: MSG to Queue : 1334324535822 Here you can clearly see the round-robin way messages are processed. Setup publish and subscribe scenario Once we've got the above code running, adding publish / subscribe functionality is very trivial. Instead of the SendingActor we now use a PublishingActor: class PublishingActor(channel: Channel, exchange: String) extends Actor { /** * When we receive a message we sent it using the configured channel */ def receive = { case some: String => { val msg = (some + " : " + System.currentTimeMillis()); channel.basicPublish(exchange, "", null, msg.getBytes()); Logger.info(msg); } case _ => {} } } An exchange is used by RabbitMQ to allow multiple recipients to receive the same message (and a whole lot of other advanced functionality). The only change in the code from the other actor is that this time we send the message to an exchange instead of to a queue. The listener code is exactly the same, the only thing we need to do is connect a queue to a specific exchange. So that listeners on that queue receive the messages sent to to the exchange. We do this, once again, from the setup method we used earlier. ... // create a new sending channel on which we declare the exchange val sendingChannel2 = connection.createChannel(); sendingChannel2.exchangeDeclare(Config.RABBITMQ_EXCHANGEE, "fanout"); // define the two callbacks for our listeners val callback3 = (x: String) => Logger.info("Recieved on exchange callback 3: " + x); val callback4 = (x: String) => Logger.info("Recieved on exchange callback 4: " + x); // create a channel for the listener and setup the first listener val listenChannel1 = connection.createChannel(); setupListener(listenChannel1,listenChannel1.queueDeclare().getQueue(), Config.RABBITMQ_EXCHANGEE, callback3); // create another channel for a listener and setup the second listener val listenChannel2 = connection.createChannel(); setupListener(listenChannel2,listenChannel2.queueDeclare().getQueue(), Config.RABBITMQ_EXCHANGEE, callback4); // create an actor that is invoked every two seconds after a delay of // two seconds with the message "msg" Akka.system.scheduler.schedule(2 seconds, 1 seconds, Akka.system.actorOf(Props( new PublishingActor(channel = sendingChannel2 , exchange = Config.RABBITMQ_EXCHANGEE))), "MSG to Exchange"); ... We also created an overloaded method for setupListener, which, as an extra parameter, also accepts the name of the exchange to use. private def setupListener(channel: Channel, queueName : String, exchange: String, f: (String) => Any) { channel.queueBind(queueName, exchange, ""); Akka.system.scheduler.scheduleOnce(2 seconds, Akka.system.actorOf(Props(new ListeningActor(channel, queueName, f))), ""); } In this small piece of code you can see that we bind the supplied queue (which is a random name in our example) to the specified exchange. After that we create a new listener as we've seen before. Running this code now will result in the following output: [info] play - Application started (Dev) [info] application - MSG to Exchange : 1334325448907 [info] application - MSG to Queue : 1334325448907 [info] application - Recieved on exchange callback 3: MSG to Exchange : 1334325448907 [info] application - Recieved on exchange callback 4: MSG to Exchange : 1334325448907 [info] application - MSG to Exchange : 1334325450006 [info] application - MSG to Queue : 1334325450006 [info] application - Recieved on exchange callback 4: MSG to Exchange : 1334325450006 [info] application - Recieved on exchange callback 3: MSG to Exchange : 1334325450006 As you can see, in this scenario both listeners receive the same message. That pretty much wraps it up for this article. As you've seen using the Java based client api for RabbitMQ is more than sufficient, and easy to use from Scala. Note though that this example is not production ready, you should take care to close connections, nicely shutdown listeners and actors. All this shutdown code isn't shown here.

Below is an implementation of a Regular Expression HashMap. It works with key-value pairs which the key is a regular expression. It compiles the key (regular expression) while adding (i.e. putting), so there is no compile time while getting. Once getting an element, you don't give regular expression; you give any possible value of a regular expression. As a result, this behaviour provides to map numerous values of a regular expression into the same value. The class does not depend to any external libraries, uses only default java.util. So, it will be used simply when a behaviour like that is required. import java.util.ArrayList; import java.util.HashMap; import java.util.regex.Pattern; /** * This class is an extended version of Java HashMap * and includes pattern-value lists which are used to * evaluate regular expression values. If given item * is a regular expression, it is saved in regexp lists. * If requested item matches with a regular expression, * its value is get from regexp lists. * * @author cb * * @param : Key of the map item. * @param : Value of the map item. */ public class RegExHashMap extends HashMap { // list of regular expression patterns private ArrayList regExPatterns = new ArrayList(); // list of regular expression values which match patterns private ArrayList regExValues = new ArrayList(); /** * Compile regular expression and add it to the regexp list as key. */ @Override public V put(K key, V value) { regExPatterns.add(Pattern.compile(key.toString())); regExValues.add(value); return value; } /** * If requested value matches with a regular expression, * returns it from regexp lists. */ @Override public V get(Object key) { CharSequence cs = new String(key.toString()); for (int i = 0; i < regExPatterns.size(); i++) { if (regExPatterns.get(i).matcher(cs).matches()) { return regExValues.get(i); } } return super.get(key); } }

In my recent projects I've had to do a lot with certificates, java and HTTPS with client-side authentication. In most of these projects, either during testing, or setting up a new environment, I've run into various SSL configuration errors that often resulted in a rather uncomprehensive error such as: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) In most of the cases it was misconfiguration where keystores didn't containt the correct certificates, the certificate chain was incomplete or the client didn't supply a valid certificate. So in the last project I decided to document what was happening and what caused specific errors during the SSL handshake. In this article I'll show you why specific SSL errors occur, how you can detect them by analyzing the handshake information, and how to solve them. For this I use the following scenario: Server uses a certificate issued by a CA and requires client authentication. The server uses a simple truststore that lists this CA as trusted. Client connects using a certificate issued by this single trusted CA and has it's own trustore that also contains this certificate from the server. Not a very complicated situation, but one you often see. Note that the following information can also be used to identify problems when you don't work with client certificates or use self-signed certificates. The way to determine the problem in those cases, is pretty much the same. Happy Flow First we'll look at the happy flow, what happens in the handshake when we use client certificates. We won't look at the complete negotiation phase, but only until both the client and the server have exchanged their certificates and have validated the received certificate. If everything goes well until that point, the rest should work. The following is what you see when you run the client and the server using the java VM parameter: -Djavax.net.debug=ssl:handshake. The first thing that happens is that the client sends a ClientHello message using the TLS protocol version he supports, a random number and a list of suggested cipher suites and compression methods. From our client this looks like this: Client sends: *** ClientHello, TLSv1 RandomCookie: GMT: 1331663143 bytes = { 141, 219, 18, 140, 148, 60, 33, 241, 10, 21, 31, 90, 88, 145, 34, 153, 238, 105, 148, 72, 163, 210, 233, 49, 99, 224, 226, 64 } Session ID: {} Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] Compression Methods: { 0 } *** The server responds, very originally, with a ServerHello message, that contains the choices made based on the information provided by the client another random number and (optionally) a session id. Server sends: *** ServerHello, TLSv1 RandomCookie: GMT: 1331663143 bytes = { 172, 233, 79, 197, 14, 21, 187, 161, 114, 206, 7, 38, 188, 228, 120, 102, 115, 214, 155, 86, 211, 41, 156, 179, 138, 2, 230, 81 } Session ID: {79, 96, 145, 39, 203, 136, 206, 69, 170, 46, 194, 17, 154, 175, 13, 138, 143, 199, 162, 193, 110, 86, 113, 109, 248, 187, 220, 169, 47, 180, 44, 68} Cipher Suite: SSL_RSA_WITH_RC4_128_MD5 Compression Method: 0 Extension renegotiation_info, renegotiated_connection: *** So in this case we're going to use SSL_RSA_WITH_RC4_128_MD5 as Cipher Suite. The next step is also done by the server. The server next sends a Certificate message that contains its complete certificate chain: Server sends: *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=server, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 1024 bits modulus: 143864428144045085986129639694300995179398936575198896494655652087658861594939489453166811774109137006267822033915476680673848164790815913192075840268069822357600376998775923266017630332239546722181180383155088413406178660120548292599278819762883993031950564327152510982887716901499177102158407884939613382007 public exponent: 65537 Validity: [From: Wed Mar 14 13:32:04 CET 2012, To: Thu Mar 14 13:32:04 CET 2013] Issuer: CN=Application CA, OU=GKD, O=Smartjava, L=Maasland, ST=ZH, C=NL SerialNumber: [ a881d144 5e631f21] ] Algorithm: [SHA1withRSA] Signature: 0000: C3 56 81 7F 33 91 8A FF 84 5E 0B BA 7A 01 D8 41 .V..3....^..z..A 0010: 6B 47 B2 F7 8F FB B5 77 23 D8 FB B2 35 19 6E C4 kG.....w#...5.n. 0020: A4 6A BC 23 BB 69 92 F6 85 5A 1E CB FE 23 C6 98 .j.#.i...Z...#.. 0030: A0 57 F8 FB E9 DB B0 40 BD 8E F8 35 F8 77 E1 09 .W.....@...5.w.. 0040: 5A 2E 45 71 80 F6 89 E7 0B 93 E2 48 EB 40 92 13 Z.Eq.......H.@.. 0050: 14 AA 1F 59 AA 98 67 46 9B 52 33 49 9A 3C 91 9B ...Y..gF.R3I.<.. 0060: F1 CB 8A BD 7D D4 DD 76 C4 15 00 36 A3 B2 87 A7 .......v...6.... 0070: D5 FF 52 E3 68 D4 F0 E0 32 86 74 02 DD 92 EC 1D ..R.h...2.t..... ] chain [1] = [ [ Version: V3 Subject: CN=Application CA, OU=SL, O=SmartJava, L=Waalwijk, ST=ZH, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 1024 bits modulus: 159927271510058538658170959055540487654246676457579822126433656091883150307639380685203152841988861440546492270915750324654620063428634486478674507234742748515614639629692189315918046446256610037776978028900716455223387878926383828815082154427031884246429239077082613371662803582187768145965112751392402313823 public exponent: 65537 Validity: [From: Mon Mar 12 13:35:16 CET 2012, To: Wed Apr 11 14:35:16 CEST 2012] Issuer: CN=Application CA, OU=CA, O=Blaat, L=Waalwijk, ST=ZH, C=NL SerialNumber: [ fe7636c5 6804e69c] Certificate Extensions: 3 [1]: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 6C CC 48 03 E4 BE 07 D6 9E F6 4C 78 53 54 A2 B8 l.H.......LxST.. 0010: 7B DA 40 09 ..@. ] ] [2]: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 6C CC 48 03 E4 BE 07 D6 9E F6 4C 78 53 54 A2 B8 l.H.......LxST.. 0010: 7B DA 40 09 ..@. ] ] [3]: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:true PathLen:2147483647 ] ] Algorithm: [SHA1withRSA] Signature: 0000: 1A 30 08 15 01 8E A6 36 5F 38 22 C6 81 5E 69 B1 .0.....6_8"..^i. 0010: 42 9A 1E FF 0F C4 D7 40 5F 85 0E 42 35 E0 CC 00 B......@_..B5... 0020: 6E A5 2E 70 6B 79 64 C5 99 AE A4 29 CB 26 DE 60 n..pkyd....).&.` 0030: 0B A6 AB 19 06 6F 19 54 6C 1A 88 9E 3A 6A D4 BB .....o.Tl...:j.. 0040: CB 28 85 2F 72 4D DE 35 C0 9B F4 2F EF 8E 6D E8 .(./rM.5.../..m. 0050: 30 AC 12 7D B4 0D A3 08 DA D4 60 46 94 BD 12 AF 0.........`F.... 0060: 44 F7 C3 B8 9D 69 2D 6A 32 C8 4D AE 12 60 05 09 D....i-j2.M..`.. 0070: FE AE D0 1A 72 6D 91 CE DA 7C 8E D5 31 14 31 4C ....rm......1.1L ] In this message you can see that the issuer of this certificate is our example CA. Our client checks to see if this certificate is trusted, which it is in this case. Since we require the client to authenticate itself the server requests a certificate from the client and after that sends a helloDone. Server sends: *** CertificateRequest Cert Types: RSA, DSS Cert Authorities: *** ServerHelloDone In this message you can see that the server provides a list of Cert Authorities it trusts. The client will use this information to determine if it has a keypair that matches this CA. In our happy flow, it has one and responds with a Certificate message. Client sends: *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=Application 3, OU=Smartjava, O=Smartjava, L=NL, ST=ZH, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 1024 bits modulus: 90655907749318585147523875906892969031300830816947226352221659107570169820452561428696751943383590982109524990627182456571533992582229229163232831159652561902456847954385746762477844009336466314872376131553489447601649924116778337873632641536164462534398137791450495316700015095054427027256393580022887087767 public exponent: 65537 Validity: [From: Mon Mar 12 15:13:24 CET 2012, To: Tue Mar 12 15:13:24 CET 2013] Issuer: CN=Application CA, OU=Smartjava, O=Smartjava, L=Maasland, ST=ZH, C=NL SerialNumber: [ b247ffb2 ce060768] ] Algorithm: [SHA1withRSA] Signature: 0000: 97 58 36 C5 28 87 B3 16 9B DD 31 0C E0 C6 23 76 .X6.(.....1...#v 0010: 72 82 5B 13 4D 23 B6 0E A9 2F 9F 0C 3F 97 15 6E r.[.M#.../..?..n 0020: 7B 38 EC DE E2 57 D7 AA 07 12 E3 98 B7 86 A7 CE .8...W.......... 0030: 57 8E A1 29 96 C9 F0 30 57 67 C7 F1 F2 98 90 64 W..)...0Wg.....d 0040: 6C B9 6C 05 24 8B 56 3F B1 FF 03 62 3D 81 DB 45 l.l.$.V?...b=..E 0050: D3 1F C1 B2 DD 77 CF 74 54 EB 9D 82 23 89 1A 70 .....w.tT...#..p 0060: F8 C4 68 6A B7 41 C7 DE 7B B6 3A 0C 17 E7 FA 98 ..hj.A....:..... 0070: 19 0C D8 91 FB 5E FE D2 B3 92 FD 2D 2A 6B 51 10 .....^.....-*kQ. ] chain [1] = [ [ Version: V3 Subject: CN=Application CA, OU=Smartjava, O=Smartjava, L=Maasland, ST=ZH, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 1024 bits modulus: 159927271510058538658170959055540487654246676457579822126433656091883150307639380685203152841988861440546492270915750324654620063428634486478674507234742748515614639629692189315918046446256610037776978028900716455223387878926383828815082154427031884246429239077082613371662803582187768145965112751392402313823 public exponent: 65537 Validity: [From: Mon Mar 12 13:35:16 CET 2012, To: Wed Apr 11 14:35:16 CEST 2012] Issuer: CN=Application CA, OU=Smartjava, O=Smartjava, L=Maasland, ST=ZH, C=NL SerialNumber: [ fe7636c5 6804e69c] Certificate Extensions: 3 [1]: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 6C CC 48 03 E4 BE 07 D6 9E F6 4C 78 53 54 A2 B8 l.H.......LxST.. 0010: 7B DA 40 09 ..@. ] ] [2]: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 6C CC 48 03 E4 BE 07 D6 9E F6 4C 78 53 54 A2 B8 l.H.......LxST.. 0010: 7B DA 40 09 ..@. ] ] [3]: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:true PathLen:2147483647 ] ] Algorithm: [SHA1withRSA] Signature: 0000: 1A 30 08 15 01 8E A6 36 5F 38 22 C6 81 5E 69 B1 .0.....6_8"..^i. 0010: 42 9A 1E FF 0F C4 D7 40 5F 85 0E 42 35 E0 CC 00 B......@_..B5... 0020: 6E A5 2E 70 6B 79 64 C5 99 AE A4 29 CB 26 DE 60 n..pkyd....).&.` 0030: 0B A6 AB 19 06 6F 19 54 6C 1A 88 9E 3A 6A D4 BB .....o.Tl...:j.. 0040: CB 28 85 2F 72 4D DE 35 C0 9B F4 2F EF 8E 6D E8 .(./rM.5.../..m. 0050: 30 AC 12 7D B4 0D A3 08 DA D4 60 46 94 BD 12 AF 0.........`F.... 0060: 44 F7 C3 B8 9D 69 2D 6A 32 C8 4D AE 12 60 05 09 D....i-j2.M..`.. 0070: FE AE D0 1A 72 6D 91 CE DA 7C 8E D5 31 14 31 4C ....rm......1.1L ] This certificate is checked on the server side and if all is well, the final steps in the handshake are executed to setup the secured connection. Note that there is a CertificateVerify step. In this step the client signs a message with its private key. This is done so the server can verify the client has access to its private key. This might seem a step where things can go wrong in an incorrectly configured environment. In the default java implementation this won't happen. In the phase where the client has to determine which certificate to present to the server, the java implementation already checks if the privatekey is available. What could possibly go wrong So what could possibly go wrong in this handshake? In the next couple of sections we'll look at some scenarios, and how to detect them. Passwords Now that we've seen what happens when things go right, lets look at a couple of scenarios where things go wrong. We'll start simple with the following exception, that we get at the moment we start up the client application: Exception in thread "main" java.security.UnrecoverableKeyException: Cannot recover key at sun.security.provider.KeyProtector.recover(KeyProtector.java:311) at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:121) at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:38) at java.security.KeyStore.getKey(KeyStore.java:763) at com.sun.net.ssl.internal.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:113) at com.sun.net.ssl.internal.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:48) at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:239) at org.apache.http.conn.ssl.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:186) at org.apache.http.conn.ssl.SSLSocketFactory.(SSLSocketFactory.java:260) This very helpful message is thrown when (from the javadoc) " .. a key in the keystore cannot be recovered". There are a couple of reasons this can happen, but normally this occurs when the key in the keystore is accessed with the wrong password. Usually when you use the keytool to create and manage your keys, the keystore password is usually the same as the key password. However, if you import keys from a PKCS#12 type keystore, the password of the keystore can be easily set to a different value. Not all the SSL client allow you to specify a different password for the key and the keystore. If that is the case you can use the following command, to change the password of the key: keytool -keypasswd -alias -keystore It is also possible to set an incorrect password for the keystore. Luckily in that case the error message that is thrown is much more helpful: Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:771) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:38) at java.security.KeyStore.load(KeyStore.java:1185) ... Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:769) ... 3 more If this occurs at the server side, we can see the same message when the SSL listener is being set up. Incomplete CA Chains Now lets look at the first of the "peer not authenticated" exceptions. In the logging we see this exception at the client side: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) So enable SSL logging, run again, and we'll start with analyzing the handshake. We'll start by looking from the client side. If we look through the logging we find the following CertificateRequest message from the server and the ServerHelloDone. *** CertificateRequest Cert Types: RSA, DSS Cert Authorities: *** ServerHelloDone So thus far, everything went ok. The server has already sent its certificate, and since our client doesn't throw an error on that part, we can assume it is trusted by the client. So something seems to be wrong with the steps that come after this message from the server. If you look closer at this message, you can see that the server doesn't specify a set of Cert Authorities it trusts. This could be a misconfiguration at the server side, or it could just be that the server expects one of the trusted Root CAs. In any case, the client is free to send any certificate he wants. So the client sends the following certificate: *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=Application4, OU=Smartjava, O=Smartjava, L=NL, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 ... ] chain [1] = [ [ Version: V3 Subject: EMAILADDRESS=jos.dirksen@gmail.com, CN=CA2, OU=Smartjava, O=Smartjava, L=Waalwijk, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 ... ] According to the specification the client now continues with the key exchange and generates secrets to exchange. Somewhere along the lines we can see the following: pool-1-thread-1, WRITE: TLSv1 Handshake, length = 32 pool-1-thread-1, READ: TLSv1 Alert, length = 2 pool-1-thread-1, RECV TLSv1 ALERT: fatal, internal_error pool-1-thread-1, called closeSocket() This means we've received an internal error. So something at the server side went wrong. Looking at the server we see the following in the SSL dump: *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=Application4, OU=Smartjava, O=Smartjava, L=NL, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 ... ] chain [1] = [ [ Version: V3 Subject: EMAILADDRESS=jos.dirksen@gmail.com, CN=CA2, OU=Smartjava, O=Smartjava, L=Waalwijk, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 ... ] *** qtp1735121130-17, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty qtp1735121130-17, SEND TLSv1 ALERT: fatal, description = internal_error qtp1735121130-17, WRITE: TLSv1 Alert, length = 2 You can see that we received the certificate from the client, and directly after that we get this error. This error however doesn't really tell us anything. We do however have enough information to at least limit the possible errors. We know that the server didn't sent a list of CAs, we can see that the client sent a valid certificate, and that server somehow isn't able to process it. It looks like a problem with the server truststore. In this case the best approach is to look at the certificates the server trusts. Either in the cacerts file or in it's own truststore. Validate whether the CA certificate our client sends is in the server's truststore, and the server actually loads the stores we expect. It's of course also possible that the client has an incomplete chain of trust for the certificate received from the server. In that case we once again get the "peer not authenticated" error at the client side. If we look at the SSL debug logging, we see the following exception occuring at the client side: pool-1-thread-1, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty pool-1-thread-1, SEND TLSv1 ALERT: fatal, description = internal_error pool-1-thread-1, WRITE: TLSv1 Alert, length = 2 This exception occured directly after the server has sent its certificate using a "Certificate message": *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=server, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Following the same reasoning as for the server we can conclude that there is something wrong with the client side truststore. For completeness sake, the server receives this error message when this situation occurs at the client: qtp1500389297-17, READ: TLSv1 Alert, length = 2 qtp1500389297-17, RECV TLSv1 ALERT: fatal, internal_error qtp1500389297-17, called closeSocket() qtp1500389297-17, handling exception: javax.net.ssl.SSLException: Received fatal alert: internal_error qtp1500389297-17, called close() qtp1500389297-17, called closeInternal(true) Invalid keys For the next exercise lets look at the following error that occurs during this handshake. In the logging at the client side we see the following error message in the SSL output: ool-1-thread-1, WRITE: TLSv1 Handshake, length = 32 pool-1-thread-1, READ: TLSv1 Alert, length = 2 pool-1-thread-1, RECV TLSv1 ALERT: fatal, internal_error pool-1-thread-1, called closeSocket() pool-1-thread-1, handling exception: javax.net.ssl.SSLException: Received fatal alert: internal_error pool-1-thread-1, IOException in getSession(): javax.net.ssl.SSLException: Received fatal alert: internal_error Which results in the very unhelpful: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) When you receive an internal error, there is usually something wrong at the server side. So looking at the serverside, lets see what caused this error. *** qtp2044601711-16, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty qtp2044601711-16, SEND TLSv1 ALERT: fatal, description = internal_error Hmm.. somewhat more useful. It seems that there is something wrong with the algorithm we used, the client seems to have provided an incorrect certificate. But what is wrong? If you look back at the happy flow, you can send that at a certain time the server asks the client for a certificate using a "Certificate" message. Lets look a bit closer at this message and the response: *** CertificateRequest Cert Types: RSA, DSS Cert Authorities: *** ServerHelloDone matching alias: application4 *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=Application4, OU=Smartjava, O=Smartjava, L=NL, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 Key: Sun DSA Public Key ... What you can see here is that the server specifies the cert types it accepts, and the authorities it accepts. The client responses in this case however with a DSA public key. Depending on the server implementation this can cause this strange message. Another possible scenario I've seen (especially with self-signed certificates) is that with a "CertificateRequest" message like this: *** CertificateRequest Cert Types: RSA, DSS Cert Authorities: *** ServerHelloDone This client won't respond with a certificate at all, if you only have DSA based keys in your keystore. It won't throw an error on the client side, but will cause a "null certificate chain" message as the server side. I haven't seen this scenario, though, when you don't use self-signed certificates. Certificate expiration So far we've seen how you can analyze the SSL handshake to determine where to look for configuration errors. In this last example we'll look at what happens when a certificate expires. In this case we once again see the very cryptic message at the client side: pool-1-thread-1, READ: TLSv1 Alert, length = 2 pool-1-thread-1, RECV TLSv1 ALERT: fatal, certificate_unknown pool-1-thread-1, called closeSocket() pool-1-thread-1, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown pool-1-thread-1, IOException in getSession(): javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown pool-1-thread-1, called close() pool-1-thread-1, called closeInternal(true) pool-1-thread-1, called close() pool-1-thread-1, called closeInternal(true) javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) If we look at the phase of the SSL handshake we're in, we can see that we've already sent our client certificate and finishing up the handshake when we receive this error. The error on the serverside is actually pretty helpful. After receiving the invalid certificate, in the debug logging, it shows us the following: *** qtp1735121130-17, SEND TLSv1 ALERT: fatal, description = certificate_unknown qtp1735121130-17, WRITE: TLSv1 Alert, length = 2 [Raw write]: length = 7 0000: 15 03 01 00 02 02 2E ....... qtp1735121130-17, called closeSocket() qtp1735121130-17, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed qtp1735121130-17, called close() qtp1735121130-17, called closeInternal(true) It tells us that during the validation of the certificate, a timestamp check failed. This tells us that we should look at the validity of the certificates in our certificate chain to see what is happening. Summary In this article you've seen a couple of common causes for SSL exceptions and ways to identify the exception. Their can be many causes for these kind of exceptions, the most common though are the following: Incorrect certificate chains in the client truststore Incorrect certificate chains in the server truststore Invalid key algorithm used for private keys Expired certificate or expired CA certificate Incorrect passwords used to access the keys Multiple private keys to choose from If you're presented with a such an exception a good general approach is this. You first check the keystores that are involved. Use the java keytool for this: keytool -list -v -keystore This will print out all the certificates and keys in the keystore. Check whether the keys are of a supported type, the required CA certificates are stored and that your application is using the correct one (spent hours figuring out an issue because I was looking into a truststore for my private key). If everything seems to be OK at first glance it's time to enable ssl debugging (-Djavax.net.debug=ssl:handshake) and check the handshake messages that are sent. Wikipedia has a nice overview of which message is sent at a specific time. For more information on the content of the messages look at the RFC 5246 (or the one of the SSL/TLS version you're using, but the handshake changes are minimal between versions). Using the messages and the handshake, determine at what place in the handshake things go wrong, taking into account that the client will continue with the handshake, while the server is processing it's certificate.

Here is a good and simple anti cross-site scripting (XSS) filter written for Java web applications. What it basically does is remove all suspicious strings from request parameters before returning them to the application. It’s an improvement over my previous post on the topic. You should configure it as the first filter in your chain (web.xml) and it’s generally a good idea to let it catch every request made to your site. The actual implementation consists of two classes, the actual filter is quite simple, it wraps the HTTP request object in a specialized HttpServletRequestWrapper that will perform our filtering. public class XSSFilter implements Filter { @Override public void init(FilterConfig filterConfig) throws ServletException { } @Override public void destroy() { } @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { chain.doFilter(new XSSRequestWrapper((HttpServletRequest) request), response); } } The wrapper overrides the getParameterValues(), getParameter() and getHeader() methods to execute the filtering before returning the desired field to the caller. The actual XSS checking and striping is performed in the stripXSS() private method. import java.util.regex.Pattern; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequestWrapper; public class XSSRequestWrapper extends HttpServletRequestWrapper { private static Pattern[] patterns = new Pattern[]{ // Script fragments Pattern.compile("", Pattern.CASE_INSENSITIVE), // src='...' Pattern.compile("src[\r\n]*=[\r\n]*\\\'(.*?)\\\'", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), Pattern.compile("src[\r\n]*=[\r\n]*\\\"(.*?)\\\"", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), // lonely script tags Pattern.compile("", Pattern.CASE_INSENSITIVE), Pattern.compile("", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), // eval(...) Pattern.compile("eval\\((.*?)\\)", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), // expression(...) Pattern.compile("expression\\((.*?)\\)", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), // javascript:... Pattern.compile("javascript:", Pattern.CASE_INSENSITIVE), // vbscript:... Pattern.compile("vbscript:", Pattern.CASE_INSENSITIVE), // onload(...)=... Pattern.compile("onload(.*?)=", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL) }; public XSSRequestWrapper(HttpServletRequest servletRequest) { super(servletRequest); } @Override public String[] getParameterValues(String parameter) { String[] values = super.getParameterValues(parameter); if (values == null) { return null; } int count = values.length; String[] encodedValues = new String[count]; for (int i = 0; i < count; i++) { encodedValues[i] = stripXSS(values[i]); } return encodedValues; } @Override public String getParameter(String parameter) { String value = super.getParameter(parameter); return stripXSS(value); } @Override public String getHeader(String name) { String value = super.getHeader(name); return stripXSS(value); } private String stripXSS(String value) { if (value != null) { // NOTE: It's highly recommended to use the ESAPI library and uncomment the following line to // avoid encoded attacks. // value = ESAPI.encoder().canonicalize(value); // Avoid null characters value = value.replaceAll("\0", ""); // Remove all sections that match a pattern for (Pattern scriptPattern : patterns){ value = scriptPattern.matcher(value).replaceAll(""); } } return value; } } Notice the comment about the ESAPI library, I strongly recommend you check it out and try to include it in your projects. If you want to dig deeper on the topic I suggest you check out the OWASP page about XSS and RSnake’s XSS (Cross Site Scripting) Cheat Sheet.

Stamping the version number and the build time of an application in a properties file so that it could be displayed by an application at runtime seemed like it should be a pretty straightforward task, although it took a bit of time to find a solution that didn’t require the timestamp, version, or ant-run plugins. I started with a version.txt file at the default package level in src/main/resources of my project, which looks as follows. version=${pom.version} build.date=${timestamp} By default the Maven resources plug-in will not do text substitution (filtering), so it will need to be enabled within the section of the pom.xml file. src/main/resources true Maven does actually define a ${maven.build.timestamp} property which in theory could have been used in the version.txt file, rather than the ${timestamp} property, but unfortunately a bug within Maven prevents the ${maven.build.timestamp} property from getting passed to the resource filtering mechanism. (issue:http://jira.codehaus.org/browse/MRESOURCES-99). The workaround is to create another property within the pom.xml file and set that new property to the timestamp value, in this case, the property name is “timestamp”, which is used above in the version.txt file. The maven.build.timestamp.format is an optional property for (obviously) defining the timestamp format. ${maven.build.timestamp} yyyy-MM-dd HH:mm Now, when the build is executed we end up with the version number and build time of the application in the version.txt file. Easy! version=1.0.2-SNAPSHOT build.date=2012-03-16 15:42

i want to like javafx, really i do. the return of the applet reminds me of the 90s which is nice. i also like the idea of being able to drag an applet into windows, ubuntu , and mac to run it as a desktop application. it's a whole new take on their "write once, run anywhere" promise and breathing some life into a platform that needs it. java used to be so trendy and cool, it was the "ruby on rails of the 90s" now it's seemingly destined to be the "cobol of the 20s". if javafx lives up to its promise it could turn things around. well, i guess android is technically leading a java revival today unless oracle's lawsuit forces google to move to a different language. so far though i've been a little disappointed with javafx. it's like an el camino, a strange combination of awt and swing that doesn't quite feel natural. i'm going to keep trying it anyway and hope that one day it catches up to c# 1.0. look, i know this sounds terribly cynical so far but you have to believe me when i say i'm trying to like it. the reality is, even if javafx is a little clunky now it's still a considerable improvement over swing. over the next few months i'm going to upgrade all my ugly swing applications to javafx. the first one is something called debigulator . it's a batch archive program that i wrote for myself but has been downloaded more than i expected. it's also one of the ugliest programs ever created. just look at this monstrosity: besides being unattractive it also doesn't resize well. javafx addresses both of those so i'm porting it. the first thing to go is that awful file browser in the top left region, i'm embarrassed to look at it. i think i'll replace it with a simple treeview. to create a treeview we first have to create a treeitem subclass to store in the tree. the api documentation for the javafx treeitem class includes a partial implementation of a file browser. i looked at it but went a different direction because it recursively populates the entire tree up front and doesn't deal with things like folder & file icons. instead i wanted to dynamically populate a node when it's expanded. the treeitem also needs to store the path to the file represented by each item but only show the folder or file name. alright, let's get our treeitem implementation started. the constructor and class members look a little something like: public class filepathtreeitem extends treeitem{ public static image foldercollapseimage=new image(classloader.getsystemresourceasstream("com/huguesjohnson/javafxfilebrowsedemo/folder.png")); public static image folderexpandimage=new image(classloader.getsystemresourceasstream("com/huguesjohnson/javafxfilebrowsedemo/folder-open.png")); public static image fileimage=new image(classloader.getsystemresourceasstream("com/huguesjohnson/javafxfilebrowsedemo/text-x-generic.png")); //this stores the full path to the file or directory private string fullpath; public string getfullpath(){return(this.fullpath);} private boolean isdirectory; public boolean isdirectory(){return(this.isdirectory);} public filepathtreeitem(path file){ super(file.tostring()); this.fullpath=file.tostring(); next we want to set the icon, full path, and isdirectory members. this would be a good time to mention that all the icons in this demo come from the tango library . //test if this is a directory and set the icon if(files.isdirectory(file)){ this.isdirectory=true; this.setgraphic(new imageview(foldercollapseimage)); }else{ this.isdirectory=false; this.setgraphic(new imageview(fileimage)); //if you want different icons for different file types this is where you'd do it } //set the value if(!fullpath.endswith(file.separator)){ //set the value (which is what is displayed in the tree) string value=file.tostring(); int indexof=value.lastindexof(file.separator); if(indexof>0){ this.setvalue(value.substring(indexof+1)); }else{ this.setvalue(value); } } now let's add the event handler for the node expanded event. that check for source.isexpanded() sure seems unnecessary. man that was a fun piece of unexpected behavior to track down. this.addeventhandler(treeitem.branchexpandedevent(),new eventhandler(){ @override public void handle(event e){ filepathtreeitem source=(filepathtreeitem)e.getsource(); if(source.isdirectory()&&source.isexpanded()){ imageview iv=(imageview)source.getgraphic(); iv.setimage(folderexpandimage); } try{ if(source.getchildren().isempty()){ path path=paths.get(source.getfullpath()); basicfileattributes attribs=files.readattributes(path,basicfileattributes.class); if(attribs.isdirectory()){ directorystream dir=files.newdirectorystream(path); for(path file:dir){ filepathtreeitem treenode=new filepathtreeitem(file); source.getchildren().add(treenode); } } }else{ //if you want to implement rescanning a directory for changes this would be the place to do it } }catch(ioexception x){ x.printstacktrace(); } } }); we'll wrap up this treeitem implementation with an handler for the node collapsed event. again the source.isexpanded() check really shouldn't be needed but just go ahead and remove it to see the goofiness that follows. this.addeventhandler(treeitem.branchcollapsedevent(),new eventhandler(){ @override public void handle(event e){ filepathtreeitem source=(filepathtreeitem)e.getsource(); if(source.isdirectory()&&!source.isexpanded()){ imageview iv=(imageview)source.getgraphic(); iv.setimage(foldercollapseimage); } } }); now we can go to work on the main program. here's all the basic stuff. public class javafxfilebrowsedemoapp extends application{ private treeview treeview; public static void main(string[] args){ launch(args); } @override public void start(stage primarystage){ //create tree pane vbox treebox=new vbox(); treebox.setpadding(new insets(10,10,10,10)); treebox.setspacing(10); now it's time to start populating the tree. we'll use the computer name as the root node. although i might go back and hide the root node since it's kind of pointless for this application. it's really just showing off how to get the name from the inetaddress class which you either already knew or didn't care about. //setup the file browser root string hostname="computer"; try{hostname=inetaddress.getlocalhost().gethostname();}catch(unknownhostexception x){} treeitem rootnode=new treeitem<>(hostname,new imageview(new image(classloader.getsystemresourceasstream("com/huguesjohnson/javafxfilebrowsedemo/computer.png")))); one nifty addition to jdk7 is the ability to list all the drives on the system. that comes in handy for the next step where we need to add all the drives under the root node. iterable rootdirectories=filesystems.getdefault().getrootdirectories(); for(path name:rootdirectories){ filepathtreeitem treenode=new filepathtreeitem(name); rootnode.getchildren().add(treenode); } rootnode.setexpanded(true); all that's left is to add the treeview to the window and show it. //create the tree view treeview=new treeview<>(rootnode); //add everything to the tree pane treebox.getchildren().addall(new label("file browser"),treeview); vbox.setvgrow(treeview,priority.always); //setup and show the window primarystage.settitle("javafx file browse demo"); stackpane root=new stackpane(); root.getchildren().addall(treebox); primarystage.setscene(new scene(root,400,300)); primarystage.show(); here's what the final product looks like, much cleaner than the awful swing version and less than half the code:

Marker Interfaces in Java have special significance because of the fact that they have no methods declared in them which means that the classes implementing these interfaces don't have to override any of the methods. A few of the marker interfaces already exist in the JDK like Serializable and Cloneable. One can also create their own custom interfaces which doesn't have any method. The purpose of these interfaces is to force some kind of functionality in the classes by providing some functionality to a class if it implements the marker interface. A common question asked very frequently is about Runnable interface being marker or not. Runnable interface is not marker because Runnable interface has the public void run() method declared inside it. A very good example of marker interface is Serializable where the class implements can be used with ObjectOutputStream and ObjectInputStream classes. The Java language specification doesn't itself define the term marker interface and the term has been coined by authors, developers and designers. One common question asked is if we can create a marker interface or not and the answer is yes because of following reason: We can't create marker interface similar to Serializable or Cloneable but we can simulate the functionality by writing extra code around the custom marker interface.

One of my favorite subjects is real-time communication. A chat component is one of the most basic forms of real-time communication. In the past, I've blogged about how to create a JavaFX 1.3 based chat application using a Comet Server. Meanwhile, we updated the internal Chat Application we are using at LodgON and it is now using JavaFX 2 and RedFX 2. The most basic JavaFX Chat Application is described at http://redfx.org/samples/chat/index.html and the required binaries can also be downloaded from that site. The example shown on the RedFX samples page is very very basic and of course not very useful in real-world cases. However, the basics about sending and receiving messages, processing them and visualizing them, are very similar. We use the exact same principles for a more complex Chat Application for focus groups that we are currently migrating from JavaFX 1.3 to JavaFX 2.1. A Chat application requires a client component and a server component. If you look at the client code that can be downloaded from the RedFX.org download section, you'll probably agree that JavaFX is the perfect candidate platform for writing chat applications. Very few lines of code are needed, and any Java developer can easily extend this application. The server component, which can be downloaded here contains a Java EE 6 Web Archive. It includes the RedFX server components, a few configuration files, and it runs out of the box on Glassfish 3.1.2 (if you enable Comet and/or Web sockets -- those are disabled by default unfortunately). A couple of weeks ago, we made the beta-release of RedFX available in a binary form. It is our intention to make available all source code that is required to run the basic samples. However, this takes time. A couple of months ago, we open-sourced the DaliCore platform, and that takes time. Making a project open-source involves much much more than putting the code in a zip and make it available online. It is my goal to make the RedFX client part of the open-source JFXtras.org project. We still have to figure out how we will deal with dependencies, and how/where we can host the RedFX server components. We will do this as fast as we can, but there are only 25 hours in a day...

My prefered solution was to use the Maven ‘update snapshots’ command line argument.

Use short enough and long enough variable names in each scope of code. Generally length may be 1 char for loop counters, 1 word for condition/loop variables, 1-2 words for methods, 2-3 words for classes, 3-4 words for globals. Use specific names for variables, for example "value", "equals", "data", ... are not valid names for any case. Use meaningful names for variables. Variable name must define the exact explanation of its content. Don't start variables with o_, obj_, m_ etc. A variable does not need tags which states it is a variable. Obey company naming standards and write variable names consistently in application: e.g. txtUserName, lblUserName, cmbSchoolType, ... Otherwise readability will reduce and find/replace tools will be unusable. Obey programming language standards and don't use lowercase/uppercase characters inconsistently: e.g. userName, UserName, USER_NAME, m_userName, username, ... use Camel Case (aka Upper Camel Case) for classes: VelocityResponseWriter use Lower Case for packages: com.company.project.ui use Mixed Case (aka Lower Camel Case) for variables: studentName use Upper Case for constants : MAX_PARAMETER_COUNT = 100 use Camel Case for enum class names and Upper Case for enum values. don't use '_' anywhere except constants and enum values (which are constants). For example for Java, Don't reuse same variable name in the same class in different contexts: e.g. in method, constructor, class. So you can provide more simplicity for understandability and maintainability. Don't use same variable for different purposes in a method, conditional etc. Create a new and different named variable instead. This is also important for maintainability and readability. Don't use non-ASCII chars in variable names. Those may run on your platform but may not on others. Don't use too long variable names (e.g. 50 chars). Long names will bring ugly and hard-to-read code, also may not run on some compilers because of character limit. Decide and use one natural language for naming, e.g. using mixed English and German names will be inconsistent and unreadable. Use meaningful names for methods. The name must specify the exact action of the method and for most cases must start with a verb. (e.g. createPasswordHash) Obey company naming standards and write method names consistently in application: e.g. getTxtUserName(), getLblUserName(), isStudentApproved(), ... Otherwise readability will reduce and find/replace tools will be unusable. Obey programming language standards and don't use lowercase/uppercase characters inconsistently: e.g. getUserName, GetUserName, getusername, ... For example for Java, use Mixed Case for method names: getStudentSchoolType use Mixed Case for method parameters: setSchoolName(String schoolName) Use meaningful names for method parameters, so it can documentate itself in case of no documentation.

In a recent project, I had a requirement of connecting to multiple databases using hibernate. As tapestry-hibernate module does not provide an out-of-box support, I thought of adding one. https://github.com/tawus/tapestry5 Now that the application is in production, I thought of writing a simple “How to”. I have cloned the latest stable(5.3.2) tapestry project at https://github.com/tawus/tapestry5 and have added multiple database support to it. Single Database It is almost fully compatible with the previous integration when using a single database except for a few things 1) HibernateConfigurer has changed public interface HibernateConfigurer { /** * Passed the configuration so as to make changes. */ void configure(Configuration configuration); /** * Factory Id for which this configurer is meant for */ Class getMarker(); /** * Entity package names * * @return */ String[] getPackageNames(); } 2) There is no HibernateEntityPackageManager, as the packages can be contributed by adding more HibernateConfigurers with the same Markers. Multiple databases For multiple database, a marker has to be used for accessing Session or HibernateSessionManager @Inject @XDB private Session session; @Inject @YDB private HibernateSessionManager sessionManager; @XDB @CommitAfter void myMethod(){ } Also you have to define a HibernateSessionManager and a Session for the secondary database in the Module class. @Scope(ScopeConstants.PERTHREAD) @Marker(DatabaseTwo.class) public static HibernateSessionManager buildHibernateSessionManagerForFinacle( HibernateSessionSource sessionSource, PerthreadManager perthreadManager) { HibernateSessionManagerImpl service = new HibernateSessionManagerImpl(sessionSource, DatabaseTwo.class); perthreadManager.addThreadCleanupListener(service); return service; } @Marker(DatabaseTwo.class) public static Session buildSessionForFinacle( @Local HibernateSessionManager sessionManager, PropertyShadowBuilder propertyShadowBuilder) { return propertyShadowBuilder.build(sessionManager, "session", Session.class); } Notice an annotation @DatabaseTwo.class. This is a Factory marker and is used to identify a service related to a particular SessionFactory. @Retention(RetentionPolicy.RUNTIME) @Target( {ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD}) @FactoryMarker @Documented public @interface DatabaseTwo { } A typical AppModule for two databases will be public class AppModule { public static void bind(ServiceBinder binder) { binder.bind(DemoService.class, DemoServiceImpl.class); } @Contribute(HibernateSessionSource.class) public static void configureHibernateSources(OrderedConfiguration configurers) { configurers.add("databaseOne", new HibernateConfigurer() { public void configure(org.hibernate.cfg.Configuration configuration) { configuration.configure("/databaseOne.xml"); } public Class getMarker() { return DefaultFactory.class; } public String[] getPackageNames() { return new String[] {"org.example.demo.one"}; } }); configurers.add("databaseTwo", new HibernateConfigurer() { public void configure(org.hibernate.cfg.Configuration configuration) { configuration.configure("/databaseTwo.xml"); } public Class getMarker() { return DatabaseTwo.class; } public String[] getPackageNames() { return new String[] {"org.example.demo.two"}; } }); } @Contribute(SymbolProvider.class) @ApplicationDefaults public static void addSymbols(MappedConfiguration configuration) { configuration.add(HibernateSymbols.DEFAULT_CONFIGURATION, "false"); configuration.add("tapestry.app-package", "org.example.demo"); } @Scope(ScopeConstants.PERTHREAD) @Marker(DatabaseTwo.class) public static HibernateSessionManager buildHibernateSessionManagerForFinacle( HibernateSessionSource sessionSource, PerthreadManager perthreadManager) { HibernateSessionManagerImpl service = new HibernateSessionManagerImpl(sessionSource, DatabaseTwo.class); perthreadManager.addThreadCleanupListener(service); return service; } @Marker(DatabaseTwo.class) public static Session buildSessionForFinacle( @Local HibernateSessionManager sessionManager, PropertyShadowBuilder propertyShadowBuilder) { return propertyShadowBuilder.build(sessionManager, "session", Session.class); } } Injecting into Services You can inject a session in a service using the marker. As DatabaseOne is being used as the default configuration, in order to inject its Session, you have to annotate it with @DefaultFactory. For DatabaseTwo, you can use @DatabaseTwo annotation. public class DemoServiceImpl implements DemoService { private Session sessionOne; private Session sessionTwo; public DemoServiceImpl( @DefaultFactory Session sessionOne, @DatabaseTwo Session sessionTwo) { this.sessionOne = sessionOne; this.sessionTwo = sessionTwo; } @SuppressWarnings("unchecked") public List listOnes() { return sessionOne.createCriteria(EntityOne.class).list(); } @SuppressWarnings("unchecked") public List listTwos() { return sessionTwo.createCriteria(EntityTwo.class).list(); } public void save(EntityOne entityOne) { sessionOne.saveOrUpdate(entityOne); } public void save(EntityTwo entityTwo) { sessionTwo.saveOrUpdate(entityTwo); } } Using @CommitAfter You can add an advice the same way you used to. The only change is in @CommitAfter. You have to additionally annotate the method with the respective marker. public interface DemoService { List listOnes(); List listTwos(); @CommitAfter @DefaultFactory void save(EntityOne entityOne); @CommitAfter @DatabaseTwo void save(EntityTwo entityTwo); } Here is an example. From http://tawus.wordpress.com/2012/03/03/tapestry-hibernate-multiple-databases/

The JavaFX 2 class javafx.scene.paint.Color includes several fields that are static Color members. I have taken advantage of the convenience of these publicly available static fields in many of my JavaFX 2 examples shown in this blog. There is a lengthy list of these predefined Color fields ranging (in alphabetical order) from Color.ALICEBLUE to Color.YELLOWGREEN. I have sometimes thought it would be nice to quickly see what some of the less obvious colors look like and the simple JavaFX 2 application featured in this post provides a sampling of those colors. The sample JavaFX 2 application shown here uses simple Java reflection to introspect the JavaFX Color class for its public fields that themselves of type Color. The application then iterates over those public fields, providing information about each color such as the color's field name, a sample of the color, and the red/green/blue components of that color. The final row allows the user to specify red/green/blue values to see how such a color is rendered. This is useful if the user sees a standard color that is close to what he or she wants and the user wants to try adjusting it slightly. To ensure meaningful values for displaying a color based on provided red/green/blue values, the application ensures that entered values are treated as doubles between 0.0 and 1.0 even if they are not numbers or are numbers outside of that range. The simple JavaFX 2 application showing standard Color fields is shown in the next code listing. JavaFxColorDemo.java package dustin.examples; import static java.lang.System.err; import java.lang.reflect.Field; import javafx.application.Application; import javafx.event.EventHandler; import javafx.scene.Group; import javafx.scene.Scene; import javafx.scene.control.*; import javafx.scene.input.MouseEvent; import javafx.scene.layout.HBox; import javafx.scene.layout.Pane; import javafx.scene.layout.VBox; import javafx.scene.paint.Color; import javafx.scene.shape.Rectangle; import javafx.scene.shape.RectangleBuilder; import javafx.stage.Stage; /** * Simple JavaFX 2 application that prints out values of standardly available * Color fields. * * @author Dustin */ public class JavaFxColorDemo extends Application { /** Width of label for colorn name. */ private final static int COLOR_NAME_WIDTH = 150; /** Width of rectangle that displays color. */ private final static int COLOR_RECT_WIDTH = 50; /** Height of rectangle that displays color. */ private final static int COLOR_RECT_HEIGHT = 25; private final TextField redField = TextFieldBuilder.create() .text("Red Value").build(); private final TextField greenField = TextFieldBuilder.create() .text("Green Value").build(); private final TextField blueField = TextFieldBuilder.create() .text("Blue Value").build(); private final Rectangle customColorRectangle = RectangleBuilder.create() .width(COLOR_RECT_WIDTH).height(COLOR_RECT_HEIGHT) .fill(Color.WHITE).stroke(Color.BLACK).build(); /** * Build a pane containing details about the instance of Color provided. * * @param color Instance of Color about which generated Pane should describe. * @return Pane representing information on provided Color instance. */ private Pane buildColorBox(final Color color, final String colorName) { final HBox colorBox = new HBox(); final Label colorNameLabel = new Label(colorName); colorNameLabel.setMinWidth(COLOR_NAME_WIDTH); colorBox.getChildren().add(colorNameLabel); final Rectangle colorRectangle = new Rectangle(COLOR_RECT_WIDTH, COLOR_RECT_HEIGHT); colorRectangle.setFill(color); colorRectangle.setStroke(Color.BLACK); colorBox.getChildren().add(colorRectangle); final String rgbString = String.valueOf(color.getRed()) + " / " + String.valueOf(color.getGreen()) + " / " + String.valueOf(color.getBlue()) + " // " + String.valueOf(color.getOpacity()); final Label rgbLabel = new Label(rgbString); rgbLabel.setTooltip(new Tooltip("Red / Green / Blue // Opacity")); colorBox.getChildren().add(rgbLabel); return colorBox; } /** * Extracts a double between 0.0 and 1.0 inclusive from the provided String. * * @param colorString String from which a double is extracted. * @return Double between 0.0 and 1.0 inclusive based on provided String; * will be 0.0 if provided String cannot be parsed. */ private double extractValidColor(final String colorString) { double colorValue = 0.0; try { colorValue = Double.valueOf(colorString); } catch (Exception exception) { colorValue = 0.0; err.println("Treating '" + colorString + "' as " + colorValue); } finally { if (colorValue < 0) { colorValue = 0.0; err.println("Treating '" + colorString + "' as " + colorValue); } else if (colorValue > 1) { colorValue = 1.0; err.println("Treating '" + colorString + "' as " + colorValue); } } return colorValue; } /** * Build pane with ability to specify own RGB values and see color. * * @return Pane with ability to specify colors. */ private Pane buildCustomColorPane() { final HBox customBox = new HBox(); final Button button = new Button("Display Color"); button.setPrefWidth(COLOR_NAME_WIDTH); button.setOnMouseClicked(new EventHandler() { @Override public void handle(MouseEvent t) { final Color customColor = new Color(extractValidColor(redField.getText()), extractValidColor(greenField.getText()), extractValidColor(blueField.getText()), 1.0); customColorRectangle.setFill(customColor); } }); customBox.getChildren().add(button); customBox.getChildren().add(this.customColorRectangle); customBox.getChildren().add(this.redField); customBox.getChildren().add(this.greenField); customBox.getChildren().add(this.blueField); return customBox; } /** * Build the main pane indicating JavaFX 2's pre-defined Color instances. * * @return Pane containing JavaFX 2's pre-defined Color instances. */ private Pane buildColorsPane() { final VBox colorsPane = new VBox(); final Field[] fields = Color.class.getFields(); // only want public for (final Field field : fields) { if (field.getType() == Color.class) { try { final Color color = (Color) field.get(null); final String colorName = field.getName(); colorsPane.getChildren().add(buildColorBox(color, colorName)); } catch (IllegalAccessException illegalAccessEx) { err.println( "Securty Manager does not allow access of field '" + field.getName() + "'."); } } } colorsPane.getChildren().add(buildCustomColorPane()); return colorsPane; } /** * Start method overridden from parent Application class. * * @param stage Primary stage. * @throws Exception JavaFX application exception. */ @Override public void start(final Stage stage) throws Exception { final Group rootGroup = new Group(); final Scene scene = new Scene(rootGroup, 700, 725, Color.WHITE); final ScrollPane scrollPane = new ScrollPane(); scrollPane.setPrefWidth(scene.getWidth()); scrollPane.setPrefHeight(scene.getHeight()); scrollPane.setContent(buildColorsPane()); rootGroup.getChildren().add(scrollPane); stage.setScene(scene); stage.setTitle("JavaFX Standard Colors Demonstration"); stage.show(); } /** * Main function for running JavaFX application. * * @param arguments Command-line arguments; none expected. */ public static void main(final String[] arguments) { Application.launch(arguments); } } The snapshots shown next demonstrate this simple application. The first snapshot shows the application after loading. The second snapshot shows the application after scrolling down to the bottom and demonstrates use of the Tooltip and of the TextField.setPrompt(String) method. The third image shows the results of providing red/green/blue values and clicking on the button to see the corresponding color. The simple JavaFX 2 application shown in this post makes it easy to get an idea of what colors are available as standard JavaFX 2 public static Color fields. It also allows one to enter red/green/blue values that might be provided to the Color constructor to obtain an instance of Color. From http://marxsoftware.blogspot.com/2012/02/viewing-javafx-2-standard-colors.html