Dropwizard is a bunch of superb frameworks glued together to provide a fast way of building web applications including REST APIs. We'll talk about great frameworks which are parts of Dropwizard as we go over our examples. The full code for the examples can be obtained here . The simplest way to create a Dropwizard project is to use the Maven archetype called java-simple which is a part of Dropwizard. This can be accomplished either via the command line or using your favorite IDE. Let's stick to the former approach. The command below is formatted for clarity but it should be pasted as a single line to the terminal. mvn archetype:generate -DgroupId=com.javaeeeee -DartifactId=DWGettingStarted -Dname=DWGettingStarted -Dpackage=com.javaeeeee.dwstart -DarchetypeGroupId=io.dropwizard.archetypes -DarchetypeArtifactId=java-simple -DinteractiveMode=false After the project was created it can be modified using an editor or opened in an IDE. All the three IntelliJ, Eclipse and Netbeans have Maven support. An important parameter to note in the pom.xml file of the newly created project is dropwizard.version, which at the time of writing was automatically set to 0.8.0-rc2-SNAPSHOT and the latest stable version was 0.7.1. We'll discuss later the important consequences of the version for project development. The directory structure of the project is shown below. There are multiple sub-packages and two classes that were created for us in a package com.javaeeeee.dwstart. Now let's create a simple resource class that produces a string “Hello world!”, start the application and test the results using both browser and cURL. The snippet below shows the resource class. @Path("/hello") public class HelloResource { @GET @Produces(MediaType.TEXT_PLAIN) public String getGreeting() { return "Hello world!"; } } There is a special package, com.javaeeeee.dwstart.recources for placing resource classes or an appropriate folder can be found from the screenshot above if you use a text editor and not an IDE. One of the main tenets of the REST architecture style is that each resource is assigned a URL. In our case the URL of our resource would be http://localhost:8080/hello, that is we access it from our local machine, the default port which is used by Dropwizard is 8080 and the final part of the URL is enshrined in @Path annotation in our class, in other words, the aforementioned annotation helps to describe the URL used to access a resource. Moving on to method getGreeting() it can be seen that it is a simple method which returns the desired string “Hello world!”, although it is marked with two annotations: first, @GET prescribes that this method is called when the aforementioned URL is accessed using HTTP GET method, second, annotation @Produces specifies what media types can be produced when the method is accessed by the client, such as browser, cURL or some other program. In our case it is plain text. These annotations are part of Java API for RESTful Web Services (JAX-RS), and its reference implementation Jersey is the cornerstone of Dropwizard. Other annotations include @POST for HTTP POST method, @DELETE for DELETE and so on and media types include MediaType.APPLICATION_JSON and not so popular MediaType.APPLICATION_XML among others. Now, to see the result of our coding, we should do some configuration. Let's open one of the created by Maven files, DWGettingStartedApplication.java, and register our resource. public void run(final DWGettingStartedConfiguration configuration, final Environment environment) { environment.jersey().register(new HelloResource()); } We added a single line environment.jersey().register(new HelloResource()); to help Dropwizard find our resource class. Now we have to create a jar-file which contains an embedded Jersey web server to serve the incoming requests, as well as all the necessary libraries. Dropwizard applications are packaged as executable jar-files rather than war-files which are deployed to an application server. The kind of jar files used by Dropwizard applications are also called “fat” because they include all the .class files necessary to run the application as this leads to a situation that the versions of all libraries are the same in both development and production environments which leads to less problems. One shouldn't worry about creating such jar-files as Maven creates them for us using instructions in the generated pom.xml file of the project. The jar-file can be created using command-line command issued from the project's folder mvn package or using an IDE. After that we should start the application, which could be accomplished either from the command line by issuing a command java -jar target/DWGettingStarted-1.0-SNAPSHOT.jar server or from the IDE which can be instructed to pass a “server” argument to the executable jar file. To stop the application it is sufficient to press Ctrl+C in the terminal. Now it is time to check if it works. The simplest way to achieve this is to navigate your browser to the URL mentioned earlier. You should see the greeting string in the main window of a browser. If you cannot see the greeting, there is probably some problems and the program’s output is the place to look for clues for what went wrong. The same greeting can be seen in the terminal window with help of cURL. curl -w "\n" localhost:8080/hello Protocol HTTP and GET method are defaults, so it is not necessary that they be included explicitly in the command. As the result a message "Hello world!" without quotes should be printed. The w-option is used to add a trailing newline to prettify the output. It is a good idea to start using tests as early as possible, so let's write a test that checks the work of our resource class using in-memory Jersey. We'll create the test in test packages and as we are testing a resource an appropriate package to place our test is com.javaeeeee.dwstart.resources (or src/test/com/javaeeeee/dwstart/resources folder). It is necessary that the following dependency be added to the pom-file of the project. (It works without explicit jUnit dependency.) io.dropwizard dropwizard-testing ${dropwizard.version} test An example test for our greeting resource is shown below. public class HelloResourceTest { @Rule public ResourceTestRule resource = ResourceTestRule.builder() .addResource(new HelloResource()).build(); @Test public void testGetGreeting() { String expected = "Hello world!"; //Obtain client from @Rule. Client client = resource.client(); //Get WebTarget from client using URI of root resource. WebTarget helloTarget = client.target("http://localhost:8080/hello"); //To invoke response we use Invocation.Builder //and specify the media type of representation asked from resource. Invocation.Builder builder = helloTarget.request(MediaType.TEXT_PLAIN); //Obtain response. Response response = builder.get(); //Do assertions. assertEquals(Response.Status.OK, response.getStatusInfo()); String actual = response.readEntity(String.class); assertEquals(expected, actual); } } The annotation @Rule is from jUnit realm and could be placed on public non-static fields and methods that return a value. This annotation may perform some initial setup and clean-up like @Before and @After methods, but it is more powerful as it allows to share functionality between classes and even projects. This rule is needed to allow us to access our resources from the code of the test using in-memory Jersey. At this point of time we should pause and discuss the version of our product. There is a pom.xml file in the dropwizard folder on the GitHub, which can readily be accessed by cloning the project, and this file specifies the version of constituent products among other things. It can be seen from it that the version of Jersey in the snapshot version of Dropwizard is 2.13 and in the version 0.7.1 it is 1.18.1. The version of Jersey influences the code of the tests as 1.x and 2.x versions of Jersey have different client APIs, that is the code to access the service programmatically from Java depends on the version of Jersey framework. Let's stick to the latest 2.x version of Jersey as it is liable to be included in following 0.8.x releases of Dropwizard and if you are interested how to test using the older version of Jersey client, please check the project repository , there is a special branch v0.7.x. Another observation concerning testing is that Fest matchers were superseded by AssertJ counterparts. First, we obtain an instance of Jersey client to make requests to our service. Second, we create an instance of a class that implements WebTarget interface for the resource using its URL. After that we use Invocation.Builder that helps to build request and send it to the server. The Invocation.Builder interface extends SynchInvoker interface which defines a bunch of get(...) methods used to invoke HTTP GET method synchronously. Finally, we check the results, namely that the HTTP status code was 200 and the desired string was returned. The same result could be obtained by chaining the methods as in the snippet below. actual = resource.client() .target("http://localhost:8080/hello") .request(MediaType.TEXT_PLAIN) .get(String.class); assertEquals(expected, actual); The replacement of the two assertions by a single is possible due to the fact that the get(...) method we used throws a WebApplicationException if the HTTP status code is not in the 2xx range, in other words if there is a problem. Now let's improve our resource to take a name and return a bespoke greeting. It could be done in a couple of ways using Jersey's annotations. The first is to use so-called path parameters, that is you pass the name using a URL like http://localhost:8080/hello/path_param/your_name. The application should return “Hello your_name”. Let's see a code snippet. @Path("/path_param/{name}") @GET @Produces(MediaType.TEXT_PLAIN) public String getNamedGreeting(@PathParam(value = "name") String name) { return "Hello " + name; } Now we see a @Path annotation on our method. It adds its content to those produced by class level annotation as seen from the URL. Methods marked with this annotation are called sub-resource methods. The name of the parameter, name, is in curly braces. Something interesting happens inside the parenthesis of the getNamedGreeting method. There is an annotation which prescribes to extract the content of the url after “/path_param/” to the method as a value. It should be noted that if you find yourself marking each method with the same annotation @Produces(MediaType.TEXT_PLAIN), the latter can be removed from methods and used to mark the class, then all the methods produce the desired media types, although this can be overruled by an annotations with a different media type for some of the methods. The second way to pass a name is to use query parameters and the URL could look like http://localhost:8080/hello/query_param?name=your_name. The output should be “Hello your_name” without quotes. The snippet is shown below. @Path("/query_param") @GET @Produces(MediaType.TEXT_PLAIN) public String getNamedStringWithParam(@DefaultValue("world") @QueryParam("name") String name) { return "Hello " + name; } Once again, the @Path annotation adds something to our URL, then the default value of the parameter is set in case there is no question mark and following it symbols in the URL using @DefaultValue annotation. In other words, if the parameter is omitted in the URL and it looks like http://localhost:8080/hello/query_param, a phrase “Hello world” is printed. The last annotation injects a parameter from the URL to the method's parameter. By the way, all these cases can be tested as was shown above. To test a method using query parameters one can use a queryParam(...) method of WebTarget interface. There is a couple of points to pay attention to. Firstly, there are other @*Param annotations available and secondly, the @DefaultValue annotation will not work with path parameters and to process the absence of the parameter a special sub-resource method should be introduced. You were patient enough to bear all this plain text staff, but it is not what REST APIs are about. Let's create another sub-resource method which is preprogrammed to return JSON representation. The first step is to create a representation class, which should be placed to com.javaeeeee.dwstart.core package. Let's limit ourselves to extremely simple example which is shown below. public class Greeting { @JsonProperty private String greeting; public Greeting() { } public Greeting(String greeting) { this.greeting = greeting; } public String getGreeting() { return greeting; } } The class above relies on one more important part of the Dropwizard stack namely Jackson , which can turn Java objects to JSON and vice versa. A @JsonProperty annotation is used to do the job. The sub-resource method looks like pretty much the same as our previously discussed methods. @Path("/hello_json") @GET @Produces(MediaType.APPLICATION_JSON) public Greeting getJSONGreeting() { return new Greeting("Hello world!"); } The difference is that we changed the media type and the method returns an instance of the greeting class. If you navigate your browser to to a URL http://localhost:8080/hello/hello_json or use cURL, you should see a JSON representation of the object {"greeting":"Hello world!"}. What if we remove the @Path annotation from the aforementioned method altogether? Well, if you try to access the localhost:8080/hello resource you'll see the text representation. How one could access the JSON-producing method? There is a special HTTP header called Accept which is used to instruct the server what representation to return, the process called content negotiation. The Jersey inside the Dropwizard will choose for us what method to use based on the content of the Accept header. Let's try cURL to do this. curl -w "\n" -H 'Accept: application/json' localhost:8080/hello/ The H-option is used to send headers and we instructed cURL to ask the JSON response. Other possible types to try could include text/plain and application/xml. The former should return the plain-text representation and the letter should result in error as there is no method in our resource class that can produce the latter media type response. Another way to engage in content negotiation is to use a REST client for your browser. There are several clients for each of major browsers but we will use Postman extension for Chrome browser. There are special fields to input headers as the screenshot below shows. One can enter the resource URL, headers and press Send button and the response will show up in the lower part of the window. To sum up we created a simple REST-like API using Dropwizard, which can greet its users. We learned how to create a Dropwizard project using Maven archetype and created a simple resource class to accomplish the task of greeting and a representation class to produce JSON response. An important ingredient of REST architecture style, HATEOAS, was omitted for now for simplicity reasons. Also a problem of testing resource classes was touched as well as the ways to access resources both from browser and command line. There are some other ideas for creating sub-resources such as returning the current date and adding two numbers using query parameters which can be easily implemented using the information presented in this article. References Dropwizard on GitHub Maven Archetypes Dropwizard Getting Started JAX-RS Resources and Subresources Dropwizard Testing Jersey Client API 2.x Jersey Client API 1.x Jackson annotations

originally written by david ducos mydumper is known as the faster (much faster) mysqldump alternative. so, if you take a logical backup you will choose mydumper instead of mysqldump. but what about the restore? well, who needs to restore a logical backup? it takes ages! even with myloader. but this could change just a bit if we are able to take advantage of fast index creation. as you probably know, mydumper and mysqldump export the struct of a table, with all the indexes and the constraints, and of course, the data. then, myloader and mysql import the struct of the table and import the data. the most important difference is that you can configure myloader to import the data using a certain amount of threads. the import steps are: create the complete struct of the table import the data when you execute myloader, internally it first creates the tables executing the “-schema.sql” files and then takes all the filenames without “schema.sql” and puts them in a task queue. every thread takes a filename from the queue, which actually is a chunk of the table, and executes it. when finished it takes another chunk from the queue, but if the queue is empty it just ends. this import procedure works fast for small tables, but with big tables with large indexes the inserts are getting slower caused by the overhead of insert the new values in secondary indexes. another way to import the data is: split the table structure into table creation with primary key, indexes creation and constraint creation create tables with primary key per table do: load the data create index create constraints this import procedure is implemented in a branch of myloader that can be downloaded from here or directly executing bzr with the repository: bzr branch lp:~david-ducos/mydumper/mydumper the tool reads the schema files and splits them into three separate statements which create the tables with the primary key, the indexes and the constraints. the primary key is kept in the table creation in order to avoid the recreation of the table when a primary key is added and the “key” and “constraint” lines are removed. these lines are added to the index and constraint statements, respectively. it processes tables according to their size starting with the largest because creating the indexes of a big table could take hours and is single-threaded. while we cannot process other indexes at the time, we are potentially able to create other tables with the remaining threads. it has a new thread (monitor_process) that decides which chunk of data will be put in the task queue and a communication queue which is used by the task processes to tell the monitor_process which chunk has been completed. i run multiple imports on an aws m1.xlarge machine with one table comparing myloader and this branch and i found that with large indexes the times were: as you can see, when you have less than 150m rows, import the data and then create the indexes is higher than import the table with the indexes all at once. but everything changes after 150m rows, import 200m takes 64 minutes more for myloader but just 24 minutes for the new branch. on a table of 200m rows with a integer primary key and 9 integer columns, you will see how the time increases as the index gets larger: where: 2-2-0: two 1-column and two 2-column index 2-2-1: two 1-column, two 2-column and one 3-column index 2-3-1: two 1-column, three 2-column and one 3-column index 2-3-2: two 1-column, three 2-column and two 3-column index conclusion this branch can only import all the tables with this same strategy, but with this new logic in myloader, in a future version it could be able to import each table with the best strategy reducing the time of the restore considerably.

Recently I was asked about diagnosing deadlocks in SQL Server – I’ve done a lot of work in this area way back in 2008, so I figure it’s time for a refresher. If there’s a lot of interest in exploring SQL Server and deadlocks further, I’m happy to write an extended article going into far more detail. Just let me know. Before we get into diagnosis and investigation, it’s a good time to pose the question: “what is a deadlock?”: From TechNet: A deadlock occurs when two or more tasks permanently block each other by each task having a lock on a resource which the other tasks are trying to lock. The following graph presents a high level view of a deadlock state where: Task T1 has a lock on resource R1 (indicated by the arrow from R1 to T1) and has requested a lock on resource R2 (indicated by the arrow from T1 to R2). Task T2 has a lock on resource R2 (indicated by the arrow from R2 to T2) and has requested a lock on resource R1 (indicated by the arrow from T2 to R1). Because neither task can continue until a resource is available and neither resource can be released until a task continues, a deadlock state exists. The SQL Server Database Engine automatically detects deadlock cycles within SQL Server. The Database Engine chooses one of the sessions as a deadlock victim and the current transaction is terminated with an error to break the deadlock. Basically, it’s a resource contention issue which blocks one process or transaction from performing actions on resources within SQL Server. This can be a serious condition, not just for SQL Server as processes become suspended, but for the applications which rely on SQL Server as well. The T-SQL Approach A fast way to respond is to execute a bit of T-SQL on SQL Server, making use of System Views. The following T-SQL will show you the “victim” processes, much like activity monitor does: select * from sys.sysprocesses where blocked > 0 Which is not particularly useful (but good to know, so you can see the blocked count). To get to the heart of the deadlock, this is what you want (courtesy of this SO question/answer): SELECT Blocker.text –, Blocker.*, * FROM sys.dm_exec_connections AS Conns INNER JOIN sys.dm_exec_requests AS BlockedReqs ON Conns.session_id = BlockedReqs.blocking_session_id INNER JOIN sys.dm_os_waiting_tasks AS w ON BlockedReqs.session_id = w.session_id CROSS APPLY sys.dm_exec_sql_text(Conns.most_recent_sql_handle) AS Blocker This will show you line and verse (the actual statement causing the resource block) – see the attached screenshot for an example. However, the generally accepted way to determine and diagnose deadlocks is through the use of SQL Server trace flags. SQL Trace Flags They are (usually) set temporarily, and they cause deadlocking information to be dumped to the SQL management logs. The flags that are useful are flags 1204 and 1222. From TechNet: https://technet.microsoft.com/en-us/library/ms178104%28v=sql.105%29.aspx Trace flags are set on or off by using either of the following methods: · Using the DBCC TRACEON and DBCC TRACEOFF commands. For example, DBCC TRACEON 2528: To enable the trace flag globally, use DBCC TRACEON with the -1 argument: DBCC TRACEON (2528, -1). To turn off a global trace flag, use DBCC TRACEOFF with the -1 argument. · Using the -T startup option to specify that the trace flag be set on during startup. The -T startup option enables a trace flag globally. You cannot enable a session-level trace flag by using a startup option. So to enable or disable deadlock trace flags globally, you’d use the following T-SQL: DBCC TRACEON (1204, -1) DBCC TRACEON (1222, -1) DBCC TRACEOFF (1204, -1) DBCC TRACEOFF (1222, -1) Due to the overhead, it’s best to enable the flag at runtime rather than on start up. Note that the scope of a non-startup trace flag can be global or session-level. Basic Deadlock Simulation By way of a very simple scenario, you can make use of SQL Management Studio (and breakpoints) to roughly simulate a deadlock scenario. Given the following basic table schema: CREATE TABLE [dbo].[UploadedFile]( [Id] [int] NOT NULL, [Filename] [nvarchar](50) NOT NULL, [DateCreated] [datetime] NOT NULL, [DateModified] [datetime] NULL, CONSTRAINT [PK_UploadedFile] PRIMARY KEY CLUSTERED ( [Id] ASC )WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ) With some basic test data in it: If you create two separate queries in SQL Management Studio, use the following transaction (Query #1) to lock rows in the table: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE BEGIN TRANSACTION SELECT [Id],[Filename],[DateCreated],[DateModified] FROM [dbo].[UploadedFile] WHERE DateCreated > ‘2015-01-01′ ROLLBACK TRANSACTION Now add a “victim” script (Query #2) in a separate query session: UPDATE [dbo].[UploadedFile] SET [DateModified] = ‘2014-12-31′ WHERE DateCreated > ‘2015-01-01′ As long as you set a breakpoint on the ROLLBACK TRANSACTION statement, you’ll block the second query due to the isolation level of the transaction which wraps query #1. Now you can use the diagnostic T-SQL to examine the victim and the blocking transaction. Enjoy!

Recently I had a dispute with my colleagues regarding performance penalty of lazy vals in Scala. It resulted in a set of microbenchmarks which compare lazy and non-lazy vals performance. All the sources can be found at http://git.io/g3WMzA. But before going to the benchmark results let's try to understand what can cause the performance penalty. For my JMH benchmark I created a very simple Scala class with lazy val in it: @State(Scope.Benchmark) class LazyValCounterProvider { lazy val counter = SlowInitializer.createCounter() } Now let's take a look at what is hidden under the hood of lazy keyword. At first, we need to compile given code with scalac, and then it can be decompiled to correspondent Java code. For this sake I used JD decompiler. It produced the following code: @State(Scope.Benchmark) @ScalaSignature(bytes="...") public class LazyValCounterProvider { private SlowInitializer.Counter counter; private volatile boolean bitmap$0; private SlowInitializer.Counter counter$lzycompute() { synchronized (this) { if (!this.bitmap$0) { this.counter = SlowInitializer.createCounter(); this.bitmap$0 = true; } return this.counter; } } public SlowInitializer.Counter counter() { return this.bitmap$0 ? this.counter : counter$lzycompute(); } } As it's seen, the lazy keyword is translated to a classical double-checked locking idiom for delayed initialization. Thus, most of the time the only performance penalty may come from a single volatile read per lazy val read (except for the time it takes to initialize lazy val instance since its very first usage). Let's finally measure its impact in numbers. My JMH-based microbenchmark is as simple as: public class LazyValsBenchmarks { @Benchmark public long baseline(ValCounterProvider eagerProvider) { return eagerProvider.counter().incrementAndGet(); } @Benchmark public long lazyValCounter(LazyValCounterProvider provider) { return provider.counter().incrementAndGet(); } } A baseline method access a final counter object and increments an integer value by 1 by calling incrementAndGet . And as we've just found out, the main benchmark method - lazyValCounter - in addition to what baseline method does also does one volatile read. Note: all measurements are performed on MBA with Core i5 1.7GHz CPU. All results were obtained by running JMH in a throughput mode. Both score and score error columns show operations/second. Each JMH run made 10 iterations and took 50 seconds. I performed 6 measurements with the different JVM and JMH options: client VM, 1 thread Benchmark Score Score error baseline 412277751.619 8116731.382 lazyValCounter 352209296.485 6695318.185 client VM, 2 threads Benchmark Score Score error baseline 542605885.932 15340285.497 lazyValCounter 383013643.710 53639006.105 client VM, 4 threads Benchmark Score Score error baseline 551105008.767 5085834.663 lazyValCounter 394175424.898 3890422.327 server VM, 1 thread Benchmark Score Score error baseline 407010942.139 9004641.910 lazyValCounter 341478430.115 18183144.277 server VM, 2 threads Benchmark Score Score error baseline 531472448.578 22779859.685 lazyValCounter 428898429.124 24720626.198 server VM, 4 threads Benchmark Score Score error baseline 549568334.970 12690164.639 lazyValCounter 374460712.017 17742852.788 The numbers show that lazy vals performance penalty is quite small and can be ignored in practice. For further reading about the subject I would recommend SIP 20 - Improved Lazy Vals Initialization, which contains very interesting in-depth analysis of existing issues with lazy initialization implementation in Scala.

After we introduced locked thread detection to Plumbr couple of months ago, we have started to receive queries similar to “hey, great, now I understand what is causing my performance issues, but what I am supposed to do now?” We are working hard to build the solution instructions into our own product, but in this post I am going to share several common techniques you can apply independent of the tool used for detecting the lock. The methods include lock splitting, concurrent data structures, protecting the data instead of the code and lock scope reduction. Locking is not evil, lock contention is Whenever you face a performance problem with the threaded code there is a chance that you will start blaming locks. After all, common “knowledge” is that locks are slow and limit scalability. So if you are equipped with this “knowledge” and start to optimize the code and getting rid of locks there is a chance that you end up introducing nasty concurrency bugs that will surface later on. So it is important to understand the difference between contended and uncontended locks. Lock contention occurs when a thread is trying to enter the synchronized block/method currently executed by another thread. This second thread is now forced to wait until the first thread has completed executing the synchronized block and releases the monitor. When only one thread at a time is trying to execute the synchronized code, the lock stays uncontended. As a matter of fact, synchronization in JVM is optimized for the uncontended case and for the vast majority of the applications, uncontended locks pose next to no overhead during execution. So, it is not locks you should blame for performance, but contended locks. Equipped with this knowledge, lets see what we can do to reduce either the likelihood of contention or the length of the contention. Protect the data not the code A quick way to achieve thread-safety is to lock access to the whole method. For example, take look at the following example, illustrating a naive attempt to build an online poker server: class GameServer { public Map<> tables = new HashMap>(); public synchronized void join(Player player, Table table) { if (player.getAccountBalance() > table.getLimit()) { List tablePlayers = tables.get(table.getId()); if (tablePlayers.size() < 9) { tablePlayers.add(player); } } } public synchronized void leave(Player player, Table table) {/*body skipped for brevity*/} public synchronized void createTable() {/*body skipped for brevity*/} public synchronized void destroyTable(Table table) {/*body skipped for brevity*/} } The intentions of the author have been good - when new players join() the table, there must be a guarantee that the number of players seated at the table would not exceed the table capacity of nine. But whenever such a solution would actually be responsible for seating players to tables - even on a poker site with moderate traffic, the system would be doomed to constantly trigger contention events by threads waiting for the lock to be released. Locked block contains account balance and table limit checks which potentially can involve expensive operations both increasing the likelihood and length of the contention. First step towards solution would be making sure we are protecting the data, not the code by moving the synchronization from the method declaration to the method body. In the minimalistic example above, it might not change much at the first place. But lets consider the whole GameServerinterface, not just the single join() method: class GameServer { public Map> tables = new HashMap>(); public void join(Player player, Table table) { synchronized (tables) { if (player.getAccountBalance() > table.getLimit()) { List tablePlayers = tables.get(table.getId()); if (tablePlayers.size() < 9) { tablePlayers.add(player); } } } } public void leave(Player player, Table table) {/* body skipped for brevity */} public void createTable() {/* body skipped for brevity */} public void destroyTable(Table table) {/* body skipped for brevity */} } What originally seemed to be a minor change, now affects the behaviour of the whole class. Whenever players were joining tables, the previously synchronized methods locked on theGameServer instance (this) and introduced contention events to players trying to simultaneouslyleave() tables. Moving the lock from the method signature to the method body postpones the locking and reduces the contention likelihood. Reduce the lock scope Now, after making sure it is the data we actually protect, not the code, we should make sure our solution is locking only what is necessary - for example when the code above is rewritten as follows: public class GameServer { public Map> tables = new HashMap>(); public void join(Player player, Table table) { if (player.getAccountBalance() > table.getLimit()) { synchronized (tables) { List tablePlayers = tables.get(table.getId()); if (tablePlayers.size() < 9) { tablePlayers.add(player); } } } } //other methods skipped for brevity } then the potentially time-consuming operation of checking player account balance (which potentially can involve IO operations) is now outside the lock scope. Notice that the lock was introduced only to protect against exceeding the table capacity and the account balance check is not anyhow part of this protective measure. Split your locks When we look at the last code example, you can clearly notice that the whole data structure is protected by the same lock. Considering that we might hold thousands of poker tables in this structure, it still poses a high risk for contention events as we have to protect each table separately from overflowing in capacity. For this there is an easy way to introduce individual locks per table, such as in the following example: public class GameServer { public Map> tables = new HashMap>(); public void join(Player player, Table table) { if (player.getAccountBalance() > table.getLimit()) { List tablePlayers = tables.get(table.getId()); synchronized (tablePlayers) { if (tablePlayers.size() < 9) { tablePlayers.add(player); } } } } //other methods skipped for brevity } Now, if we synchronize the access only to the same table instead of all the tables, we have significantly reduced the likelihood of locks becoming contended. Having for example 100 tables in our data structure, the likelihood of the contention is now 100x smaller than before. Use concurrent data structures Another improvement is to drop the traditional single-threaded data structures and use data structures designed explicitly for concurrent usage. For example, when picking ConcurrentHashMapto store all your poker tables would result in code similar to following: public class GameServer { public Map> tables = new ConcurrentHashMap>(); public synchronized void join(Player player, Table table) {/*Method body skipped for brevity*/} public synchronized void leave(Player player, Table table) {/*Method body skipped for brevity*/} public synchronized void createTable() { Table table = new Table(); tables.put(table.getId(), table); } public synchronized void destroyTable(Table table) { tables.remove(table.getId()); } } The synchronization in join() and leave() methods is still behaving as in our previous example, as we need to protect the integrity of individual tables. So no help from ConcurrentHashMap in this regards. But as we are also creating new tables and destroying tables in createTable() and destroyTable()methods, all these operations to the ConcurrentHashMap are fully concurrent, permitting to increase or reduce the number of tables in parallel. Other tips and tricks Reduce the visibility of the lock. In the example above, the locks are declared public and are thus visible to the world, so there is there is a chance that someone else will ruin your work by also locking on your carefully picked monitors. Check out java.util.concurrent.locks to see whether any of the locking strategies implemented there will improve the solution. Use atomic operations. The simple counter increase we are actually conducting in example above does not actually require a lock. Replacing the Integer in count tracking withAtomicInteger would most suit this example just fine. Hope the article helped you to solve the lock contention issues, independent of whether you are using Plumbr automatic lock detection solution or manually extracting the information from thread dumps.

Part 1 – Configuring Ehcache with Spring Ehcache is most widely used open source cache for boosting performance, offloading your database, and simplifying scalability. It is very easy to configure Ehcache with Spring context xml. Assuming you have a working Spring configured application, here we discuss how to integrate Ehcache with Spring. In order to configure it, we need to add a CacheManager into the Spring context file. Code snippet added below In ‘ehcache’ bean, we are referring to ‘Ehcache.xml’ file which will contain all ehcache configurations. Sample ehcache file is added below. You can configure this file according to your project needs. Here you can two caches, defaultCache and one defined with name ‘TestCache’. DefaultCache configuration is applied to any cache that is not explicitly configured. Part 2 - Enabling Caching Annotations In order to used Spring provided java annotations , we need to declarative enable cache annotations in Spring config file. Code snippet added below. In order to use above annotation declaration, we need to add cache namespace into the spring file. Code snippet added below. Spring Annotations for caching @Cacheable triggers cache population @CacheEvict triggers cache eviction @CachePut updates the cache without interfering with the method execution I will give brief description and practical implementation of these annotations. For detailed explanations please refer spring documentation for caching . @Cacheable - This annotation means that the return value of the corresponding method can be cached and multiple invocation of this method with same arguments must populate the return value from cache without executing the method. Code snippet added below. @Cacheable("customer") public Customer findCustomer(String custId) {...} Each time the method is called, the cache with name ‘customer’ is checked to see whether the invocation has been already executed for the same ‘custId’. @CachePut - This annotation is used to update the cache without interfering the method execution. Method will always be executed and its result will be replaced in the cache. This annoatation is used for cache population. @CachePut("customer") public Customer updateBook(String custId){...} @CacheEvict – This annotation is used for cache eviction, used to remove stale or unused data from the cache. @CacheEvict(value="customer", allEntries=true) public void unloadCustomer(String newBatch) With above configurations, all entries in the ‘customer’ cache will be removed. Above description will give head start to configure Ehcachewith spring and use some of the caching annotations.

Originally Written byAndrew Moore MySQL table alterations can interrupt production traffic causing bad customer experience or in worst cases, loss of revenue. Not all DBAs, developers, syadmins know MySQL well enough to avoid this pitfall. DBAs usually encounter these kinds of production interruptions when working with upgrade scripts that touch both application and database or if an inexperienced admin/dev engineer perform the schema change without knowing how MySQL operates internally. Truths * Direct MySQL ALTER table locks for duration of change (pre-5.6) * Online DDL in MySQL 5.6 is not always online and may incurr locks * Even with Percona Toolkit‘s pt-online-schema-change there are several workloads that can experience blocking Here on the Percona MySQL Managed Services team we encourage our clients to work with us when planning and performing schema migrations. We aim to ensure that we are using the best method available in their given circumstance. Our intentions to avoid blocking when performing DDL on large tables ensures that business can continue as usual whilst we strive to improve response time or add application functionality. The bottom line is that a business relying on access to its data cannot afford to be down during core trading hours. Many of the installations we manage are still below MySQL 5.6, which requires us to seek workarounds to minimize the amount of disruption a migration can cause. This may entail slave promotion or changing the schema with an ‘online schema change’ tool. MySQL version 5.6 looks to address this issue by reducing the number of scenarios where a table is rebuilt and locked but it doesn’t yet cover all eventualities, for example when changing the data type of a column a full table rebuild is necessary. The topic of 5.6 Online Schema Change was discussed in great detail last year in the post, “Schema changes – what’s new in MySQL 5.6?” by Przemysław Malkowski With new functionality arriving in MySQL 5.7, we look forward to non-blocking DDL operations such as; OPTIMIZE TABLE and RENAME INDEX. (More info) The best advice for MySQL 5.6 users is to review the matrix to familiarize with situations where it might be best to look outside of MySQL to perform schema changes, the good news is that we’re on the right path to solving this natively. Truth be told, a blocking alter is usually going to go unnoticed on a 30MB table and we tend to use a direct alter in this situation, but on a 30GB or 300GB table we have some planning to do. If there is a period of time where activity is low and the this is permissive of locking the table then sometimes it is better execute within this window. Frequently though we are reactive to new SQL statements or a new performance issue and an emergency index is required to reduce load on the master in order to improve the response time. To pt-osc or not to pt-osc? As mentioned, pt-online-schema-change is a fixture in our workflow. It’s usually the right way to go but we still have occasions where pt-online-schema-change cannot be used, for example; when a table already uses triggers. It’s an important to remind ourselves of the the steps that pt-online-schema-change traverses to complete it’s job. Lets look at the source code to identify these; [moore@localhost]$ egrep 'Step' pt-online-schema-change # Step 1: Create the new table. # Step 2: Alter the new, empty table. This should be very quick, # Step 3: Create the triggers to capture changes on the original table and <--(metadata lock) # Step 4: Copy rows. # Step 5: Rename tables: orig -> old, new -> orig <--(metadata lock) # Step 6: Update foreign key constraints if there are child tables. # Step 7: Drop the old table. [moore@localhost]$egrep'Step'pt-online-schema-change # Step 1: Create the new table. # Step 2: Alter the new, empty table. This should be very quick, # Step 3: Create the triggers to capture changes on the original table and <--(metadata lock) # Step 4: Copy rows. # Step 5: Rename tables: orig -> old, new -> orig <--(metadata lock) # Step 6: Update foreign key constraints if there are child tables. # Step 7: Drop the old table. I pick out steps 3 and 5 from above to highlight a source of a source of potential downtime due to locks, but step 6 is also an area for concern since foreign keys can have nested actions and should be considered when planning these actions to avoid related tables from being rebuilt with a direct alter implicitly. There are several ways to approach a table with referential integrity constraints and they are detailed within the pt-osc documentation a good preparation step is to review the structure of your table including the constraints and how the ripples of the change can affect the tables around it. Recently we were alerted to an incident after a client with a highly concurrent and highly transactional workload ran a standard pt-online-schema-change script over a large table. This appeared normal to them and a few hours later our pager man was notified that this client was experiencing max_connections limit reached. So what was going on? When pt-online-schema-change reached step 5 it tried to acquire a metadata lock to rename the the original and the shadow table, however this wasn’t immediately granted due to open transactions and thus threads began to queue behind the RENAME command. The actual effect this had on the client’s application was downtime. No new connections could be made and all existing threads were waiting behind the RENAME command. Metadata locks Introduced in 5.5.3 at server level. When a transaction starts it will acquire a metadata lock (independent of storage engine) on all tables it uses and then releases them when it’s finished it’s work. This ensures that nothing can alter the table definition whilst a transaction is open. With some foresight and planning we can avoid these situations with non-default pt-osc options, namely –nodrop-new-table and –no-swap-tables. This combination leaves both the shadow table and the triggers inplace so that we can instigate an atomic RENAME when load permits. EDIT: as of percona-toolkit version 2.2 we have a new variable –tries which in conjunction with –set-vars has been deployed to cover this scenario where various pt-osc operations could block waiting for a metadata lock. The default behaviour of pt-osc (–set-vars) is to set the following session variables when it connects to the server; wait_timeout=10000 innodb_lock_wait_timeout=1 lock_wait_timeout=60 when using –tries we can granularly identify the operation, try count and the wait interval between tries. This combination will ensure that pt-osc will kill it’s own waiting session in good time to avoid the thread pileup and provide us with a loop to attempt to acquire our metadata lock for triggers|rename|fk management; –tries swap_tables:5:0.5,drop_triggers:5:0.5 The documentation is here http://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html#cmdoption-pt-online-schema-change–tries This illustrates that even with a tool like pt-online-schema-change it is important to understand the caveats presented with the solution you think is most adequate. To help decide the direction to take use the flow chart to ensure you’re taking into account some of the caveats of the MySQL schema change. Be sure to read up on the recommended outcome though as there are uncharted areas such as disk space, IO load that are not featured on the diagram. Choosing the right DDL option Ensure you know what effect ALTER TABLE will have on your platform and pick the right method to suit your uptime. Sometimes that means delaying the change until a period of lighter use or utilising a tool that will avoid holding a table locked for the duration of the operation. A direct ALTER is sometimes the answer like when you have triggers installed on a table. – In most cases pt-osc is exactly what we need – In many cases pt-osc is needed but the way in which it’s used needs tweaking – In few cases pt-osc isn’t the right tool/method and we need to consider native blocking ALTER or using failovers to juggle the change into place on all hosts in the replica cluster. If you want to learn more about avoiding avoidable downtime please tune into my webinar Wednesday, November 19 at 10 a.m. PST. It’s titled “Tips from the Trenches: A Guide to Preventing Downtime for the Over-Extended DBA.” Register now!(If you miss it, don’t worry: Catch the recording and download the slides from that same registration page.)

In this post you can see an approach for conditionally enabling/disabling a set of checkboxes. For this we can use the ng-disabled directive and some CSS clases of typeclassName-true and className-false: ENABLE/DISABLE CHECKBOXES USING ANGULAR JS Select the maximum prize money: Select one prize money{{item.prizemoney} {{item.name}

In this tip you can see how to use the Angular Websocket module for connecting client applications to servers.

In this article I am going to present a solution about sending SMS messages in Java for those developers and marketers who think – like me – that SMS is not dead.

I’ve lost count of the number of times I’ve seen code which fail-fast validates the state of something, using an approach like public class PersonValidator { public boolean validate(Person person) { boolean valid = person != null; if (valid) valid = person.givenName != null; if (valid) valid = person.familyName != null; if (valid) valid = person.age != null; if (valid) valid = person.gender != null; // ...and many more } } It works, but it’s a brute force approach that’s filled with repetition due to the valid check. If your code style enforces braces for if statements (+1 for that), your method is also three times longer and growing every time a new check is added to the validator. Using Java 8’s new stream API, we can improve this by taking the guard condition of if (valid) and making a generic validator that handles the plumbing for you. import java.util.LinkedList; import java.util.List; import java.util.function.Predicate; public class GenericValidator implements Predicate { private final List> validators = new LinkedList<>(); public GenericValidator(List> validators) { this.validators.addAll(validators); } @Override public boolean test(final T toValidate) { return validators.parallelStream() .allMatch(predicate -> predicate.test(toValidate)); } } Using this, we can rewrite the Person validator to be a specification of the required validations. public class PersonValidator extends GenericValidator { private static final List> VALIDATORS = new LinkedList<>(); static { VALIDATORS.add(person -> person.givenName != null); VALIDATORS.add(person -> person.familyName != null); VALIDATORS.add(person -> person.age != null); VALIDATORS.add(person -> person.gender != null); // ...and many more } public PersonValidator() { super(VALIDATORS); } } PersonValidator, and all your other validators, can now focus completely on validation. The behaviour hasn’t changed – the validation still fails fast. There’s no boiler plate, which is A Good Thing. This one’s going in the toolbox.

My objective here is to recreate a similar set-up in a smaller unit test mode.

Recently I wanted to extract certain data from an output log. Here’s part of the log file: 2015-01-06 11:33:03 b.s.d.task [INFO] Emitting: eVentToRequestsBolt __ack_ack [-6722594615019711369 -1335723027906100557] 2015-01-06 11:33:03 c.s.p.d.PackagesProvider [INFO] ===---> Loaded package com.foo.bar 2015-01-06 11:33:04 b.s.d.executor [INFO] Processing received message source: eventToManageBolt:2, stream: __ack_ack, id: {}, [-6722594615019711369 -1335723027906100557] 2015-01-06 11:33:04 c.s.p.d.PackagesProvider [INFO] ===---> Loaded package co.il.boo 2015-01-06 11:33:04 c.s.p.d.PackagesProvider [INFO] ===---> Loaded package dot.org.biz I decided to do it using the Java8 Stream and Lambda Expression features. Read the file First, I needed to read the log file and put the lines in a Stream: Stream lines = Files.lines(Paths.get(args[1])); Filter relevant lines I needed to get the packages names and write them into another file. Not all lines contained the data I need, hence filter only relevant ones. lines.filter(line -> line.contains("===---> Loaded package")) Parsing the relevant lines Then, I needed to parse the relevant lines. I did it by first splitting each line to an array of Strings and then taking the last element in that array. In other words, I did a double mapping. First a line to an array and then an array to a String. .map(line -> line.split(" ")) .map(arr -> arr[arr.length - 1]) Writing to output file The last part was taking each string and write it to a file. That was the terminal operation. .forEach(package -> writeToFile(fw, package)); writeToFile is a method I created. The reason is that Java File System throws IOException. You can’t use checked exceptions in lambda expressions. Here’s a full example (note, I don’t check input) import java.io.FileWriter; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; import java.util.Arrays; import java.util.List; import java.util.stream.Stream; public class App { public static void main(String[] args) throws IOException { Stream lines = null; if (args.length == 2) { lines = Files.lines(Paths.get(args[1])); } else { String s1 = "2015-01-06 11:33:03 b.s.d.task [INFO] Emitting: adEventToRequestsBolt __ack_ack [-6722594615019711369 -1335723027906100557]"; String s2 = "2015-01-06 11:33:03 b.s.d.executor [INFO] Processing received message source: eventToManageBolt:2, stream: __ack_ack, id: {}, [-6722594615019711369 -1335723027906100557]"; String s3 = "2015-01-06 11:33:04 c.s.p.d.PackagesProvider [INFO] ===---> Loaded package com.foo.bar"; String s4 = "2015-01-06 11:33:04 c.s.p.d.PackagesProvider [INFO] ===---> Loaded package co.il.boo"; String s5 = "2015-01-06 11:33:04 c.s.p.d.PackagesProvider [INFO] ===---> Loaded package dot.org.biz"; List rows = Arrays.asList(s1, s2, s3, s4, s5); lines = rows.stream(); } new App().parse(lines, args[0]); } private void parse(Stream lines, String output) throws IOException { final FileWriter fw = new FileWriter(output); //@formatter:off lines.filter(line -> line.contains("===---> Loaded package")) .map(line -> line.split(" ")) .map(arr -> arr[arr.length - 1]) .forEach(package -> writeToFile(fw, package)); //@formatter:on fw.close(); lines.close(); } private void writeToFile(FileWriter fw, String package) { try { fw.write(String.format("%s%n", package)); } catch (IOException e) { throw new RuntimeException(e); } } } If you enjoyed this article and want to learn more about Java Streams, check out this collection of tutorials and articles on all things Java Streams.

Byte Buddy is a code generation library for creating Java classes during the runtime of a Java application and without the help of a compiler. Other than the code generation utilities that ship with the Java Class Library, Byte Buddy allows the creation of arbitrary classes and is not limited to implementing interfaces for the creation of runtime proxies. In order to use Byte Buddy, one does not require an understanding of Java byte code or the class file format. In contrast, Byte Buddy’s API aims for code that is concise and easy to understand for everybody. Nevertheless, Byte Buddy remains fully customizable down to the possibility of defining custom byte code. Furthermore, the API was designed to be as non-intrusive as possible and as a result, Byte Buddy does not leave any trace in the classes that were created by it. For this reason, the generated classes can exist without requiring Byte Buddy on the class path. Because of this feature, Byte Buddy’s mascot was chosen to be a ghost. Byte Buddy is written in Java 6 but supports the generation of classes for any Java version. Byte Buddy is a light-weight library and only depends on the visitor API of the Java byte code parser library ASM which does itself not require any further dependencies. At first sight, runtime code generation can appear to be some sort of black magic that should be avoided and only few developers write applications that explicitly generate code during their runtime. However, this picture changes when creating libraries that need to interact with arbitrary code and types that are unknown at compile time. In this context, a library implementer must often choose between either requiring a user to implement library-proprietary interfaces or to generate code at runtime when the user’s types becomes first known to the library. Many known libraries such as for example Spring or Hibernate choose the latter approach which is popular among their users under the term of using Plain Old Java Objects. As a result, code generation has become an ubiquitous concept in the Java space. Byte Buddy is an attempt to innovate the runtime creation of Java types in order to provide a better tool set to those relying on code generation. Hello World Saying Hello World with Byte Buddy is as easy as it can get. Any creation of a Java class starts with an instance of the ByteBuddy class which represents a configuration for creating new types: Class dynamicType = new ByteBuddy() .subclass(Object.class) .method(named("toString")).intercept(FixedValue.value("Hello World!")) .make() .load(getClass().getClassLoader(), ClassLoadingStrategy.Default.WRAPPER) .getLoaded(); assertThat(dynamicType.newInstance().toString(), is("Hello World!")); The default ByteBuddy configuration which is used in the above example creatse a Java class in the newest version of the class file format that is understood by the processing Java virtual machine. As hopefully obvious from the example code, the created type will extend the Object class and intercept its toString method which should return a fixed value of Hello World!. The method to be intercepted is identified by a so-called method matcher. In the above example, a predefined method matcher named(String) is used which identifies a method by its exact name. Byte Buddy comes with numerous predefined and well-tested method matchers which are collected in the MethodMatchers class. The creation of custom matchers is however as simple as implementing the (functional) MethodMatcher interface. For implementing the toString method, the FixedValue class defines a constant return value for the intercepted method. Defining a constant value is only one example of many method interceptors that ship with Byte Buddy. By implementing the Instrumentation interface, a method could however even be defined by custom byte code. Finally, the described Java class is created and then loaded into the Java virtual machine. For this purpose, a target class loader is required as well as a class loading strategy where we choose a wrapper strategy. The latter creates a new child class loader which wraps the given class loader and only knows about the newly created dynamic type. Eventually, we can convince ourselves of the result by calling the toString method on an instance of the created class and finding the return value to represent the constant value we expected. Where to go from here? Byte Buddy is a comprehensive library and we only scratched the surface of Byte Buddy's capabilities. However, Byte Buddy aims for being easy to use by providing a domain-specific language for creating classes. Most runtime code generation can be done by writing readable code and without any knowledge of Java's class file format. If you want to learn more about Byte Buddy, you can find such a tutorial on Byte Buddy's web page. Furthermore, Byte Buddy comes with a detailed in-code documentation and extensive test case coverage which can also serve as example code. you can also find the source code of Byte Buddy on GitHub .

Introduction So your thinking on designing your own DSL or to use an existing grammar from ANTLR, but how do you execute it? That is what this tutorial is about. Groovy is great for building DSL's, but the expressiveness is limited to language features already included in the Groovy syntax. ANTLR offers a much larger set of language features that you can use. Just look at the language's already available at: GitHub - ANTLR 4 Grammars. You could pick one grammar file, or two, modify them to your needs and rapidly create your DSL. One thing is missing though from all books and articles on DSL's with groovy and ANTLR, at least from the ones I've read, and that is, how to run a script written in your newly created DSL? With ANTLR, you can parse the script, but you'll need to write a Java or Groovy program with a main class to read in the file. This may be sufficient for certain use cases, for example when translating between two syntaxes, or when reading in a configuraiton file, or when you have a command language for your server, but in certain cases you just want the customers of your DSL to be able to use it as a 'real' language. In this article I will demonstrate how the powerful Groovy AST transformations can be used to execute a script written in a ANTLR grammar with a custom extension as a Groovy script. If you're new to Groovy DSL's and AST transformations I recommend to read the following articles as well: Joe's Groovy Blog on AST Transformations Global AST Transformations Groovy Global AST Transformations Groovy makes a distinction between a global and a local AST Transformation. A global AST Transformation is fired once and has the entire script, or SourceUnit, as scope. A local AST Transformation is fired when found and has the statement where it is found as scope. Global transformations are always executed when present on the classpath, every time the groovy compiler is invoked. Local transformations are only invoked when they are present in a script. Local transformations are useful when a single statement needs to be transformed. Global transformations are useful when the entire script needs to be transformed. In this example we will use global transformations. Global transformations have an extra cool feature: you can specify your own file extension. This feature will be used to specify a syntax with ANTLR, and then run it with Groovy. Our DSL: the Cymbol language Cymbol is a little language written by Terence Parr and published in his book: The Definitive ANTLR 4 Reference. Cymbol only contains the basic C-style syntax needed to write declarations and functions. It is a good point to start if you want to implement your own DSL with some extra features. The grammar file is available at GitHub: Cymbol.g4. This file is included in the grammar directory of the sample workspace. You can install the ANTLRWorks workbench, if you want to try the examples out for yourself. The workbench can be found at: Tunnel Vision Labs. The ANTLR Workbench was used to generate the necessary classes to parse the DSL. This is however not necessary to run this example and all needed classes are already included in the attached workspace zip file. A sample Cymbol program could look as follows (taken from the ANTLR 4 Reference, p. 99): // Cymbol test int g = 9; // a global variable int fact(int x) { // a factorial function if x==0 then return 1; return x * fact(x-1); } Set up the workspace Eclipse is used as IDE. To start, you can download the Cymbol_workspace.zip file and import it in Eclipse. After that, you'll need to download a few libraries, the antlr_4.4_complete.jar, hamcrest_1.3.jar and junit_4.11.jar. Due to upload size limitations I couldn't include the libraries in the zip file. Links to the download sites are provided in the README.txt file in the libs directory of the Eclipse project. Add the libraries to the build path and the project set up is complete. Now you can execute DslScriptTransformTest.java as JUnit test and you should see some output in the console. What is going on behind the scenes here? Generate the Parser Open the Cymbol.g4 grammar file in the ANTLR Workbench. Go to: Run > Generate Recognizer... Select the ouput directory, tick the boxes in front of Generate Listener and Generate Visitor. Don't forget to set the package to com.cymbol.parser. This will generate all classes that you'll need to parse a script written with your DSL's syntax. Create the ASTTransformation The DslScriptTransform.java file contains the ASTTransformation. The transformation class needs to implement the ASTTransformation interface from Groovy. An AST Transformation is annotated with: @GroovyASTTransformation(phase = CompilePhase.SEMANTIC_ANALYSIS) This annotation tells the Groovy compiler that this class is an AST Transformation that should be applied in the specified phase. Global transformations can occur in any compiler phase so here are a few words on how to select the right phase. The Groovy compiler hase nine phases. Of these, only the second, third and fourth are relevant for us: Initialization The resources and environment are initialized. Parsing The script text is parsed and a Concrete Syntax Tree (CST) is constructed according to the Groovy grammar. Conversion The Astract Syntaxt Tree (AST) is constructed from the CST. Semantic Analysis In this phase, the AST is checked, classes, static imports and variables are resolved. After this phase, the AST is complete and the input is seen as a valid source. Our DSL script has been transformed into a Groovy script with one method: getScriptText(). In the semantic analysis phase, all necessary Groovy structures have been initialised and we can safely modify the AST. The only thing that needs to be done now is to implement the run() method to call the ANTLR Parser and parse the script text. First, we check whether the file has the required extension: @Override public void visit(ASTNode[] nodes, SourceUnit source) { if(!source.getName().endsWith(".cymbol")) { return; } ... Second, the main class is retrieved from the AST: private ClassNode getMainClass(ModuleNode moduleNode) { for(ClassNode classNode : moduleNode.getClasses()) { if(classNode.getNameWithoutPackage().equals(moduleNode.getMainClassName())) { return classNode; } } return null; } And the run method's body is imlemented. The run method's body will look something like follows: @Override public Object run() { DslScriptTransformHelper helper = new DslScriptTransformHelper(); String scriptText = this.getScriptText(); helper.parse(scriptText); } The @Override annotation is not actually there but is included here for clarity. The annotation could be added as well but it is left out to keep the example simple. The implementation is kept as minimal as possible because writing AST Transformations is rather cumbersome. As much as posisble is factored out into a delegate called DslScriptTransformHelper. The helper class contains the actual code to parse the DSL script and to do something with it. The benefit is that complex logic can be implemented that can be checked by the compiler and by your IDE. Third, the AST Transformation to implement the run method's body is as follows: private void addRunMethod(ClassNode scriptClass) { List statements = new ArrayList<>(); /* * initialise the parser helper: * * DslScriptTransformHelper helper = new DslScriptTransformHelper() */ ClassNode helperClassNode = new ClassNode(DslScriptTransformHelper.class); ConstructorCallExpression initParserHelper = new ConstructorCallExpression(helperClassNode, new ArgumentListExpression()); VariableExpression helperVar = new VariableExpression(HELPER_VAR, helperClassNode); DeclarationExpression assignHelperExpr = new DeclarationExpression(helperVar, new Token(Types.EQUALS, "=", -1, -1), initParserHelper); ExpressionStatement initHelperExpr = new ExpressionStatement(assignHelperExpr); statements.add(initHelperExpr); /* * get the script text and assign it to a variable: * * String scriptText = this.getScriptText() */ MethodCallExpression getScriptTextExpr = new MethodCallExpression(new VariableExpression("this"), new ConstantExpression(GET_SCRIPT_TEXT_METHOD), MethodCallExpression.NO_ARGUMENTS); VariableExpression scriptTextVar = new VariableExpression(SCRIPT_TEXT_VAR, new ClassNode(String.class)); DeclarationExpression declareScriptTextExpr = new DeclarationExpression(scriptTextVar, new Token(Types.EQUALS, "=", -1, -1), getScriptTextExpr); ExpressionStatement getScriptTextStmt = new ExpressionStatement(declareScriptTextExpr); statements.add(getScriptTextStmt); /* * call the parse method on the helper: * * helper.parse(scriptText) */ MethodCallExpression callParserExpr = new MethodCallExpression(helperVar, PARSE_METHOD, new ArgumentListExpression(new VariableExpression(SCRIPT_TEXT_VAR))); ExpressionStatement callParserStmt = new ExpressionStatement(callParserExpr); statements.add(callParserStmt); // implement the run method's body BlockStatement methodBody = new BlockStatement(statements, new VariableScope()); MethodNode runMethod = scriptClass.getMethod(RUN_METHOD, new Parameter[0]); runMethod.setCode(methodBody); } All the above is needed to implement three lines of code! Fifth, the DslScriptTransformHelper class contains a parse method that calls the ANTLR parser and then does something with the ParseTree returned by ANTLR. You will probably want a semantic model, and I included a visitor that can iterate over the model to do something with it. This bit is left unimplemented because it requires, a lot, of design and coding to get this right. Here is a simple example: public class DslScriptTransformHelper { public SemanticModel parse(String scriptText) throws IOException { ANTLRInputStream is = new ANTLRInputStream(scriptText); CymbolLexer lexer = new CymbolLexer(is); CommonTokenStream tokens = new CommonTokenStream(lexer); CymbolParser parser = new CymbolParser(tokens); ParseTree tree = parser.file(); SemanticModel model = initSemanticModel(tree); SemanticModelVisitor visitor = initVisitor(); visitor.visit(model); return model; } Here, the generated Lexer and Parser are called. If you would want to create your own DSL with ANTLR, you will have to modify the DslScriptTransformHelper. The AST Transformation, DslScriptTransform, need not be modified. As much logic is factored out of the AST Transformation as posisble into a helper class. As you will see in the next section, creating an AST Transformation is difficult, time consuming, error prone, and generally not something that you will enjoy doing. by factoring out behaviour into a helper you can write logic and still have IDE support, and Unit testability for your logic. All you need to do is to initialize the helper and call a method on the helper with the DSL script's text as argument. I included two interfaces that are left unimplemented: SemanticModel and SemanticModelVisitor. private SemanticModel initSemanticModel(ParseTree tree) { SemanticModel model = new SemanticModel() { @Override public void init(ParseTree tree) { System.out.println("init: " + tree.toStringTree()); } }; model.init(tree); return model; } private SemanticModelVisitor initVisitor() { SemanticModelVisitor visitor = new SemanticModelVisitor() { @Override public void visit(SemanticModel model) { System.out.println("visit"); } }; return visitor; } } The purpose of these is that the semantic model represents, well, your semantic model. This is so to say your domain model. In the design of your DSL, deciding on a semantic model is probably the most important activity. The semantic model models what the DSL means. The grammar specifies how to write a valid script, but the model specifies what it means. Think of it as follows: not all valid English sentences are meaningful. Martin Fowler discusses this in his book: Domain Specific Languages. The visitor is included so that after you've constructed a valid semantic model, you can do something with it, if you want. As you can see from the above example, performing an AST Transformation is rather cumbersome, at least in Java. Surely Groovy offers something better? Groovy does, and there are several excellent examples on this available at: GitHub. To me, the major downside to using the Groovy Builder DSL to specify AST Transformations is that the types can't be checked. It is rather difficult to get everything right, especially without type checking. The Cymbol DSL implementation uses the Groovy compiler, but doesn't actually contain any Groovy code. I think that is a benefit because fiddling with compilers and AST's is difficult enough, even with strict typing and IDE support. The examples can be implemented in Groovy as well if you wish so. Create the Groovy configuration for the AST Transformations Groovy needs the following configuration files to know that an AST Transformation should be called, and that it should accept files with a custom extension. Two files must be added: META-INF/services/org.codehaus.groovy.source.Extensions The extension is just declared in the org.codehaus.groovy.source.Extensions file: .cymbol META-INF/services/org.codehaus.groovy.transform.ASTTransformation This file only contains the name with package of the class that contains the AST Transformation: com.dsl.transformation.DslScriptTransform Create a Source Pre-Processor Groovy supports custom compiler configurations. They are documented at: Advanced Compiler Configuration. We will use a custom compiler configuration to create a source pre-processor that reads the source of a script, a Cymbol script in our case, and that modifies the source. This happens before the Groovy compiler tries to parse the source and thus can be used as a source pre-processor. The text of the source is wrapped in a String and a method that returns that string. The Cymbol example is transformed into the following Groovy script: String getScriptText() { return '''// Cymbol test int g = 9; // a global variable int fact(int x) { // a factorial function if x==0 then return 1; return x * fact(x-1); }''' } The method getScriptText() now returns the DSL script as string. The Groovy source pre-processor is implemented as follows. First, a custom CustomParserPlugin is created. The CustomParserPlugin extends the Groovy AntlrParserPlugin and overrides the parseCST() method. This method is similar to the following example from Guillaume Laforge: Custom antlr parser plugin. Here the example is taken a step further however by calling ANTLR's StringTemplate to operate on the entire source, instead of modifiying the Groovy source by remapping the bindings, as is done in Guillaume's example. public class CustomParserPlugin extends AntlrParserPlugin { @Override public Reduction parseCST(final SourceUnit sourceUnit, Reader reader) throws CompilationFailedException { StringBuffer bfr = new StringBuffer(); int intValueOfChar; try { while ((intValueOfChar = reader.read()) != -1) { bfr.append((char) intValueOfChar); } } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } String text = modifyTextSource(bfr.toString()); StringReader stringReader = new StringReader(text); SourceUnit modifiedSource = SourceUnit.create(sourceUnit.getName(), text); return super.parseCST(modifiedSource, stringReader); } String modifyTextSource(String text) { ST textAsStringConstant = new ST("String getScriptText() { return '''''' }"); textAsStringConstant.add("text", text); return textAsStringConstant.render(); } } A CompilerConfiguration is equipped with a ParserPluginFactory and we also create a custom implementation that returns our CustomParserPlugin: public class SourcePreProcessor extends ParserPluginFactory { @Override public ParserPlugin createParserPlugin() { return new CustomParserPlugin(); } } A CompilerConfiguration needs to be created with our implementation of the ParserPluginFactory. Several ways of doing this are available to us: Use Groovy embedded (GroovyShell, GroovyScriptEngine, ...) and add a CompilerConfiguration Use the groovyc compiler with a configscript Both techniques will be presented. Embedded Groovy is used in the Unit test and the Groovy compiler with the configuration script is invoked from the command line. Create a Unit test with embedded groovy The Unit test simply evaluates the DSL script with a GroovyShell that has a custom CompilerConfiguration with our SourcePreProcessor: public class DslScriptTransformTest { String scriptPath = "tests"; @Test public void testPreProcessor() throws CompilationFailedException, IOException { String testScript = scriptPath + "/test1.cymbol"; File scriptFile = new File(testScript); ParserPluginFactory ppf = new SourcePreProcessor(); CompilerConfiguration cc = new CompilerConfiguration(); cc.setPluginFactory(ppf); Binding binding = new Binding(); GroovyShell shell = new GroovyShell(binding, cc); shell.evaluate(scriptFile); } } If you now call the testPreProcessor() method you should see some output on the console with the output from ANTLR's tree.toStringTree() method call and a message that the semantic models has been visited. Create a compiler configuration script First, a CompilerConfiguration is created with a custom ParserPluginFactory, and passed to the GroovyShell. Second, the groovyc compiler is invoked with the --configscript flag. The config script has a CompilerConfiguration injected before any files are compiled. The CompilerConfiguration is exposed as a variable named configuration. The compiler configuration script config.groovy looks as follows: configuration.setPluginFactory(new com.dsl.transformation.SourcePreProcessor()) We can simply set the SourcePreProcessor on the configuration as a PluginFactory. Export the Jar file With Eclipse, you can export the project as a Jar file by selecting: Export > Java > JAR File You don't need the libs folder because Jar files that are included in another Jar file aren't available on the class path. What you can do is to create a Manifest.MF file with the following class path entries: Class-Path: libs/antlr-4.4-complete.jar libs/hamcrest-core-1.3.jar This makes sure that if you run the jar file from the right location, the required libraries are on the class path. This is not bullet proof production code, but for testing it is sufficient. Now that you exported a Jar file all you need to do is to include the Jar file with your AST Transformations on the class path and Groovy will automatically understand what to do. The DSL script is invoked from the command line with the following command: groovy -cp Cymbol.jar --configscript config.groovy tests\test1.cymbol You can also invoke the groovyc compile and compile your DSL script into a Java class. The command is the same, really: groovyc -cp Cymbol.jar --configscript config.groovy tests\test1.cymbol In case you have a DSL in which you have customers that write scripts with it, you can now also compile it with a build system such as Ant or Gradle and package the results in a JAR file and deliver it to a production system. Conclusion In the previous sections you have seen a technique to develop a DSL and execute it as a Groovy script, or to compile the DSL script into a Java class file. It requires quite a lot of groovy's DSL features, but it is a good show case on what you can do with Groovy's superb DSL features and how to succesfully apply them. And I didn't even write a single line of Groovy code! The techniques that have been covered are: Generate an ANTLR Parser from a grammar file Create a Groovy source pre-processor Create Global AST Transformations that delegate to a helper class for easy implementations Execute a DSL script with a custom CompilerConfiguration with embedded Groovy or from the command line Compile a DSL script with an ANTLR grammar into a Java .class file with a build system That was a lot of ground to cover. One open point remains tool support. Now that I have my DSL, can I at least have syntax high-lighting? I hope to return to this, and other topics involving DSL design and development, in a next article.

History I think it’s important to take a look at the evolution of Interceptors in Java EE because of the simple fact that it started as an EJB-specific item and later evolved into a separate spec which is now open for extension by other Java EE specifications. Version 1.0 Interceptors were first introduced in EJB 3.0 (part of Java EE 5). Interceptors did not have a dedicated spec but they were versioned 1.0 and bought basic AOP related features to managed beans (POJOs) via simple annotations @AroundInvoke – to annotate methods containing the interception logic for target class methods @Intercerptors – to bind the the interceptor classes with their target classes/methods Capability to configure interceptors for an entire module (EJB JAR) via the deployment descriptor @ExcludeDefaultInterceptors – to mute default interceptors defined in the deployment descriptor @ExcludeClassInterceptors – to mute a globally defined (class level) interceptor for a particular method/constructor of the class Interceptors 1.1 Along came Java EE 6 with EJB 3.1 – Interceptors 1.1 was still included in the EJB spec document @InterceptorBinding – a type safe way of specifying interceptors of a class or a method. Please note that this annotation was leveraged by CDI 1.0 (another specification introduced in Java EE 6) and its details are present in the CDI 1.0 spec doc rather than EJB 3.1 (light bulb moment … at least for me) @Interceptor – Used to explicitly declare a class containing an interception logic in a specific method (annotated with @AroundInvoke etc) as an interceptor along with an appropriate Interceptor Binding. This too was mentioned in the CDI 1.0 documentation only. @AroundTimeout – used to intercept time outs of EJB timers along with a way to obtain an instance of the Timer being intercepted (viajavax.interceptor.InvocationContext.getTimer()) Interceptors 1.2 Interceptors were split off into an individual spec in Java EE 7 and thus Interceptors 1.2came into being Interceptors 1.2 was a maintenance release on top of 1.1 and hence the JSR number still remained the same as EJB 3.1 (JSR 318) Interceptor.Priority (static class) – to provide capability to define the order (priority) in which the interceptors need to invoked. @AroundConstruct – used to intercept the construction of the target class i.e. invoke logic prior to the constructor of the target class is invoked It’s important to bear in mind that Interceptors are applicable to managed beans in general. Managed Beans themselves are simple POJOs which are privileged to basic services by the container – Interceptors are one of them along with life cycle callbacks, resource injection. Memory Aid It’s helpful to think of Interceptors as components which can interpose on beans throughout their life cycle before they are even constructed – @AroundConstruct after they are constructed – @PostConstruct during their life time (method invocation) – @AroundInvoke prior to destruction – @PreDestroy time outs of EJBs – @AroundTimeout Let’s look at some of the traits of Interceptors in more detail and try to answer questions like where are they applied and what do they intercept ? how to bind interceptors to the target (class) they are supposed to intercept ? Interceptors Types (based on the intercepted component) Method Interceptors Achieved by @AroundInvoke public class MethodInterceptor{ @AroundInvoke public Object interceptorMethod(InvocationContext ictx) throws Exception{ //logic goes here } } @Stateless public class AnEJB{ @Interceptors(MethodInterceptor.class) public void bizMethod(){ //any calls to this method will be intercepted by MethodInterceptor.interceptorMethod() } } The method containing the logic can be part of separate class as well as the target class (class to be intercepted) itself. Lifecycle Callback interceptors Decorate the method with @AroundConstruct in order to intercept the constructor invocation for a class public class ConstructorInterceptor{ @AroundConstruct public Object interceptorMethod(InvocationContext ictx) throws Exception{ //logic goes here } } public class APOJO{ @Interceptors(ConstructorInterceptor.class) public APOJO(){ //any calls to this constructor will be intercepted by ConstructorInterceptor.interceptorMethod() } } The method annotated with @AroundConstruct cannot be a part of the intercepted class. It has to be defined using a separate Interceptor class Use the @PostConstruct annotation on a method in order to intercept a call back method on a managed bean. Just to clarify again – the Interceptor spec does not define a new annotation as such. One needs to reuse the @PostConstruct (part of theCommon Annotations spec) on the interceptor method. public class PostConstructInterceptor{ @PostConstruct public void interceptorMethod(InvocationContext ictx) throws Exception{ //logic goes here } } @Interceptors(PostConstructInterceptor.class) public class APOJO{ @PostConstruct public void bizMethod(){ //any calls to this method will be intercepted by PostConstructInterceptor.interceptorMethod() } } The @PreDestroy (another call back annotation defined in Common Annotations spec) annotation is used in a similar fashion Time-out Interceptors As mentioned above – @AroundTimeout used to intercept time outs of EJB timers along with a way to obtain an instance of the Timer being intercepted (viajavax.interceptor.InvocationContext.getTimer()) Applying/Binding Interceptors Using @Interceptors As shown in above examples – just use the @Interceptors annotation to specify the interceptor classes @Interceptors can be applied on a class level (automatically applicable to all the methods of a class), to a particular method or multiple methods and constructor in case of a constructor specific interceptor using @AroundConstruct Using @IntercerptorBinding Interceptor Bindings (explained above) – Use @IntercerptorBinding annotation to define a binding annotation which is further used on the interceptor class as well as the target class (whose method, constructor etc needs to be intercepted) @InterceptorBinding @Target({TYPE, METHOD, CONSTRUCTOR}) @Retention(RUNTIME) public @interface @Auditable { } @Auditable @Interceptor public class AuditInterceptor { @AroundInvoke public Object audit(InvocationContext ictx) throws Exception{ //logic goes here } } @Stateless @Auditable public class AnEJB{ public void bizMethod(){ //any calls to this method will be intercepted by AuditInterceptor.audit() } } Deployment Descriptor One can also use deployment descriptors to bind interceptors and target classes either in an explicit fashion as well as in override mode to annotations. This was a rather quick overview of Java EE interceptors. Hopefully the right trigger for you to dig deeper :-) Cheers !

This is a step by step guide on How to Configure MySQL Metastore for Hive in place of Derby Metastore (Default).

Recently at DevSKiller.com we've decided to move majority of our stuff to simple containers. It was pretty easy due to use of Spring Boot uber-jars, but the problem was in NewRelic agents which should have to be included separately. That caused uncomfortable situation so we decided to solve it by including NewRelic agent into our uber-jar applications. If you also want to simplify your life please follow provided instructions :) At first we have to add proper dependency into our pom.xml descriptor: com.newrelic.agent.java< newrelic-agent 3.12.1 provided Now since we have proper jar included into our project it's time to unpack the dependency to have all necessary classes in our application jar file: org.apache.maven.plugins maven-dependency-plugin 2.9 prepare-package unpack-dependencies newrelic-agent ${project.build.outputDirectory} After this step we've all agent related classes accessible directly from our jar. But still the file cannot be used as an agent jar. There are some important manifest entries that have to be present in every agent jar. The most important is the Premain-Class attribute specifying main agent class including premain() method. In case of NewRelic it's also important to include Can-Redefine-Classes and Can-Retransform-Classes attributes. The easiest way to do that is to extend maven-jar-plugin configuration: org.apache.maven.plugins maven-jar-plugin 2.5 com.newrelic.bootstrap.BootstrapAgent true true Now is coming the tricky part :) NewRelic agent also contains class with main() method which causes that Spring Boot repackager plugin is unable to find single main() method so build fails. It's not a problem but we have to remember to specify proper main class in spring-boot-maven-plugin (or in gradle plugin): my.custom.Application That's all! You can execute your application with following command: java -javaagent:myapp.jar -jar myapp.jar Last but not least: don't forget to include NewRelic configuration file (newrelic.yml) in the same directory as your application jar. The other solution is to set newrelic.config.file system property to point the fully qualified file name.

I was recently asked about the benefits and wisdom of using off heap memory in Java. The answers may be of interest to others facing the same choices. Off heap memory is nothing special. The thread stacks, application code, NIO buffers are all off heap. In fact in C and C++, you only have unmanaged memory as it does not have a managed heap by default. The use of managed memory or "heap" in Java is a special feature of the language. Note: Java is not the only language to do this. new Object() vs Object pool vs Off Heap memory. new Object() Before Java 5.0, using object pools was very popular. Creating objects was still very expensive. However, from Java 5.0, object allocation and garbage cleanup was made much cheaper, and developers found they got a performance speed up and a simplification of their code by removing object pools and just creating new objects whenever needed. Before Java 5.0, almost any object pool, even an object pool which used objects provided an improvement, from Java 5.0 pooling only expensive objects obviously made sense e.g. threads, socket and database connections. Object pools In the low latency space it was still apparent that recycling mutable objects improved performance by reduced pressure on your CPU caches. These objects have to have simple life cycles and have a simple structure, but you could see significant improvements in performance and jitter by using them. Another area where it made sense to use object pools is when loading large amounts of data with many duplicate objects. With a significant reduction in memory usage and a reduction in the number of objects the GC had to manage, you saw a reduction in GC times and an increase in throughput. These object pools were designed to be more light weight than say using a synchronized HashMap, and so they still helped. Take this StringInterner class as an example. You pass it a recycled mutable StringBuilder of the text you want as a String and it will provide a String which matches. Passing a String would be inefficient as you would have already created the object. The StringBuilder can be recycled. Note: this structure has an interesting property that requires no additional thread safety features, like volatile or synchronized, other than is provided by the minimum Java guarantees. i.e. you can see the final fields in a String correctly and only read consistent references. public class StringInterner { private final String[] interner; private final int mask; public StringInterner(int capacity) { int n = Maths.nextPower2(capacity, 128); interner = new String[n]; mask = n - 1; } private static boolean isEqual(@Nullable CharSequence s, @NotNull CharSequence cs) { if (s == null) return false; if (s.length() != cs.length()) return false; for (int i = 0; i < cs.length(); i++) if (s.charAt(i) != cs.charAt(i)) return false; return true; } @NotNull public String intern(@NotNull CharSequence cs) { long hash = 0; for (int i = 0; i < cs.length(); i++) hash = 57 * hash + cs.charAt(i); int h = (int) Maths.hash(hash) & mask; String s = interner[h]; if (isEqual(s, cs)) return s; String s2 = cs.toString(); return interner[h] = s2; } } Off heap memory usage Using off heap memory and using object pools both help reduce GC pauses, this is their only similarity. Object pools are good for short lived mutable objects, expensive to create objects and long live immutable objects where there is a lot of duplication. Medium lived mutable objects, or complex objects are more likely to be better left to the GC to handle. However, medium to long lived mutable objects suffer in a number of ways which off heap memory solves. Off heap memory provides; Scalability to large memory sizes e.g. over 1 TB and larger than main memory. Notional impact on GC pause times. Sharing between processes, reducing duplication between JVMs, and making it easier to split JVMs. Persistence for faster restarts or replying of production data in test. The use of off heap memory gives you more options in terms of how you design your system. The most important improvement is not performance, but determinism. Off heap and testing One of the biggest challenges in high performance computing is reproducing obscure bugs and being able to prove you have fixed them. By storing all your input events and data off heap in a persisted way you can turn your critical systems into a series of complex state machines. (Or in simple cases, just one state machine) In this way you get reproducible behaviour and performance between test and production.A number of investment banks use this technique to replay a system reliably to any event in the day and work out exactly why that event was processed the way it was. More importantly, once you have a fix you can show that you have fixed the issue which occurred in production, instead of finding an issue and hoping this was the issue.Along with deterministic behaviour comes deterministic performance. In test environments, you can replay the events with realistic timings and show the latency distribution you expect to get in production. Some system jitter can't be reproduce esp if the hardware is not the same, but you can get pretty close when you take a statistical view. To avoid taking a day to replay a day of data you can add a threshold. e.g. if the time between events is more than 10 ms you might only wait 10 ms. This can allow you to replay a day of events with realistic timing in under an hour and see whether your changes have improved your latency distribution or not. By going more low level don't you lose some of "compile once, run anywhere"? To some degree this is true, but it is far less than you might think. When you are working closer the processor and so you are more dependant on how the processor, or OS behaves. Fortunately, most systems use AMD/Intel processors and even ARM processors are becoming more compatible in terms of the low level guarantees they provide. There is also differences in the OSes, and these techniques tend to work better on Linux than Windows. However, if you develop on MacOSX or Windows and use Linux for production, you shouldn't have any issues. This is what we do at Higher Frequency Trading. What new problems are we creating by using off heap? Nothing comes for free, and this is the case with off heap. The biggest issue with off heap is your data structures become less natural. You either need a simple data structure which can be mapped directly to off heap, or you have a complex data structure which serializes and deserializes to put it off heap. Obvious using serialization has its own headaches and performance hit. Using serialization thus much slower than on heap objects.In the financial world, most high ticking data structure are flat and simple, full of primitives which maps nicely off heap with little overhead. However, this doesn't apply in all applications and you can get complex nested data structures e.g. graphs, which you can end up having to cache some objects on-heap as well.Another problem is that the JVM limits how much of the system you can use. You don't have to worry about the JVM overloading the system so much. With off heap, some limitations are lifted and you can use data structures much larger than main memory, and you start having to worry about what kind of disk sub-system you have if you do this. For example, you don't want to be paging to a HDD which has 80 IOPS, instead you are likely to want an SSD with 80,000 IOPS (Input/Ouput Operations per Second) or better i.e. 1000x faster. How does OpenHFT help? OpenHFT has a number of libraries to hide the fact you are really using native memory to store your data. These data structures are persisted and can be used with little or no garbage. These are used in applications which run all day without a minor collection Chronicle Queue - Persisted queue of events. Supports concurrent writers across JVMs on the same machine and concurrent readers across machines. Micro-second latencies and sustained throughputs in the millions of messages per second. Chronicle Map - Native or Persisted storage of a key-value Map. Can be shared between JVMs on the same machine, replicated via UDP or TCP and/or accessed remotely via TCP. Micro-second latencies and sustained read/write rates in the millions of operations per second per machine. Thread Affinity - Binding of critical threads to isolated cores or logical cpus to minimise jitter. Can reduce jitter by a factor of 1000. Which API to use? If you need to record every event -> Chronicle Queue If you only need the latest result for a unique key -> Chronicle Map If you care about 20 micro-second jitter -> Thread Affinity Conclusion Off heap memory can have challenges but also come with a lot of benefits. Where you see the biggest gain and compares with other solutions introduced to achieve scalability. Off heap is likely to be simpler and much faster than using partitioned/sharded on heap caches, messaging solutions, or out of process databases. By being faster, you may find that some of the tricks you need to do to give you the performance you need are no longer required. e.g. off heap solutions can support synchronous writes to the OS, instead of having to perform them asynchronously with the risk of data loss.The biggest gain however, can be your startup time, giving you a production system which restarts much faster. e.g. mapping in a 1 TB data set can take 10 milli-seconds, and ease of reproducibility in test by replaying every event in order you get the same behaviour every time. This allows you to produce quality systems you can rely on.

This article represents code samples representing lambda expression and the related ease with which one could print key and value of a Map object using one liner. Please feel free to comment/suggest if I missed to mention one or more important points. Also, sorry for the typos. Code Sample – Printing Map using BiConsumer Functional Interface Following is detail for Map.forEach API in Java 8. Read further on this page. default void forEach(BiConsumer action) Performs the given action for each entry in this map until all entries have been processed or the action throws an exception. Unless otherwise specified by the implementing class, actions are performed in the order of entry set iteration (if an iteration order is specified.) Exceptions thrown by the action are relayed to the caller. Pay attention to following: Traditional way of printing key & value would require one to get an iterator of Map.Entry objects and print key and values Lambda expression way represents defining a BiConsumer implementation by passing two input arguments as key and value of Map and printing their values. public static void main(String[] args) { Map map = new HashMap(); String[][] tempStrArr = {{"Chris","USA"}, {"Raju","India"}, {"Lynda","Canada"} }; // Create a Map using String Array for( int i = 0; i < tempStrArr.length; i++ ) { map.put( tempStrArr[i][0], tempStrArr[i][1] ); } // Traditional way of printing key, value Iterator> iter = map.entrySet().iterator(); if( iter != null ) { while( iter.hasNext() ) { Map.Entry entry = iter.next(); System.out.println( "Key: " + entry.getKey() + "\t" + " Value: " + entry.getValue() ); } } // Using Lambda Expression: All in One line map.forEach( (key, value) -> { System.out.println( "Key: " + key + "\t" + " Value: " + value ); }); }