I have been using Spring Scala for a toy project for the last few days and I have to say that it is a fantastic project, it simplifies Spring configuration even further when compared to the already simple configuration purely based on Spring Java Config. Let me demonstrate this by starting with the Cake Pattern based sample here: // ======================= // service interfaces trait OnOffDeviceComponent { val onOff: OnOffDevice trait OnOffDevice { def on: Unit def off: Unit } } trait SensorDeviceComponent { val sensor: SensorDevice trait SensorDevice { def isCoffeePresent: Boolean } } // ======================= // service implementations trait OnOffDeviceComponentImpl extends OnOffDeviceComponent { class Heater extends OnOffDevice { def on = println("heater.on") def off = println("heater.off") } } trait SensorDeviceComponentImpl extends SensorDeviceComponent { class PotSensor extends SensorDevice { def isCoffeePresent = true } } // ======================= // service declaring two dependencies that it wants injected trait WarmerComponentImpl { this: SensorDeviceComponent with OnOffDeviceComponent => class Warmer { def trigger = { if (sensor.isCoffeePresent) onOff.on else onOff.off } } } // ======================= // instantiate the services in a module object ComponentRegistry extends OnOffDeviceComponentImpl with SensorDeviceComponentImpl with WarmerComponentImpl { val onOff = new Heater val sensor = new PotSensor val warmer = new Warmer } // ======================= val warmer = ComponentRegistry.warmer warmer.trigger Cake pattern is a pure Scala way of specifying the dependencies. Now, if we were to specify this dependency using Spring's native Java config, but with Scala as the language, firs to define the components that need to be wired together: trait SensorDevice { def isCoffeePresent: Boolean } class PotSensor extends SensorDevice { def isCoffeePresent = true } trait OnOffDevice { def on: Unit def off: Unit } class Heater extends OnOffDevice { def on = println("heater.on") def off = println("heater.off") } class Warmer(s: SensorDevice, o: OnOffDevice) { def trigger = { if (s.isCoffeePresent) o.on else o.off } } and the configuration with Spring Java Config and a sample which makes use of this configuration: import org.springframework.context.annotation.Configuration import org.springframework.context.annotation.Bean @Configuration class WarmerConfig { @Bean def heater(): OnOffDevice = new Heater @Bean def potSensor(): SensorDevice = new PotSensor @Bean def warmer() = new Warmer(potSensor(), heater()) } import org.springframework.context.annotation.AnnotationConfigApplicationContext val ac = new AnnotationConfigApplicationContext(classOf[WarmerConfig]) val warmer = ac.getBean("warmer", classOf[Warmer]) warmer.trigger Taking this further to use Spring-Scala project to specify the dependencies, the configuration and a sample look like this: import org.springframework.context.annotation.Configuration import org.springframework.context.annotation.Bean @Configuration class WarmerConfig { @Bean def heater(): OnOffDevice = new Heater @Bean def potSensor(): SensorDevice = new PotSensor @Bean def warmer() = new Warmer(potSensor(), heater()) } import org.springframework.context.annotation.AnnotationConfigApplicationContext val ac = new AnnotationConfigApplicationContext(classOf[WarmerConfig]) val warmer = ac.getBean("warmer", classOf[Warmer]) warmer.trigger The essence of the Spring Scala project as explained in this wiki is the "bean" method derived from the `FunctionalConfiguration` trait, this method can be called to create a bean, passing in parameters to specify, if required, bean name, alias, scope and a function which returns the instantiated bean. This sample hopefully gives a good appreciation for how simple Spring Java Config is, and how much more simpler Spring-Scala project makes it for Scala based projects.

this post was originally written by muhammad irfan here on the percona mysql support team, we often see issues where a customer is complaining about replication delays – and many times the problem ends up being tied to mysql replication slave lag. this of course is nothing new for mysql users and we’ve had a few posts here on the mysql performance blog on this topic over the years (two particularly popular post in the past were: “ reasons for mysql replication lag ” and “ managing slave lag with mysql replication ,” both by percona ceo peter zaitsev) . in today’s post, however, i will share some new ways of identifying delays in replication – including possible causes of lagging slaves – and how to cure this problem. how to identify replication delay mysql replication works with two threads, io_thread & sql_thread. io_thread connects to a master, reads binary log events from the master as they come in and just copies them over to a local log file called relaylog . on the other hand, sql_thread reads events from a relay log stored locally on the replication slave (the file that was written by io thread) and then applies them as fast as possible. whenever replication delays, it’s important to discover first whether it’s delaying on slave io_thread or slave sql_thread. normally, i/o thread would not cause a huge replication delay as it is just reading the binary logs from the master. however, it depends on the network connectivity, network latency… how fast is that between the servers. the slave i/o thread could be slow because of high bandwidth usage. usually, when the slave io_thread is able to read binary logs quickly enough it copies and piles up the relay logs on the slave – which is one indication that the slave io_thread is not the culprit of slave lag. on the other hand, when the slave sql_thread is the source of replication delays it is probably because of queries coming from the replication stream are taking too long to execute on the slave. this is sometimes because of different hardware between master/slave, different schema indexes, workload. moreover, the slave oltp workload sometimes causes replication delays because of locking. for instance, if a long-running read against a myisam table blocks the sql thread, or any transaction against an innodb table creates an ix lock and blocks ddl in the sql thread. also, take into account that slave is single threaded prior to mysql 5.6, which would be another reason for delays on the slave sql_thread. let me show you via master status/slave status example to identify either slave is lagging on slave io_thread or slave sql_thread. mysql-master> show master status; +------------------+--------------+------------------+------------------------------------------------------------------+ | file | position | binlog_do_db | binlog_ignore_db | executed_gtid_set | +------------------+--------------+------------------+------------------------------------------------------------------+ | mysql-bin.018196 | 15818564 | | bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:1-2331947 | +------------------+--------------+------------------+------------------------------------------------------------------+ mysql-slave> show slave status\g *************************** 1. row *************************** slave_io_state: queueing master event to the relay log master_host: master.example.com master_user: repl master_port: 3306 connect_retry: 60 master_log_file: mysql-bin.018192 read_master_log_pos: 10050480 relay_log_file: mysql-relay-bin.001796 relay_log_pos: 157090 relay_master_log_file: mysql-bin.018192 slave_io_running: yes slave_sql_running: yes replicate_do_db: replicate_ignore_db: replicate_do_table: replicate_ignore_table: replicate_wild_do_table: replicate_wild_ignore_table: last_errno: 0 last_error: skip_counter: 0 exec_master_log_pos: 5395871 relay_log_space: 10056139 until_condition: none until_log_file: until_log_pos: 0 master_ssl_allowed: no master_ssl_ca_file: master_ssl_ca_path: master_ssl_cert: master_ssl_cipher: master_ssl_key: seconds_behind_master: 230775 master_ssl_verify_server_cert: no last_io_errno: 0 last_io_error: last_sql_errno: 0 last_sql_error: replicate_ignore_server_ids: master_server_id: 2 master_uuid: bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:2-973166 master_info_file: /var/lib/mysql/i1/data/master.info sql_delay: 0 sql_remaining_delay: null slave_sql_running_state: reading event from the relay log master_retry_count: 86400 master_bind: last_io_error_timestamp: last_sql_error_timestamp: master_ssl_crl: master_ssl_crlpath: retrieved_gtid_set: bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:2-973166 executed_gtid_set: bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:2-973166, ea75c885-c2c5-11e3-b8ee-5cf3fcfc9640:1-1370 auto_position: 1 this clearly suggests that the slave io_thread is lagging and obviously because of that the slave sql_thread is lagging, too, and it yields replication delays. as you can see the master log file is mysql-bin.018196 (file parameter from master status) and slave io_thread is on mysql-bin.018192 ( master_log_file from slave status ) which indicates slave io_thread is reading from that file, while on master it’s writing on mysql-bin.018196 , so the slave io_thread is behind by 4 binlogs. meanwhile, the slave sql_thread is reading from same file i.e. mysql-bin.01819 2 (relay_master_log_file from slave status) this indicates that the slave sql_thread is applying events fast enough, but it’s lagging too, which can be observed from the difference between read_master_log_pos & exec_master_log_pos from show slave status output. you can calculate slave sql_thread lag from read_master_log_pos – exec_master_log_pos in general as long as master_log_file parameter output from show slave status and relay_master_log_file parameter from show slave status output are the same. this will give you rough idea how fast slave sql_thread is applying events. as i mentioned above, the slave io_thread is lagging as in this example then off course slave sql_thread is behind too. you can read detailed description of show slave status output fields here. also, the seconds_behind_master parameter shows a huge delay in seconds. however, this can be misleading, because it only measures the difference between the timestamps of the relay log most recently executed, versus the relay log entry most recently downloaded by the io_thread. if there are more binlogs on the master, the slave doesn’t figure them into the calculation of seconds_behind_master. you can get a more accurate measure of slave lag using pt-heartbeat from percona toolkit. so, we learned how to check replication delays – either it’s slave io_thread or slave sql_thread. now, let me provide some tips and suggestions for what exactly causing this delay. tips and suggestions what causing replication delay & possible fixes usually, the slave io_thread is behind because of slow network between master/slave. most of the time, enabling slave_compressed_protocol helps to mitigate slave io_thread lag. one other suggestion is to disable binary logging on slave as it’s io intensive too unless you required it for point in time recovery. to minimize slave sql_thread lag, focus on query optimization. my recommendation is to enable the configuration option log_slow_slave_statements so that the queries executed by slave that take more than long_query_time will be logged to the slow log. to gather more information about query performance, i would also recommend setting the configuration option log_slow_verbosity to “full”. this way we can see if there are queries executed by slave sql_thread that are taking long time to complete. you can follow my previous post about how to enable slow query log for specific time period with mentioned options here . and as a reminder, log_slow_slave_statements as variable were first introduced in percona server 5.1 which is now part of vanilla mysql from version 5.6.11 in upstream version of mysql server log_slow_slave_statements were introduced as command line option. details can be found here while log_slow_verbosity is percona server specific feature. one another reason for delay on slave sql_thread if you use row based binlog format is that if your any database table missing primary key or unique key then it will scan all rows of the table for dml on slave and causes replication delays so make sure all your tables should have primary key or unique key. check this bug report for details http://bugs.mysql.com/bug.php?id=53375 you can use below query on slave to identify which of database tables missing primary or unique key. mysql> select t.table_schema,t.table_name,engine from information_schema.tables t inner join information_schema .columns c on t.table_schema=c.table_schema and t.table_name=c.table_name group by t.table_schema,t.table_name having sum(if(column_key in ('pri','uni'), 1,0)) =0; one improvement is made for this case in mysql 5.6, where in memory hash is used slave_rows_search_algorithms comes to the rescue. note that seconds_behind_master is not updated while we read huge rbr event, so, “lagging” may be related to just that – we had not completed reading of the event. for example, in row based replication huge transactions may cause delay on slave side e.g. if you have 10 million rows table and you do “delete from table where id < 5000000″ 5m rows will be sent to slave, each row separately which will be painfully slow. so, if you have to delete oldest rows time to time from huge table using partitioning might be good alternative for this for some kind of workloads where instead using delete use drop old partition may be good and only statement is replicated because it will be ddl operation. to explain it better, let suppose you have partition1 holding rows of id’s from 1 to 1000000 , partition2 – id’s from 1000001 to 2000000 and so on so instead of deleting via statement “delete from table where id<=1000000;” you can do “alter table drop partition1;” instead. for alter partitioning operations check manual – check this wonderful post too from my colleague roman explaining possible grounds for replication delays here pt-stalk is one of finest tool from percona toolkit which collects diagnostics data when problems occur. you can setup pt-stalk as follows so whenever there is a slave lag it can log diagnostic information which we can be later analyze to check to see what exactly causing the lag. here is how you can setup pt-stalk so that it captures diagnostic data when there is slave lag: ------- pt-plug.sh contents #!/bin/bash trg_plugin() { mysqladmin $ext_argv ping &> /dev/null mysqld_alive=$? if [[ $mysqld_alive == 0 ]] then seconds_behind_master=$(mysql $ext_argv -e "show slave status" --vertical | grep seconds_behind_master | awk '{print $2}') echo $seconds_behind_master else echo 1 fi } # uncomment below to test that trg_plugin function works as expected #trg_plugin ------- -- that's the pt-plug.sh file you would need to create and then use it as below with pt-stalk: $ /usr/bin/pt-stalk --function=/root/pt-plug.sh --variable=seconds_behind_master --threshold=300 --cycles=60 --notify-by-email=muhammad@example.com --log=/root/pt-stalk.log --pid=/root/pt-stalk.pid --daemonize you can adjust the threshold, currently its 300 seconds, combining that with –cycles, it means that if seconds_behind_master value is >= 300 for 60 seconds or more then pt-stalk will start capturing data. adding –notify-by-email option will notify via email when pt-stalk captures data. you can adjust the pt-stalk thresholds accordingly so that’s how it triggers to collect diagnostic data during problem. conclusion a lagging slave is a tricky problem but a common issue in mysql replication. i’ve tried to cover most aspects of replication delays in this post. please share in the comments section if you know of any other reasons for replication delay.

So when I heard about a new feature in Java 8 that was supposed to help mitigate bugs, I was excited.

Tony Hoare called the invention of the null reference the “billion dollars mistake”. May be the use of primitives in Java could be called the million dollars mistake. Primitives where created for one reason: performance. Primitives have nothing to do in an Object language. Introduction of auto boxing/unboxing was a good thing, but much more should have been done. It probably will be done (it is sometimes said to be on the Java 10 road map). In the meanwhile, we have to deal with primitives, and this is a hassle, specially when using functions. Functions in Java 5/6/7 Before Java 8, one could create functions like this: public interface Function { U apply(T t); } Function addTax = new Function() { @Override public Integer apply(Integer x) { return x / 100 * (100 + 10); } }; System.out.println(addTax.apply(100)); This code produces the following result: 110 What Java 8 gives us is the Function interface and the lambda syntax. We do not need anymore to define our own functional interface, and we may use the following syntax: Function addTax = x -> x / 100 * (100 + 10); System.out.println(addTax.apply(100)); Note that in the first example, we used an anonymous class to create a named function. In the second example, using the lambda syntax does not change anything about this. There is still an anonymous class, and a named function. One interesting question is “What is the type of x?” The type was manifest in the first example. Here, it is inferred because of the type of the function. Java knows the function argument type is an Integer because the type of the function is explicitly Function. The first Integer is the type of the argument, and the second Integer is the return type. Boxing is automatically used to convert int to Integer and back as needed. More on this later. Could we use an anonymous function? Yes, but we would have a problem with type. This does not work: System.out.println((x -> x / 100 * (100 + 10)).apply(100)); which means we can't substitute the identifier addTax with its value (the addTax function). We have to restore the type information that is now missing because Java 8 is simply not able to infer the type in this case. The most visible thing which has no explicit type here is the identifier x. So we might try: System.out.println((Integer x) -> x / 100 * 100 + 10).apply(100)); After all, int the first example, we could have written: Function addTax = (Integer x) -> x / 100 * 100 + 10; so it should be enough for Java to infer the type. But this does not work. What we have to do is specifying the type of the function. Specifying the type of its argument is not enough, even if the return type may be inferred. And there is a serious reason for this: Java 8 does not know anything about functions. Functions are ordinary object with ordinary methods that we may call. Nothing more. So we have to specify the type like this: System.out.println(((Function) x -> x / 100 * 100 + 10).apply(100)); Otherwise, it could translate to: System.out.println(((Whatever) x -> x / 100 * 100 + 10).whatever(100)); So the lambda is only syntactic sugar to simplify the Function (or Whatever) interface implementation by an anonymous class. It has in fact absolutely nothing to do with functions. Should Java had only the Function interface with its apply method, this would not be a big deal. But what about primitives? The Function interface would be fine if Java was an object language. But it is not. It is only vaguely oriented toward the use of objects (hence the name Object Oriented). The most important types in Java are the primitives. And primitives do not fit well in OOP. Auto boxing has been introduced in Java 5 to help us deal with this problem, but auto boxing as severe limitations in terms of performance, and this is related to how thing are evaluated in Java. Java is a strict language, so eager evaluation is the rule. The consequence is that each time we have a primitive and need an object, the primitive has to be boxed. And each time we have an object and need a primitive, it has to be unboxed. If we rely upon automatic boxing an unboxing, we may end with much overhead for multiple boxing and unboxing. Other languages have solved this problem differently, allowing only objects and dealing with conversion in the background. They may have “value classes”, which are objects that are backed with primitives. With this functionality, programmers only use objects and the compiler only use primitives (this is over simplified, but it gives an idea of the principle). By allowing programmers to explicitly manipulate primitives, Java makes things much more difficult and much less safe, because programmers are encouraged to use primitives as business types, which is total nonsense either in OOP or in FP. (I will come back to this in another article.) Let's say it abruptly: we should not care about the overhead of boxing and unboxing. If Java programs using this feature are too slow, the language should be fixed. We should not use bad programming techniques to work around language weaknesses. By using primitives, we make the language work against us, and not for us. If this problem is not solved through fixing the language, we should just use another language. But we probably can't for lot of bad reasons, the most important being that we are payed to program in Java and not in any other language. The result is that instead of solving business problems, we find ourselves solving Java problems. And using primitives is a Java problem, and a big one. Lets rewrite our example using primitives instead of objects. Our function takes an argument of type Integer and returns an Integer. To replace this, Java has the type IntUnaryOperator. Wow, this smells! And guess what, it is defined as: public interface IntUnaryOperator { int applyAsInt(int operand); ... } It would probably have been too simple to call the method apply. So, our example using primitives may be rewritten as: IntUnaryOperator addTax = x -> x / 100 * (100 + 10); System.out.println(addTax.applyAsInt(100)); or, using an anonymous function: System.out.println(((IntUnaryOperator) x -> x / 100 * (100 + 10)).applyAsInt(100)); If only for functions of int returning int, this would be simple. But it is much more complex. Java 8 has 43 (functional) interfaces in the java.util.function package. In reality, they do not all represent functions. They can be grouped as follows: 21 one argument functions, among which 2 are functions of object returning object and 19 are various cases of object to primitive and primitive to object functions. One of the two object to object functions is for the specific case when both argument and return value are of the same type. 9 two arguments functions, among which 2 are functions of (object, object) to object, and 7 are various cases of (object, object) to primitive or (primitive, primitive) to primitive. 7 are effects, and not functions, since they do not return any value and are supposed to be used only for their side effect. (It's somewhat strange to call these “functional interfaces”.) 5 are “suppliers”, which means functions that do not take an argument but return a value. These could be functions. In the functional world, these are special functions called nullary functions (to indicate that their arity, or number of arguments, is zero). As functions, their return value may never change, so they allow treating constants as functions. In Java 8, their role is to depend upon mutable context to return variable values. So, they are not functions. What a mess! And furthermore, the methods of these interfaces have different names. Object functions have a method named apply, where methods returning numeric primitives have method name applyAsInt, applyAsLong, or applyAsDouble. Functions returning boolean have a method called test, and suppliers have methods called get, or getAsInt, getAsLong, getAsDouble, or getAsBoolean. (They did not dare calling BooleanSupplier “Predicate” with a test method taking no argument. I really wonder why!) One thing to note is that there are no functions for byte, char, short and float. Nor are there functions for arity greater that two. Needless to say, this is totally ridiculous. But we have to stick with it. As long as Java can infer the type, we may think we have no problem. However, if you want to manipulate functions in a functional way, you will soon face the problem of Java being unable to infer a type. Worst, Java will sometime infer the type and stay silent while using a type which is no the one you intended. How to help discovering the right type Let's say we want to use a three arguments function. As there are no such functional interfaces in Java 8, you are left with a choice: create you own functional interface, or use currying, as we have seen in a previous article (What's wrong with Java 8 part I ). Creating a three object arguments functional interface returning object is straightforward: interface Function { R apply(T, t, U, u, V, v); } However, we may face two problems. The first one is that we may need to process primitives. Parametric types will not help us for this. You may create special versions of the function using primitives instead of objects. After all, with eight type of primitives, three arguments and one return value, there are only 6 561 different versions of this function. Why do you think Oracle did not put TriFunction in Java 8? (To be precise, they only put a very limited number of BiFunction where arguments are Object and return type int, long or double, or when argument and return types are of the same type int, long or Object, leading to a total of 9 out of 729 possible.) A much better solution is to use autoboxing. Just use Integer, Long, Boolean and so on and let Java handle this. Doing whatever else would be the root of all evil, i.e. premature optimization (see http://c2.com/cgi/wiki?PrematureOptimization). Another way to go (beside creating three arguments functional interface) is to use currying. This is mandatory if the arguments may not be evaluated at the same time. Furthermore, it allows using only functions of one argument, which limits the number of possible functions to 81. If we restrict ourselves to boolean, int, long and double, the number falls to 25 (four primitive types plus Object in two places equals 5 x 5). The problem is that it may be somewhat difficult to use currying with functions returning primitives or taking primitives as their argument. As an example, here is the same example used in our previous article (What's wrong with Java 8 part I ), but using primitives: IntFunction> intToIntCalculation = x -> y -> z -> x + y * z; private IntStream calculate(IntStream stream, int a) { return stream.map(intToIntCalculation.apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); Note that the result is not “a stream containing the values 5, 8, 11, 14 and 17”, no more than the initial stream would have contained the value 1, 2, 3, 4 and 5. newStream in not evaluated at this stage, so it does not contain values. (We'll talk about this in a next article). To see the result, we have to evaluate the stream, which may be forced by binding it to a terminal operation. This may be done through a call to the collect method. But before doing this, we will bind the result to one more non terminal function using the method boxed. The boxed methods binds to the stream a function converting primitives to the corresponding objects. This will simplify evaluation: System.out.println(newStream.boxed().collect(toList())); This prints: [5, 8, 11, 14, 17] We could as well use an anonymous function. However, Java is not be able to infer the type, so we must help it: private IntStream calculate(IntStream stream, int a) { return stream.map(((IntFunction>) x -> y -> z -> x + y * z).apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); Currying in itself is very easy. Just remember, as I said in a previous article, that: (x, y, z) -> w translates to x -> y -> z -> w Finding the right type is slightly more complicated. You have to remember that each time you apply an argument, you are returning a function, so you need a function from the type of the argument to an object type (because functions are objects). Here, each argument is of type int, so we need to use IntFunction parameterized with the type of the returned function. As the final type is IntUnaryOperator (as required by the map method of the IntStream class), the result is: IntFunction>> Here, we are applying two of the three parameters and all parameters are of type int, so the type is: IntFunction> This may be compared to the version using autoboxing: Function>> If you have problems determining the right type, start with the version using autoboxing, just replacing the final type you know you need (since it is the type of the argument of map): Function> Note that you may perfectly use this type in your program: private IntStream calculate(IntStream stream, int a) { return stream.map(((Function>) x -> y -> z -> x + y * z).apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); You may then replace each Function>) x -> y -> z -> x + y * z).apply(b).apply(a)); } and then to: private IntStream calculate(IntStream stream, int a) { return stream.map(((IntFunction>) x -> y -> z -> x + y * z).apply(b).apply(a)); } Note that all three versions compile and run. The only difference is whether autoboxing is used or not. When to be anonymous So, as we saw in the examples above, lambdas are very good at simplifying anonymous class creation, but there is rarely good reason not to name the instance that is created. Naming functions allows: function reuse function testing function replacement program maintenance program documentation Naming function plus currying will make your function completely independent from the environment (“referential transparency”), making you programs safer and more modular. There is however a difficulty. Using primitives makes it difficult to figure the type of curried function. And worst, primitive are not the right business types to use, so the compiler will not be able to help you in this area. To see why, look at this example: double tax = 10.24; double limit = 500.0; double delivery = 35.50; DoubleStream stream3 = DoubleStream.of(234.23, 567.45, 344.12, 765.00); DoubleStream stream4 = stream3.map(x -> { double total = x / 100 * (100 + tax); if ( total > limit) { total = total + delivery; } return total; }); To replace the anonymous “capturing” function by a named curried one, determining the correct type is not so difficult. There will be four arguments and it will return a DoubleUnaryOperator, so the type will be DoubleFunction>>. However, it is very easy to misplace the arguments: DoubleFunction>> computeTotal = x -> y -> z -> w -> { double total = w / 100 * (100 + x); if (total > y) { total = total + z; } return total; }; DoubleStream stream2 = stream.map(computeTotal.apply(tax).apply(limit).apply(delivery)); How can you be sure what x, y, z and w are ? There is in fact a simple rule: the arguments that are evaluated through the explicit use of the apply method come first, in the order they are applied, i.e. tax, limit, delivery, corresponding to x, y and z. The argument coming from the stream is applied last, so it corresponds to w. However, we are still having a problem: once the function is tested, we now that it is correct, but there is no way to be sure it will be used right. For example if we apply the parameters in the wrong order: DoubleStream stream2 = stream.map(computeTotal.apply(limit).apply(tax).apply(delivery)); we get [1440.8799999999999, 3440.2000000000003, 2100.2200000000003, 4625.5] instead of: [258.215152, 661.05688, 379.357888, 878.836] This means we have to test not only the function, but each use of it. Wouldn't it be nice if we could be sure that using the parameters in the wrong order would not compile? This is what using the right type system is about. Using primitives for business types is not good. It has never be. But now, with functions, we have one more reason not to do this. This will be the subject of another article. What's next We have seen how using primitives is somewhat more complicated that using objects. Functions using primitives are a real mess in Java 8. But the worst is to come. In a next article, we will talk about using primitives with streams.

Earlier I had blogged about using Scala with Spring Boot and how the combination just works. There was one issue with the previous approach though - the only way to run the earlier configuration was to build the project into a jar file and run the jar file. ./gradlew build java -jar build/libs/spring-boot-scala-web-0.1.0.jar Spring boot comes with a gradle based plugin which should have allowed the project to run with a "gradle bootRun" command, this unfortunately gives an error for scala based projects. A good workaround is to use sbt for building and running Spring-boot based projects. The catch though is that with gradle and maven, the versions of the dependencies would have been managed through a parent pom, now these have to be explicitly specified. This is how a sample sbt build file with the dependencies spelled out looks: name := "spring-boot-scala-web" version := "1.0" scalaVersion := "2.10.4" sbtVersion := "0.13.1" seq(webSettings : _*) libraryDependencies ++= Seq( "org.springframework.boot" % "spring-boot-starter-web" % "1.0.2.RELEASE", "org.springframework.boot" % "spring-boot-starter-data-jpa" % "1.0.2.RELEASE", "org.webjars" % "bootstrap" % "3.1.1", "org.webjars" % "jquery" % "2.1.0-2", "org.thymeleaf" % "thymeleaf-spring4" % "2.1.2.RELEASE", "org.hibernate" % "hibernate-validator" % "5.0.2.Final", "nz.net.ultraq.thymeleaf" % "thymeleaf-layout-dialect" % "1.2.1", "org.hsqldb" % "hsqldb" % "2.3.1", "org.springframework.boot" % "spring-boot-starter-tomcat" % "1.0.2.RELEASE" % "provided", "javax.servlet" % "javax.servlet-api" % "3.0.1" % "provided" ) libraryDependencies ++= Seq( "org.apache.tomcat.embed" % "tomcat-embed-core" % "7.0.53" % "container", "org.apache.tomcat.embed" % "tomcat-embed-logging-juli" % "7.0.53" % "container", "org.apache.tomcat.embed" % "tomcat-embed-jasper" % "7.0.53" % "container" ) Here I am also using xsbt-web-plugin which is plugin for building scala web applications. xsbt-web-plugin also comes with commands to start-up tomcat or jetty based containers and run the applications within these containers, however I had difficulty in getting these to work. What worked is the runMain command to start up the Spring-boot main program through sbt: runMain mvctest.SampleWebApplication and xsbt-web-plugin allows the project to be packaged as a war file using the "package" command, this war deploys and runs without any issues in a standalone tomcat container. Here is a github project with these changes: https://github.com/bijukunjummen/spring-boot-scala-web.git

Learn all about Jersey/Jax RS streaming with JSON.

System Requirements: Eclipse Kepler Service Release 2 JDK 1.5 or above Installed Google App Engine SDK on eclipse – this is required for second version of this example. Create a google spreadsheet – login to your google account and create a new spreadsheet, if you want to read existing one then put that url in SPREADSHEET_URL. Once you create a new spreadsheet our url will be like – https://docs.google.com/spreadsheets/d/1L8xtAJfOObsXL-XemliUV10wkDHQNxjn6jKS4XwzYZ8/ but don’t put this url in SPREADSHEET_URL, Use the below url simply change the bold part. public static final String SPREADSHEET_URL = “https://spreadsheets.google.com/feeds/spreadsheets/1L8xtAJfOObsXL-XemliUV10wkDHQNxjn6jKS4XwzYZ8“; // Fill in google spreadsheet URI package org.gopaldas.readsps; import java.io.IOException; import java.net.URL; import com.google.gdata.client.spreadsheet.SpreadsheetService; import com.google.gdata.data.spreadsheet.ListEntry; import com.google.gdata.data.spreadsheet.ListFeed; import com.google.gdata.data.spreadsheet.SpreadsheetEntry; import com.google.gdata.data.spreadsheet.WorksheetEntry; import com.google.gdata.util.ServiceException; public class ReadSpreadsheet { public static final String GOOGLE_ACCOUNT_USERNAME = "xxxx@gmail.com"; // Fill in google account username public static final String GOOGLE_ACCOUNT_PASSWORD = "xxxx"; // Fill in google account password public static final String SPREADSHEET_URL = "https://spreadsheets.google.com/feeds/spreadsheets/1L8xtAJfOObsXL-XemliUV10wkDHQNxjn6jKS4XwzYZ8"; //Fill in google spreadsheet URI public static void main(String[] args) throws IOException, ServiceException { /** Our view of Google Spreadsheets as an authenticated Google user. */ SpreadsheetService service = new SpreadsheetService("Print Google Spreadsheet Demo"); // Login and prompt the user to pick a sheet to use. service.setUserCredentials(GOOGLE_ACCOUNT_USERNAME, GOOGLE_ACCOUNT_PASSWORD); // Load sheet URL metafeedUrl = new URL(SPREADSHEET_URL); SpreadsheetEntry spreadsheet = service.getEntry(metafeedUrl, SpreadsheetEntry.class); URL listFeedUrl = ((WorksheetEntry) spreadsheet.getWorksheets().get(0)).getListFeedUrl(); // Print entries ListFeed feed = (ListFeed) service.getFeed(listFeedUrl, ListFeed.class); for(ListEntry entry : feed.getEntries()) { System.out.println("new row"); for(String tag : entry.getCustomElements().getTags()) { System.out.println(" "+tag + ": " + entry.getCustomElements().getValue(tag)); } } } }

wanted to go through some of the basic tenets, the technical terminology related to java ee. for many people, java ee/j2ee still mean servlets, jsps or maybe struts at best. no offence or pun intended! this is not a java ee 'bible' by any means. i am not capable enough of writing such a thing! so let us line up the 'keywords' related to java ee and then look at them one by one java ee java ee apis (specifications) containers services multitiered applications components let's try to elaborate on the above mentioned points. ok. so what is java ee? 'ee' stands for enterprise edition. that essentially makes java ee - java enterprise edition. if i had to summarize java ee in a couple of sentences, it would go something like this "java ee is a platform which defines 'standard specifications/apis' which are then implemented by vendors and used for development of enterprise (distributed, 'multi-tired', robust) 'applications'. these applications are composed of modules or 'components' which use java ee 'containers' as their run-time infrastructure." what is this 'standardized platform' based upon? what does it constitute? the platform revolves around 'standard' specifications or apis . think of these as contracts defined by a standard body e.g. enterprise java beans (ejb), java persistence api (jpa), java message service (jms) etc. these contracts/specifications/apis are implemented by different vendors e.g. glassfish, oracle weblogic, apache tomee etc alright. what about containers? containers can be visualized as 'virtual/logical partitions' . each container supports a subset of the apis/specifications defined by the java ee platform they provide run-time 'services' to the 'applications' which they host the java ee specification lists 4 types of containers ejb container web container application client container applet container java ee containers i am not going to dwell into details of these containers in this post. services?? well, 'services' are nothing but a result of the vendor implementations of the standard 'specifications' (mentioned above). examples of specifications are - jersey for jax-rs (restful services), tyrus (web sockets), eclipselink (jpa), weld (cdi) etc. the 'container' is the interface between the deployed application ('service' consumer) and the application server. here is a list of 'services' which are rendered by the 'container' to the underlying 'components' (this is not an exhaustive list) persistence - offered by the java persistence api (jpa) which drives object relational mapping (orm) and an abstraction for the database operations. messaging - the java message service (jms) provides asynchronous messaging between disparate parts of your applications. contexts & dependency injection - cdi provides loosely coupled and type safe injection of resources. web services - jaxrs and jaxws provide support for rest and soap style services respectively transaction - provided by the java transaction api (jta) implementation what is a typical java ee 'application'? what does it comprise of? applications are composed of different ' components ' which in turn are supported by their corresponding ' container ' supported 'component' types are: enterprise applications - make use of the specifications like ejb, jms, jpa etc and are executed within an ejb container web applications - they leverage the servlet api, jsp, jsf etc and are supported by a web container application client - executed in client side. they need an application client container which has a set of supported libraries and executes in a java se environment. applets - these are gui applications which execute in a web browser. how are java ee applications structured? as far as java ee 'application' architecture is concerned, they generally tend follow the n-tier model consisting of client tier, server tier and of course the database (back end) tier client tier - consists of web browsers or gui (swing, java fx) based clients. web browsers tend to talk to the 'web components' on the server tier while the gui clients interact directly with the 'business' layer within the server tier server tier - this tier comprises of the dynamic web components (jsp, jsf, servlets) and the business layer driven by ejbs, jms, jpa, jta specifications. database tier - contains 'enterprise information systems' backed by databases or even legacy data repositories. generic 3-tier java ee application architecture java ee - bare bones, basics.... as quickly and briefly as i possibly could. that's all for now! :-) stay tuned for more java ee content, specifically around the latest and greatest version of the java ee platform --> java ee 7 happy reading!

splunk vs. sumo logic vs. logstash vs. graylog vs. loggly vs. papertrails vs. splunk>storm splunk, sumo logic, logstash, graylog, loggly, papertrails - did i miss someone? i’m pretty sure i did. logs are like fossil fuels - we’ve been wanting to get rid of them for the past 20 years, but we’re not quite there yet. well, if that's the case i want a bmw! to deal with the growth of log data a host of log management & analysis tools have been built over the last few years to help developers and operations make sense of the growing data. i thought it’d be interesting to look at our options and what are each tools’ selling point, from a developer’s standpoint . splunk as the biggest tool in this space, i decided to put splunk in a category of its own. that’s not to say it’s the best tool for what you need, but more to give credit to a product who essentially created a new category. pros splunk is probably the most feature rich solution in the space. it’s got hundreds of apps (i counted 537 ) to make sense of almost every format of log data, from security to business analytics to infrastructure monitoring. splunk’s search and charting tools are feature rich to the point that there’s probably no set of data you can’t get to through its ui or apis. cons splunk has two major cons. the first, that is more subjective, is that it’s an on-premise solution which means that setup costs in terms of money and complexity are high. to deploy in a high-scale environment you will need to install and configure a dedicated cluster. as a developer, it’s usually something you can't or don’t want to do as your first choice. splunk’s second con is that it’s expensive. to support a real-world application you’re looking at tens of thousands of dollars, which most likely means you’ll need sign offs from high-ups in your organization, and the process is going to be slow. if you’ve got a new app and you want something fast that you can quickly spin up and ramp as things progress - keep reading. some more enterprise log analyzers can be found here . saas log analyzers sumo logic sumo was founded as a saas version of splunk, going so far as to imitate some of splunk’s features and visuals early on. having said that, sl has developed to a full fledged enterprise class log management solution. pros sl is chock-full of features to reduce, search and chart mass amounts of data. out of all the saas log analyzers, it’s probably the most feature rich. also, being a saas offering it inherently means setup and ongoing operation are easier. one of sumo logic’s main points of attraction is the ability to establish baselines and to actively notify you when key metrics change after an event such as a new version rollout or a breach attempt. cons this one is shared across all saas log analyzers, which is you need to get the data to the service to actually do something with it. this means that you’ll be looking at possible gbs (or more) uploaded from your servers. this can create issues on multiple fronts - as a developer, if you're logging sensitive or pii you need to make sure it’s redacted. there may be a lag between the time data is logged and the time it’s visible to to the service. there’s additional overhead on your machines transmitting gbs of data, which really depends on your logging throughput. sumo’s pricing is also not transparent , which means you might be looking at a buying process which is more complex than swiping your team’s credit card to get going. loggly loggly is also a robust log analyzer, focusing on simplicity and ease of use for a devops audience. pros whereas sumo logic has a strong enterprise and security focus, loggly is geared more towards helping devops find and fix operational problems. this makes it very developer-friendly. things like creating custom performance and devops dashboards are super-easy to do. pricing is also transparent, which makes start of use easier. cons don't expect loggly to scale into a full blown infrastructure, security or analytics solution. if you need forensics or infrastructure monitoring you’re in the wrong place. this is a tools mainly for devops to parse data coming from your app servers. anything beyond that you’ll have to build yourself. papertrails papertrails is a simple way to look and search through logs from multiple machines, in one consolidated easy-to-use interface. think of it like tailing your log in the cloud, and you won't be too far off. pros pt is what it is. a simple way to look at log files from multiple machines in a singular view in the cloud. the ux itself is very similar to looking at a log on your machine, and so are the search commands. it aims to do something simple and useful, and does it elegantly. it’s also very affordable . cons pt is mostly text based. looking for any advanced integrations, predictive or reporting capabilities? you're barking up the wrong tree. splunk>storm this is splunk’s little (some may say step) saas brother. it’s a pretty similar offering that’s hosted on splunk’s servers. pros storm lets you experiment with splunk without having to install the actual software on-premise, and contains much of the features available in the full version. cons this isn't really a commercial offering, and you're limited in the amount of data you can send. it seems to be more of an online limited version of splunk meant to help people test out the product without having to deploy first. a new service called splunk cloud is aimed at providing a full-blown splunk saas experience. open source analyzers logstash logstash is an open source tool for collecting and managing log files. it’s part of an open-source stack which includes elasticsearch for indexing and searching through data and kibana for charting and visualizing data. together they form a powerful log management solution. pros being an open-source solution means you're inherently getting a lot of a control and a very good price. logstash uses three mature and powerful components, all heavily maintained, to create a very robust and extensible package. for an open-source solution it’s also very easy to install and start using. we use logstash and love it. cons as logstash is essentially a stack, it means you're dealing with three different products. that means that extensibility also becomes complex. logstash filters are written in ruby, kibana is pure javascript and elasticsearch has its own rest api as well as json templates. when you move to production, you’ll also need to separate the three into different machines, which adds to the complexity. graylog2 a fairly new player in the space, gl2 is an open-source log analyzer backed by mongodb as well as elasticsearch (similar to logstash) for storing and searching through log errors. it’s mainly focused on helping developers detect and fix errors in their apps. also in this category you can find fluentd and kafka whose one of its main use-cases is also storing log data. phew, so many choices! takipi for logs while this post is not about takipi, i thought there’s one feature it has which you might find relevant to all of this. the biggest disadvantage in all log analyzers and log files in general, is that the right data has to be put there by you first. from a dev perspective, it means that if an exception isn’t logged, or the variable data you need to understand why it happened isn't there, no log file or analyzer in the world can help you. production debugging sucks. one of the things we’ve added to takipi is the ability to jump into a recorded debugging session straight from a log file error. this means that for every log error you can see the actual source code and variable values at the moment of error. you can learn more about it here . this is one post where i would love to hear from you guys about your experiences with some of the tools mentioned (and some that i didn’t). i’m sure there are things you would disagree with or would like to correct me on - so go ahead, the comment section is below and i would love to hear from you. originally posted on takipi blog

Get a rundown on the Tomcat NIO Connector as well as a tutorial on how to set it up.

Running Groovy scripts with GroovyShell is easy. We can for example incorporate a Domain Specific Language (DSL) in our application where the DSL is expressed in Groovy code and executed by GroovyShell. To limit the constructs that can be used in the DSL (which is Groovy code) we can apply a SecureASTCustomizer to the GroovyShell configuration. With the SecureASTCustomizer the Abstract Syntax Tree (AST) is inspected, we cannot define runtime checks here. We can for example disallow the definition of closures and methods in the DSL script. Or we can limit the tokens to be used to just a plus or minus token. To have even more control we can implement the StatementChecker andExpressionChecker interface to determine if a specific statement or expression is allowed or not. In the following sample we first use the properties of the SecureASTCustomizer class to define what is possible and not within the script: package com.mrhaki.blog import org.codehaus.groovy.control.customizers.SecureASTCustomizer import org.codehaus.groovy.control.CompilerConfiguration import org.codehaus.groovy.ast.stmt.* import org.codehaus.groovy.ast.expr.* import org.codehaus.groovy.control.MultipleCompilationErrorsException import static org.codehaus.groovy.syntax.Types.PLUS import static org.codehaus.groovy.syntax.Types.MINUS import static org.codehaus.groovy.syntax.Types.EQUAL // Define SecureASTCustomizer to limit allowed // language syntax in scripts. final SecureASTCustomizer astCustomizer = new SecureASTCustomizer( // Do not allow method creation. methodDefinitionAllowed: false, // Do not allow closure creation. closuresAllowed: false, // No package allowed. packageAllowed: false, // White or blacklists for imports. importsBlacklist: ['java.util.Date'], // or importsWhitelist staticImportsWhitelist: [], // or staticImportBlacklist staticStarImportsWhitelist: [], // or staticStarImportsBlacklist // Make sure indirect imports are restricted. indirectImportCheckEnabled: true, // Only allow plus and minus tokens. tokensWhitelist: [PLUS, MINUS, EQUAL], // or tokensBlacklist // Disallow constant types. constantTypesClassesWhiteList: [Integer, Object, String], // or constantTypesWhiteList // or constantTypesBlackList // or constantTypesClassesBlackList // Restrict method calls to whitelisted classes. // receiversClassesWhiteList: [], // or receiversWhiteList // or receiversClassesBlackList // or receiversBlackList // Ignore certain language statement by // whitelisting or blacklisting them. statementsBlacklist: [IfStatement], // or statementsWhitelist // Ignore certain language expressions by // whitelisting or blacklisting them. expressionsBlacklist: [MethodCallExpression] // or expresionsWhitelist ) // Add SecureASTCustomizer to configuration for shell. final conf = new CompilerConfiguration() conf.addCompilationCustomizers(astCustomizer) // Create shell with given configuration. final shell = new GroovyShell(conf) // All valid script. final result = shell.evaluate ''' def s1 = 'Groovy' def s2 = 'rocks' "$s1 $s2!" ''' assert result == 'Groovy rocks!' // Some invalid scripts. try { // Importing [java.util.Date] is not allowed shell.evaluate ''' new Date() ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('Indirect import checks prevents usage of expression') } try { // MethodCallExpression not allowed shell.evaluate ''' println "Groovy rocks!" ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('MethodCallExpressions are not allowed: this.println(Groovy rocks!)') } To have more fine-grained control on which statements and expression are allowed we can implement the StatementChecker andExpressionChecker interfaces. These interfaces have one method isAuthorized with a boolean return type. We return true if a statement or expression is allowed and false if not. package com.mrhaki.blog import org.codehaus.groovy.control.customizers.SecureASTCustomizer import org.codehaus.groovy.control.CompilerConfiguration import org.codehaus.groovy.ast.stmt.* import org.codehaus.groovy.ast.expr.* import org.codehaus.groovy.control.MultipleCompilationErrorsException import static org.codehaus.groovy.control.customizers.SecureASTCustomizer.ExpressionChecker import static org.codehaus.groovy.control.customizers.SecureASTCustomizer.StatementChecker // Define SecureASTCustomizer. final SecureASTCustomizer astCustomizer = new SecureASTCustomizer() // Define expression checker to deny // usage of variable names with length of 1. def smallVariableNames = { expr -> if (expr instanceof VariableExpression) { expr.variable.size() > 1 } else { true } } as ExpressionChecker astCustomizer.addExpressionCheckers smallVariableNames // In for loops the collection name // can only be 'names'. def forCollectionNames = { statement -> if (statement instanceof ForStatement) { statement.collectionExpression.variable == 'names' } else { true } } as StatementChecker astCustomizer.addStatementCheckers forCollectionNames // Add SecureASTCustomizer to configuration for shell. final CompilerConfiguration conf = new CompilerConfiguration() conf.addCompilationCustomizers(astCustomizer) // Create shell with given configuration. final GroovyShell shell = new GroovyShell(conf) // All valid script. final result = shell.evaluate ''' def names = ['Groovy', 'Grails'] for (name in names) { print "$name rocks! " } def s1 = 'Groovy' def s2 = 'rocks' "$s1 $s2!" ''' assert result == 'Groovy rocks!' // Some invalid scripts. try { // Variable s has length 1, which is not allowed. shell.evaluate ''' def s = 'Groovy rocks' s ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('Expression [VariableExpression] is not allowed: s') } try { // Only names as collection expression is allowed. shell.evaluate ''' def languages = ['Groovy', 'Grails'] for (name in languages) { println "$name rocks!" } ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('Statement [ForStatement] is not allowed') } Code written with Groovy 2.2.2.

Testing is crucial. While many different kinds and levels of testing exist, there’s good library support only for unit tests (the Python unittest package and its moral equivalents in other languages). However, unit testing does not cover all kinds of testing we may want to do – for example, all kinds of whole program tests and integration tests. This is where we usually end up with a custom "test runner" script. Having written my share of such custom test runners, I’ve recently gravitated towards a very convenient approach which I want to share here. In short, I’m actually using Python’s unittest, combined with the dynamic nature of the language, to run all kinds of tests. Let’s assume my tests are some sort of data files which have to be fed to a program. The output of the program is compared to some "expected results" file, or maybe is encoded in the data file itself in some way. The details of this are immaterial, but seasoned programmers usually encounter such testing rigs very frequently. It commonly comes up when the program under test is a data-transformation mechanism of some sort (compiler, encryptor, encoder, compressor, translator etc.) So you write a "test runner". A script that looks at some directory tree, finds all the "test files" there, runs each through the transformation, compares, reports, etc. I’m sure all these test runners share a lot of common infrastructure – I know that mine do. Why not employ Python’s existing "test runner" capabilities to do the same? Here’s a very short code snippet that can serve as a template to achieve this: import unittest class TestsContainer(unittest.TestCase): longMessage = True def make_test_function(description, a, b): def test(self): self.assertEqual(a, b, description) return test if __name__ == '__main__': testsmap = { 'foo': [1, 1], 'bar': [1, 2], 'baz': [5, 5]} for name, params in testsmap.iteritems(): test_func = make_test_function(name, params[0], params[1]) setattr(TestsContainer, 'test_{0}'.format(name), test_func) unittest.main() What happens here: The test class TestsContainer will contain dynamically generated test methods. make_test_function creates a test function (a method, to be precise) that compares its inputs. This is just a trivial template – it could do anything, or there can be multiple such "makers" fur multiple purposes. The loop creates test functions from the data description in testmap and attaches them to the test class. Keep in mind that this is a very basic example. I hope it’s obvious that testmap could really be test files found on disk, or whatever else. The main idea here is the dynamic test method creation. So what do we gain from this, you may ask? Quite a lot. unittest is powerful – armed to its teeth with useful tools for testing. You can now invoke tests from the command line, control verbosity, control "fast fail" behavior, easily filter which tests to run and which not to run, use all kinds of assertion methods for readability and reporting (why write your own smart list comparison assertions?). Moreover, you can build on top of any number of third-party tools for working with unittest results – HTML/XML reporting, logging, automatic CI integration, and so on. The possibilities are endless. One interesting variation on this theme is aiming the dynamic generation at a different testing "layer". unittest defines any number of "test cases" (classes), each with any number of "tests" (methods). In the code above, we generate a bunch of tests into a single test case. Here’s a sample invocation to see this in action: $ python dynamic_test_methods.py -v test_bar (__main__.TestsContainer) ... FAIL test_baz (__main__.TestsContainer) ... ok test_foo (__main__.TestsContainer) ... ok ====================================================================== FAIL: test_bar (__main__.TestsContainer) ---------------------------------------------------------------------- Traceback (most recent call last): File "dynamic_test_methods.py", line 8, in test self.assertEqual(a, b, description) AssertionError: 1 != 2 : bar ---------------------------------------------------------------------- Ran 3 tests in 0.001s FAILED (failures=1) As you can see, all data pairs in testmap are translated into distinctly named test methods within the single test case TestsContainer. Very easily, we can cut this a different way, by generating a whole test case for each data item: import unittest class DynamicClassBase(unittest.TestCase): longMessage = True def make_test_function(description, a, b): def test(self): self.assertEqual(a, b, description) return test if __name__ == '__main__': testsmap = { 'foo': [1, 1], 'bar': [1, 2], 'baz': [5, 5]} for name, params in testsmap.iteritems(): test_func = make_test_function(name, params[0], params[1]) klassname = 'Test_{0}'.format(name) globals()[klassname] = type(klassname, (DynamicClassBase,), {'test_gen_{0}'.format(name): test_func}) unittest.main() Most of the code here remains the same. The difference is in the lines within the loop: now instead of dynamically creating test methods and attaching them to the test case, we create whole test cases – one per data item, with a single test method. All test cases derive from DynamicClassBase and hence from unittest.TestCase, so they will be auto-discovered by the unittest machinery. Now an execution will look like this: $ python dynamic_test_classes.py -v test_gen_bar (__main__.Test_bar) ... FAIL test_gen_baz (__main__.Test_baz) ... ok test_gen_foo (__main__.Test_foo) ... ok ====================================================================== FAIL: test_gen_bar (__main__.Test_bar) ---------------------------------------------------------------------- Traceback (most recent call last): File "dynamic_test_classes.py", line 8, in test self.assertEqual(a, b, description) AssertionError: 1 != 2 : bar ---------------------------------------------------------------------- Ran 3 tests in 0.000s FAILED (failures=1) Why would you want to generate whole test cases dynamically rather than just single tests? It all depends on your specific needs, really. In general, test cases are better isolated and share less, than tests within one test case. Moreover, you may have a huge amount of tests and want to use tools that shard your tests for parallel execution – in this case you almost certainly need separate test cases. I’ve used this technique in a number of projects over the past couple of years and found it very useful; more than once, I replaced a whole complex test runner program with about 20-30 lines of code using this technique, and gained access to many more capabilities for free. Python’s built-in test discovery, reporting and running facilities are very powerful. Coupled with third-party tools they can be even more powerful. Leveraging all this power for any kind of testing, and not just unit testing, is possible with very little code, due to Python’s dynamism. I hope you find it useful too.

See how HashMap performance has been improved with new features of Java 8.

Recently needed to update the last inserted row of a table but didn't have anyway in knowing what the highest ID in the table was. I can easily do this by using the max() function to select the highest ID in the table. SELECT MAX(id) FROM table; Then I can use the result of this query in the UPDATE query to edit the record with the highest ID. But this is quite a easy query so I should be able to do this in one query by using a nested select query on the UPDATE. UPDATE table SET name='test_name' WHERE id = (SELECT max(id) FROM table) But the problem with this is that the MAX() function doesn't work inside a nested select so had to find another way of doing this. I found out that you can use an ORDER BY and a LIMIT in an UPDATE query therefore I can use a combination of these in the UPDATE query to make sure I only update the record with the highest ID, by doing a descendant order on the ID and limiting the return to only 1 record. UPDATE table SET name='test_name' ORDER BY id DESC LIMIT 1;

with the recent release of java 8, developers are still just beginning to asses the strengths and weaknesses of the new platform. the most pressing question is: does java 8 have the fastest jvm so far? a good way to asses the progress of java 8 is to test its ability to work with something that was new to java 7... fork/join. oleg shelajev uses the "infamous" java microbenchmark harness project (jmh) to create a benchmark test for the two most recent versions of java. but before implementing the benchmark, he takes the time to give a brief overview of fork/join and how it changes between java 7 and 8. here is a graph of the results : based on these results, oleg recommends taking a chance and upgrading to java 8, especially if you are working with mapreducing or fork/join. this is his interpretation of the data that lead him to that conclusion: one can see that the baseline results, which show the throughput of running the math directly in a single thread do not differ between the jdk 7 and 8. however, when we include the overhead of managing recursive tasks and going through a forkjoin execution then java 8 is much faster. the numbers for this simple benchmark suggest that the overhead of managing forkjoin tasks is around 35% more performant in the latest release. check out the full article ! it is very informative, has great visuals, and tackles complexity with clarity.

Visa Card ^4[0-9]{12}(?:[0-9]{3})?$^5[1-5][0-9]{14}$ Amex Card ^3[47][0-9]{13}$ Carte Blanche Card ^389[0-9]{11}$ Diners Club Card ^3(?:0[0-5]|[68][0-9])[0-9]{11}$ Discover Card ^65[4-9][0-9]{13}|64[4-9][0-9]{13}|6011[0-9]{12}|(622(?:12[6-9]|1[3-9][0-9]|[2-8][0-9][0-9]|9[01][0-9]|92[0-5])[0-9]{10})$ JCB Card ^(?:2131|1800|35\d{3})\d{11}$ Visa Master Card ^(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14})$ Insta Payment Card ^63[7-9][0-9]{13}$ Laser Card ^(6304|6706|6709|6771)[0-9]{12,15}$ Maestro Card ^(5018|5020|5038|6304|6759|6761|6763)[0-9]{8,15}$ Solo Card ^(6334|6767)[0-9]{12}|(6334|6767)[0-9]{14}|(6334|6767)[0-9]{15}$ Switch Card ^(4903|4905|4911|4936|6333|6759)[0-9]{12}|(4903|4905|4911|4936|6333|6759)[0-9]{14}|(4903|4905|4911|4936|6333|6759)[0-9]{15}|564182[0-9]{10}|564182[0-9]{12}|564182[0-9]{13}|633110[0-9]{10}|633110[0-9]{12}|633110[0-9]{13}$ Union Pay Card ^(62[0-9]{14,17})$ KoreanLocalCard ^9[0-9]{15}$ BCGlobal ^(6541|6556)[0-9]{12}$ As you can see, regular expressions are incredibly powerful and the above examples are very basic. Regular expressions are essential for all sorts of application from web scraping, form validation and pattern matching. Hopefully you will find this resource useful and would come back to refer again and again.

Facts and Terminology As you probably know, Java uses UTF-16 to represent Strings. In order to understand the confusion about String.length(), you need to be familiar with some Encoding/Unicode terms. Code Point: A unique integer value which represents a character in the code space. Code Unit: A bit sequence used to encode characters (Code Points). One or more Code Units may be required to represent a Code Point. UTF-16 Unicode Code Points are logically divided into 17 planes. The first plane, the Basic Multilingual Plane (BMP) contains the “classic” characters (from U+0000 to U+FFFF). The other planes contain the supplementary characters (from U+10000 to U+10FFFF). Characters (Code Points) from the first plane are encoded in one 16-bit Code Unit with the same value. Supplementary characters (Code Points) are encoded in two Code Units (encoding-specific, see Wiki for the explanation). Example Character: A Unicode Code Point: U+0041 UTF-16 Code Unit(s): 0041 Character: Mathematical double-struck capital A Unicode Code Point: U+1D538 UTF-16 Code Unit(s): D835 DD38 As you can see here, there are characters which are encoded in two Code Units. String.length() Let’s take a look at the Javadoc of the length() method: public int length() Returns the length of this string. The length is equal to the number of Unicode code units in the string. So if you have one supplementary character which consists of two code units, the length of that single character is two. // Mathematical double-struck capital A String str = "\uD835\uDD38"; System.out.println(str); System.out.println(str.length()); //prints 2 Which is correct according to the documentation, but maybe it’s not expected. ~Solution You need to count the code points not the code units: String str = "\uD835\uDD38"; System.out.println(str); System.out.println(str.codePointCount(0, str.length())); See: codePointCount(int beginIndex, int endIndex) References/Sources The Java Language Specification Unicode Glossary: Code Point Wiki: Code Point Unicode Glossary: Code Unit Wiki: Code Unit Wiki: Unicode Wiki: UTF-16 Supplementary Characters in the Java Platform Wiki: Unicode Planes

The Map interface provides some handy new methods in JDK 8. Because the Map methods I cover in this post are implemented as default methods, all existing implementations of the Map interface enjoy the default behaviors defined in the default methods without any new code. The JDK 8 introduced Map methods covered in this post are getOrDefault(Object, V), putIfAbsent(K, V), remove(Object, Object), remove(Object, Object), replace(K, V), and replace(K, V, V). Example Map for Demonstrations I will be using the Map declared and initialized as shown in the following code throughout the examples in this blog post. The statesAndCapitals field is a class-level static field. I intentionally have only included a small subset of the fifty states in the United States for reading clarity and to allow easier demonstration of some of the new JDK 8 Map default methods. private final static Map statesAndCapitals; static { statesAndCapitals = new HashMap<>(); statesAndCapitals.put("Alaska", "Anchorage"); statesAndCapitals.put("California", "Sacramento"); statesAndCapitals.put("Colorado", "Denver"); statesAndCapitals.put("Florida", "Tallahassee"); statesAndCapitals.put("Nevada", "Las Vegas"); statesAndCapitals.put("New Mexico", "Sante Fe"); statesAndCapitals.put("Utah", "Salt Lake City"); statesAndCapitals.put("Wyoming", "Cheyenne"); } Map.getOrDefault(Object, V) Map's new method getOrDefault(Object, V) allows the caller to specify in a single statement to get the value of the map that corresponds to the provided key or else return a provided "default value" if no match is found for the provided key. The next code listing compares how checking for a value matching a provided key in a map or else using a default if no match is found was implemented before JDK 8 and how it can now be implemented with JDK 8. /* * Demonstrate Map.getOrDefault and compare to pre-JDK 8 approach. The JDK 8 * addition of Map.getOrDefault requires fewer lines of code than the * traditional approach and allows the returned value to be assigned to a * "final" variable. */ // pre-JDK 8 approach String capitalGeorgia = statesAndCapitals.get("Georgia"); if (capitalGeorgia == null) { capitalGeorgia = "Unknown"; } // JDK 8 approach final String capitalWisconsin = statesAndCapitals.getOrDefault("Wisconsin", "Unknown"); The Apache Commons class DefaultedMap provides functionality similar to the newMap.getOrDefault(Object, V) method. The Groovy GDK includes a similar method for Groovy,Map.get(Object, Object), but that one's behavior is a bit different because it not only returns the provided default if the "key" is not found, but also adds the key with the default value to the underlying map. Map.putIfAbsent(K, V) Map's new method putIfAbsent(K, V) has Javadoc advertising its default implementation equivalent: The default implementation is equivalent to, for this map: V v = map.get(key); if (v == null) v = map.put(key, value); return v; This is illustrated with another code sample that compares the pre-JDK 8 approach to the JDK 8 approach. /* * Demonstrate Map.putIfAbsent and compare to pre-JDK 8 approach. The JDK 8 * addition of Map.putIfAbsent requires fewer lines of code than the * traditional approach and allows the returned value to be assigned to a * "final" variable. */ // pre-JDK 8 approach String capitalMississippi = statesAndCapitals.get("Mississippi"); if (capitalMississippi == null) { capitalMississippi = statesAndCapitals.put("Mississippi", "Jackson"); } // JDK 8 approach final String capitalNewYork = statesAndCapitals.putIfAbsent("New York", "Albany"); Alternate solutions in the Java space before the addition of this putIfAbsent method are discussed in theStackOverflow thread Java map.get(key) - automatically do put(key) and return if key doesn't exist?. It's worth noting that before JDK 8, the ConcurrentMap interface (extends Map) already provided a putIfAbsent(K, V)method. Map.remove(Object, Object) Map's new remove(Object, Object) method goes beyond the long-available Map.remove(Object) method to remove a map entry only if both the provided key and provided value match an entry in the map (the previously available version only looked for a "key" match to remove). The Javadoc comment for this method explains the how the default method's implementation works in terms of equivalent pre-JDK 8 Java code: The default implementation is equivalent to, for this map: if (map.containsKey(key) && Objects.equals(map.get(key), value)) { map.remove(key); return true; } else return false; A concrete comparison of the new approach to the pre-JDK 8 approach is shown in the next code listing. /* * Demonstrate Map.remove(Object, Object) and compare to pre-JDK 8 approach. * The JDK 8 addition of Map.remove(Object, Object) requires fewer lines of * code than the traditional approach and allows the returned value to be * assigned to a "final" variable. */ // pre-JDK 8 approach boolean removed = false; if ( statesAndCapitals.containsKey("New Mexico") && Objects.equals(statesAndCapitals.get("New Mexico"), "Sante Fe")) { statesAndCapitals.remove("New Mexico", "Sante Fe"); removed = true; } // JDK 8 approach final boolean removedJdk8 = statesAndCapitals.remove("California", "Sacramento"); Map.replace(K, V) The first of the two new Map "replace" methods sets the specified value to be mapped to the specified key only if the specified key already exists with some mapped value. The Javadoc comment explains the Java equivalent of this default method implementation: The default implementation is equivalent to, for this map: if (map.containsKey(key)) { return map.put(key, value); } else return null; The comparison of this new approach to the pre-JDK 8 approach is shown next. /* * Demonstrate Map.replace(K, V) and compare to pre-JDK 8 approach. The JDK 8 * addition of replace(K, V) requires fewer lines of code than the traditional * approach and allows the returned value to be assigned to a "final" * variable. */ // pre-JDK 8 approach String replacedCapitalCity; if (statesAndCapitals.containsKey("Alaska")) { replacedCapitalCity = statesAndCapitals.put("Alaska", "Juneau"); } // JDK 8 approach final String replacedJdk8City = statesAndCapitals.replace("Alaska", "Juneau"); Map.replace(K, V, V) The second newly added Map "replace" method is more narrow in its interpretation of which existing values are replaced. While the method just covered replaces any value in a value available for the specified key in the mapping, this "replace" method that accepts an additional (third) argument will only replace the value of a mapped entry that has both a matching key and a matching value. The Javadoc comment shows the default method's implementation: The default implementation is equivalent to, for this map: if (map.containsKey(key) && Objects.equals(map.get(key), value)) { map.put(key, newValue); return true; } else return false; My comparison of this approach to the pre-JDK 8 approach is shown in the next code listing. /* * Demonstrate Map.replace(K, V, V) and compare to pre-JDK 8 approach. The * JDK 8 addition of replace(K, V, V) requires fewer lines of code than the * traditional approach and allows the returned value to be assigned to a * "final" variable. */ // pre-JDK 8 approach boolean replaced = false; if ( statesAndCapitals.containsKey("Nevada") && Objects.equals(statesAndCapitals.get("Nevada"), "Las Vegas")) { statesAndCapitals.put("Nevada", "Carson City"); replaced = true; } // JDK 8 approach final boolean replacedJdk8 = statesAndCapitals.replace("Nevada", "Las Vegas", "Carson City"); Observations and Conclusion There are several observations to make from this post. The Javadoc methods for these new JDK 8 Map methods are very useful, especially in terms of describing how the new methods behave in terms of pre-JDK 8 code. I discussed these methods' Javadoc in a more general discussion on JDK 8 Javadoc-based API documentation. As the equivalent Java code in these methods' Javadoc comments indicates, these new methods do not generally check for null before accessing map keys and values. Therefore, one can expect the same issues with nulls using these methods as one would find when using "equivalent" code as shown in the Javadoc comments. In fact, the Javadoc comments generally warn about the potential forNullPointerException and issues related to some Map implementations allowing null and some not for keys and values. The new Map methods discussed in this post are "default methods," meaning that implementations of Map "inherit" these implementations automatically. The new Map methods discussed in this post allow for cleaner and more concise code. In most of my examples, they allowed the client code to be converted from multiple state-impacting statements to a single statement that can set a local variable once and for all. The new Map methods covered in this post are not ground-breaking or earth-shattering, but they are conveniences that many Java developers previously implemented more verbose code for, wrote their own similar methods for, or used a third-party library for. JDK 8 brings these standardized methods to the Java masses without need for custom implementation or third-party frameworks. Because default methods are the implementation mechanism, even Map implementations that have been around for quite a while suddenly and automatically have access to these new methods without any code changes to the implementations.

A few days ago the second beta of Groovy 2.3 got released. One of the major new Groovy 2.3 features are traits. A trait is a reusable set of methods and fields that can be added to one or more classes. A class can be composed out of multiple traits without using multiple inheritance (and therefore avoiding the diamond problem). Basic usage The following piece of code shows a basic definition of a trait in Groovy 2.3. trait SwimmingAbility { def swim() { println "swimming.." } } A trait definition looks very similar to a class definition. This trait defines a single method swim(). We can add this trait to a class using the implements keyword: class Goldfish implements SwimmingAbility { .. } Now we are able to call the swim() methods on Goldfish objects: def goldfish = new Goldfish() goldfish.swim() So far we could have accomplished the same using inheritance. The difference is that we can add multiple traits to a single class. So let's define another trait: trait FlyingAbility { def fly() { println "flying.." } } We can now create a new class that makes use of both traits: class Duck implements SwimmingAbility, FlyingAbility { .. } Ducks can swim and fly now: def duck = new Duck() duck.swim() duck.fly() this inside traits Inside traits the this keyword represents the implementing instance. This means you can write the following: trait FlyingAbility { def fly() { println "${this.class.name} is flying.." } } In case of the Duck class from above this will print Duck is flying.. if we call duck.fly(). A more complex example Now let's look at an example that shows some more features of Groovy traits trait Trader { int availableMoney = 0 private int tradesDone = 0 def buy(Item item) { if (item.price <= availableMoney) { availableMoney -= item.price tradesDone += 1 println "${getName()} bought ${item.name}" } } def sell(Item item) { .. } abstract String getName() } Like Groovy classes traits support properties. Here the property availableMoney will become private and public getter / setter methods will be generated. These methods can be accessed on implementing classes.tradesDone is a private variable that cannot be accessed outside the Trader trait. Within this trait we defined an abstract method getName(). This method has to be implemented by classes that make use of this trait. Let's create a class that implements our Trader trait: class Merchant implements Trader { String name String getName() { return this.name } } A Merchant is now be able to buy items: def bike = new Item(name: 'big red bike', price: 750) def paul = new Merchant(name: 'paul') paul.availableMoney = 2000 paul.buy(bike) // prints "paul bought big red bike" println paul.availableMoney // 1250 Extending from traits, Overriding and conflict resolution A trait can inherit functionality from another trait using the extends keyword: trait Dealer { def getProfit() { ... } def deal() { ... } } trait CarDealer extends Dealer { def deal() { ... } } Here the CarDealer trait extends Dealer and overrides the deal() method of Dealer. Trait methods can also be overwritten in implementing classes: class OnlineCarDealer implements CarDealer { def deal() { ... } } If a class implements more than one trait it is possible to create a conflict. That's the case if two or more traits define a method with an identical signature: trait Car { def drive() { ... } } trait Bike { def drive() { ... } } class DrivingThing implements Car, Bike { ... } In such a situation the last declared trait wins (Bike in this example). Conclusion I think traits are a very useful concept and I am happy to see them in Groovy. Other than Groovy mixins traits work at compile time and can therefore be accessed from Java code. For further reading I can recommend the Groovy 2.3 Trait documentation.

If you need to compare the java.io.file.Path object, be aware that Path.endsWith(String) will ONLY match another sub-element of Path object in your original path, not the path name string portion! If you want to match the string name portion, you would need to call the Path.toString() first. For example // Match all jar files. Files.walk(dir).forEach(path -> { if (path.toString().endsWith(".jar")) System.out.println(path); }); With out the "toString()" you will spend many fruitless hours wonder why your program didn't work.