A couple of weeks ago we looked at how to do a quick "health check" of an agile team. We saw that a great deal can be learned just by attending one of their daily stand-ups and by inspecting the state of their Scrum and Kanban boards. Of course these are nothing more than cursory examinations, and serious ailments can lie behind an apparently robust façade of agile practice. If you have reason to believe that a team is dysfunctional, you might have to dig deeper than the superficial evidence suggested by its apparent morphology. In my experience the next thing to examine is the team's "Definition of Done". This is the standard to which all work is put before it can be considered to be complete. Each team is collectively responsible for its own Definition of Done. It's up to them to make sure that it is adequate, and that it is applied by all members to all of the work they do. Without such professional oversight there can be no assurance that any deliverable will truly be fit for release. A spiffy stand-up and a cracker of a Kanban board might suggest rude team health, but they are no guarantee that the Definition of Done is solid, or that it isn't being undercut by someone along the way. Technical debt and rework are the main symptoms to look for. The consequences of backsliding on a Definition of Done might not become apparent until long after the events that caused it. By then, that rework or debt can be difficult to trace to the specific behaviors of those who cheated the system. You see, unfortunately a Definition of Done is a bit like personal hygiene. If there is no oversight, everyone can pretend that they are following the rules for the sake of the team, even though the presence of E. Coli on the office keyboards will tell its own tale about compliance. Everyone knows that it has to be coming from somewhere, but won't admit to their own liability or involvement, perhaps not even to themselves. Just as team vomiting will follow one member's dubious hand-washing practices, a short-changed Definition of Done will lead to rework by the team or the creation of technical debt. This is why team ownership and enforcement of what "Done" means is so important. An effective Definition of Done has to be founded on a healthy balance between due diligence and professional trust. What does this mean for agile development? You can think of a Definition of Done as the key defensive bulwark in software development epidemiology. If you balance the right level of team oversight with the right level of trust, severe outbreaks of technical debt or rework will become rare. High levels of oversight may be needed to start with, since team members might not have bought in to the idea of "done" yet. Once people become conditioned to do the right thing and see themselves as stakeholders in the team and its success, the balance can swing more towards trust. People become reluctant to renege on a team investment if they can see that it adds value for everyone including them. What's more, a Definition of Done improves the more it is respected, and becomes better respected the more it improves. In terms of agile best practice a Definition of Done will be used to determine whether or not User Story implementations are release-ready. However, each team can implement many User Stories over the course of a sprint, and making sure that all of these stories meet the Definition of Done can be challenging. Teams that are new to agile methods often have more modest ambitions. For example, their Definition of Done may only extend as far as delivery into a pre-production environment. Of course, anything less than "fully release ready" incurs the risk of technical debt and the need to pay it back later. Yet like a sloppy approach to hand-washing, it has to be admitted that something is better than nothing at all. Applying a Definition of Done consistently to even a sub-optimal standard will at least permit the delivery of each User Story to a known level of quality. It might not be great but at least it's there, and it's something that can be built upon and improved. The Lessons of Lean-Kanban Lean-Kanban methodologies have an instructive relationship with the Definition of Done. In these approaches the optimization of the value stream is of great significance. Naturally though, if a value stream is to be optimized it must first be understood. This means breaking the stream down into multiple discrete steps that can be studied for bottlenecks and any other occurrences of waste. For example, "Work in Progress" can be broken down into finer-grained stations such as "In Development", "Peer Review", "QA Test", "Knowledge Transfer", and "In Deployment". Team members will be cross-trained and will move freely across those stations in order to expedite as smooth a workflow as possible. Now here's where it gets interesting. If a Lean-Kanban operation has multiple well-defined stations, the case for having a Definition of Done can seem rather harder to make. After all, by the time a User Story gets to "Done", you already know that it has gone through the key steps you care about in the development process. What value can a Definition of Done really add in such a situation? Doesn't it just become waste itself? To find the answer, we need to look back to the manufacturing roots of Lean-Kanban. In a car plant for example, the steps of construction are exceptionally well defined and team members can move freely over several dozen stations. Some of those stations will be for the chassis, others for the interior, others for the engine block and electrics and so on. Yet despite this the Definition of Done will be an absolute corker, and much of the process of verification will be automated. Each station might even have its own Definition of Done so inspection can occur as close as possible to the point of action and potential remedy. The total number of checks that happen before each vehicle leaves the factory will be exhaustive. Why is this thought to be necessary? Because the manufacturers know perfectly well that the verification of "done" adds value. Merely having well-defined stations is no guarantee that everything will be done well. Quality is built in and validated by inspection. One thing's for sure: no-one in IT should accuse car manufacturers of having a weak understanding of what "done" means. The Definition of Done versus Acceptance Criteria However, software projects have a wild-card to deal with that car manufacturers don't have to worry about. Unlike the car doors and carburettors that roll down an assembly line, each User Story is different and has to be treated as a special case. To deal with this, each User Story has Acceptance Criteria that are agreed by the team members and the Product Owner as part of a Sprint Planning Session. Acceptance Criteria have to be quite specific to particular User Stories, because each story can be unique. The Definition of Done, on the other hand, applies to all of the User Stories being worked on by a team. The associated conditions must be invariant. For example, if all work has to be peer reviewed and subjected to QA testing prior to release, then those criteria would be enumerated in the Definition of Done rather than being repeated in each User Story's Acceptance Criteria. If the definition is enforced properly, a developer could never claim that a User Story was “Done” if it hadn't both been reviewed and passed QA testing. Writing a Definition of Done The Scrum Guide describes a Definition of Done as a "shared understanding of what it means for work to be complete". No process is suggested for writing a Definition of Done, nor in fact is there any suggestion that one should be written down at all. However, a documented definition may go some way towards providing that shared understanding. Here's how you can set about eliciting one: Review Acceptance Criteria: Gather the Acceptance Criteria for work completed so far Look for common criteria that can be abstracted out and applied across work in general Use these common criteria as the basis for a Definition of Done Assess Technical Debt Identify any rework that needs to be done Identify the reasons why it wasn't done properly the first time Identify what measures can be put in place to stop similar rework from occurring Add these measures to the Definition of Done (DoD) Continually update the DoD In each Sprint Review, identify which (if any) work was rejected or which caused rework to be done, then In each Sprint Retrospective, challenge the DoD for relevance and completeness There isn't a prescribed format for a Definition of Done, but it can be beneficial to use the same as that which is used for Acceptance Criteria, either in whole or in part. This allows a flexible definition based on story type. Alternatively they can be written as simple lists. Here are some examples: Example of a Definition of Done in Acceptance Criteria Format Given that a user story has required a code change When BDD and unit tests have been written for the story and the code change has been reviewed and the code change has been approved by a peer and all BDD and unit tests have been run and no BDD or unit tests have broken (green bar) and the code change has been committed to the repository and QA testing has passed satisfactorily and the Product Owner has approved the change Then the user story will be deployed to production and it will be considered Done. Given that a user story has required the authoring of documentation When the documentation has been reviewed and approved by a peer and the documentation has been approved by the Product Owner Then a new version of the documentation will be committed and the user story will be considered Done. Example of a Definition of Done in List Format Code changes: BDD tests written and pass Unit tests written and pass Code peer reviewed & approved Code committed to repository QA testing done Product Owner signed off Documentation: Documentation has been peer reviewed & approved Documentation approved by Product Owner Version number updated Definitions of Done for IT Infrastructure Support We've seen that having a good Definition of Done is important, although in IT we also need Acceptance Criteria that address the particulars of each User Story. When used in combination they can approach the levels of rigor that have been shown to be possible in manufacturing. Those working in software development can adopt a similar commitment to quality. Now we need to turn our attention to another function within the IT department...Infrastructure Support. Infrastructure support teams are increasingly expected to work in an agile way. As part of an enterprise transformation that does not seem unreasonable. After all, the rest of the organization is highly dependent upon them. Their scope includes such things as deploying new workstations and laptops (possibly across entire sites), installing networks, performing miscellaneous diagnostics and repairs, and maintaining and upgrading local server resources. Clearly they will also need Definitions of Done and Acceptance Criteria if they are to be stakeholders in a joined-up agile enterprise. The question is, how on earth can a meaningful Definition of Done be abstracted across wildly different physical tasks? How can a Definition of Done cover everything from a phone installation to a printer driver upgrade or a memory enhancement, or a firewall configuration to a keyboard replacement? The answer is to focus on the value chain that is represented by each user story. Work is not "released" as such, but rather it is handed over to someone who can derive benefit from it (i.e. the person occupying the user story role). This is the key to understanding "done" in an infrastructure context. If you can identify the parties who derive value, and demonstrably pass that value on to them, then your work is done. Here's an example of a Definition of Done that might be used to close out a support ticket: The receiver of the work has been identified Handover instructions have been completed and given to the receiver The receiver has been notified of the intention to close the ticket, and has not raised an objection A security assessment has been conducted and approved There are three things to point out here. The absence of any reference to a Product Owner. This is because infrastructure teams have to support multiple products, and prioritization of work is traditionally handled not through any sense of ownership of those products, but rather through help-desk triage. It's certainly possible for work to be represented by Product Owners, but it would have to be ownership of the business support function rather than ownership of the actual products being supported. The need to identify and work with the actual receivers of value is still there. The "acceptance by default" position. Receivers typically have little incentive to sign work off as being complete. On the contrary, their incentive is to defer acceptance as long as possible, for potential use as a "banker" in case a requirement for additional unforeseen work transpires. They might hope to ride this new work on an existing ticket instead of having to raise a new one. Receivers can be expected to care about their own support needs, not about the big picture of enterprise delivery. If a Product Owner can be identified - even if it is just the most likely owner of the business support function - then this situation can be improved. A Product Owner can apply leverage for appropriate and timely sign-off, such as by not accepting new work from certain parties while their approval (or justified rejection) of prior work remains outstanding. The elicitation of solid Acceptance Criteria can help the Product Owner immensely. Security implications need to be given careful consideration. The reworking of organizational infrastructure offers great potential for security to be compromised. Approval from Information Security should be obtained for all work, either directly or through authorized agents. One approach is for each team to have a designated "security champion" who provides this function. Conclusion Teams that appear to be healthy can still incur rework and technical debt. A poor understanding of what "done" means often underlies such problems, and this should be one of the first things to be looked at if problems are suspected. Having a meaningful Definition of Done encourages a team's sense of ownership of their own process, and helps instil self-discipline into its members to follow agile best practices. The application of this standard requires finding the right balance between team oversight and trust.

Since Grails 2.1 we can create a Grails wrapper. The wrapper allows developer to run Grails commands in a project without installing Grails first. The wrapper concept is also available in other projects from the Groovy ecosystem like Gradle or Griffon. A wrapper is a shell script for Windows, OSX or Linux named grailsw.bat or grailsw and a couple of JAR files to automatically download a specific version of Grails. We can check in the shell scripts and supporting files into a version control system and make it part of the project. Developers working on the project simply check out the code and execute the shell script. If there is no Grails installation available then it will be downloaded. To create the shell scripts and supporting files someone on the project must run the wrapper command for the first time. This developer must have a valid Grails installation. The files that are generated can then be added to version control and from then one developers can use the grailsw or grailsw.bat shell scripts. $ grails wrapper | Wrapper installed successfully $ In the root of the project we have two new files grailsw and grailsw.bat. Windows users can uss grailsw.bat and on other operating systems we use grailsw. Also a new directory wrapper is created with three files: grails-wrapper-runtime-2.2.0.jar grails-wrapper.properties springloaded-core-1.1.1.jar When we run the grailsw or grailsw.bat scripts for the first time we see how Grails is downloaded and installed into the $USER_HOME/.grails/wrapper directory. The following output shows that the file is downloaded and extracted when we didn't run the grailsw script before: $ ./grailsw --version Downloading http://dist.springframework.org.s3.amazonaws.com/release/GRAILS/grails-2.2.0.zip to /Users/mrhaki/.grails/wrapper/grails-2.2.0-download.zip ..................................................................................... ................................................................ Extracting /Users/mrhaki/.grails/wrapper/grails-2.2.0-download.zip to /Users/mrhaki/.grails/wrapper/2.2.0 Grails version: 2.2.0 When we want to use a new version of Grails one of the developers needs to run to run $ grails upgrade followed by $ grails wrapper with the new Grails version. Notice this developer needs to have a locally installed Grails installation of the version we want to create a wrapper for. The newly generated files can be checked in to version control and all developers on the project will have the new Grails version when they run the grails or grailsw.bat shell scripts. $ ./grailsw --version Downloading http://dist.springframework.org.s3.amazonaws.com/release/GRAILS/grails-2.2.1.zip to /Users/mrhaki/.grails/wrapper/grails-2.2.1-download.zip ..................................................................................... ... ................................................................ Extracting /Users/mrhaki/.grails/wrapper/grails-2.2.1-download.zip to /Users/mrhaki/.grails/wrapper/2.2.1 Grails version: 2.2.1 We can change the download location of Grails to for example a company intranet URL. In the wrapper/ directory we see the file grails-wrapper.properties. The file has one property wrapper.dist.url, which by default refers to http://dist.springframework.org.s3.amazonaws.com/release/GRAILS/. We can change this to another URL, add the change to version control so other developers will get the change automatically. And when the grailsw shell script is executed the download location will be another URL. To set a different download URL when generating the wrapper we can use the command-line option --distributionUrl: $ grails wrapper --distributionUrl=http://company.intranet/downloads/grails-releases/ If we don't like the default name for the directory to store the supporting files we can use the command-line option --wrapperDir. The files are then stored in the given directory and the grailsw and grailsw.bat shell scripts will contain the given directory name. Written with Grails 2.2.0 and 2.2.1

it happens to me many times: i’m stepping with the debugger through my code, and ups! i made one step too far! debugging, and made one step over too far what now? restart the whole debugging session? actually, there is a way to go ‘backwards’ gdb has a ‘reverse debugging’ feature, described here . i’m using the eclipse based codewarrior debugger, and this debug engine is not using gdb. the codewarrior debugger in mcu10.3 supports an eclipse feature: i select a code line in the editor view and use move to line : move to line what it does: it changes the current pc (program counter) of the program to that line: performed move to line now i can continue debugging from that line, e.g. stepping into that function call. yes, this is not true backward debugging. but it is simple and very effective. to perform true backward stepping, the debugger would need to reverse all operations, typically with a rather heavy state machine and data recording. but for the usual case where i simply need to go back a few lines, the ‘move to line’ is perfect. of course there are a few points to consider: this only changes the program counter. any variable changes/etc are not affected or reverted. in case of highly optimized code, there might be multiple sequence points per source line. so doing this for highly optimized code might not work correctly. it works ok within a function. it is not recommended to use it e.g. to set the pc outside of a function. because the context/stack frame is not set up. i use the ‘move to line’ frequently to ‘advance’ the program execution. e.g. to bypass some long sequences i’m not interested in, or to get out of an ‘endless’ loop. the same ‘move to line’ as available while doing assembly stepping too. see this post for details. happy line moving

n this post i’m going to cover how to do job chaining in quartz versus obsidian scheduler . both are java job schedulers, but they have different approaches so i thought i’d highlight them here and give some guidance to users using both options. it’s very common when using a job scheduler to need to chain one job to another. chaining in this case refers to executing a specific job after a certain job completes (or maybe even fails). often we want to do this conditionally, or pass on data to the target job so it can receive it as input from the original job. we’ll start with demonstrating how to do this in quartz, which will take a fair bit of work. obsidian will come after since it’s so simple. chaining in quartz quartz is the most popular job scheduler out there, but unfortunately it doesn’t provide any way to give you chaining without you writing some code. quartz is a low-level library at heart, and it doesn’t try to solve these types of problems for you, which in my mind is unfortunate since it puts the onus on developers. but despite this, many teams still end up using quartz, so hopefully this is useful to some of you. i’m going to outline probably the most basic way to perform chaining. it will allow a job to chain to another, passing on its jobdatamap (for state). this is simpler than using listeners, which would require extra configuration, but if you want to take a look, check out this listener for a starting point. sample code this will rely on an abstract class that will provided basic flow and chaining functionality to any subclasses. it acts as a very simple template class. first, let’s create the abstract class that gives us chaining behaviour: import static org.quartz.jobbuilder.newjob; import static org.quartz.triggerbuilder.newtrigger; import org.quartz.*; import org.quartz.impl.*; public abstract class chainablejob implements job { private static final string chain_job_class = "chainedjobclass"; private static final string chain_job_name = "chainedjobname"; private static final string chain_job_group = "chainedjobgroup"; @override public void execute(jobexecutioncontext context) throws jobexecutionexception { // execute actual job code doexecute(context); // if chainjob() was called, chain the target job, passing on the jobdatamap if (context.getjobdetail().getjobdatamap().get(chain_job_class) != null) { try { chain(context); } catch (schedulerexception e) { e.printstacktrace(); } } } // actually schedule the chained job to run now private void chain(jobexecutioncontext context) throws schedulerexception { jobdatamap map = context.getjobdetail().getjobdatamap(); @suppresswarnings("unchecked") class jobclass = (class) map.remove(chain_job_class); string jobname = (string) map.remove(chain_job_name); string jobgroup = (string) map.remove(chain_job_group); jobdetail jobdetail = newjob(jobclass) .withidentity(jobname, jobgroup) .usingjobdata(map) .build(); trigger trigger = newtrigger() .withidentity(jobname + "trigger", jobgroup + "trigger") .startnow() .build(); system.out.println("chaining " + jobname); stdschedulerfactory.getdefaultscheduler().schedulejob(jobdetail, trigger); } protected abstract void doexecute(jobexecutioncontext context) throws jobexecutionexception; // trigger job chain (invocation waits for job completion) protected void chainjob(jobexecutioncontext context, class jobclass, string jobname, string jobgroup) { jobdatamap map = context.getjobdetail().getjobdatamap(); map.put(chain_job_class, jobclass); map.put(chain_job_name, jobname); map.put(chain_job_group, jobgroup); } } there’s a fair bit of code here, but it’s nothing too complicated. we create the basic flow for job chaining by creating an abstract class which calls a doexecute() method in the child class, then chains the job if it was requested by calling chainjob() . so how do we use it? check out the job below. it actually chains to itself to demonstrate that you can chain any job and that it can be conditional. in this case, we will chain the job to another instance of the same class if it hasn’t already been chained, and we get a true value from new random().nextboolean() . import java.util.*; import org.quartz.*; public class testjob extends chainablejob { @override protected void doexecute(jobexecutioncontext context) throws jobexecutionexception { jobdatamap map = context.getjobdetail().getjobdatamap(); system.out.println("executing " + context.getjobdetail().getkey().getname() + " with " + new linkedhashmap(map)); boolean alreadychained = map.get("jobvalue") != null; if (!alreadychained) { map.put("jobtime", new date().tostring()); map.put("jobvalue", new random().nextlong()); } if (!alreadychained && new random().nextboolean()) { chainjob(context, testjob.class, "secondjob", "secondjobgroup"); } } } the call to chainjob() at the end will result in the automatic job chaining behaviour in the parent class. note that this isn’t called immediately, but only executes after the job completes its doexecute() method. here’s a simple harness that demonstrates everything together: import org.quartz.*; import org.quartz.impl.*; public class test { public static void main(string[] args) throws exception { // start up scheduler stdschedulerfactory.getdefaultscheduler().start(); jobdetail job = jobbuilder.newjob(testjob.class) .withidentity("firstjob", "firstjobgroup").build(); // trigger our source job to triggers another trigger trigger = triggerbuilder.newtrigger() .withidentity("firstjobtrigger", "firstjobbtriggergroup") .startnow() .withschedule( simpleschedulebuilder.simpleschedule().withintervalinseconds(1) .repeatforever()).build(); stdschedulerfactory.getdefaultscheduler().schedulejob(job, trigger); thread.sleep(5000); // let job run a few times stdschedulerfactory.getdefaultscheduler().shutdown(); } } sample output executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=5420204983304142728, jobtime=sat mar 02 15:19:29 pst 2013} executing firstjob with {} executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=-2361712834083016932, jobtime=sat mar 02 15:19:31 pst 2013} executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=7080718769449337795, jobtime=sat mar 02 15:19:32 pst 2013} executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=7235143258790440677, jobtime=sat mar 02 15:19:33 pst 2013} executing firstjob with {} deficiencies well, we’re up and chaining, but there are some problems with this approach: it doesn’t integrate with a container like spring to use configured jobs. more code would be required. it forces you to know up front which jobs you want to chain, and write code for it. configuration is fixed, unless, once again, you write more code. no real-time changes (unless you write more code). a fair bit of code to maintain , and high likelihood you will have to expand it for more functionality. the theme here is that it’s doable, but it’s up to you to do the work to make it happen. obsidian avoids these problems by making chaining configurable, instead of it being a feature of the job itself. read on to find out how. chaining in obsidian in contrast to quartz, chaining in obsidian requires no code and no up-front knowledge of which jobs will chain or how you might want to chain them later. chaining is a form of configuration, and like all job-related configuration in obsidian, you can make live changes at any time without a build or any code at all. job configuration can use a native rest api or the web ui that’s included with obsidian. the following chaining features are available for free: no code and no redeploy to add or remove chains. you can chain specific configurations of job classes. you can chain only on certain states, including failure. chain conditionally based on source job saved state (equivalent to quartz’s jobdatamap), including multiple conditions. regexp/equals/greater than, etc. chain only when matching a schedule. check out the feature and ui documentation to find out more. now that we know what’s possible, let’s see an example. once you have your jobs configured , just create a new chain using the ui. rest api support will be here shortly but as of 1.5.1 chaining isn’t included in the api. if you need to script this right now, we can provide pointers . in the ui, it looks like the following: easy, huh? all configuration is stored in a database, so it’s easy to replicate it in various environments or to automate it via scripting. as a bonus, obsidian tracks and shows you all chaining state including what job triggered a chained job. it will even tell you why a job chain didn’t fire, whether it’s because the job status didn’t match, or one of your conditions didn’t. conclusion that summarizes how you can go about chaining in quartz and obsidian. quartz definitely has a minimalist approach, but that leaves developers with a lot of work to do. meanwhile, obsidian provides rich functionality out of the box to keep developers working on their own rich functionality, instead of the plumbing that so often seems to consume their time. if you have any suggestions or feature requests for obsidian, drop us a note by leaving a comment or by contacting us .

Here are some simple steps to get you fully started with Quartz Scheduler on MySQL database using Groovy. The script below will allow you to quickly experiment different Quartz configuration settings using an external file. First step is to setup the database with tables. Assuming you already have installed MySQL and have access to create database and tables. bash> mysql -u root -p sql> create database quartz2; sql> create user 'quartz2'@'localhost' identified by 'quartz2123'; sql> grant all privileges on quartz2.* to 'quartz2'@'localhost'; sql> exit; bash> mysql -u root -p quartz2 < /path/to/quartz-dist/docs/dbTables/tables_mysql.sql The tables_mysql.sql can be found from Quartz distribution download, or directly from their source here. Once the database is up, you need to write some code to start up the Quartz Scheduler. Here is a simply Groovy script quartzServer.groovy that will run as a tiny scheduler server. // Run Quartz Scheduler as a server // Author: Author: Zemian Deng, Date: 2012-12-15_16:46:09 @GrabConfig(systemClassLoader=true) @Grab('mysql:mysql-connector-java:5.1.22') @Grab('org.slf4j:slf4j-simple:1.7.1') @Grab('org.quartz-scheduler:quartz:2.1.6') import org.quartz.* import org.quartz.impl.* import org.quartz.jobs.* config = args.length > 0 ? args[0] : "quartz.properties" scheduler = new StdSchedulerFactory(config).getScheduler() scheduler.start() // Register shutdown addShutdownHook { scheduler.shutdown() } // Quartz has its own thread, so now put this script thread to sleep until // user hit CTRL+C while (!scheduler.isShutdown()) { Thread.sleep(Long.MAX_VALUE) } And now you just need a config file quartz-mysql.properties that looks like this: # Main Quartz configuration org.quartz.scheduler.skipUpdateCheck = true org.quartz.scheduler.instanceName = DatabaseScheduler org.quartz.scheduler.instanceId = NON_CLUSTERED org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate org.quartz.jobStore.dataSource = quartzDataSource org.quartz.jobStore.tablePrefix = QRTZ_ org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = 5 # JobStore: JDBC jobStoreTX org.quartz.dataSource.quartzDataSource.driver = com.mysql.jdbc.Driver org.quartz.dataSource.quartzDataSource.URL = jdbc:mysql://localhost:3306/quartz2 org.quartz.dataSource.quartzDataSource.user = quartz2 org.quartz.dataSource.quartzDataSource.password = quartz2123 org.quartz.dataSource.quartzDataSource.maxConnections = 8 You can run the Groovy script as usual bash> groovy quartzServer.groovy quartz-mysql.properties Dec 15, 2012 6:20:26 PM com.mchange.v2.log.MLog INFO: MLog clients using java 1.4+ standard logging. Dec 15, 2012 6:20:27 PM com.mchange.v2.c3p0.C3P0Registry banner INFO: Initializing c3p0-0.9.1.1 [built 15-March-2007 01:32:31; debug? true; trace:10] [main] INFO org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor [main] INFO org.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl [main] INFO org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.1.6 created. [main] INFO org.quartz.core.QuartzScheduler - JobFactory set to: org.quartz.simpl.SimpleJobFactory@1a40247 [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Using thread monitor-based data access locking (synchronization). [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - JobStoreTX initialized. [main] INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.1.6) 'DatabaseScheduler' with instanceId 'NON_CLUSTERED' Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally. NOT STARTED. Currently in standby mode. Number of jobs executed: 0 Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 5 threads. Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is not clustered. [main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'DatabaseScheduler' initialized from the specified file : 'quartz-mysql.properties' from the class resource path. [main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.1.6 Dec 15, 2012 6:20:27 PM com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 1hge16k8r18mveoq1iqtotg|1486306, debugUnreturnedConnectionStackTraces -> fals e, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1hge16k8r18mveoq1iqtotg|1486306, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://localhost:3306/quartz2, lastAcquisitionFailureDefaultUser -> null, maxAdministrativeTaskTime -> 0 , maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 8, maxStatements -> 0, maxStatementsPerConnection -> 120, minPoolSize -> 1, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, pref erredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ] [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Freed 0 triggers from 'acquired' / 'blocked' state.[main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Recovering 0 jobs that were in-progress at the time of the last shut-down. [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Recovery complete. [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Removed 0 'complete' triggers. [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Removed 0 stale fired job entries. [main] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED started. ... CTRL+C [Thread-6] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED shutting down. [Thread-6] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED paused. [Thread-6] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED shutdown complete. That's a full run of above setup. Go ahead and play with different config. Read http://quartz-scheduler.org/documentation/quartz-2.1.x/configuration for more details. Here I will post couple more easy config that will get you started in a commonly used config set. A MySQL cluster enabled configuration. With this, you can start one or more shell terminal and run different instance of quartzServer.groovy with the same config. All the quartz scheduler instances should cluster themselve and distribute your jobs evenly. # Main Quartz configuration org.quartz.scheduler.skipUpdateCheck = true org.quartz.scheduler.instanceName = DatabaseClusteredScheduler org.quartz.scheduler.instanceId = AUTO org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate org.quartz.jobStore.dataSource = quartzDataSource org.quartz.jobStore.tablePrefix = QRTZ_ org.quartz.jobStore.isClustered = true org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = 5 # JobStore: JDBC jobStoreTX org.quartz.dataSource.quartzDataSource.driver = com.mysql.jdbc.Driver org.quartz.dataSource.quartzDataSource.URL = jdbc:mysql://localhost:3306/quartz2 org.quartz.dataSource.quartzDataSource.user = quartz2 org.quartz.dataSource.quartzDataSource.password = quartz2123 org.quartz.dataSource.quartzDataSource.maxConnections = 8 Here is another config set for a simple in-memory scheduler. # Main Quartz configuration org.quartz.scheduler.skipUpdateCheck = true org.quartz.scheduler.instanceName = InMemoryScheduler org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = 5 Now, if you need more fancy UI management of Quartz, give MySchedule a try.

Sometimes a programmer will come to me and explain that they don’t like the design of something and that “we’re gonna need to do a whole bunch of refactoring” to make it right. Oh Oh. This doesn’t sound good. And it doesn’t sound like refactoring either…. CHECK OUT THE NEW REFACTORING REFCARD! --DZone curator interruption Refactoring, as originally defined by Martin Fowler and Kent Beck, is A change made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behavior… It is a disciplined way to clean up code that minimizes the chances of introducing bugs. Refactoring is done to fill in short-cuts, eliminate duplication and dead code, and to make the design and logic clear. To make better and clearer use of the programming language. To take advantage of information that you have now but that the programmer didn’t have then – or that they didn’t take advantage of then. Always to simplify the code and to make it easier to understand. Always to make it easier and safer to change in the future. Fixing any bugs that you find along the way is not refactoring. Optimization is not refactoring. Tightening up error handling and adding defensive code is not refactoring. Making the code more testable is not refactoring – although this may happen as the result of refactoring. All of these are good things to do. But they aren’t refactoring. Programmers, especially programmers maintaining code, have always cleaned up code as part of their job. It’s natural and often necessary to get the job done. What Martin Fowler and others did was to formalize the practices of restructuring code, and to document a catalog of common and proven refactoring patterns – the goals and steps. Refactoring is simple. Protect yourself from making mistakes by first writing tests where you can. Make structural changes to the code in small, independent and safe steps, and test the code after each of these steps to ensure that you haven’t changed the behavior – it still works the same, just looks different. Refactoring patterns and refactoring tools in modern IDEs make refactoring easy, safe and cheap. Refactoring Isn’t an End in Itself Refactoring is supposed to be a practice that supports making changes to code. You refactor code before making changes, so that you can confirm your understanding of the code and make it easier and safer to put your change in. Regression test your refactoring work. Then make your fix or changes. Test again. And afterwards maybe refactor some more of the code to make the intent of the changes clearer. And test everything again. Refactor, then change. Or change, then refactor. You don’t decide to refactor, you refactor because you want to do something else, and refactoring helps you do that other thing. The scope of your refactoring work should be driven by the change or fix that you need to make – what do you need to do to make the change safer and cleaner? In other words: Don’t refactor for the sake of refactoring. Don’t refactor code that you aren’t changing or preparing to change. Scratch Refactoring to Understand There’s also Scratch Refactoring from Michael Feather’s Working Effectively with Legacy Code book; what Martin Fowler calls “Refactoring to Understand”. This is where you take code that you don’t understand (or can’t stand) and clean it up so that you can get a better idea of what is going on before you start to actually work on changing it for real, or to help in debugging it. Rename variables and methods once you figure out what they really mean, delete code that you don’t want to look at (or don’t think works), break complex conditional statements down, break long routines into smaller ones that you can get your head around. Don't bother reviewing and testing all of these changes. The point is to move fast – this is a quick and dirty prototype to give you a view into the code and how it works. Learn from it and throw it away. Scratch refactoring also lets you test out different refactoring approaches and learn more about refactoring techniques. Michael Feathers recommends that you keep notes during this on anything that wasn’t obvious or that was especially useful, so that you can come back and do a proper job later - in small, disciplined steps, with tests. What About “Large Scale” Refactoring? You can get a big return in understandability and maintainability from making simple and obvious refactoring changes: eliminating duplication, changing variable and method names to be more meaningful, extracting methods to make code easier to understand and more reusable, simplifying conditional logic, replacing a magic number with a named constant, moving common code together. There is a big difference between minor, inline refactoring like this, and more fundamental design restructuring – what Martin Fowler refers to as “Big Refactoring”. Big, expensive changes that carry a lot of technical risk. This isn’t cleaning up code and improving the design while you are working: this is fundamental redesign. Some people like to call redesign or rewriting or replatforming or reengineering a system “Large Scale Refactoring” because technically you aren’t changing behavior – the business logic and inputs and outputs stay the same, it’s “only” the design and implementation that’s changing. The difference seems to be that you can rewrite code or even an entire system, and as long as you do it in steps, you can still call it “refactoring”, whether you are slowly Strangling a legacy system with new code, or making large-scale changes to the architecture of a system. “Large Scale Refactoring” changes can be ugly. They can take weeks or months (or years) to complete, requiring changes to many different parts of the code. They need to be broken down and released in multiple steps, requiring temporary scaffolding and detours, especially if you are working in short Agile sprints. This is where practices like Branch by Abstraction come in to play, to help you manage changes inside the code over a long period of time. In the meantime you have to keep working with the old code and new code together, making the code harder to follow and harder to change, more brittle and buggy - the opposite of what refactoring is supposed to achieve. Sometimes this can go on forever – the transition work never gets completed because most of the benefits are realized early, or because the consultant who came up with the idea left to go on to something else, or the budget got cut, and you’re stuck maintaining a Frankensystem. This Is Refactoring — That Isn't Mixing this kind of heavy project work up with the discipline of refactoring-as-you-go is wrong. They are fundamentally different kinds of work, with very different costs and risks. It muddies up what people think refactoring is, and how refactoring should be done. Refactoring can and should be folded in to how you write and maintain code – a part of the everyday discipline of development, like writing tests and reviewing code. It should be done quietly, continuously and implicitly. It becomes part of the cost of doing work, folded in to estimates and risk assessments. Done properly, it doesn’t need to be explained or justified. Refactoring that takes a few minutes or an hour or two as part of a change is just part of the job. Refactoring that can take several days or longer is not refactoring; it is rewriting or redesigning. If you have to set aside explicit blocks of time (or an entire sprint!) to refactor code, if you have to get permission or make a business case for code cleanup, then you aren’t refactoring – even if you are using refactoring techniques and tools, you’re doing something else. Some programmers believe it is their right and responsibility to make fundamental and significant changes to code, to reimagine and rewrite it, in the name of refactoring and for the sake of the future and for their craft. Sometimes redesigning and rewriting code is the right thing to do. But be honest and clear. Don’t hide this under the name of refactoring.

Option 1: JMX Many people asked can they manage Quartz via JMX, and that the documentation on this is not clear enough to help them get started. So, let me highlight couple ways you can do this. Yes you can enable JMX in quartz with the following in quartz.properties org.quartz.scheduler.jmx.export = true After this, you use standard JMX client such as $JAVA_HOME/bin/jconsole to connect and manage remotely. Option 2: RMI Another way to manage quartz remotely is to enable RMI in Quartz. If you use this, you basically run one instance of Quartz as RMI server, and then you can create second Quartz instance as RMI client. These two can talk remotely via a TCP port. For server scheduler instance, you want to add these in quartz.properties org.quartz.scheduler.rmi.export = true org.quartz.scheduler.rmi.createRegistry = true org.quartz.scheduler.rmi.registryHost = localhost org.quartz.scheduler.rmi.registryPort = 1099 org.quartz.scheduler.rmi.serverPort = 1100 And for client scheduler instance, you want to add these in quartz.properties org.quartz.scheduler.rmi.proxy = true org.quartz.scheduler.rmi.registryHost = localhost org.quartz.scheduler.rmi.registryPort = 1099 The RMI feature is mentioned in Quartz doc here. Quartz doesn't have a client API, but use the same org.quartz.Scheduler for both server and client. It's just the configuration are different. By different configuration, you get very different behavior. For server, your scheduler is running all the jobs, while for client, it's simply a proxy. Your client scheduler instance will not run any jobs! You must be really careful when shutting down client because it does allow you to bring down the server! These configurations have been highlighted in the MySchedule project. If you run the webapp, you should see a screen like this demo, and you will see it provided many sample of quartz configurations with these remote managment config properties. If configure with RMI option, you can actually still use MySchedule web UI to manage the Quartz as proxy. You can view and drill down jobs, and you can even stop or shutdown remote server! Based on my experience, there is a down side of using Quartz RMI feature though. That is it creates a single point of failure. There is no fail over if your RMI server port is down!

Sometimes Quartz is not capable of running your job at the time when you desired. There are three reasons for that: all worker threads were busy running other jobs (probably with higher priority) the scheduler itself was down the job was scheduled with start time in the past (probably a coding error) You can increase the number of worker threads by simply customizing the org.quartz.threadPool.threadCount in quartz.properties (default is 10). But you cannot really do anything when the whole application/server/scheduler was down. The situation when Quartz was incapable of firing given trigger is called misfire. Do you know what Quartz is doing when it happens? Turns out there are various strategies (called misfire instructions) Quartz can take and also there are some defaults if you haven't thought about it. But in order to make your application robust and predictable (especially under heavy load or maintenance) you should really make sure your triggers and jobs are configured conciously. There are different configuration options (available misfire instructions) depending on the trigger chosen. Also Quartz behaves differently depending on trigger setup (so called smart policy). Although the misfire instructions are described in the documentation, I found it hard to understand what do they really mean. So I created this small summary article. Before I dive into the details, there is yet another configuration option that should be described. It is org.quartz.jobStore.misfireThreshold (in milliseconds), defaulting to 60000 (a minute). It defines how late the trigger should be to be considered misfired. With default setup if trigger was suppose to be fired 30 seconds ago, Quartz will happily just run it. Such delay is not considered misfiring. However if the trigger is discovered 61 seconds after the scheduled time - the special misfire handler thread takes care of it, obeying the misfire instruction. For test purposes we will set this parameter to 1000 (1 second) so that we can test misfiring quickly. Simple trigger without repeating In our first example we will see how misfiring is handled by simple triggers scheduled to run only once: val trigger = newTrigger(). startAt(DateUtils.addSeconds(new Date(), -10)). build() The same trigger but with explicitly set misfire instruction handler: val trigger = newTrigger(). startAt(DateUtils.addSeconds(new Date(), -10)). withSchedule( simpleSchedule(). withMisfireHandlingInstructionFireNow() //MISFIRE_INSTRUCTION_FIRE_NOW ). build() For the purpose of testing I am simply scheduling the trigger to run 10 seconds ago (so it is 10 seconds late by the time it is created!) In real world you would normally never schedule triggers like that. Instead imagine the trigger was set correctly but by the time it was scheduled the scheduler was down or didn't have any free worker threads. Nevertheless, how will Quartz handle this extraordinary situation? In the first code snippet above no misfire handling instruction is set (so called smart policy is used in that case). The second code snippet explicitly defines what kind of behaviour do we expect when misfiring occurs. See the table: Instruction Meaning smart policy - default See: withMisfireHandlingInstructionFireNow withMisfireHandlingInstructionFireNow MISFIRE_INSTRUCTION_FIRE_NOW The job is executed immediately after the scheduler discovers misfire situation. This is the smart policy. Example scenario: you have scheduled some system clean up at 2 AM. Unfortunately the application was down due to maintenance by that time and brought back on 3 AM. So the trigger misfired and the scheduler tries to save the situation by running it as soon as it can - at 3 AM. withMisfireHandlingInstructionIgnoreMisfires MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICY QTZ-283 See: withMisfireHandlingInstructionFireNow withMisfireHandlingInstructionNextWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_EXISTING_COUNT See: withMisfireHandlingInstructionNextWithRemainingCount withMisfireHandlingInstructionNextWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_REMAINING_COUNT Does nothing, misfired execution is ignored and there is no next execution. Use this instruction when you want to completely discard the misfired execution. Example scenario: the trigger was suppose to start recording of a program in TV. There is no point of starting recording when the trigger misfired and is already 2 hours late. withMisfireHandlingInstructionNowWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_EXISTING_REPEAT_COUNT See: withMisfireHandlingInstructionFireNow withMisfireHandlingInstructionNowWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_REPEAT_COUNT See: withMisfireHandlingInstructionFireNow Simple trigger repeating fixed number of times This scenario is much more complicated. Imagine we have scheduled some job to repeat fixed number of times: val trigger = newTrigger(). startAt(dateOf(9, 0, 0)). withSchedule( simpleSchedule(). withRepeatCount(7). withIntervalInHours(1). WithMisfireHandlingInstructionFireNow() //or other ). build() In this example the trigger is suppose to fire 8 times (first execution + 7 repetitions) every hour, beginning at 9 AM today (startAt(dateOf(9, 0, 0)). Thus the last execution should occur at 4 PM. However assume that due to some reason the scheduler was not capable of running jobs at 9 and 10 AM and it discovered that fact at 10:15 AM, i.e. 2 firings misfired. How will the scheduler behave in this situation? Instruction Meaning smart policy - default See: withMisfireHandlingInstructionNowWithExistingCount withMisfireHandlingInstructionFireNow MISFIRE_INSTRUCTION_FIRE_NOW See: withMisfireHandlingInstructionNowWithRemainingCount withMisfireHandlingInstructionIgnoreMisfires MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICYQTZ-283 Fires all triggers that were missed as soon as possible and then goes back to ordinary schedule. Example scenario: With this strategy in our example the scheduler will fire jobs scheduled at 9 and 10 AM immediately. Then it will wait to 11 AM and go back to ordinary schedule. Note: When handling misfires it is equally important to realize that the actual job execution time might be way after the scheduled time. This means you cannot simply rely on current system date, but you need to use JobExecutionContext .getScheduledFireTime(): def execute(context: JobExecutionContext) { val date = context.getScheduledFireTime //... } withMisfireHandlingInstructionNextWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_EXISTING_COUNT The scheduler won't do anything immediately. Instead it will wait for next scheduled time and run all triggers with scheduled intervals. See also: withMisfireHandlingInstructionNextWithRemainingCount Example scenario: at 10:15 the scheduler discovers 2 misfired executions. It waits until next scheduled time (11 AM) and fires all 8 scheduled executions every hour, stopping at 6 PM (the trigger should have stopped at 4 PM). withMisfireHandlingInstructionNextWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_REMAINING_COUNT The scheduler discards misfired executions and waits for the next scheduled time. The total number of trigger executions will be less then configured. Example scenario: at 10:15 two misfired executions are discarded. The scheduler waits for next scheduled time (11 AM) and fires remaining triggers up to 4 PM. Effectively it behaves as if misfire never occurred. withMisfireHandlingInstructionNowWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_EXISTING_REPEAT_COUNT First misfired trigger is executed immediately. Then the scheduler waits desired interval and executes all remaining triggers. Effectively the first fire time of the misfired trigger is moved to current time with no other changes. Example scenario: at 10:15 the scheduler runs the first misfired execution. Then it waits 1 hour and fires the second one at 11:15 AM. All 8 executions are performed, the last one at 5:15 PM withMisfireHandlingInstructionNowWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_REPEAT_COUNT First misfired execution runs immediately. Remaining misfired executions are discarded. Triggers that were not misfired are executed with desired interval. Example scenario: at 10:15 the scheduler runs the first misfired execution (from 9 AM). It discards remaining misfired executions (the one from 10 AM) and waits 1 hour to execute six more triggers: 11:15, 12:15, … 4:15 PM Simple trigger repeating infinitely In this scenario trigger repeats infinite number of times at a given interval: val trigger = newTrigger(). startAt(dateOf(9, 0, 0)). withSchedule( simpleSchedule(). withRepeatCount(SimpleTrigger.REPEAT_INDEFINITELY). withIntervalInHours(1). WithMisfireHandlingInstructionFireNow() //or other ). build() Once again trigger should fire on every hour, beginning at 9 AM today (startAt(dateOf(9, 0, 0)). However the scheduler was not capable of running jobs at 9 and 10 AM and it discovered that fact at 10:15 AM, i.e. 2 firings misfired. This is a more general situation compared to simple trigger running fixed number of times. Instruction Meaning smart policy - default See: withMisfireHandlingInstructionNextWithRemainingCount withMisfireHandlingInstructionFireNow MISFIRE_INSTRUCTION_FIRE_NOW See: withMisfireHandlingInstructionNowWithRemainingCount withMisfireHandlingInstructionIgnoreMisfires MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICYQTZ-283 The scheduler will immediately run all misfired triggers, then continue on schedule. Example scenario: the triggers scheduled at 9 and 10 AM are executed immediately. Future invocations (next scheduled at 11 AM) are executed according to the plan. withMisfireHandlingInstructionNextWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_EXISTING_COUNT See: withMisfireHandlingInstructionNextWithRemainingCount withMisfireHandlingInstructionNextWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_REMAINING_COUNT Does nothing, misfired executions are discarded. Then the scheduler waits for next scheduled interval and goes back to schedule. Example scenario: Misfired execution at 9 and 10 AM are discarded. The first execution occurs at 11 AM. withMisfireHandlingInstructionNowWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_EXISTING_REPEAT_COUNT See: withMisfireHandlingInstructionNowWithRemainingCount withMisfireHandlingInstructionNowWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_REPEAT_COUNT The first misfired execution is run immediately, remaining are discarded. Next execution happens after desired interval. Effectively the first execution time is moved to current time. Example scenario: the scheduler fires misfired trigger immediately at 10:15 AM. Then waits an hour and runs the second one at 11:15 AM and continues with 1 hour interval. CRON triggers CRON triggers are the most popular ones amongst Quartz users. However there are also two other available triggers: DailyTimeIntervalTrigger (e.g. fire every 25 minutes) and CalendarIntervalTrigger (e.g. fire every 5 months). They support triggering policies not possible in both CRON and simple triggers. However they understand the same misfire handling instructions as CRON trigger. val trigger = newTrigger(). withSchedule( cronSchedule("0 0 9-17 ? * MON-FRI"). withMisfireHandlingInstructionFireAndProceed() //or other ). build() In this example the trigger should fire every hour between 9 AM and 5 PM, from Monday to Friday. But once again first two invocations were missed (so the trigger misfired) and this situation was discovered at 10:15 AM. Note that available misfire instructions are different compared to simple triggers: Instruction Meaning smart policy - default See: withMisfireHandlingInstructionFireAndProceed withMisfireHandlingInstructionIgnoreMisfires MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICYQTZ-283 All misfired executions are immediately executed, then the trigger runs back on schedule. Example scenario: the executions scheduled at 9 and 10 AM are executed immediately. The next scheduled execution (at 11 AM) runs on time. withMisfireHandlingInstructionFireAndProceed MISFIRE_INSTRUCTION_FIRE_ONCE_NOW Immediately executes first misfired execution and discards other (i.e. all misfired executions are merged together). Then back to schedule. No matter how many trigger executions were missed, only single immediate execution is performed. Example scenario: the executions scheduled at 9 and 10 AM are merged and executed only once (in other words: the execution scheduled at 10 AM is discarded). The next scheduled execution (at 11 AM) runs on time. withMisfireHandlingInstructionDoNothing MISFIRE_INSTRUCTION_DO_NOTHING All misfired executions are discarded, the scheduler simply waits for next scheduled time. Example scenario: the executions scheduled at 9 and 10 AM are discarded, so basically nothing happens. The next scheduled execution (at 11 AM) runs on time. QTZ-283Note: QTZ-283: MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICY not working with JDBCJobStore - apparently there is a bug when JDBCJobStore is used, keep an eye on that issue. As you can see various triggers behave differently based on the actual setup. Moreover, even though the so called smart policy is provided, often the decision is based on business requirements. Essentially there are three major strategies: ignore, run immediately and continue and discard and wait for next. They all have different use-cases: Use ignore policies when you want to make sure all scheduled executions were triggered, even if it means multiple misfired triggers will fire. Think about a job that generates report every hour based on orders placed during that last hour. If the server was down for 8 hours, you still want to have that reports generated, as soon as you can. In this case the ignore policies will simply run all triggers scheduled during that 8 hour as fast as scheduler can. They will be several hours late, but will eventually be executed. Use now* policies when there are jobs executing periodically and upon misfire situation they should run as soon as possible, but only once. Think of a job that cleans /tmp directory every minute. If the scheduler was busy for 20 minutes and finally can run this job, you don't want to run in 20 times! One is enough, but make sure it runs as fast it can. Then back to your normal one-minute intervals. Finally next* policies are good when you want to make sure your job runs at particular points in time. For example you need to fetch stock prices quarter past every hour. They change rapidly so if your job misfired and it is already 20 minutes past full hour, don't bother. You missed the correct time by 5 minutes and now you don't really care. It is better to have a gap rather than an inaccurate value. In this case Quartz will skip all misfired executions and simply wait for the next one.

Although briefly described in the official documentation, I believe Quartz plugins aren't known enough, looking at how useful they are. Essentially plugins in Quartz are convenient classes wrapping registration of underlying listeners. You are free to write your own plugins but we will focus on existing ones shipped with Quartz. LoggingTriggerHistoryPlugin First some background. Two main abstractions in Quartz are jobs and triggers. Job is a piece of code that we would like to schedule. Trigger instructs the scheduler when this code should run. CRON (e.g. run every Friday between 9 AM and 5 PM until November) and simple (run 100 times every 2 hours) triggers are most commonly used. You associate any number of triggers to a single job. Believe it or not, Quartz by default provides no logging or monitoring whatsoever of executed jobs and triggers. There is an API, but no built-in logging is implemented. It won't show you that it now executes this particular job due to this trigger firing. So the first thing you should do is adding the following lines to your quartz.properties: org.quartz.plugin.triggerHistory.class=org.quartz.plugins.history.LoggingTriggerHistoryPlugin org.quartz.plugin.triggerHistory.triggerFiredMessage=Trigger [{1}.{0}] fired job [{6}.{5}] scheduled at: {2, date, dd-MM-yyyy HH:mm:ss.SSS}, next scheduled at: {3, date, dd-MM-yyyy HH:mm:ss.SSS} org.quartz.plugin.triggerHistory.triggerCompleteMessage=Trigger [{1}.{0}] completed firing job [{6}.{5}] with resulting trigger instruction code: {9}. Next scheduled at: {3, date, dd-MM-yyyy HH:mm:ss.SSS} org.quartz.plugin.triggerHistory.triggerMisfiredMessage=Trigger [{1}.{0}] misfired job [{6}.{5}]. Should have fired at: {3, date, dd-MM-yyyy HH:mm:ss.SSS} The first line (and the only required) loads the plugin class LoggingTriggerHistoryPlugin. The remaining lines are configuring the plugin, customizing the logging messages. I found the built-in defaults not very well thought, e.g. they display current time which is already part of the logging framework message. You are free to construct any logging message, see the API for details. Adding these extra few lines makes debugging and monitoring much easier: LoggingTriggerHistoryPlugin | Trigger [Demo.Every-few-seconds] fired job [Demo.Print-message] scheduled at: 04-04-2012 23:23:47.036, next scheduled at: 04-04-2012 23:23:51.036 //...job output LoggingTriggerHistoryPlugin | Trigger [Demo.Every-few-seconds] completed firing job [Demo.Print-message] with resulting trigger instruction code: DO NOTHING. Next scheduled at: 04-04-2012 23:23:51.036 You see now why naming your triggers (Demo.Every-few-seconds) and jobs (Demo.Print-message) is so important. LoggingJobHistoryPlugin There is another handy plugin related to logging: org.quartz.plugin.jobHistory.class=org.quartz.plugins.history.LoggingJobHistoryPlugin org.quartz.plugin.jobHistory.jobToBeFiredMessage=Job [{1}.{0}] to be fired by trigger [{4}.{3}], re-fire: {7} org.quartz.plugin.jobHistory.jobSuccessMessage=Job [{1}.{0}] execution complete and reports: {8} org.quartz.plugin.jobHistory.jobFailedMessage=Job [{1}.{0}] execution failed with exception: {8} org.quartz.plugin.jobHistory.jobWasVetoedMessage=Job [{1}.{0}] was vetoed. It was to be fired by trigger [{4}.{3}] at: {2, date, dd-MM-yyyy HH:mm:ss.SSS} The rule is the same - plugin + extra configuration. See JavaDoc of LoggingJobHistoryPlugin for details and possible placeholders. Quick look at logs reveals very descriptive output: Trigger [Demo.Every-few-seconds] fired job [Demo.Print-message] scheduled at: 04-04-2012 23:34:53.739, next scheduled at: 04-04-2012 23:34:57.739 Job [Demo.Print-message] to be fired by trigger [Demo.Every-few-seconds], re-fire: 0 //...job output Job [Demo.Print-message] execution complete and reports: null Trigger [Demo.Every-few-seconds] completed firing job [Demo.Print-message] with resulting trigger instruction code: DO NOTHING. Next scheduled at: 04-04-2012 23:34:57.739 I have no idea why these plugins aren't enabled by default. After all, if you don't want such a verbose output, you can turn it off in your logging framework. Never mind, I think it is a good idea to have them in place when troubleshooting Quartz execution. XMLSchedulingDataProcessorPlugin This is a pretty comprehensive plugin. It reads XML file (by default named quartz_data.xml) containing jobs and triggers definitions and adds them to the scheduler. This is especially useful when you have a global job that you need to add once. Plugin can either update the existing jobs/triggers or ignore the XML file if they already exist - very useful when JDBCJobStore is used. org.quartz.plugin.xmlScheduling.class=org.quartz.plugins.xml.XMLSchedulingDataProcessorPlugin In the aforementioned article we have been manually adding job to the scheduler: val trigger = newTrigger(). withIdentity("Every-few-seconds", "Demo"). withSchedule( simpleSchedule(). withIntervalInSeconds(4). repeatForever() ). build() val job = newJob(classOf[PrintMessageJob]). withIdentity("Print-message", "Demo"). usingJobData("msg", "Hello, world!"). build() scheduler.scheduleJob(job, trigger) The same can be achieved with XML configuration, just place the following quartz_data.xml in your CLASSPATH: false true Every-few-seconds Demo Print-message Demo -1 4000 Print-message Demo com.blogspot.nurkiewicz.quartz.demo.PrintMessageJob msg Hello, World! The file supports both simple and CRON triggers and is well described using XML Schema. It is even possible to point out to an XML files somewhere in the file system and periodically scan them for changes (!) (see: XMLSchedulingDataProcessorPlugin.setScanInterval(). Guess what is Quartz using to schedule periodic scanning? org.quartz.plugin.xmlScheduling.fileNames=/etc/quartz/system-jobs.xml,/home/johnny/my-jobs.xml org.quartz.plugin.xmlScheduling.scanInterval=60 ShutdownHookPlugin Last but not least, ShutdownHookPlugin. Small but probably useful plugin that register shutdown hook in the JVM in order to gently stop the scheduler. However I recommend turning cleanShutdown off - if the system already tries to abruptly stop the application (typically scheduler shutdown is called by Spring via SchedulerFactoryBean) or the user hit Ctrl+C - waiting for currently running jobs seems like a bad idea. After all, maybe we are killing the application because some jobs are running for too long/hanging? org.quartz.plugin.shutdownHook.class=org.quartz.plugins.management.ShutdownHookPlugin org.quartz.plugin.shutdownHook.cleanShutdown=false As you can see Qurtz ships with few quite interesting plugins. For some reason they aren't described in detail in the official documentation, but they work pretty well and are a valuable addition to scheduler. The source code with applied plugins is available on GitHub.

I am starting a little series about Quartz scheduler internals, tips and tricks, this is chapter 0 - how to configure persistent job store.

If you need to schedule jobs in Java, it is fairly common in the industry to use Quartz directly or via Spring integration, but you might want to think twice.

Right. Let’s have a look at the massive technical implications of the Fix Puppet idea. As I mentioned in my earlier blogpost, in order to fix puppet in a sensible way, we’ll have to review all, and overhaul some of the underlying infrastructure that allows it all to run. The interlinks and dependencies between all the parts are a little tricky to visualise. So, here’s a picture. Anything in red needs attention, and the stuff in green *just works*. Things in blue are install stages, and these are what we’re working on making perfect. Right, so we’ve basically got a directed graph, representing the steps and stages that have to happen to a new machine before users can log in. The steps taken to build a machine, roughly look like this: Unbox. Plug in. Configure Netboot. Hand MAC Address to DHCP server and assign a hostname. Client PXEBoots. Client downloads a preseed file. Client installs itself. Client Reboots. Puppet runs on First Boot. Puppet completes. Client Reboots again. Users login That’s about it, really. The first 4 steps are a hell of a lot easier with the support and co-operation of the supplier. It’s nice to have systems preconfigured to PXE boot as the BIOS default, and even cooler if they can send the MAC addresses as labels on each physical machine. If we’re going to build out a new infrastructure, we’re going to need to review and reinstall the servers that provide this infrastructure, before we can build any workstations. I’m a massive massive fan of puppet, and believe that it should be used for the configuration of all servers and workstations. As such, I didn’t want to rebuild anything without using puppet, so the first step, had to be getting puppet working again. So, without further ado, let’s take a look at the Puppet portion of this, well, one of them. My predecessor saw fit that all nodes should be defined with puppet-dashboard, which is itself, a fine piece of software, but I think more for reporting than specification. Initially, at least, I rebuilt the puppet manifest from a known-good configuration. Namely the base configs I wrote for a blogpost about a year ago; base configs that I’m going to update soon. I’m a bit of an old fashioned puppet user. I like my nodes defined in nodes.pp, not some External Node Classifier service. Reason being, I like to be able to look in one place and find exactly what I want. It’s not a massive ballache to clone down the puppet git repo, make a change and push it back up. In fact, it’s better than having a web interface for your node classifications, because git provides you with an intrinsic log of what was changed, and it’s easy to revert to an old version, because everything’s stored in source control. You can also test what you’re about to do, because again, it’s just a source control repo. I’m a fan of having Jenkins run a few sanity checks on your puppet repo, but that’s a digression for another blogpost. I’m not going to go into great depth about how to install DHCP and DNS, and how to make it work with puppet, at least, not here. What I will say, though is that Puppet Module Tool is the most fantastically easy way to generate boilerplate modules for puppet. All you need to do is run puppet-module generate tomoconnor-dhcp and you get a full puppet module folder called tomoconnor-dhcp which contains all the structure according to the best practice guidelines. Excellent. As part of the review process, it became quite apparent that Bind9 has no sensible admin/management interface, or at least, there wasn’t one installed, and frankly, anything that has such horrific config files should be shot. Having had good experience and results using PowerDNS in the past, we decided that this would be a valid upgrade from BIND. PowerDNS relies on a SQL backend for storing the record data in. You can use either MySQL or PostgreSQL, or possibly some others. Since MySQL can be a bitch, and is, to all serious purposes, a toy database, Postgres seems like a better choice. 9.1 is stable, and there are deb package available for it. 9.1 also does hot-standby replication, which is a miracle, because Postgres replication used to be a massive pain in the testicles. There were, initially some mysterious problems with the TFTPd server being generally crappy, mostly regarding timeouts, which was because the storage of the TFTP data was on a painfully slow disk. Moving it from there to the NFS mount dramatically increased performance and stopped TFTP going crazy. In the TFTP'd config, there's a block for configuring the boot options of the preseed install. This is how PXE hands over the details of the preseed server, and the classes of preseed file to run (basically, which modules) label lucid_ws menu label ^2) Auto Install Ubuntu Lucid WorkStation text help Start hands off install of a workstation. endtext menu default kernel ubuntu-1004-installer/amd64/linux append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false vga=normal initrd=ubuntu-1004-installer/amd64/initrd.gz -- quiet auto debian-installer/country=GB debian-installer/language=en debian-installer/keymap=us debian-installer/locale=en_GB.UTF8 netcfg/choose_interface=eth0 netcfg/get_hostname=ubuntu netcfg/get_domain=installdomain.wibblesplat.com url=http://autoserver/d-i/lucid/preseed.cfg classes=wibblesplat;workstation DEBCONF_DEBUG=1 Initially, the Preseed files contained all sorts of crazy hacky shit in the d-i late-command setting. late-command is cool. It’s basically the last thing to run before the first reboot when you build a new debian/ubuntu system. You can tell it to do all sorts of stuff in there. You probably shouldn’t, though. Especially when what you’re doing in there is better done elsewhere. The previous Preseed file contained a whole bunch of “inject these source files into /etc/apt/sources.list”, which is utter bullshit, because you can do exactly the same thing with d-i local repositories, which does the same thing, only far far cleaner. That’s not to say that my refactored preseed files don’t use late-command at all. I’ve chosen to insert some lines into /etc/rc.local on the freshly built system that ensures a puppet run at first boot. On the preseed server, there’s a file called “firstboot.sh” which gets dropped into /usr/local/bin by way of a wget command in late-command. The next thing that happens in late-command is a line to remove “exit 0” from /etc/rc.local and replace it with a thing that calls “/usr/local/bin/firstboot.sh” When firstboot runs, it runs puppet, checks for sanity, and then removes itself from /etc/rc.local. The code to actually do that looks like this: d-i preseed/late_command string \ wget -q -O /target/root/firstboot.sh http://autoserver/d-i/bin/firstboot.sh && \ chmod +x /target/root/firstboot.sh && \ sed -i 's_exit 0_sh /root/firstboot.sh_' /target/etc/rc.local This relies on having something on http://autoserver that is basically just apache hosting some files for the preseeder to retrieve during installation. Cool huh? That ensures that the first thing that happens once the new machine has been built and rebooted, is a puppet run. Some stuff we do here relies on our hand-rolled deb packages, which are stored in our own, internal APT repo. We’ve also got an APT cache, created and maintained by apt-cacher-ng, which at least means that when you’re rebuilding systems frequently, that all the packages you would otherwise download from archive.ubuntu.com come straight over the LAN. The major problem initially with this was the speed, or lack of. It certainly wasn’t performing anywhere near speeds you’d expect from a 1GE LAN, and the reason was again, slow disks. Moving the apt-cache files to the NFS highspeed storage again helped performance. If we struggle in future, I’m going to look at a SSD cache for this, but I think that the performance of the SAS/SATA disks on massively parallel storage provided by our NFS servers will be adequate for the forseeable future. Next up, the Puppetmaster. Again, I was pretty keen on building this from scratch, but using puppet itself to configure it’s own master. Sounds pretty counter-intuitive, right? But the puppet client can bootstrap the master quite easily by using files as it’s source. The first step is to clone down the latest puppet manifests from git, so you either need to git export elsewhere, or install git-core. Your choice. Once you’ve got those, all you need to do is install puppet-client, and run: puppet apply /path/to/your/manifests/site.pp If you’ve written the manifests right, and you’ve got your master defined as a node, you should find that puppet will install puppetmaster, and so on, and then you get a ready and working puppetmaster that just configured itself. I used puppet-module tool to generate modules for the following services/items: “applications” - which actually contains a bunch of custom/proprietary application install rules, a declassified example is there’s a googlechrome.pp file that installs chrome from a PPA. Other modules: dhcp, kernel, ldap, network, nfs, nscd, ntp, nvidia, postgres, powerdns and ssmtp. As is the trend with puppet, and modern DevOps, a vast majority of the code in the entire manifest repository has been gleaned and researched from other puppet modules on github. Acknowledgement is in place where it’s due, and the working copies we’re using are frequently forked on github from the original. It’s great, this, actually. If you search on PuppetForge http://forge.puppetlabs.com/ the array of modules available is staggering. It makes bootstrapping a new manifest set remarkably quick and easy. The NFS module contains a bunch of requirements for mounting NFS shares, and the definitions for an NFS share to be mounted. All pretty simple stuff, but modularised for ease of use. I’m particularly proud of the postgres module which has a master class, and a slave class, which installs and configures the required files and packages to enable streaming hot-standby replication on Postgres9.1 I will release the declassified fork of this soon. I’m going to wrap this post up here. It’s a massively long one, and there’s still lots more left to write. Source: tomoconnor.eu/blogish/low-level-infrastructure-puppet-dns-and-dhcp/

A collection of short how-to's for enabling JMX in several popular Java technologies. Continuing our journey with JMX (see: ...JMX for human beings) we will learn how to enable JMX support (typically statistics and monitoring capabilities) in some popular frameworks. Most of this information can be found on project's home pages, but I decided to collect it with few the addition of some useful tips. Hibernate (with Spring support) Exposing Hibernate statistics with JMX is pretty simple, however some nasty workarounds are requires when JPA API is used to obtain underlying SessionFactory class JmxLocalContainerEntityManagerFactoryBean() extends LocalContainerEntityManagerFactoryBean { override def createNativeEntityManagerFactory() = { val managerFactory = super.createNativeEntityManagerFactory() registerStatisticsMBean(managerFactory) managerFactory } def registerStatisticsMBean(managerFactory: EntityManagerFactory) { managerFactory match { case impl: EntityManagerFactoryImpl => val mBean = new StatisticsService(); mBean.setStatisticsEnabled(true) mBean.setSessionFactory(impl.getSessionFactory); val name = new ObjectName("org.hibernate:type=Statistics,application=spring-pitfalls") ManagementFactory.getPlatformMBeanServer.registerMBean(mBean, name); case _ => } } } Note that I have created a subclass of Springs built-in LocalContainerEntityManagerFactoryBean. By overriding createNativeEntityManagerFactory() method I can access EntityManagerFactory and by trying to downcast it to org.hibernate.ejb.EntityManagerFactoryImpl we were able to register Hibernate Mbean. One more thing has left. Obviously we have to use our custom subclass instead of org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean. Also, in order to collect the actual statistics instead of just seeing zeroes all the way down we must set the hibernate.generate_statistics flag. @Bean def entityManagerFactoryBean() = { val entityManagerFactoryBean = new JmxLocalContainerEntityManagerFactoryBean() entityManagerFactoryBean.setDataSource(dataSource()) entityManagerFactoryBean.setJpaVendorAdapter(jpaVendorAdapter()) entityManagerFactoryBean.setPackagesToScan("com.blogspot.nurkiewicz") entityManagerFactoryBean.setJpaPropertyMap( Map( "hibernate.hbm2ddl.auto" -> "create", "hibernate.format_sql" -> "true", "hibernate.ejb.naming_strategy" -> classOf[ImprovedNamingStrategy].getName, "hibernate.generate_statistics" -> true.toString ).asJava ) entityManagerFactoryBean } Here is a sample of what can we expect to see in JvisualVM (don't forget to install all plugins!): In addition we get a nice Hibernate logging: HQL: select generatedAlias0 from Book as generatedAlias0, time: 10ms, rows: 20 EhCache Monitoring caches is very important, especially in application where you expect values to generally be present there. I tend to query the database as often as needed to avoid unnecessary method arguments or local caching. Everything to make code as simple as possible. However this approach only works when caching on the database layer works correctly. Similar to Hibernate, enabling JMX monitoring in EhCache is a two-step process. First you need to expose provided MBean in MBeanServer: @Bean(initMethod = "init", destroyMethod = "dispose") def managementService = new ManagementService(ehCacheManager(), platformMBeanServer(), true, true, true, true, true) @Bean def platformMBeanServer() = ManagementFactory.getPlatformMBeanServer def ehCacheManager() = ehCacheManagerFactoryBean.getObject @Bean def ehCacheManagerFactoryBean = { val ehCacheManagerFactoryBean = new EhCacheManagerFactoryBean ehCacheManagerFactoryBean.setShared(true) ehCacheManagerFactoryBean.setCacheManagerName("spring-pitfalls") ehCacheManagerFactoryBean } Note that I explicitly set CacheManager name. This is not required but this name is used as part of the Mbean name and a default one contains hashCode value, which is not very pleasant. The final touch is to enable statistics on a cache basis: Now we can happily monitor various caching characteristics of every cache separately: As we can see the percentage of cache misses increases. Never a good thing. If we don't enable cache statistics, enabling JMX is still a good idea since we get a lot of management operations for free, including flushing and clearing caches (useful during debugging and testing). Quartz scheduler In my humble opinion Quartz scheduler is very underestimated library, but I will write an article about it on its own. This time we will only learn how to monitor it via JMX. Fortunately it's as simple as adding: org.quartz.scheduler.jmx.export=true To quartz.properties file. The JMX support in Quartz could have been slightly broader, but still one can query e.g. which jobs are currently running. By the way the new major version of Quartz (2.x) brings very nice DSL-like support for scheduling: val job = newJob(classOf[MyJob]) val trigger = newTrigger(). withSchedule( repeatSecondlyForever() ). startAt( futureDate(30, SECOND) ) scheduler.scheduleJob(job.build(), trigger.build()) Apache Commons DBCP Apache Commons DBCP is the most reasonable JDBC pooling library I came across. There is also c3p0, but it doesn't seem like it's actively developed any more. Tomcat JDBC Connection Pool looked promising, but since it's bundled in Tomcat, your JDBC drivers can no longer be packaged in WAR. The only problem with DBCP is that it does not support JMX. At all (see this two and a half year old issue). Fortunately this can be easily worked around. Besides we will learn how to use Spring built-in JMX support. Looks like the standard BasicDataSource has all what we need, all we have to do is to expose existing metrics via JMX. With Spring it is dead-simple – just subclass BasicDataSource and add @ManagedAttribute annotation over desired attributes: @ManagedResource class ManagedBasicDataSource extends BasicDataSource { @ManagedAttribute override def getNumActive = super.getNumActive @ManagedAttribute override def getNumIdle = super.getNumIdle @ManagedAttribute def getNumOpen = getNumActive + getNumIdle @ManagedAttribute override def getMaxActive: Int= super.getMaxActive @ManagedAttribute override def setMaxActive(maxActive: Int) { super.setMaxActive(maxActive) } @ManagedAttribute override def getMaxIdle = super.getMaxIdle @ManagedAttribute override def setMaxIdle(maxIdle: Int) { super.setMaxIdle(maxIdle) } @ManagedAttribute override def getMinIdle = super.getMinIdle @ManagedAttribute override def setMinIdle(minIdle: Int) { super.setMinIdle(minIdle) } @ManagedAttribute override def getMaxWait = super.getMaxWait @ManagedAttribute override def setMaxWait(maxWait: Long) { super.setMaxWait(maxWait) } @ManagedAttribute override def getUrl = super.getUrl @ManagedAttribute override def getUsername = super.getUsername } Here are few data source metrics going crazy during load-test: JMX support in the Spring framework itself is pretty simple. As you have seen above exposing arbitrary attribute or operation is just a matter of adding an annotation. You only have to remember about enabling JMX support using either XML or Java (also see: SPR-8943 : Annotation equivalent to with @Configuration): or: @Bean def annotationMBeanExporter() = new AnnotationMBeanExporter() This article wasn't particularly exciting. However, the knowledge of JMX metrics will enable us to write simple yet fancy dashboards in no time. Stay tuned! From http://nurkiewicz.blogspot.com/2011/12/enabling-jmx-in-hibernate-ehcache-qurtz.html

For most large web applications, uptime is of foremost importants. Any outage can be seen by customers as a frustration, or opportunity to move to a competitor. What's more for a site that also includes e-commerce, it can mean real lost sales. Zero Downtime describes a site without service interruption. To achieve such lofty goals, redundancy becomes a critical requirement at every level of your infrastructure. If you're using cloud hosting, are you redundant to alternate availability zones and regions? Are you using geographically distributed load balancing? Do you have multiple clustered databases on the backend, and multiple webservers load balanced. All of these requirements will increase uptime, but may not bring you close to zero downtime. For that you'll need thorough testing. The solution is to pull the trigger on sections of your infrastructure, and prove that it fails over quickly without noticeable outage. The ultimate test is the outage itself. Sean Hull on Quora: What is zero downtime and why is it important? Source: http://www.iheavy.com/2011/06/23/zero-downtime-what-is-it-and-why-is-it-important/

This article shows *one* way to mock the JMS infrastructure in a Spring JMS application. This allows us to test our JMS infrastructure without actually having to depend on a physical connection being available. If you are reading this article, chances are that you are also frustrated with failing tests in your continuous integration environment due to a JMS server being (temporarily) unavailable. By mocking the JMS provider, developers are left free to test not only the functionality of their API (unit tests) but also the plumbing of the different components, e.g. in a Spring container. In this article I show how a Spring JMS Hello World application can be fully tested without the need of a physical JMS connection. I would like to stress the fact that the code in this article is by no means meant for production and that the approach shown is just one of many. The infrastructure For this article I use the following infrastructure: Apache ActiveMQ, an open source JMS provider, running on an Ubuntu installation Spring 3 Java 6 MockRunner Eclipse as development environment, running on Windows 7 The Spring configuration It's my belief that using what I define as Spring Configuration Strategy Pattern (SCSP) is the right solution in almost all cases when there is the need for a sound testing infrastructure. I will dedicate an entire article to SCSP, for now this is how it looks: The Spring application context Here follows the content of jemosJms-appContext.xml The only important thing to note here is that there are some services which rely on an existing bean named jmsConnectionFactory but that such bean is not defined in this file. This is key to the SCSP and I will illustrate this in one of my future articles. The Spring application context implementation Here follows the content of jemosJms-appContextImpl.xml which could be seen as an implementation of the Spring application context defined above This Spring context file imports the Spring application context defined above and it is this application context which declared the connection factory. This decoupling of the bean requirement (in the super context) from its actual declaration (Spring application context implementation) represents the cornerstore of SCSP. Mocking the JMS provider - The Spring Test application context and MockRunner Following the same approach I used above, I can now declare a fake connection factory which does not require a physical connection to a JMS provider. Here follows the content of jemosJmsTest-appContext.xml. Please note that this file should reside in the test resources of your project, i.e. it should never make it to production. Here the Spring test application context file imports the Spring application context (not its implementation) and it declares a fake connection factory, thanks to the MockRunner MockQueueConnectionFactory class. A POJO listener The job of handling the message is delegated to a simple POJO, which happens to be declared also as a bean: package uk.co.jemos.experiments; public class HelloWorldHandler { /** The application logger */ private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger .getLogger(HelloWorldHandler.class); public void handleHelloWorld(String msg) { LOG.info("Received message: " + msg); } } There is nothing glamorous about this class. In real life this should have probably be the implementation of an interface, but here I wanted to keep things simple. A simple JMS message producer Here follows an example of a JMS message producer, which would use the real JMS infrastructure to send messages: package uk.co.jemos.experiments; import org.springframework.context.ApplicationContext; import org.springframework.context.support.ClassPathXmlApplicationContext; import org.springframework.jms.core.JmsTemplate; public class JmsTest { /** The application logger */ private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger .getLogger(JmsTest.class); /** * @param args */ public static void main(String[] args) { ApplicationContext ctx = new ClassPathXmlApplicationContext( "classpath:jemosJms-appContextImpl.xml"); JmsTemplate jmsTemplate = ctx.getBean(JmsTemplate.class); jmsTemplate.send("jemos.tests", new HelloWorldMessageCreator()); LOG.info("Message sent successfully"); } } The only thing of interest here is that this class retrieves the real JmsTemplate to send a message to the queue. Now if I was to run this class as is, I would obtain the following: 2011-07-31 17:09:46 ClassPathXmlApplicationContext [INFO] Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@19e0ff2f: startup date [Sun Jul 31 17:09:46 BST 2011]; root of context hierarchy 2011-07-31 17:09:46 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemosJms-appContextImpl.xml] 2011-07-31 17:09:46 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemosJms-appContext.xml] 2011-07-31 17:09:46 DefaultListableBeanFactory [INFO] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@3479e304: defining beans [helloWorldConsumer,jmsTemplate,org.springframework.jms.listener.DefaultMessageListenerContainer#0,jmsConnectionFactory]; root of factory hierarchy 2011-07-31 17:09:46 DefaultLifecycleProcessor [INFO] Starting beans in phase 2147483647 2011-07-31 17:09:47 HelloWorldHandler [INFO] Received message: Hello World 2011-07-31 17:09:47 JmsTest [INFO] Message sent successfully Writing the integration test There are various interpretations as to what different types of tests mean and I don't pretend to have the only answer; my interpreation is that an integration test is a functional test which also wires up different components together but which does not interact with real external infrastructure (e.g. a Dao integration test fakes data, a JMS integration test fakes the JMS physical connection, an HTTP integration test fakes the remote Web host, etc). Whereas in my opinion, the main purpose of a unit (aka functional) test is to let the API emerge from the tests, the main goal of an integration test is to test that the plumbing amongst components works as expected so as to avoid surprises in a production environment. Both unit (functional) and integration tests should run very fast (e.g. under 10 minutes) as they constitute what can be considered the "development token". If unit and integration tests are green one should feel pretty confident that 90% of the functionality works as expected; in my projects when both unit and integration tests are green I let developers free to release the token. This does not mean that the other 10% (e.g. the interaction with the real infrastructure) should not be tested, but this can be delegated to system tests which run nightly and don't require the development token. Because unit and integration tests need to run fast, interaction with external infrastructure should be mocked whenever possible. Here follows an integration test for the Hello World handler: package uk.co.jemos.experiments.test.integration; import javax.annotation.Resource; import javax.jms.TextMessage; import junit.framework.Assert; import org.junit.Before; import org.junit.Test; import org.springframework.jms.core.JmsTemplate; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.AbstractJUnit4SpringContextTests; import uk.co.jemos.experiments.HelloWorldHandler; import uk.co.jemos.experiments.HelloWorldMessageCreator; import com.mockrunner.jms.DestinationManager; import com.mockrunner.mock.jms.MockQueue; /** * @author mtedone * */ @ContextConfiguration(locations = { "classpath:jemosJmsTest-appContextImpl.xml" }) public class HelloWorldHandlerIntegrationTest extends AbstractJUnit4SpringContextTests { @Resource private JmsTemplate jmsTemplate; @Resource private DestinationManager mockDestinationManager; @Resource private HelloWorldHandler helloWorldHandler; @Before public void init() { Assert.assertNotNull(jmsTemplate); Assert.assertNotNull(mockDestinationManager); Assert.assertNotNull(helloWorldHandler); } @Test public void helloWorld() throws Exception { MockQueue mockQueue = mockDestinationManager.createQueue("jemos.tests"); jmsTemplate.send(mockQueue, new HelloWorldMessageCreator()); TextMessage message = (TextMessage) jmsTemplate.receive(mockQueue); Assert.assertNotNull("The text message cannot be null!", message.getText()); helloWorldHandler.handleHelloWorld(message.getText()); } } And here follows the output: 2011-07-31 17:17:26 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemosJmsTest-appContextImpl.xml] 2011-07-31 17:17:26 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemosJms-appContext.xml] 2011-07-31 17:17:26 GenericApplicationContext [INFO] Refreshing org.springframework.context.support.GenericApplicationContext@f01a1e: startup date [Sun Jul 31 17:17:26 BST 2011]; root of context hierarchy 2011-07-31 17:17:27 DefaultListableBeanFactory [INFO] Pre-instantiating singletons in org.springframework.beans.factory.support. DefaultListableBeanFactory@39478a43: defining beans [helloWorldConsumer,jmsTemplate,org.springframework.jms.listener.DefaultMessageListener Container#0,destinationManager,configurationManager,jmsConnectionFactory,org.springframework.context.annotation.internalConfigurationAnnotation Processor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequired AnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy 2011-07-31 17:17:27 DefaultLifecycleProcessor [INFO] Starting beans in phase 2147483647 2011-07-31 17:17:27 HelloWorldHandler [INFO] Received message: Hello World 2011-07-31 17:17:27 GenericApplicationContext [INFO] Closing org.springframework.context.support.GenericApplicationContext@f01a1e: startup date [Sun Jul 31 17:17:26 BST 2011]; root of context hierarchy 2011-07-31 17:17:27 DefaultLifecycleProcessor [INFO] Stopping beans in phase 2147483647 2011-07-31 17:17:32 DefaultMessageListenerContainer [WARN] Setup of JMS message listener invoker failed for destination 'jemos.tests' - trying to recover. Cause: Queue with name jemos.tests not found 2011-07-31 17:17:32 DefaultListableBeanFactory [INFO] Destroying singletons in org.springframework.beans.factory.support. DefaultListableBeanFactory@39478a43: defining beans [helloWorldConsumer,jmsTemplate,org.springframework.jms.listener.DefaultMessageListener Container#0,destinationManager,configurationManager,jmsConnectionFactory,org.springframework.context.annotation.internalConfigurationAnnotationProcessor ,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context. annotation.internalCommonAnnotationProcessor]; root of factory hierarchy In this test, although we simulated a message roundtrip to a JMS queue, the message never left the current JVM and it the whole execution did not depend on a JMS infrastructure being up. This gives us the power to simulate the JMS infrastructure, to test the integration of our business components without having to fear a red from time to time due to JMS infrastructure being down or inaccessible. Please note that in the output there are some warnings because the JMS listener container declared in the jemosJms-appContext.xml does not find a queue named "jemos.test" in the fake connection factory, but this is fine; it's a warning and does not impede the test from running successfully. The Maven configuration Here follows the Maven pom.xml to compile the example: 4.0.0 uk.co.jemos.experiments jmx-experiments 0.0.1-SNAPSHOT Jemos JMS experiments junit junit 4.8.2 test com.mockrunner mockrunner 0.3.1 test log4j log4j 1.2.16 compile org.slf4j slf4j-api 1.6.1 compile org.slf4j slf4j-simple 1.6.1 compile org.apache.activemq activemq-all 5.5.0 compile org.springframework spring-beans 3.0.5.RELEASE org.springframework spring-context 3.0.5.RELEASE org.springframework spring-core 3.0.5.RELEASE org.springframework spring-jms 3.0.5.RELEASE org.springframework spring-test 3.0.5.RELEASE test From http://tedone.typepad.com/blog/2011/07/mocking-spring-jms-with-mockrunner.html

In the old days... You would have a closet in your startup company with a rack of computers. Provisioning involved: Deciding on your architectural direction, what, where & how Ordering the new hardware Waiting weeks for the packages to arrive Setup the hardware, wire things together, power up Discover some component is missing, or failed and order replacement Wait longer... Finally get all the pieces setup Configure software components and go Along came some industrious folks who realized power and data to your physical location wasn't reliable. So datacenters sprang up. With data centers, most of the above steps didn't change except between steps 3 & 4 you would send your engineers out to the datacenter location. Trips back and forth ate up time and energy. Then along came managed hosting. Managed hosting saved companies a lot of headache, wasted man hours, and other resources. They allowed your company to do more of what it does well, run the business, and less on managing hardware and infrastructure. Provisioning now became: Decide on architecture direction Call hosting provider and talk to sales person Wait a day or two Setup & configure software components and go Obviously this new state of affairs improved infrastructure provisioning dramatically. It simplified the process and sped it up as well. What's more a managed hosting provider could keep spare parts and standard components on hand in much greater volume than a small firm. That's a big plus. This evolution continued because it was a win-win for everyone. The only downside was when engineers made mistakes, and finger pointing began. But despite all of that, a managed hosting provider which does only that, can do it better, and more reliably than you can yourself. So where are we in present day? We are all either doing, or looking out cloud provisioning of infrastructure. What's cloud provisioning? It is a complete paradigm shift, but along the same trajectory as what we've described above. Now you removed all the waiting. No waiting for sales team, or the ordering process. That's automatic. No waiting for engineers to setup the servers, they're already setup. They are allocated by your software and scripts. Even the setup and configuration of software components, Operating System and services to run on that server - all automatic. This is such a dramatic shift, that we are still feeling the affects of it. Traditional operations teams have little experience with this arrangement, and perhaps little trust in virtual servers. Business units are also not used to handing the trigger to infrastructure spending over to ops teams or to scripts and software. However the huge economic pressures continue to push firms to this new model, as well as new operational flexibility. Gartner predicts this trend will only continue. The advantages of cloud infrastructure provisioning include: Metered payment - no huge outlay of cash for new infrastructure Infrastructure as a service - scripted components automate & reduced manual processes Devops - Manage infrastructure like code with version control and reproduceability Take unused capacity offline easily & save on those costs Disaster Recovery is free - reuse scripts to build standard components Easily meet seasonal traffic requirements - spinup additional servers instantly On Quora Sean Hull asks - What is infrastructure provisioning and why is it important?

It seems that lots of projects/products/services want to expose a REST API these days. But I have found very few that actually follow the REST constraints, and in a lot of the cases it doesn't even make sense for them to follow REST constraints in the first place. One of the main constraints that is commonly violated is the hypertext constraint. Basically, all state changes have to be done by following links, starting from a bookmarked URL. But almost noone does that. However, should they? This article will outline various layers that REST API's can be implemented in, and when it makes sense, and when not. To begin with, in a typical enterprise app there are three options for layers that you might want to expose using a REST API. These are the infrastructure layer, the domain layer, and the application layer. Infrastructure layer If we start with the infrastructure layer, we are typically talking about a database vendor that wants to allow developers to access it using "REST". The API would allow you to create/remove databases, and then insert/update/delete data. Typically it's pretty normal stuff, and the API doesn't change all that much between versions. Accessing this over HTTP maybe makes sense, but is it RESTful? I'll give you an example. I installed CouchDB, and given the hypermedia constraint I should then be able to go to "http://localhost:5984/", and it will tell me what I can do next (like create a database). But when I do a GET on that URL I get this: {"couchdb":"Welcome","version":"1.0.1"} So now what? The hypermedia doesn't tell me what I can do, so therefore as a REST client I will assume there's nothing I can do. This very simple test shows that the HTTP API for CouchDB isn't really RESTful at all. The question is: should it be? That is obviously up to the developers to decide. But if I were the architect I would maybe say, no, it shouldn't be RESTful. Why? Because I want to allow URL templates to be used, so that the client, given the server URL and a document id, is allowed to construct a URL on its own and GET the document. If this was truly RESTful the client would have to do a query in a form first, with the id, in order to get the URL of the document to be retrieved. That might be inefficient for a database, so I might opt not to do this. Which is, in effect, what they already have done. The only problem is that they call it RESTful, when it isn't, so it gives me as a developer the wrong impression of what I can expect from it. This line of reasoning could be done for pretty much most infrastructure layer API's. They're not RESTful, though many say they are, and most likely they shouldn't try to be! IT'S OK! Just say "Accessible over HTTP, see docs for URL templates and whatnot", and be done with it. Domain layer The next potential layer to be exposed over REST is the domain layer. This typically means that you take your domain entities and expose their data straight on the web, through CRUD operations. Very straightforward. There are tons of articles and blogs that show how to do this. But is it RESTful? Or is it even a good idea in the first place? The first test, again, would be to see if the app follows the hypermedia constraint. In this option it is technically possible to allow queries that will list the various URL's to entities in your domain, which you then can update/delete. So on the surface it might seem like you are following the hypermedia constraint. The problem usually comes with the fact that you are exposing domain state rather than application state. Let me explain through a simple example. Let's say you are building an issue tracker. You can access individual issues through links like: /issue/123 which on GET gives you documents such as: {"status":"OPEN","description":"Some issue"} Awesome. Now a client can change the status to "CLOSED" and PUT that. Tada! Case closed. Or is it? What if a client then decides to reopen it, by simply posting a new status of "OPEN" to it. Ok, that worked. But should it? Maybe your domain model really would have wanted it to only go to "REOPENED" from the "CLOSED" state. But how do you express that? How is the client to know that this is the only valid transition? And what happens when we have many versions of clients, each of which has a slightly different set of rules for what you are allowed to do when? Basically, chaos is ensured. And this is the problem with exposing your domain model using a REST API. The client has to own the application logic, and there's no way the server can be sure that it has the "right" logic. And the client, even if it *wants* to play nice (if code ever wants anything is debatable), will have a hard time knowing whether it is playing by the rules or not. It might even get a bit neurotic, trying to do the right thing, whatever that means. In summary, exposing your domain model does not help the client know what the valid state transitions are, and makes it very hard to do other things like role-based security authorization (maybe only an admin is allowed to REOPEN a CLOSED case?). I would therefore recommend that noone exposes their domain models using a REST API. Application layer Finally we come to the application layer. The application layer is designed to implement usecases of the domain model, and has all the context and logic needed to ensure that only valid state transitions are made. In short, it seems like it is especially appropriate to being exposed through a REST API, as it can at all points tell the client what it can do (either based on state or authorization rules or any other type of rules it might have). If we go back to the issue tracker, what would this mean in practice? It could mean that when you do a GET on /issue/123 you get something like this back: {"data":{"status":"OPEN","description":"Some issue"},"links":[{"close":"/issue/123/close.json"}]} This now instead of referring to viewing the domain state of an issue refers to the usecase of viewing an issue with the intent of working on it. There might be other URL's and other queries that only return the data, or maybe a table of the data, or somesuch. But this one, specifically, refers to the usecase of working with the issue. So, as a REST client I can now inspect the data, and then look at what links are available. If the client has a UI it can enable a button that says "Close issue" based on the available link, since it detected a link relation "close" that it understands. The client can then do GET on that link, find out whether the server expects any form to be filled in, and then submit it using POST, thereby letting the server application layer logic transition the issue to the "CLOSED" state. We are no longer relying on the client to contain the logic of knowing when to allow what, and the client also does not have to know how to construct the URL. As long as it can parse the hypertext (and we might use a custom JSON mediatype to indicate what "data" and "links" mean) and do something with it, we're fine. If we in the future change the domain model to also allow the "resolve" link relation for "OPEN" issues, old clients can ignore it, and new clients can enable new actions in the UI that uses it. In summary, the application layer is a very good candidate to be exposed through a REST API. It encapsulates the application rules for when the various state transitions are allowed, and can make use of user authorization to further enable/disable actions. This takes away a lot of responsibilities from the client, which now also can be "dynamic" in the sense that it can easily react to what state changes are available when by simply checking link availability in the hypermedia returned from the server. The main issue with exposing the application layer through a REST API is that there are pretty much no available frameworks that help you do all this in an easy way. But this is not REST's "fault", obviously, but rather that the "REST" community hasn't yet matured to understand what it should and what it should not do. In the Streamflow project we rolled our own simple framework for doing the above, and I'm very happy with that, but unfortunately most other frameworks seems to be in the "expose your domain model" camp, which means that a lot of this link management is non-trivial to do. This is a fixable situation though. I hope that this post has somewhat clarified what the issues are with exposing infrastructure and domain models through REST API's, and why it's not really a good idea in the first place, and why exposing the application layer really is the logical and simpler option. From http://www.jroller.com/rickard/entry/rest_api_for_infrastructure_domain

Continuous Integration commonly known as CI is a process that consists of continuously compiling, testing, inspecting, and deploying source code. In any typical CI environment, this means running a new build every time code changes within a version control repository. Martin Fowler describes CI as: A software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. While CI is actually a process, the term Continuous Integration often is associated with three important tools in particular. As shown in the image the three pillars of CI are: 1. A version control repository like Subversion, or CVS. 2. A CI Server such as Hudson, or Cruise Control 3. An automated build process like Ant or Nant So, let’s look at each of these in detail: Version Control Repository: Version control repositories also known as SCM (source code management) play a crucial role in any software development environment. They also play a very important role for a successful CI process. The SCM is a central place for the team to store every needed artifact for the project. It is mandatory for the teams to put everything needed for a successful build into this repository. This includes the build scripts, property files, database scripts, all the libraries required to build the software and so on. The CI Server: For CI to function properly, we also need to have an automated process that monitors a version control repository and runs a build when any changes are detected. There are several CI servers available, both open source and commercial. Most of them are similar in their basic configuration and monitor a particular version control repository and run builds when any changes are detected. Some of the most commonly used open source CI servers are; Cruise Control, Continuum, and Hudson. Hudson is particularly interesting because of its ease of configuration and compelling plug-ins, which makes integration with test and static analysis tools much easier. Automated Build: The process of CI is about building software often, which is accomplished through the use of a build. A sturdy build strategy is by far the most important aspect of a successful CI process. In the absence of a solid build that does more than compile your code, CI withers. With automated builds, teams can reliably perform (in an automated fashion) otherwise manual tasks like compilation, testing, and even more interesting things like software inspection and deployment. Now that we have seen the important tools in our CI process, let’s see how a typical CI scenario looks like for a developer: CI server is configured to poll the version control repository continuously for changes. Developer commits code to the repository. CI server detects this change, and retrieves the latest code from the repository. This causes the CI server to invoke the build script with the given targets and options. If configured, CI Server will send out an e-mail to the specified recipients when a certain important event occurs. The CI server continues to poll for changes. Why is CI Important? This is one of the most frequently asked questions, and here are a few points to note about this powerful technique: Building software often greatly increases the likelihood that you will spot defects early, when they still are relatively manageable. Extends defect visibility. CI ensures that you have production ready software at every change. CI also ensures that you have reduced the risk of integration issues by building software at every change. CI server can also be configured to run continuous inspection which can assist the development team in finding potential bugs, bad programming practice, automatically check coding standards, and also provide valuable feedback on the quality of code being written. Over the past several months, I have assisted several companies in implementing CI. There was a little bit of resistance from the developers in the early stages when we implemented continuous feedback. But, never heard a single negative comment about this approach. If you already have a version control repository and automated builds, you are very close to the CI process. Download one of the open source CI servers, configure and setup a simple project. It should take less than an hour if you have automated build scripts. Start adding additional features like code inspections, generating reports, metrics, documentation and so on. Most important, send continuous feedback to your team. Give this process a try, you sure will be surprised to see how effective it is. And, as always share your thoughts, concerns or questions.

Learn how to properly configure logs in JBoss.

Using patches is a popular way to share changes between the teammates or supply updates to software products to the customers. With IntelliJ IDEA, creating and applying versioned patches is quite simple and intuitive: you can do it from the main Version Control menu, or from the Changes tool window. However, IntelliJ IDEA suggests an additional way to create and apply your “personal” patches. As we have discussed earlier, numerous changes pass unnoticed by the version control systems, because you just do not check in every change you make to your files while working. You know that IntelliJ IDEA keeps your own “personal version control” – the local history. Besides the possibility to roll back to a certain revision, you can also create a patch on the base of a revision or action, share it with your colleagues, and apply it when necessary. Local history applies to the folders, files, members and fragments of text, but the technique of creating a patch is common in all cases. Let’s see how it’s done. Select a folder in the Project tool window, and choose Local History on its context menu. In the Local History view, right-click the desired revision, and choose Create Patch: [img_assist|nid=1204|title=|desc=|link=none|align=left|width=505|height=357] In the dialog box that opens, specify the name and location of the patch file: [img_assist|nid=1205|title=|desc=|link=none|align=left|width=415|height=136] An interesting possibility is suggested by the Reverse patch checkbox. If you check this option, IntelliJ IDEA will create a patch that rolls back the selected action. For example, it you have created a file, the patch will delete it. Applying your “personal” patch is done as usual, using the Apply Patch command on the main Version Control menu. If a patch file is stored in project, you can invoke this command on the context menu of the patch file in the Project tool window: [img_assist|nid=1206|title=|desc=|link=none|align=left|width=249|height=195]