technoChord

I recently started working in an environment where we have to develop software using teams that are geographically dispersed over several timezones. There are many challenges (and benefits) of such a set up. The one I will address today is to do with team collaboration and source control.
I have been a user of Subversion and the Atlassian suite of tools for some time now but discovered that when it comes to distributed teams, JIRA and Cruicible fall short. One obvious shortcoming is the inability for team members to do code reviews without checking in un-reviewed code into Source Control. I will use this as the basis of making a case to switch to a Distributed Version Control System, and in doing so, will describe how to use Git.

Git is complex. But distributed version control is a complex problem so it is understandable that the internals of Git makes you feel like you should have a PhD in Graph Theory. However, in true Agile fashion, I approached the problem of understanding Git with a User Story in mind. And my User Story is the following:

“How can I use Git to allow and enforce peer-code reviews, maintain a couple of branches of the product and not have to remember hundreds of Git commands on my finger tips”

Of the several of ways to choose from, to achieve the above, here’s one possible way that worked for me.

First I will describe the overall approach and then discuss each step in detail.

1. Use EGit to create a local repo and to switch and create branches.
2. Use GitHub to host remote repository (It’s free)
3. Use EGit to push a feature branch to the remote repo
4. Use GitHub to send out Pull Requests to potential reviewers based on the push.
5. Use Github to view Pull Requests that already exist.
6. Use EGit to confirm that the changes in the pull request are good and to  merge into master locally
7. Use EGit to push the master to remote.

As you can see, I’ve used EGit, an Eclipse plugin for the Git client and GitHub.com for hosting a remote repository. This choice has two benefits:

1. Keep away from the command line interface of Git with it’s formidable array of commands and options.
2. And the bigger reason: Get introduced to GIT best practices that these tools default to.

Steps:

1. Get EGit. This is an Eclipse client for Git and can be downloaded as a plugin by using this update site. Note, that there is no need to download and install Git itself.
2. Create a project in Eclipse and add a few files to it.
3. Click on that project and go to Team -> Share and select Git as the SCM provider and then create a new repository. Note that you can also clone an existing Git repository, which typically means that you can “copy” a remote repository to your local environment. But we will start upstream of that event just for completeness.
   It is stated in EGit documentation that it is not a good idea to create a Git repository in the eclipse workspace. Since I am not totally convinced of the reasons given, I will create a directory in my eclipse workspace called gitprojects and create the repository within that (YMMV).
   
4. Press Create to create a new repository. Git allows us to have as many code repositories as we like. If you have used the git clone command for downloading an open source project, you have essentially copied an Git repository to your local machine. One nice feature of Git is that it doesn’t clutter up  each directory of your code base with dot directories (hidden directories) like Subversion does. But it does create one .git directory at the top level of your directory structure. This .git directory represents a repository, not a project. You can have as many projects as you like within that repository.
5. Make sure that you have added gitignores by going to Team|Ignore. Then ensure that you also check in .gitignore for others to use. If, you goof up and ignore .gitignore, you can simply edit the .gitignore file and delete the /.gitignore entry. A far sight better than SVN’s handling of svn-ignores!
   
6. Now let us do the initial commit of our project like so:
   
7. By doing all above, you will have a local project and it’s shared to your local Git repo. Now go ahead and make a change to a file or files. By doing so, you are working on your local master trunk. No branches yet.  Now, I’d like to find out what I’ve changed. For that, I can simply do a Team|Synchronize View which will show me the diffs between my workspace and my local Git. Go ahead and commit the changes, just like you would, in any source control system. Just to make it look real, make another change (like adding a file) and commit again.
8. Now let’s say that you need to start working on a User Story called Add Validation.  In the SVN world, you would have continued to work in your synched up workspace to make changes by say, modifying a file and adding another. But not so in Git. Here is where we start to use Git’s cool feature of branching.  First confirm that you are on the master branch on your local like so:
   Next, select Team|Switch To| New Branch to get the dialog shown below
   
   Notice that we are branching off of the master. We see that the current ‘head’ of our project has changed:
   So now we can start to develop in the for-validation branch.
9. This time, you can modify the United.java class and add another class called Validation.java. And check that in. So now you have a commit on the new branch. Now the interesting part. Select Team|SwitchTo| [master | for-validation] and see the code in your workspace switch. That is pretty neat because by merely changing a branch, an entire new code base is swapped in, including added and dropped files.
10. At this point, I would like to visually see the changes I have made. So I go to Team|Show In History. The view shows me two commits on the master branch and then one on the for-validation branch.
    
11. However, to see true branching behavior, let’s  start working on yet another story for say, changing-theme. But you don’t want to make that change on the validation code you just added. So we will create another branch off of master by going to Team|Switch To|master and then Team|Switch To | New Branch. This results in a dialog as below:
    
    Note that the source-ref represents the place you branch out from. Since we want to branch from before we started working on for-validation, select master as your source-ref. Specify a new branch called for-world-peace. By doing so, eclipse (EGit actually) will implicitly check out the new branch and make that current (aka head).
12. Make changes on that branch (to implement world-peace), commit a couple of times, so that we can see some nice nodes. Now let’s see how it looks in a graphical representation by checking out the history:
    Here we see two commits off of the for-world-peace branch.
13. Next I want to  have someone review my changes that I have made on the FEATURE-123-Use-Validation branch.  To do that I will need to push that branch over to github and issue a pull-request to all contributors of the repository.
    But I do not yet have a repository on github to which I can push. So I can go over to github, log into my account and create a “New Repository”. In the case of a corporate repository, this step will not be necessary as, hopefully that repository will already have been created by your company admin, and you would have been given access to it.
    Once you have created that repository, you can cut-n-paste it’s URL and then in your local Eclipse, go over to Team|Remote|Push and enter it like so:
    

    Upon hitting “next” I see the following dialog that basically askes me to map my local branch to a remote branch (defaulting to the same name):
    
    Finally getting:
    
    Hop over to github.com to see if your files actually made it… they should have.

14. On github.com, you can now switch over to the just checked in branch (from a drop down) and issue a pull-request to the committers of the repository.
15. Let’s switch roles now, and assume that you are the one reviewing the pull request. Github will give you nice tools to do diffs on files modified and you are able to add comments to each file. After this interaction with the developer, let us suppose that you are ok with the changes and would like to merge feature-123-use-validation to the master branch on the remote.
    The point to note is that the merge will be done on a local developer’s machine and then pushed back to remote.
16. Go to Team|Remote|Fetch From.. and fill in the following dialog
    Here we are pulling the remote master into the local master branch.
17. Check Team|History to ensure that the picture of the repository looks good. Ensure that the current branch is master and merge feature-123-use-validaton into it.
18. Select for-validation and hit ok. There are no conflicts here so we have a clean merge and GitX displays as below:
    As we can see, master and feature-123-for-validation now sit at the same head.
19. Now let’s merge for-world-peace and master. Do the same thing, switch to master (if not already there) and merge for-world-peace into it.
    Here we see a slight problem. There is a conflict because the same file was modified in the two branches.
    Correspondingly, Git modifies the file(s) in conflict by showing the following diff (The stuff above the ===== is in the head, the rest is in the branch being merged

    public class SouthWest {
    <<<<<<< HEAD //A line to suport validation ======= //To support worldPeace //Some more world peace >>>>>>> refs/heads/for-world-peace
    }

    So fix the file as you see fit and then select “Add to Index” from the menu below. Follow the above action with a commit so that changes are actually merged. At this point we have merged feature-123-use-validation to the master locally. Ensure that Team|Show In History shows the right branches

20. Now it’s time to push the master to the remote repository so that it can be subsequently built and deployed (by a Continuous Integration tool, say). Use Team | Remote|Push to push the master branch to the remote much like we had pushed the feature-123-use-validaton branch earlier on.

Gotchas:

1. When accessing a remote repository via EGit you may see:
   “Exception caught during execution of ls-remote command”

   I found that this happens if you have NOT used EGit’s project import wizard to pull in the project initially from the remote repo. (Instead, you may have used the command line option of git clone <url>)
   To fix, this, if possible, use File|Import|From Git … and get the project that way.
   Now, if you go to Team|Remote|Fetch From… you will be able to complete the dialog that would otherwise issue the error above.

Resources:

• EGit: This is a comprehensive EGit guide: http://wiki.eclipse.org/EGit/User_Guide#Getting_Started
• The Git Book (I found Chapter 5 is especially very useful where usage scenarios are discussed): http://git-scm.com/book/en/Distributed-Git-Contributing-to-a-Project

Using Java properties to specify “boundary conditions” or user preferences of any non-trivial application is standard practice in many development shops. However, even though using properties is a good abstraction, if the property files are packaged with the application, they are not exactly a 100% configurable at runtime.

On the other hand, if they are not packaged with the application and interjected on the application’s classpath, they can be configured at runtime. But that is not a really palatable option from source code management point of view. In this post, I will talk about the pros and cons of these two approaches, and then describe a hybrid approach that works well and keeps everyone happy!

There are two technologies that I will rely on for the proposed solution:

• Maven for resource filtering
• Springframework for it’s very helpful PropertyPlaceHolderConfigurer class.

Let us assume that the following build environments exist:

• DEV – local developer’s environment.
• NIGHTLY – build server’s continuous build
• TEST – internal testing
• UAT – User Acceptance Testing
• STAGE – pre-production
• PROD – production

Approach 1: Build-time Injection of Properties

This approach uses Maven’s Resource filtering and Profiles.

First create the following property files and place them in src/main/filters of your mavenized project:

• acme-common.properties – Holds properties that are not environment specific, like customer.telephone.number, company.tag.line etc
• acme-${env}.properties – There should be as many files as there are build environments. For example, acme-dev.properties, acme-nightly.properties etc. These files contain properties that are environment specific. For example, datasource urls including userId and passwords, JMS endpoints, email flags etc.

Optionally create one more file as below and place it in ${user.home}

• my-acme.properties – This holds all properties that are specific to the developer. Examples could be email address of the developer, local db url etc.

Since this property file is developer specific, notice that it is not in the source tree (and therefore not in source control).

Now configure the filter section of the pom of your project like so:

    <build>
        ...
	    <filters>
	      <filter> src/main/filters/acme-common.properties </filter>
	      <filter> src/main/filters/acme-${env}.properties </filter>
	    </filters>
		<resources>
			<resource>
				<directory>src/main/resources</directory>
				<filtering>true</filtering>
			</resource>
		</resources>
          ...
	</build>

Note that files specified later in the filters section override the earlier ones for the same property name.

In addition, set up a profile section like so:

    <profiles>
        <profile>
            <id>DEV</id>
             <properties><env>dev</env></properties>
        </profile>
        <profile>
            <id>NIGHTLY</id>
            <properties><env>nightly</env></properties>
        </profile>
		...
        <profile>
            <id>PROD</id>
            <properties><env>prod</env></properties>
        </profile>
    </profiles>

By doing the above and issuing a maven package -PUAT will ensure that values specified in acme-common.properties, acme-uat.properties and my-acme.properties are injected appropriately in the (xml files) that reside in the resources directory. Therefore the war or jar or ear (whatever the final artifact is) will have the correct values in them.

Note,that in this approach, there should not be any property file in the final classpath of the artifact, because properties are already injected into appropriate files at build time.

Pros:

• Since property files are part of the code base, and the code base is built for a specific environment, there is a lower chance of the wrong file being used in the wrong environment.
• There is no deploy-time  modification of the build environment. There is only one process that modifies the run-time environment, and that is the deploy process.

Cons:

• Properties cannot be changed without re-building and re-deploying the application. Changing property file values in a packaged war can lead to all sort of problems from cached properties issues to re-loading of the webapp automatically when the timestamp on the property file changes (depends on the servlet container).
• Secure passwords have to be checked into source control.
• The application has to be re-built for every environment.

Approach 2: Run-time substitution of Properties

In this approach, we will use Spring’s PropertyPlaceHolderConfigurer and a property file that is placed on the application’s classpath.

First create one property file (or several, for that matter) that hold all the properties used by the application and place it in a location outside source control.

The add the following bean definition in Spring’s application context file:

<bean id="propertyPlaceholderConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
  <property name="locations">
	    <list>
	      <value>classpath:acme-runtime.properties</value>
  </property>
  <property name="ignoreResourceNotFound" value="true"/>
</bean>

Or, if you prefer, the more recent:

<context:property-placeholder system-properties-mode="OVERRIDE" location="acme-runtime.properties"/>

By doing he above, we will have ensured that at run-time, all properties are substituted with the correct values.

Pros:

• Properties can be changed on the fly, without having to build and re-deploy the application.
• The property file(s) are not under source control, so secure passwords are not in source control
• Password management can happen for all environments by a trusted source instead of the development group.

Cons:

• Since the files are not under source control, there is a chance of the wrong file getting into the wrong environment. This is especially cumbersome in development environments where a property file will have to be passed around to each developer separately to get her up and running.
• A separate process will have to be implemented (apart from the deploy process) to mange these property files.

Approach 3: Combine the two approaches

Here we will use Maven filtering and Spring’s PropertyPlaceHolderConfigurer together.
See figure below:

However, the key is the balance between how much is achieved via Build time Injection vs Run time substitution. Depending on what is done using each of these technologies, the above diagonal line can be made to move to the right (for more build-time behavior) or to the left (for more run-time behavior).

Here are the steps for one possible configuration:

1. Set Maven filtering on on the resources directory like so. Note that there are no filter files.

	<build>
	    ...
		<resources>
			<resource>
				<directory>src/main/resources</directory>
				<filtering>true</filtering>
			</resource>
		</resources>
		...
	</build>

2. Create the following property files in your resources directory. All the property files will be in the packaged artifact.

      src/main/resources/
                         |
                         +-- properties/
                                     |
                                     +-- acme-common.properties
                                     +-- acme-dev.properties
                                     +-- acme-qas.properties
                                     +-- acme-prd.properties

3. Specify the following profiles and property values in the pom.xml

    <profiles>
        <profile>
            <id>DEV</id>
             <properties>
                <env>dev</env>
                <log-file-location>${user.home}/logs/acme-dev.log</log-file-location>
                <acme-runtime-properties-location>${user.home}/acme-runtime.properties</acme-runtime-properties-location>
            </properties>
        </profile>
        <profile>
            <id>QAS</id>
             <properties>
                <env>qas</env>
                <log-file-location>/var/tmp/logs/acme-qas.log</log-file-location>
                <acme-runtime-properties-location>/var/tmp/acme-runtime.properties</acme-runtime-properties-location>
            </properties>
        </profile>
        <profile>
            <id>PRD</id>
             <properties>
                <env>prd</env>
                <log-file-location>/opt/app/logs/acme-prd.log</log-file-location>
                <acme-runtime-properties-location>/opt/app/acme-runtime.properties</acme-runtime-properties-location>
            </properties>
        </profile>
    </profiles>

4. Lastly, place the spring contexts in src/main/resources and introduce the PropertyPlaceHolderConfigurer in the bootstrap context like so:

    	<context:property-placeholder system-properties-mode="OVERRIDE"
			location="classpath:properties/acme-common.properties,
			classpath:properties/acme-${env}.properties,
			file://${acme-runtime-properties-location}"
			ignore-resource-not-found="true"
			/>

Tying it all together

By specifying the configuration like above, we will have achieved:

1. The ability to place environment specific properties in the environment specific property file(s) and have the correct file passed on to the PropertyPlaceHolderConfigurer at build time. Note that even though, all the property files will exist in the final artifact, only the correct file will be specified to the PropertyPlaceHolderConfigrer to enable run time substitution.
2. The PropertyPlaceHolderConfigurer will substitute the correct values in the rest of the spring contexts at run time from the passed in common and environment specific files.
3. Those properties that are not specified in the common or environment specific property files are picked up from the runtime-properties file. (Even if the same property is specified in the common or environment specific files AND the runtime property file, the runtime property will trump the similarly named property in the other files). The runtime property file, therefore becomes a good place to specify passwords and other secure information that will not be checked into source control.
4. Note that the run-time property file has been specified using the file protocol. This implies that that file can be placed anywhere in the file system, outside the application or system classpath. And certainly out of source control.
5. Since log4j.xml file (that has the log file location specified via a property) is NOT managed by spring (in this configuration), it’s log-file-location property is specified outside the property files and directly in the pom profiles.
6. Note the use of ${user.home} indirection for log file location. This is to prevent unnecessary changes in the pom which would occur if each developer were to specify her own log-file-location differently.

Now that we know the order the files above are accessed in (at run time) we can decide for which environment we would like to place secure information in source control by placing them in environment specific property files, and for which, not, by placing them in the runtime property file.

I recently had the need to write a RESTful web service. Spring has re-engineered the MVC project to support REST based services (because the large overlap in annotation support), whereas it has left Spring Web Services as it’s own project with a distinct life cycle.

In Spring MVC, there are two ways to render a response, one using the conventional ViewResolver interface and the other using the HttpMessageConverter interface. I was a bit cloudy on when to use which, when I stumbled on this post from Keith Donald which brings out the difference succinctly:

Views vs. @ResponseBody (aka HttpMessageConverters)

Two different systems exist for rendering responses
• ViewResolver + View
• HttpMessageConverter
Triggered in different ways
• Render a view by returning a String
• Write a message by returning a @ResponseBody Object or ResponseEntity
Which one do I use?
• Use views to generate documents for display in a web browser
• HTML, PDF, etc
• Use @ResponseBody to exchange data with web service clients
• JSON, XML, etc

That cleared up things nicely for me and I hope it helps others too!

After working on a GRAILS project, I decided to dig a little deeper into the underlying technology: Groovy, a 21st century language with a 70′s name . There are many good things that can be written about Groovy being a dynamic, interpreted language built on Java, but in this post I’ll focus on two things :

1. How to  access a bundled jar and all it’s dependencies from a Groovy script.
2. How to add methods to classes in that bundled jar without modifying the jar.

What that means is that I can take any Java functionality that is bundled as a jar file, and surgically pick a class from that jumble, modify it and run it in a scripted environment without having to write another bootstrap application! Let us see a real-world use of this feature.

The problem to solve: At a certain client’s site, there is a flurry of activity every 4 months when release time comes around. The company’s flagship application is installed at 100+ client sites but along with the application are installed file system resources and database resources. And even though there are automated tools to carry out this activity, once done, every install is slightly different because of environmental reasons. So to really make sure that all went well, programmers have to do the unthinkable… log into each individual site and putz around till they are sure that the installation looks good. And do that a 100 times! That, is what I call a PROBLEM DOMAIN! Although, we will not be using Groovy to create a DSL to solve this problem (next post), I will demonstrate some of Groovy’s grooviest features to alleviate the problem.

What already existed

The flagship application already had an integration test framework. As a separate maven module, the integration-test project had a pom in which dependencies were declared with all the other modules that were being integration tested. The maven powered integration test environment fired up a jetty server and a Selenium server in the pre-integration-test phase ran the Selenium JUnit tests and then stopped both the servers in the post-integration-test phase. The results of the tests were available via the maven-surefire-plugin reporting mechanism. The packaging of this module was of type jar.When packaged or installed, the jar file produced contained all the Selenium based JUnit tests inside of it.

What would be nice to have

Since we already have a jar with all Selenium tests written for one site, it would be great if I could re-use that code for each of the 100 sites that have to be sanity checked every 4 months. However, the URLs and credentials for running the tests are read from property files that are a part of the code base. It would be great if we could pass in new URLs and credentials to those tests and yet not break the Integration tests that run with nightly builds.

Also, it would be great if we could access classes from that jar in a Groovy Script without having to write yet another maven module with dependencies and the whole nine yards.

So here’s a Groovy Solution!

In Groovy we can either write a script or a class. Although both end up being compiled into a .class file, I find writing a script file more liberating (or ad-hoc) so that’s what I used.

First let’s tackle the issue of using a class from any arbitrary jar file in my Groovy script.

The initial thought is to place that jar file in

<GROOVY_HOME>/lib or to specify it on the command line with the -cp flag like so:

>groovy -cp ~/jars/arbitrary.jar MyScript.groovy

However, it is more than likely that arbitrary.jar doesn’t play by itself and lives in an jar-soup of some sort with tons of collaborators and dependencies to do it’s work (see figure). So the classpath option is no good. We will need to make certain assumptions about the jar file:

1. Arbitrary.jar is available in a Maven repository (local or remote)
2. Arbitrary.jar has dependencies that are specified in a pom.xml buried inside it’s META-INF directory OR that it has no dependencies and has all the classes it needs to run bundled within the jar.

If both of the above are true, then the solution of the above problem can be solved by using a nifty tool from the Groovy camp called Grape. Grape allows you to Grab dependencies by using an annotation right above your import statement (yes, you can have imports in a groovy script). And in keeping with scripting’s ad-hoc nature, no xml files for specifying dependencies, thank God!

@Grab (group='com.acme.widgets', module='sales-integration-test', version='1.2.0')
import com.acme.test.SmokeTests

By doing the above, you have loaded that artifact (jar file) into the GroovyClassLoader that is used by the script.

Grape uses Ivy under the covers. So a little primer on Ivy may be in order. Ivy uses a config file (an xml file this time ) that tells it where to resolve it’s dependencies from. Since Grape is using Ivy, Grape is configured to look for that ivy config in <USER_HOME>/.groovy/grapeConfig.xml. Here is what it looks like:

<ivysettings>
  <settings defaultResolver="downloadGrapes"/>
  <resolvers>
    <chain name="downloadGrapes">
      <filesystem name="cachedGrapes">
        <ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
        <artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/>
      </filesystem>

      <ibiblio name="local" root="file:/usr/jay/maven2repo" m2compatible="true"/>
      <ibiblio name="local2" root="http://nexus.acme.com:9000/nexus/repository/repoGroup/" m2compatible="true"/>
      <ibiblio name="codehaus" root="http://repository.codehaus.org/" m2compatible="true"/>
      <ibiblio name="ibiblio" m2compatible="true"/>
      <ibiblio name="java.net2" root="http://download.java.net/maven/2/" m2compatible="true"/>
    </chain>
  </resolvers>
</ivysettings>

The things to note here:

1. “local” tells Ivy to look first in the local maven repository (optional). “local2″ tells Ivy to look in the company configured maven artifact repository. Also optional. And lastly to go out on the net and get the artifact from remote repositories.
2. “cachedGrapes” tells Ivy where to store the downloaded artifacts and in what format.

So when the import statement in the Groovy script looks for SmokeTests , not finding it in the Groovy ClassLoader, it looks at the annotation on the import. The @Grab annotation specifies the coordinates of the artifact which is resolved using the resolvers specified in the ivysettings file. The downside to this approach is that we end up with two local caches, one for Maven and another for Ivy. But the good thing is that the @Grab annotation passes the resolved artifacts directly to the Groovy ClassLoader so now SmokeTests is available directly  in the script.

I must point out that I had a classpath problem with slf4j classes (that were dependencies in the jar). I used a sledgehammer approach by making ALL classes bump up to the SystemClassLoader by specifying the following:

@Grapes ([
@Grab (group='com.acme.widgets', module='sales-integration-test', version='1.2.0'),
@GrabConfig(systemClassLoader=true)
])
import com.acme.tests.SmokeTests

So that takes care of the dependency issue.

–

The other issue we had identified was that of needing to change some code on the fly without modifying the jar file.

Let’s begin by looking at the existing JUnit code that exists in the jar file.

public class SmokeTests {
	private static String url = PropertyUtils.getPropertyFromFile("url", "test.properties");
	private static String context = PropertyUtils.getPropertyFromFile("context", "test.properties");
	private static String username = PropertyUtils.getPropertyFromFile("user", "test.properties");
	private static String password = PropertyUtils.getPropertyFromFile("password", "test.properties");
        ...
	@Test
	public void testLogin() throws Exception {
		try {
			selenium.open(context + "/login.jsp");
			selenium.type("j_username", username);
			selenium.type("j_password", password);
			selenium.click("login");
			selenium.waitForPageToLoad("10000");
			String xpathToCustomerTab = "//*[@id='span4']";
			SeleniumUtils.clickWhenElementLoads(selenium, xpathToCustomerTab);
			String xPathToLogoutLink = "//html/body/div[3]/a[4]";
			SeleniumUtils.clickWhenElementLoads(selenium, xPathToLogoutLink);
		} catch (ElementNotFoundException e){
			Assert.fail(e.getMessage());
		}
	}

Listing 1 – Original JUnit Test Class

In the above code we see how test.properties are being read for the context, url and credentials. We need to be able to change those values by passing them externally and yet not causing the Integration tests to break. Here is where we will use a trick from the Groovy meta-programming toolkit called ExpandoMetaClass (or EMC).

In my Groovy Script, I will create an instance of SmokeTests, get at it’s metaClass that Groovy wraps around every Java and Groovy class and add a method to it. That method will be available for the scope of the script. Here’s how I would do it:

@Grapes ([
@Grab (group='com.acme.widgets', module='sales-integration-test', version='1.2.0'),
@GrabConfig(systemClassLoader=true)
])
import com.acme.tests.SmokeTests

def smokeTests = new SmokeTests()

smokeTests.metaClass.changeStuff = {url, context, userId, password ->
delegate.url = url
delegate.context = context
delegate.username = userId
delegate.password = password
}
...

//Read url, credentials from file:
//For each line loop
smokeTests.changeStuff (url, context, userId, password) //change the values on this instance
smokeTests.testLogin() //call the original test on the class
//EndLoop

Listing 2 – Groovy script file

By doing the above, I’ve been able to interject new values for those private static member variables for the duration of the script, all without changing the original class.

Conclusion

We have seen how Groovy can help in ad-hoc scripting by being able to access a class(es) from a packaged jar and we have seen how we can change the functionality of existing classes. I have used the use case of Testing to illustrate this but this technique can be used in any situation where existing behavior has to be quickly modified.

In the next post, I will talk about using Groovy for creating a DSL for integration testing.

Resources:

http://www.ibm.com/developerworks/java/library/j-pg06239/index.html

http://jira.codehaus.org/browse/GROOVY-3755?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel#issue-tabs

http://mrhaki.blogspot.com/2010/04/groovy-goodness-configuring-grape-to.html

Today there are far too many choices in the technology stack to build a modern enterprise application. Altho’, for the most part, that’s a good thing, when it comes to building a plain vanilla solution that is mainly a CRUD application, that range of choices may be unnecessary. Convention over Configuration (CoC) frameworks are an answer to that problem. And altho’ Ruby On Rails was the first modern attempt (meaning web based enterprise applications) to address that issue, GRAILS seems to be the emerging winner.  Here’s why:

1. It has the backing of a avant garde organization like the Spring Framework.
2. It is built on Spring MVC which is already very widely used.
3. It is based on Groovy that is, in turn based on Java (Groovy offers all the benefits of a Dynamic Programming Language,which are absent in Java)
4. It has the momentum of a freight train judging from the community support and traffic.

So I experimented with GRAILS and was pretty impressed. So here’s a brief look at this technology.

What is GRAILS?

The picture below shows the technology involved in enabling GRAILS (Groovy on Rails).

Gant - Allows Ant Taskdefs to be embedded in Groovy instead of of XML. More powerful because the logical language constructs that Groovy offers are absent in XML

Gradle - A build language using a Groovy supported DSL

Grape - Provides annotations for making project dependencies available in your source code.

Grails uses Gant to build a conventional project structure. Gant, in turn, can access all the Ant taskdefs to do it’s work. At present Grails uses GAnt to run it’s commands, but indends to switch to Gradle in the near future.

Actually Gant is a subset of Gradle. The main difference is that Gradle has a plugin architecture that is absent in Gant.

Gradle may be used to build the Grails project and Grape provides annotations in the Groovy code. The several (hundred) plugins that the Grails eco-system offers, may use Gant for macro functions (like file manipulation) and Groovy for fine grained functions.

As we can see,  Groovy offers the glue that holds everything together.Groovy can:

1. Invoke Java programs, including Spring managed beans because it is based on Java and built on Spring
2. Invoke Ant based Taskdefs by using Gant scripts that are Ant taskdefs embedded in Groovy
3. Use Lambda functions (closures) being dynamic

How to build an app with GRAILS:

Here are the typical steps to begin a new project. Install Grails, set GRAILS_HOME and place GRAILS_HOME/bin in your classpath. Then do the following at the grails prompt :

1. To build a project  – Type create-app testapp: Grails creates the basic directory structure of the project including defaults for the look-and-feel (css) of the web application.
2. To create Domain Objects – Type create-domain-class com.acme.IncomeStatement: Grails creates a Groovy class that represents IncomeStatement in the correct directory. Since the details of the domain object are not yet defined, it’ll just be a stub for now.
3. To create Controller – Type create-controller com.acme.IncomeStatement : Grails creates a stubbed out controller called IncomeStatementController.
4. To provide app specific info – Edit the domain object by placing some member variables in it.  Edit the controller and specify a static scaffold=true in it.
5. To build out the project – Type generate-all to let Grails create the controller and the views (GSPs) for this use case (IncomeStatement)
6. To run the app, type run-app and visit a browser by pointing to testapp on port 8080 and go from there.

When issuing each of these commands, GRAILS runs Gant based scripts that use conventions to build out the application objects.

At this point you should be able to Create, Read, Update and  Delete an IncomeStatement from a web page right down to the database. The details on how to build the application can be gotten from several tutorials and the documentation.

What’s GRAILS value proposition:

Doing the above is pretty painless and after the dev environment is set up, would typically take no more than 5 minutes. Assuming that building validation and modifying the look-and-feel would probably take another 10 minutes, fifteen minutes to create a full-fledged CRUD application is not bad!

Looking under the covers to see how GRAILS actually achieves this then it boils down to just two things (remember, GRAILS, not Groovy):

1. Scaffolding: Here is where Grails uses Gant and Groovy to create CRUD code for you in the controller and views as GSPs with appropriate tags in it.
   To help the creation of scripts you can use the Domain Specific Language aspect of Groovy to specify constraints and bindings. This is a big time-saver when all you want is CRUD operations on your domain objects and properly laid out css for your web pages. The code that is generated by scaffolding, includes, among other great features,  validation in the controller and supports pagination in the views. So although it may sound trivial. scaffolding does do a lot for you!
2. GORM: Grails ORM uses a hibernate plugin under the covers to interact with the persistent store. All domain objects, by default are persistent objects. To change things you have to use an ORM DSL in either the application wide configuration or domain specific groovy file. Most of the features of Hibernate are available with the addition of Dynamic Finders. Because Groovy is interpreted at run-time, these methods are not physically available on the domain object but are made available at run time.

In addition to above, Grails has a VERY busy developer community with tons of plugins, that you can check out here.

How to introduce GRAILS in your organization:

Identifying a narrow use-case of an existing application and building an app that reuses the existing application’s domain model and database schema is probably a good way to ease into GRAILS development.

Let’s use an example of an existing application that runs a company’s lawn care business. To illustrate reuse, let us assume that the domain objects in that existing application are persisted using Hibernate and the application is built using Maven. (Note that if no code re-use is intended, then the existing application does not even have to be written in Java; just build the GRAILS app parallel to the existing one and the only touch point would be the database).

We now want to add a feature where lawn care workers can update where they worked on a particular day and for how long. Let’s call this new feature WorkRecorder and use GRAILS to build it.

WorkRecorder will need the customer and equipment domain objects from the existing app, but will need to define it’s own domain object called Work. Therefore it will need to Create, Read, Update and Delete a Work domain object and Read  customer and equipment domain objects from the existing application.

In the picture above we can see that our brand new GRAILS application WorkRecorder creates a GRAILS object called Work and also accesses the Customer and Equipment objects from the exiting application.

For the CRUD on Work, GRAILS commands like create-domain-class can be used followed by  create-controller and create-views. With the use of scaffolding, you can get full fledged CRUD operations on the Work object. But when you have to show a list of equipment that is used to execute the work, then you will have to make those domain classes available to your Groovy classes so that you can include them in the constraints therein. There is a nice blog entry here and and Grails docs talk of it here.  In Peter Ledbrook’s article, he suggests that we can either make classes from our LawnCare app available via source code (by placing in src/java under grails-app directory), or by placing the jar file in the lib directory.

If the LawnCare application is fairly well-baked, then making it’s  jar file available to WorkRecorder may make more sense. But instead of placing it in the GRAILS project’s lib directory, I found it cleaner to make it a dependency of  the WorkRecorder app by specifying in the BuildConfig DSL’s dependencies section.
For example:

...
    dependencies {
        compile 'com.acme.greenhouse.LawnCareServices'
        runtime 'ojdbc6.ojdbc6.11.1.0.7.0'
    }
...

In theory, you could build out a parallel CRUD application (UI to db) using the “imported” domain objects Customer and Equipment. That would probably not be a good thing because now you have two ways to persist Customer and Equipment (and you’ll have bypassed any business logic embedded in the LawnCare app), but in the event that you would like this to happen, you could still use GRAILS commands create-controller and create-views commands and pass them these domain objects. With scaffolding turned on and constraints defined in CustomerConstraints.groovy and EquipmentConstraints.groovy, you could build out a modern validation UI also by using the command generate-all.

Maven and GRAILS:

Having been a Maven user for several years now, I have to admit (with a heavy heart , that Maven and GRAILS do not play well together. With GRAILS, Gradle is the build tool of choice (which I’ll talk about it in a later post). Gradle uses Ivy for dependency management and so, in our example, since LawnCare was built using Maven, we will end up with two local repositories of dependencies, one from Maven and another from Ivy.

By default, the Ivy repository lives under <UserName>/.grails/ivy-cache whereas the Maven one lives in <UserName>/.m2/repository. That’s not a good thing, especially for the first time when the dependencies are being downloaded.

Also, when developing behind a firewall, a proxy may need to be specified. This can be done via a grails command called add-proxy and corresponding remove-proxy. The add-proxy command creates a file called ProxySettings.groovy which contains:

myproxy=["http.proxyHost":"a.b.c.d", "http.proxyPort":"8080", "http.proxyUser":"", "http.proxyPassword":""]
currentProxy="myproxy"

To disable the proxy, you can simply comment out the currentProxy line.

In the next post I’ll cover the impedance mismatch between Gradle/Ivy and Maven.

When I wrote this post about checking code coverage while testing Java applications, I got feedback from users who complained about the cobertura plugin running twice when run with the check goal.

Looking closer, I realized that this is indeed so.The tests are run once with the classes of the project and a second time with the instrumented classes.

So I did some digging and found out that the cobertura maven plugin spawns a separate lifecycle (called cobertura), that does the following, among other things:

• Bind the instrument goal to the process-classes phase
• Interject the instrumented classes on the test classpath in the test phase.

If tests are enabled (skipTests=false, or maven.test.skip=false) and the check goal is run, the tests run once using testClasses and then again, using ${project.build.directory}/generated-classes/cobertura which are the classes that are generated by instrumentation.

However, if tests are disabled (skipTests=true or maven.test.skip=true), the check goal does not run the tests at all because the cobertura lifecycle piggybacks off of the test phase that has tests turned off! This returns a 0% coverage for the check goal because the instrumented classes were never used!

To me, this looks like a bug. I reported this to the development team of the plugin but got an ambivalent response mostly to the tune of “I am no longer active on this project, ask the dev mailing list”. I did, but nothing came of it.

So here’s a simple solution for this problem. Force tests to run in the custom lifecycle and make it independent of the setting in the user’s pom.

I have created this plugin and bumped up the bug version from 2.4 to 2.4.1.

It can be downloaded from here.

Here’s the patch file relative to the source of the plugin v2.4.

I have tested it to the best of my ability and it works great!

Java EE is more than a decade old; old enough to assume that there are several applications out there that have acquired the infamous “legacy” status. One of the most challenging aspects of such an application is it’s brittle nature. Adding a feature or changing a database flag, can cause an not-so-frequently-used part of the system to break. The only way to fix this situation is to add comprehensive and automated tests.

In this post, I will tell you how I solved this hairy issue for a 10 year old Java EE  application that I “inherited”. Like any good Java EE application, this application was nicely layered into the web tier, presentation logic, application logic and the Data Access layer. Developers, with all good intentions over the years had done their part to introduce the technology that was in vogue at the time (including, but not limited to Spring and  EJBs (<2.0)     ).

Unfortunately, because of the age of the app, business logic was strewn across all layers: from the front-end in JavaScript – (gasp!) to scriptlets in JSPs – (double gasp!), to business logic in the presentation layer; in the services layer (yay!) and most certainly… in the database, in stored procedures!

Since I was not familiar with all the existing business logic, I used what is called “black-box” testing. This simply compares before and after states of an application. In the case of a database application, we will need to compare data in selected database tables, before and after the test is run. In addition, we will need a set of data that needs to be run for the entire test suite.

On a positive note, most, if not all services were nicely defined using a Spring application context. So as to not attempt too much at once, I decided to ensure that all functionality, service layer on down is tested. That does leave untested code in the presentation (and possibly web layer) which is important if there is a lot of business logic in there. But that’s for another day… today, I will talk about how I introduced functional testing in a Java EE application, downstream of the services layer.

Broadly, here are the steps:

1. Create test data
2. Create and configure annotations to be used in tests
3. Implement the TestExecutionListener interface such that:
   1. The beforeTestClass method accesses the class level annotation, creates a new db connection, stores it on the testContext, and inserts setup data.
   2. The beforeTestmethod method accesses the method level annotation and uses the Spring db connection to insert test input data.
   3. The afterTestClass method accesses the connection from the testContext and rolls back the transaction, thereby rolling back the setup data.
4. Check post-run database data with data in the expected dataset to determine pass/fail status of the test.

The technologies I used for this are:

• Spring Testing Framework
• DBUnit
• Annotations
• Maven

and inspiration from an earlier post where I had talked about applying cross-cutting concerns to tests.

Here is a picture showing the moving pieces:

Create test data

There are three sets of data that are needed:

1. Seed data required for the entire test suite
2. Data that forms the input for one test case
3. Data that forms the expected output of the same test case

This may seem like a daunting task at first but here is a process that will make is much, much simpler, tho not completely painless.

Since we are talking about a JEE database application, we will attempt to capture the before and after states of the tests in terms of data.  We will ‘freeze’ that state of data in XML files and check that into source control as a part of the code-base (in test/resources/dbunit typically). I can see people going… XML file creation… that’s a show stopper. But with the maven-nddbunit-plugin described here , you will see that it’s not that difficult at all!

So, as a one time activity, to test your sevice, you will need to create data in a database (any database, could be your own development database). This database should have data to test your service. Typically this will exist when you are developing anyway, so if you haven’t discarded/modified that data, then you are half way there already. Note that at this stage the database will contain data not only for the service use case but also set-up data for the entire application.

Remember that the XML files will need to run against different database instance (typically a test database) than the one they were extracted from. And we cannot assume that any data is present in that test database. We will assume tho’ that the objects in the test database are current with those in development in that, the tables/views and database procedures in both databases are identical.

Next we will use the maven plugin mentioned earlier to create the XML files.Detailed documentation can be found on the plugin’s website, but here’s the gist:

You configure the plugin in your project’s pom, just like any other maven plugin, in the build section. The configuration specifies a jdbc URL, a username and a password. It has two goals, export and autoExport.

The export goal can be used to extract a set of data as specified by a SQL query into an XML file. With this goal you are responsible for the order of the extract. So if tableA depends on tableB, then you have to ensure that the order is maintained by specifying the order of the queries in the configuration section of the plugin. So, the query pulling data for tableB, should precede the query for pulling data from tableA.

The more sophisticated autoExport goal can be used to specify a base table and a where-clause. The plugin, then chases the foreign keys in the database and extracts all the data in the right order so that you do not get a constraint violation when (later on) upserting that data in your test database.

Let’s look at an example: Assume you are testing a Teller services’ balanceCheckBook() method. This method needs rows in the MONEY_TRANSACTION table that has credit and debit rows over several months because the balanceCheckBook() method needs to be able to exercise some part of the code that reconciles over quarters, say. So having those transactions in the MONEY_TRANSACTION table is crucial for you to be able to test your service. You’ve taken the trouble to create transactions in your development database. In doing so, all the data in dependent tables have to be already existent (or you would have to have created them in the past). When you extract data from the MONEY_TRANSACTION table, you have to ensure that you also extract data from dependent tables.  And that could start you on the slippery slope where TABLEA depends on TABLEB which depends on TABLEC which loops back to depend on TABLEA! It would be a nightmare to extract all those rows of data, and before you know it, you’ll find yourself making your way to the DBA to ask for the permissions (or syntax) to extract the entire database! That’s not a good idea.

That’s where the autoExport goal of the maven-nddbunit-plugin comes in useful. By specifying a base table, in our example MONEY_TRANSACTION and an optional WHERE clause, you can rely on the plugin to create an XML file of minimal data (across several dependent tables) that are needed to be inserted first to insert the required rows in the base table

Here is an example of the interaction with that plugin with using the export goal:

C:\projects\Acme> mvn nddbunit:export
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Acme-Core
[INFO]    task-segment: [nddbunit:export]
[INFO] ------------------------------------------------------------------------
[INFO] [nddbunit:export {execution: default-cli}]
[INFO] 1. sales-input-account
[INFO] 2. sales-input-create-account
[INFO] 3. sales-input-delete-account
[INFO] 4. sales-input-update-account
[INFO] 5. sales-result-one-account
[INFO] 6. sales-result-more-than-one-account
[INFO] 7. sales-result-domestic-account
[INFO] 8. sales-result-international-account
[INFO] 9. sales-seed-countries
[INFO] 10. sales-seed-states
[INFO] 11. sales-seed-currencies
[INFO] 12. sales-seed-users
[INFO] Enter the number next to the Id that you want to export, 0 to quit
3
[INFO] Exporting to DataSetPath: C:\projects\Acme/src/test/resources/dbunit/sales/sales-input-delete-account.xml using URL: jdbc:oracle:thin:@somehost:1521:somesid...
Successfully wrote file 'C:\projects\Acme/src/test/resources/dbunit/sales/sales-input-delete-account.xml'

Similarly, if using the autoExport goal, here is a sample output:

C:\projects\acme-core> mvn nddbunit:autoExport
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Acme Core
[INFO]    task-segment: [nddbunit:autoExport]
[INFO] ------------------------------------------------------------------------
[INFO] [nddbunit:autoExport {execution: default-cli}]
[INFO] 1. sales-region-input
[INFO] 2. sales-market-share-input
[INFO] 3. accounts-teller-input
[INFO] Enter the number next to the Id that you want to export, 0 to quit
3
[INFO] Accessing URL: jdbc:oracle:thin:@somehost:1521:somesid as user superman...
[INFO] Ready to export:
[INFO]  LEDGER (9 rows).
[INFO]          ACCOUNT_TYPE (1 row).
[INFO]          BANK_BRANCH (1 row).
[INFO]          TEAMMATE (2 rows).
[INFO]          ACCOUNT_CODE (1 row).
[INFO]          COUNTRY_CODE (1 row).
[INFO]          US_STATES (1 row).
[INFO]                  ACCOUNT (3 rows).
[INFO]                  TELLER (1 row).
[INFO]                  CLEARING_HOUSE (5 rows).
[INFO]                  BANK_BRANCH (1 row).
[INFO]                  SHARE (1 row).
[INFO]                  TRANSACTION_CODE (1 row).
[INFO]                          MONEY_TRANSACTION (2 rows).
[INFO] Exporting to DataSet path: C:\projects\acme-core/src/test/resources/dbunit/accounts/teller/teller-input.xml...
[INFO] Do you want to continue to export 30 rows in 13 tables to this file (Y|N)?
y
[INFO] File written...

At this stage, you have a bunch of xml files that represents your test case input (including setup) and another file (or two) that represents your output. From this point on, we are not going to be needing the database data anymore and it’s data can be modified or deleted. Note that the datasets have data in the correct order (for insert) to prevent foreign key violations.

Here is a sample xml file that is generated. Note the element called table points to a name that is going to be used by DBUnit to insert into. So for input datasets, ensure that the table name is correct. For the corresponding output dataset, the table name is not important, so long as the same name is used for the compare.

<?xml version='1.0' encoding='UTF-8'?>
<dataset>
  <table name="test_table1">
    <column>COL1</column>
    <column>COL2</column>
    <row>
      <value>slo</value>
      <value>jo</value>
    </row>
    <row>
      <value>fast</value>
      <value>lane</value>
    </row>
  </table>
</dataset>

Create and configure annotations to be used in tests

Next we need to write an annotation that will be used on the test classes. I have already talked about how to write annotations here. In this case we will need two annotations. One to be used at class level will be needed to specify the seed data that is going to be run for the entire test suite . And the other to be specified at method level for the data that is needed for each test (method).

The class level annotation can be specified like so:

@AcmeDBUnitSetUp(action=AcmeDBUnitSetUp.ACTION_REFRESH,
        setUpDataSetList={"classpath:dbunit/setup/setup.xml",
                                        "classpath:dbunit/setup/setup2.xml"})
public class SalesServiceTests
          extends AbstractTransactionalJUnit4SpringContextTests {
...

And at the method level:

@Test
@AcmeDBUnitRefresh(action=AcmeDBUnitRefresh.ACTION_REFRESH,
        dataSetList={"classpath:dbunit/sales/areas.xml",
                              "classpath:dbunit/sales/territories.xml"})
public void testSomething(){
...

Since our test inherits from AbstractTransactionJUnit4SpringContextTests, transaction semantics are already configured. Therefore I do not have to annotate each test method with @Transactional or @Rollback explicitly. Tests will automatically rollback after each execution leaving the db as it was before the test was run.

Implement the TestExecutionListener interface

Now we need to configure the TestExecutionListener interface. This interface has before and after callback methods at method and class level (new in Spring 3.0).

We will need to implement the beforeTestClass method and the beforeTestMethod method. (Note that since we will use the @TestExecutionListeners annotation at class level, we should not use JUnit4‘s or TestNG‘s @BeforeClass annotation as that will conflict with the similarly named methods on the TestExecutionListener interface).

In the beforeTestClass method implementation, we will access the annotation to get the path to the dataSet and then use the following code to insert data:

DataSource ds = (DataSource)testContext.getApplicationContext().getBean("dataSource");
IDatabaseConnection connection =
                      new DatabaseDataSourceConnection(ds, schemaName);
testContext.setAttribute("connection", connection);
...
Resource dataSetResource = testContext.getApplicationContext()
                                 .getResource(pathFromAnnotation);
IDataSet dataSet = new XmlDataSet(dataSetResource.getInputStream());
...

DatabaseOperation.REFRESH.execute(connection, dataSet);

There are a couple of things that are of note:

1. The connection that is got is a new connection; not one used by Spring and therefore by the AbstractTransactionalJUnit4SpringContextTests class.
2. That connection is placed in the testContext, so that it can be rolled back later in the afterTestClass method implementation.
3. The use of DatabaseOperation.REFRESH. This causes DBUnit to leave data in the db if it already exists and add/update data that does not/changes.

In the afterTestClass method implementation, we access the connection object from the testContext and roll back the transaction such that setup data is no longer in the database.

In the beforeTestMethod implementation we do the exact same thing as in the beforeTestClass method implementation except we use Spring’s existing connection so that rollback semantics are in place.

...
Connection sqlConnection = DataSourceUtils.getConnection(ds);
IDatabaseConnection connection = new DatabaseConnection(sqlConnection, schemaName);
...

Here, since we have used DataSourceUtils.getConnection, we are guaranteed to get the connection object that is passed to the JdbcTemplate that is used by AbstractTransactionalJUnit4SpringContextTests to cause a rollback. And since DBUnit is also using the same connection, the rollback will occur as intended by Spring.

This picture will explain what’s going on:

Check post-run database data with data in the expected dataset

Now that data is setup before the test class is loaded, and data is also injected before each test appropriately, all we have to do is call our service method, have it do what it does and then check results.
The call to compare looks like:

...
public void testSales(){
  //Calculate raise for empId=1
  salesService.calculateRaise("1");
  compareDbResults("SELECT emp_salary FROM emp, dept WHERE emp_dep_id = dep_id AND emp_id = 1 AND dep_id=2",
                 "classpath:dbunit/employeeServices/calculate-raise/results/emp-salary.xml");
...

Here we see that a query is being passed in to compare the current status of the database, to a dataset that was pre-determined using the exact same query in the configuration of the maven-nddbunit-plugin.
The implementation of the compareDbResults method is something like this:

public void compareDbResults(String sqlQuery, String dataSetPath) {
...
DataSource ds = (DataSource)applicationContext.getBean("dataSource");
Connection sqlConnection = DataSourceUtils.getConnection(ds);
connection = new DatabaseConnection(sqlConnection, getSchemaName()); 

org.springframework.core.io.Resource dataSetResource = this.applicationContext.getResource(dataSetPath);
IDataSet expectedDataSet = new XmlDataSet(dataSetResource.getInputStream());

QueryDataSet actualDataSet = new QueryDataSet(connection);
actualDataSet.addTable("temp", sqlQuery);

Assertion.assertEquals(expectedDataSet, actualDataSet);
...
}

Conclusion

We have seen a powerful way to do black box testing of a JEE database application where we are comparing before and after states in the database to assert success. DBUnit offers a good mechanism to capture the state of a database in xml and allow us to apply that data using broad strokes via the REFRESH operation. The maven-nddbunit-plugin gives the added advantage of creating and managing the huge amount of xml data that will need to be produced to carry out the tests.

I realized that by now I have written several maven plugins that make my (and several other folks’) life easy and it would be useful to have one place that enumerates them. So here it is

1. Maven SQL Plugin: Used to execute SQL remotely. Often used when a bunch of DDL or DML needs to be executed from a build box without physically having to move the SQL over.
2. Maven Distribute Plugin: Used to move jar’s and tarballs over, again, from a build box to remote machines and optionally explode and establish links to them
3. Maven Authorize Plugin: Used in conjunction with this blog post where I explained the process of setting up a Maven based corporate development and production repository manager.
4. Maven NDCobertura Plugin: Used in conjunction with this blog post where I described a process to ensure maximum test coverage.
5. Maven NDJacobe Plugin: Used in conjunction with this blog post where I described a process to make an entire code base adhere to a corporate coding standard.
6. Maven NDDBUnit Plugin: Used in conjunction with this blog post where I describe a process to test a database based JEE application using black-box testing.
7. Maven Cobertura Plugin-v2.4.1: Used in conjunction with this blog post where I patched up the original plugin to prevent tests from being run twice.

In software shops, code compliance is a big deal. Most of the places I have worked in, I often have found managers (and indeed, myself) say: I don’t care what coding standards we use, let’s just be consistent, use one set and be done with it!

But there’s more to that statement than meets the eye. Timing is an issue. How to propagate the ‘chosen’ configuration amongst the developer groups is another. And lastly, how do we enforce compliance.

Here’s how I overcame this last problem, recently.

The tool I used is jacobe. It is a pretty comprehensive code ‘beautifier’, judging from the default configuration file that it runs off of. Very broadly, jacobe is passed a configuration file and based on parameters specified in it, it recursively steps through all java files specified in the input parameter and formats each according the the configuration passed. It either overwrites the existing java file (if the -overwrite flag is set), or produces a file with a ‘.jacobe’ extension in the same directory as the original file.

Jacobe can be invoked in several ways:

From the command line.

By specifying as input the root directory of your java code, jacobe picks up all the *.java files under it and format’s them.

C:\jacobe\jacobe.exe -cfg=C:\jacobe\sun.cfg
-overwrite C:\projects\AcmeProject/src/main/java/com/nayidisha/plugins/jacobe

Note that there is no way we can enforce that developers will invoke this command during development.

Using the Eclipse-Jacobe Plugin

Another way to format files is to use the eclipse jacobe plugin from within the eclipse IDE. As of the current release of the plugin, the Jacobe eclipse plugin causes an icon to appear on the eclipse toolbar which the developer has to explicitly press to format files.


There is a good chance that developers will omit to press the format button and press the (regular) save button instead. And then there’s always the issue of those folks who do not use Eclipse or (gasp) any IDE at all!

Using the Maven-jacobe Plugin

Yet another way to format files is to use the maven-jacobe-plugin. This is invoked at build time and can be applied to the entire code base in one go. The plugin is configured in the pom and then the following command can be issued from the project root.

mvn jacobe:jacobe

Again, there is no systematic way to ensure that developers will invoke this plugin during development time. It can be systematically invoked at build time, but formatting at build time doesn’t have any discernible advantage.

Using a Pre-commit trigger to Source Control

Jacobe can be invoked by configuring a pre-commit trigger on CVS, SVN, P4 (or similar source control system). However,checking code compliance on a pre-commit trigger to a source control system is not a good idea for many reasons, the main one being that what get’s checked in is different from what the developer has on her machine. Also, I doubt checking in  modified code without testing is going to pass many audits

So, what is the solution?

All the above techniques (save the last) rely on developer discipline and are highly error prone. And the last, is unsavory for many reasons.

So here’s a process that I used recently that does the trick. It involves the use of yet-another-maven-plugin called the maven-ndjacobe-plugin that I wrote. This plugin has, in addition to the format goal, a check-compliance goal too. You can download that plugin from here, and it’s documentation is here.

First, as a one-time process, find a time when your project is reasonably stable, that is, not too much churn is going on in the code base. Then run the process in the picture below:

Here, in the green box, we have run the format goal of the plugin. By doing that, jacobe will format/beautify your entire code base (as configured in the plugin configuration). Build it and test it, to make sure that nothing unexpected has happened to the code during formatting. After checking in the ‘beautified’ code, you are ready for the development process shown below:

Here, we see that developers may (or may not use) the Jacobe plugin in their IDE to format code before/while saving their code. But during the build process, binding the ndjacobe plugin’s check goal to the early phase of the build lifecycle can enforce compliance of the formatting of the code.

The ndjacobe:check plugin can be configured to fail (failOnError=true, default) if the check does not pass. This configuration can be specified in the corporate pom, or in the project’s pom and can be bound to the validate or initialize phases (pre-compile).

Another parameter that is useful, if the check fails is the keepJacobeFileForFailedMatches flag (default=false). Setting this to true will also keep the Jacobe generated .jacobe file that can then be used for comparison to see what part of the formatting failed and address it accordingly.

Using the ndjacobe:clean goal will remove any spurious .jacobe files from the codebase.

Since there is no way of getting around a build-time check, we can enforce compliance to the corporate coding standards of all java code going into source control. This proves to be a valuable on-going development process that ensures that your code is 100% compliant with the corporate standard as specified in the jacobe configuration file.

If you are developing a web application that uses a Spring IoC container, this post will give you an idea of how to apply aspects to a running application in production. This is a powerful capability, one that can quickly get out of hand if applied without giving due consideration to the implications of adding cross-cutting concerns throughout the application. But, on the other hand, if used for something  harmless but useful like logging, it can add great value.

So in this post, I will use the example of adding logging as an aspect to a web application while the application is deployed and running. Last year I had posted about managing logging on the fly in a production system. But when there are no log statements in the application or not enough log statements to debug a production problem, applying a cross-cutting logging concern across all Spring Managed Beans may prove useful.

The broad steps are:

1. Write the advice: in this case an around logging advice.
2. Write a LoggingAspectManager that will add the aspect.
3. Expose the Manager via Spring JMX
4. Invoke the method on the LoggingAspectManager using JConsole.

There is also one caveat, or rather a short comming in this approach that I will point out in the end.

Writing the Logging Advice

Your existing application could be already configured with Spring managed beans that serve as advised proxies for say, transactions or security. We will add another advice to implement our logging aspect. Spring managed beans can be advised by using annotations, namespaces, the Spring 1.2 compatible API or a combination of all three. Your existing application could be using annotations or namespaces for configuring proxies. We will configure the below advice using Spring API as we will see later on.

The advice is simple around advice that logs before and after every method call (including parameters passed to the method). Note that to log the class name we have had to determine if the class being advised is already an advised proxy or not.

...
public class LoggedMethodAdvice implements MethodInterceptor {
  if (invocation.getThis() instanceof Advised ) {
    Advised advised = (Advised)invocation.getThis();
    className = advised.getTargetClass().getCanonicalName();
  } else  {
    className = invocation.getThis().getClass().getCanonicalName();
  }
  Object[] args = invocation.getArguments();
  String methodName = method.getName();
  log.info("Going to invoke: " + className + ":" + methodName + " with params " + ArrayUtils.toString(args));
  Object ret = invocation.proceed();
  log.info("Done with method: " + className + ":" + methodName);
}
...

Write the LoggingAspectManager

The LoggingAspectManager is where we will define the bean to add the aspect to the BeanFactory. First we have to make this class ApplicationContextAware and a BeanPostProcessor. This allows the manager to register a hash of all beans in the application context at start-up.

Next we make the manager implement an MBean interface that exposes the ‘business’ methods.

public interface MethodLogableMBean{
	public void startMethodLoggingForPackage(String partialPackageName);
	public void stopMethodLoggingForAll();
}

Finally we implement the methods like so:

public void startMethodLoggingForPackage(String partialPackageName) {
  DefaultListableBeanFactory bf = this.getBeanFactory();

  //Register the advisor
  BeanDefinitionBuilder aBuilder = BeanDefinitionBuilder
       .rootBeanDefinition(new RegexpMethodPointcutAdvisor().getClass());
  aBuilder.addPropertyValue("advice", this.getLoggedMethodAdvice());
  aBuilder.addPropertyValue("patterns", new String[]{partialPackageName + "..*"});
  bf.registerBeanDefinition("loggedMethodAdvisor", aBuilder.getBeanDefinition());

  //Register proxies for spring managed beans
  //(only those that implement the passed in interface)
  Set keySet = beanManagedObjectList.keySet();
  for (String beanName : keySet) {
    Object o = beanManagedObjectList.get(beanName);
    Class[] interfaceArray = o.getClass().getInterfaces();
    Collection interfaceClassList = new ArrayList();

    //Make and array of all interfaces that need proxied
    for (int i = 0; i < interfaceArray.length; i++) {       if (interfaceArray[i].isInterface() && interfaceArray[i].getName() 	.startsWith(partialPackageName)){ 			interfaceClassList.add(interfaceArray[i]);       }     }		     Class[] selectedInterfaceArray = interfaceClassList.toArray(new Class[interfaceClassList.size()]);     if (selectedInterfaceArray.length > 0){
      //Since this bean is going to be proxied, cache it for re-instating later on
      originalManagedObjectList.put(beanName, bf.getBeanDefinition(beanName));

      ProxyFactoryBean proxy = new ProxyFactoryBean();
      BeanDefinitionBuilder builder = BeanDefinitionBuilder.rootBeanDefinition(new ProxyFactoryBean().getClass());
      builder.addPropertyValue("proxyInterfaces", selectedInterfaceArray);
      //Target the old bean
      builder.addPropertyValue("target", bf.getBean(beanName));
      //Add your interceptor that was just created.
      builder.addPropertyValue("interceptorNames", new String[]{"loggedMethodAdvisor"});
      //Register the new (proxied) bean definition against the same beanName
      bf.registerBeanDefinition(beanName, builder.getBeanDefinition());

      //Store in a hash for removal if needed.
      modifiedManagedObjectList.put(beanName, bf.getBeanDefinition(beanName));
    }
  }
}

Here we first use the BeanDefinitionBuilder to create an advisor that takes in a packageName to apply the advice to. Next we loop through the beanManagedObject list that was populated by the BeanPostProcessor implementation at startup. For each bean that implements the passed in interface we store the original bean in a hash and create a new proxy with an interceptor name that matches the name of the advisor that we created earlier.

In the corresponding stopMethodLoggingForAll() we remove the proxied objects from the modifiedManagedObjectList and restore the old beans from the originalManagedObjectList hash.

Note that in the above implementation, if a bean was already a proxy (for some other reason), we are making that proxied bean a target to the new proxy. That is ok, so long as the proxy is logged as such. This was done the the LoggedMethodAdvice class as seen above.

Expose the Manager via Spring JMX

We are almost there. Now that we have a LogAspectmanager that does the proxying, we need a way to be able to invoke it. Spring JMX comes to the rescue here. All we need to do is to expose the Manager via JMX by adding the following lines to the spring context:

...
    <bean id="exporter" class="org.springframework.jmx.export.MBeanExporter" lazy-init="false">
    <property name="beans">
      <map>
        <entry key="bean:name=acmeProperties" value-ref="jmxLogManager"/>
      </map>
    </property>
    </bean>
    <bean id="jmxLogManager" class="com.acme.jmx.LogManager">
        	<property name="loggedMethodAdvice">
	        	 <ref local="loggedMethodAdvice"/>
	        </property>
    </bean>
 ...

Invoke the method on the LoggingAspectManager using JConsole.

And now all that’s left to do is to deploy your application into a web application container (like Tomcat) that also hosts an MBeanServer (like… you guessed it… Tomcat!) and you are ready to use start logging and stop logging in your application. I will not go into details of how that’s done because it’s explained quite well here.

But once you have it up and running, here is what the JConsole looks like:

Note that the methods that are exposed in the MBean interface are now available in the console. When you specify a partially filled out package name, the startMethodLoggingForPackage method creates an advisor with that pattern and registers it in the BeanFactory. Next proxies are dynamically created for all Spring Managed beans that implement the passed in interface and assigned an interceptor name of the loggingAspectManager.

You can now run your application and be confident that all managed beans that implement the passed in interface will log appropriately. When the stopMethidLoggingForAll method is invoked, the old proxies (cached in the LoggingAspectManager bean instance of the application context) are restored thereby stopping the logging.

So what’s missing?

This design seems pretty useful and easy to implement. But before you start applying it aggressively in your application(s), I must point out a pitfall, that I don’t have a clean solution to.

When we programatically re-register or remove a Spring managed bean, what happens to clients that may already be using them? By default Spring managed beans are singletons and there is value in caching state in them which many application developers will happily take advantage of. When we re-register a bean, we do not restore that state, at least in this example. So what can we do about this?

One approach would be to proxy every spring bean and then use the HotSwapableTargetSource class for swapping out the target of each proxy. But that seems a little extreme and not well-performing.

Another approach could be to only proxy those beans that implement a certain interface (Swapable?) that do not hold state.

Thirdly, a more practical approach could be to narrow the scope of applying aspects to already proxied beans and then add an advice to the existing proxies. In the case of logging service level objects, this seems do-able because typically applications already have a Transactional proxy on these objects. It is then a simple matter to add another bit of advice to this proxy.

Here is an example of how that could be done

...
public void startMethodLoggingOnPreAdvisedObjectsForPackage(String partialPackageName) {
  //Loop thru all spring managed beans
  Set keySet = beanManagedObjectList.keySet();
  for (String beanName : keySet) {
    Object o = beanManagedObjectList.get(beanName);

    //Check if this an (already) advised object
    if (o instanceof Advised){
	//Check that this object implements the passed in interface
	Class[] interfaceArray = o.getClass().getInterfaces();
	boolean implementsArray = false;
	for (int i = 0; i < interfaceArray.length; i++) {
	  if (interfaceArray[i].isInterface() && interfaceArray[i]
	     .getName().startsWith(partialPackageName)){
		implementsArray = true;
		break;
	  }
	}
	if (implementsArray){
	  //Add loggedMethodAdvice to it
	  ((Advised)o).addAdvice(this.getLoggedMethodAdvice());
	}
    }
  }
}
...

And similarly, advice can be removed from those objects that have been so advised (sic! )

Conclusion

We have seen how an aspect can be programatically applied to a running web application using Spring AOP. The logging aspect is just an example, there is no reason that this concept cannot be extended to apply other cross cutting concerns of an application. That, however, can prove dangerous if due consideration is not given to the ramifications of such a design. On the other hand…as our webbed friend once said: With great power comes great responsibility. If used judiciously, it can be of great benefit too!

I have also pointed out a weakness in this approach that I welcome comments on.

References

1. Inspired by this great post from the Spring Blog!
2. My previous post.
3. Spring docs .

Download

A complete working example that demonstrates what’s been covered in this post is available here.