Automating testing in Coldfusion and Railo with Testbox, Gulp, and Jenkins. Pt 2

If you haven’t already read it, then part 1 of this article attempts to explain how I got to this point and in particular why I chose these particular pieces of software. Part 1 is available here : Automating testing in Coldfusion and Railo with Testbox, Gulp, and Jenkins. Pt 1

Before reading this there were a few places I checked out for inspiration which are worth a look. Adam Cameron’s TestBox: running on actual proper tests along with Sean Corfield’s TestBox – replacing MXUnit and providing BDD helped to get me started along with the TestBox documentation.

A Gist containing the code samples used below is available here : Code samples for Automating testing in Coldfusion and Railo with Testbox, Gulp, and Jenkins. Pt 2

Assumptions

For the purposes of this guide I will assume the following.

1. You have at least 1 site available with a CFML engine (Railo or Coldfusion) installed. The engine must support closures for this to work.
2. Tomcat is running as railo:railo (no prizes for guessing which CFML engine I prefer)
3. The assumed webroot for this site will be /nas/content/live/msoweb/www.testme.local/.
4. The assumed domain for this site will be www.testme.local
5. The below is done on a Redhat based linux box, for your environment these instructions may need altering. This has also been tested on an EC2 instance running Amazons version of Linux.
6. Your user has permission to do everything below. For a production environment you want to be far more careful with security.

I have created an AMI on AWS to provide an example of Jenkins and Railo configured on Tomcat which you can use to check settings if you have any problems. The AMI name is mso-railo-tomcat-public ( AMI ID ami-05e23472 )

What is needed to automate testing?

A web server

Well, hopefully this one is obvious but you never know. As mentioned above you do need a server of some kind on which to run the tests. In an ideal world you would configure your CI server to build a fresh machine for your tests, and to throw it away when it’s done. This how-to isn’t based on an ideal world, but is just intended to get you started with testing. If you’d like to look in to automating building the servers as part of your test suite then you likely need to look in to tools like Vagrant and Chef

Testing framework ( TestBox )

The testing framework provides the syntax in which to create tests. Normally these will take the form of CFCs and are able to be loaded individually within a browser should you wish to dig into a specific test. Some alternatives are MXUnit and Tiny Test

Task runner ( Gulp )

The task runner is used to fire off the unit tests. This is generally a command-line tool that is configured to fire off multiple tests and generate a report in a specific format for interpretation by the CI server.

Some alternatives are Apache Ant and Grunt

Continuous Integration (CI) Server ( Jenkins )

The CI server runs the tests and processes the results. It is able to present reports and fire alerts should a build fail. You can make an “everything is OK” alarm too if you like. This is where all the test scheduling or automatic executing happens.

Some alternatives are Hudson, Bamboo and Apache Continuum

Enough talk, lets get installing!

Install TestBox

Originally I was going to install CommandBox and install TestBox through that, however for some reason there seem to be problems in following that same route at present. Instead I’ve just gone down the zip-download route instead. These instructions were found in the TestBox documentation

Setup MXUnit tests to work in TestBox

Now that we have TestBox installed we need to make some changes to allow the MXUnit tests to be used. It’ll also be a good idea to grab the sample test if you don’t have any of your own.

You can now run the tests from the URL http://www.testme.local/demo/BaseTest.cfc?method=runRemote but there is a problem. No tests will be found. Some alterations to the MXUnit tests will also need to be made to allow them to be used with TestBox. Instructions for this are also available in the TestBox documentation regarding MXUnit compatibility.

The main difference here is that MXUnit will run all public functions within a CFC suffixed with the word “Test” (i.e. BaseTest.cfc) however this is not the case with TestBox. TestBox requires either that all tests are prefixed with the word test (i.e. function testRetrieveUUID()) or alternatively you can use function annotation to indicate that a function is a test. To do this just add alter the retrieveUUID() to read as follows

If you need to do this on a lot of tests though then this can be a bit daunting. In my case it was about 300 MXUnit tests I needed to update and so decided to go down a Regex replacement route within sublime. If you just run the following regexes then hopefully it should do all the updates you need. It is definitely worth a sanity check prior to committing though. I ran the below on the folder containing all my tests, but not any folders containing any final logic.

You are now ready to run the tests. If you load up the following URL in a browser then hopefully you will be greeted with a passing unit test. If your domain differs from the assumed domain below, then of course update that.

http://www.testme.local/demo/BaseTest.cfc?method=runRemote

The report generated is simple, but accessing the test like this can help with debugging individual failing tests if the output in your CI server is insufficient. Additionally it is a good way to ensure your environment is configured correctly, as a compilation error often offends!

Create an endpoint to run all your tests

Chances are that you won’t put all your tests into one big CFC. At least I hope you don’t put all your tests in one big CFC. That’d be ugly. Hopefully you’ve put them all in a directory though. If, like me, you came from MXUnit then chances are you have suffixed all the test case CFCs with “Test” in the name, though not to worry if you haven’t. For this we will create a simple CFM file that will set up a TestBox task with all the tests within a given directory.

Within the sample runAll.cfm you will see a function passed in against the argument filter. This exists to ensure that any files that are deemed to not contain test are not included. In this case the fact it is looking for Test.cfc at the end of the path will ensure that “Base.cfc” will be ignored while “BaseTest.cfc” will be checked for tests. If you are running a version of ColdFusion that doesn’t support closures then this part of the script will probably error. But that’s what Railo is for, right?

The use of is because run() seems to be pretty noisy. If you like a load of # signs being output then feel free to omit the , but personally I prefer just the useful output.

If you load this script up in your browser ( http://www.testme.local/demo/runAllTests.cfm ) then you should see an Ant-formatted report. The reason for choosing the Ant-formatted report is simply that I was converting an Ant task over and so this seemed sensible. You can always adjust this to other formatting methods as you see fit.

Install Node and Gulp

Now that the tests run, and are easily executed en-masse, we need something that will run the tests for us. To do this we need a task runner, and for this job I chose Gulp. This means installing node too. And the node package manager. Yep, another package manager! If you already have this installed though feel free to take a nap at this point. For this you will also probably want the EPEL repo. Here are some links for instructions for installing the EPEL repo and for getting started with gulp. I won’t include the EPEL repo instructions here though. The package.json in the Gist includes the packages we will be using.

Configure your gulpfile to run your tests

You now have gulp and all dependencies installed in the demo site. Why not try it out, just type “gulp”. Oh what’s that? An error? Gulp looks for a file named gulpfile.js which tells it what to do. Now would be a sensible time to set this up. As with the rest of this, there is one I have prepared earlier.

This gulpfile will remove any old tests, create a folder named testfolder in which to store the tests, and run the /demo/runAllTests.cfm file. This is by no means perfect, and can easily be edited as needed. For the purposes of this demo though it works. I have specified the first task in gulp should make sure that a url argument is passed in. This is to allow this same gulpfile.js to be used in multiple environments by just changing the target domain. Just try running without the argument (by just typing “gulp”) and see what happens!

You should now see an xml file in /nas/content/live/msoweb/www.testme.local/demo/testresults/ containing your test results. Happy days!

Commit your changes

Make sure to commit all these configuration files to your repository before continuing as Jenkins will be checking out your repository to get both the package.json and gulpfile.js. In this demonstration I created the repository with mercurial using the following. Within the Jenkins task (which you will come to later) these correspond to a repository URL of “/nas/content/live/msoweb/www.testme.local/demo” with a Revision of “default”. In a real repository these settings would of course change. If you would like an alternative file structure then you will need to modify the gulpfile.js accordingly.

Install Jenkins

You now hopefully have a gulp task that can generate your test results. Time to add in the final step, the ever-helpful and polite Jenkins. It’s just a shame that jenkins-ci.org seems to be less ever-helpful as when writing this it was frequently having stability problems. Don’t be surprised if it takes a few attempts to get everything downloaded. So lets get Jenkins installed and started. The instructions for this were found on the Jenkins wiki.

Now here is where I hit an issue. If I was using a standalone Tomcat installation then I would likely have followed some different instructions to install Jenkins. In fact I would have just downloaded the war file and used that. As it stands though I used the Railo installer (with its own Tomcat) and the Jenkins installer (with its own Tomcat). Idiot!

The good news is that this is fixable. If you check the logs you can find out that it’s due to the Ajp connector clashing with Railos tomcat on port 8009. So just requires a slight tweak to Jenkins.

1. Load up /etc/sysconfig/Jenkins in your favourite text editor
2. Find 8009 (probably looking for JENKINS_AJP_PORT=”8009″)
3. Edit it to an unused port (I went for 8008, pretty original right?)
4. Restart Jenkins

For future installs I will be going down the war file route, but I just thought it best to include this as it stumped me first time up. After this Jenkins was being far more polite, as he actually stuck around to offer me gentlemanly assistance.

Grab a few Jenkins plugins

What kind of person gets a new butler without adorning him in all sorts of accessories to make him look even more foolish? The same is true of Jenkins. If you use anything other than CVS or Subversion then you will need a plugin for your source control. Personaly I love mercurial. Thankfully the process of installing plugins is pretty painless so long as jenkins-ci.org remains online long enough for you to complete.

1. Load up your Jenkins admin panel
2. Click on “Manage Jenkins”
3. Click on “Manage Plugins”
4. If the list is empty then something’s gone wrong, just try restarting Jenkins and he should get his bits together. A StackOverflow post about this can be found here.
5. When you see you have loads of plugins to update just get that out of the way, be prepared to cry a bit as many of them fail to connect to the update server.
6. Click on “Available” and search for the repository of your choice (yay “mercurial”)
7. Install an appropriate plugin
8. Click on “Install without Restart”
9. When it finishes timing out connecting to update servers then you are good to go.

There is also a nodejs plugin available but realistically this didn’t work so well for me. Possibly worth looking at again later on

Configure your new job in Jenkins

I’ve been going for a while here at this point, so time to wrap it up. Lets give Jenkins something to do. He needs to be told how to do everything, but at least has a good memory.

1. Click on “New Item”
2. Choose a good name. I’ll call this one “testme”
3. Select “Build a free-style software project” and click “Ok”
4. Under “Source Code Management” select your source control method and fill out appropriate details. For mercurial this boils down to a repository URL, credentials, and revisions or tags to use.
5. You can configure “Build Triggers” if you’d like it to automatically run tests. At present I am running tests manually however if you wished to have them triggered on commit, this is where you would configure that.
6. “Build” is the section where the fun happens. To keep it simple I opted for the shell method and ran the following commands
7. “Post-build” actions is where you tell Jenkins what to do with reports and the like. I added a post-build action stating “Publish JUnit test result report” and set up the “Test report XMLs” value to be “testresults/*.xml”

There are plenty more settings you can tweak in here, but just to get something running the above should suffice. You can now save your settings and hit “Build Now”. This will run the task, and hopefully show you some nice results. If you don’t then feel free to give me a shout on twitter @SimonHooker and I’ll try and help you out.

Where do we go from here?

It should be noted that the only files Jenkins is really using at this point from your repository are gulpfile.js and package.json. Due to specifying the URL as above the actual logic and test CFCs will be accessed wherever that domain name resolves. This is of course not ideal, but for the purpose of getting a basic Jenkins install running will suffice. In a future article I will cover configuring Jenkins and Railo on a standalone Tomcat install including pulling all source code from the repo to ensure entirely fresh builds.

There are a lot of areas for improvement in the above. Jenkins at the moment is being a bit dumb, he’s not really doing anything with the results and neither is he doing things of his own accord. Again hopefully I’ll get a chance to cover these at some point in the future, but if not then hopefully this guide will give you enough of a headstart to allow you to make some progress in configuring your own environment.

I was just going to leave this here but I’ve since changed (again) how I have this all configured, and so will put together a 3rd part detailing that. This way doesn’t even need an external task runner, relying purely on TestBox and Jenkins to do the work.