The Mighty Development Cluster – Part I: The Awesomeness

Through my experience at Adaptive Computing I’ve developed a great respect for the development cluster. I’m going to be doing a series of entries about the development cluster. This part will be about why we should all want a development cluster. Before I start I want to point out that the use of the word development instead of test is deliberate. Testing is one of the biggest benefits that a development cluster provides, but it can do a lot more.

Image via http://api.ning.com

Image via http://api.ning.com

Since we’ve mentioned testing, let’s start there. Checking out a release on your dev cluster is like having a personal beta for every release. This can be your cluster’s coat of armor when used correctly. In a development cluster you likely won’t have the same number of nodes and jobs as in production, but you can replicate all of the settings and policies that you use there. You will have a place to test your unique situation, something that a regression suite, no matter how comprehensive, is unlikely to match. Ultimately, a good development cluster will result in less emergencies and less calls at 2 am.

Another tremendous benefit of the development cluster is the enhanced ability to tinker. Nobody likes the idea of changing policies on the fly in production without much idea of what will happen. Nobody. If you have a development cluster, you don’t have to wait for a downtime to test new things, because that’s what it is for. You can change things to see if a new policy really does work as advertized and most importantly, if it has any unmentioned side effects or doesn’t jive well with your settings that may or may not accompany the regression tests around it.

Being able to reproduce an issue you’re facing in a development cluster can be a good way to:

  1. Pass off the information for Adaptive to easily reproduce it and create a regression test for it.
  2. Get a developer to be able to look at it as it happens without harming production.
  3. Be able to quickly add a patch that logs more information about what is happening.
  4. Be able to efficiently test the fix for that issue.

Suppose you are waiting for a fix in a new version that comes out in two weeks, but you don’t have a scheduled outage for two more months. When that new version is released, you can put it on the dev cluster and verify that it will behave how you expect. Very similarly, let’s suppose you see a feature in a change log that says an optimization occurred; you like optimizations, but when it comes to doing the upgrade, you don’t know if the juice will be worth the squeeze. Perhaps the jobs you run aren’t big enough to experience the pain, and changing from 5 to 4 seconds isn’t worth your headache. Instead of guessing, you can prove it on the dev cluster. The same benefits also apply to changing your epilogue and prologue scripts, Moab policies, TORQUE settings, or anything else surrounding your cluster.

Finally, the development cluster allows you to contribute in a very positive way to the TORQUE community. Most of us believe in open source in a big way, and you can use your development cluster to really advance the cause. When we are coming up on a release, we make it a practice to branch and inform people that they can test it if they have the cycles. These tests are invaluable to us, and when we’ve strayed from doing them we’ve felt the pain. Obviously, having a development cluster won’t always give you the time to participate in beta-testing software – although I believe it will help you work more efficiently—but when you can it benefits all of us. One of the limiting factors in software development is the ability to adequately test it. We are constantly working internally to better our testing so that we can keep up with and stay ahead of the market. I believe we’re advancing by leaps and bounds with our internal testing, but development clusters will continue to be worth their weight in gold regardless.

I hope this makes you want to have a development cluster if you don’t have one, and that it gives you ideas on how to use yours in new ways if you already have one. Look for part two to come out next week as I’ll work on discussing strategies for making a development cluster a reality at your site.

Facebook Twitter Email