In-Depth
Build to Order
Are your software builds slowing you down? Powerful solutions streamline build and branch management.
- By Peter Varhol
- 03/01/2008
It used to be that building an application was a relatively straightforward process -- compile source code, link in a library or two, and out comes the .EXE. Configuration was the same; with only a handful of files to track, most development teams didn't even bother writing down their versions and changes. Because of this simplicity, many enterprise development groups have traditionally built only at major milestones, or even only at the end of the development process.
This approach no longer works. Today's n-tier applications typically have hundreds or even thousands of source files. There may also be dozens of libraries, modules from the open source community and Web services that make up parts of the application. You may be building for multiple platforms.
As a result, most development teams can no longer build once or even build only at major milestones. Those that still attempt this practice find that it can take days or weeks to tweak the build process and files to produce the application -- a big drain on developers, and potentially the cause of late software delivery. Configuration and build requirements can also be lost over time, making it impossible to build the application again.
The solution is simple, says Damon Poole, founder and CTO of AccuRev Inc., a software configuration management vendor: "You need to have building your software down to a science."
That means build early and build often, so that your teams are always in a position to deliver the application. Keep track of all library versions, and upgrade them and move them into the mainline only after regression testing. The same advice holds with open source modules. Dev managers need to know what versions they build with, where they're located and in what sequence. At the very least, building often identifies problem areas quickly and makes those problems easier to correct.
Poole knows a thing or two about the benefit of frequent builds. His company is a provider of software configuration management solutions, with a focus on agile processes.
"Continuous builds give you the confidence you can deliver software at a moment's notice," Poole says. "They'll help you find and fix bugs, and ultimately result in a faster and more responsive development cycle. You'll build better applications more quickly."
Poole says that building once a week won't cut it in an agile environment. If you only build at the end of a sprint, he explains, you often spend more time debugging the build than writing code.
Continuous builds also help make the development process more predictable, by removing doubt at the final steps of the process. It's a benefit, says Forrester Research Inc. analyst Jeffrey Hammond, that many development shops sorely lack.
"If you build infrequently," Hammond continues, "each build will be a painful adventure."
Controlling Source Code
The first thing most teams do is track all of the source and link files that comprise the application. For this practice, dev shops use a source code control system (SCCS) or software configuration management (SCM) tools, such as Perforce, IBM Rational ClearCase, AccuRev or the open source Concurrent Versioning System (CVS). Source code control lets teams lock down files so changes are done in an organized way; only one person can change a file at a time.
While SCM solutions help ensure file integrity, build integrity is an entirely different issue. Build problems can stem from several sources:
-
Lost or corrupted source files
- Complex and difficult to maintain build scripts
- Changes to the build environment
- Errors in branching
- Coding and dependency errors
In smaller projects, the tools within Visual Studio can address some of these problems. When you build a solution from within Visual Studio, it abstracts all of the details of the build from the developer. And Visual Studio can work within the context of source code control systems, especially the one provided by Team System, so you can still maintain file integrity.
However, getting a solution to build in Visual Studio can be a challenge. While you don't deal directly with build scripts, you still have to worry about dependencies, external libraries and other build issues. Complicated builds require that you set up your Visual Studio environment very specifically to handle the requirements, cautions Tracy Ragan, co-founder and chief operating officer of build solutions and services provider OpenMake Software Inc.
"Individual builds in Visual Studio are always different than the main build. The scripts are different, or Visual Studio uses different options," Ragan says.
For larger projects, building within Visual Studio isn't feasible. Unless you plan to start the build within the IDE, then take the resulting application and kick off smoke tests manually, much of the process simply can't be automated. 
"You can't easily automate the entire build-and-test sequence in Visual Studio," Ragan says. "It's really an individual developer environment, not a group-build tool."
The Team Build facilities in Team System provide some added build capabilities at the command line, but managing scripts and tracking source files isn't yet a feature of this environment.
OpenMake, a Visual Studio Industry Partner, integrates with Visual Studio and Team Foundation Server to provide the ability to build inside or outside of Visual Studio without manually coding any scripts. OpenMake's Meister provides a number of build-specific enhancements to Team Build.
For complex builds, development teams must typically set up one or more build systems driven by script (often in Perl), a command-line build engine, and a set of smoke tests that confirm the basic integrity and functionality of the build.
To manage software configurations, dev teams move beyond simple source code control to full software configuration management. That means knowing what files and what code versions you need for each build, and tracking them to ensure their location and date stamp.
Build scripts are yet another problem area. You can think of writing and maintaining build scripts as the soft underbelly of the development process. The process lacks the intellectual challenge of writing application code, yet it's a necessary and exacting task that doesn't allow for error. Scripts are often filled with obtuse code, hard-coded references and obsolete instructions, says Forrester's Hammond.
"Build scripts get reused from project to project. Sometimes no one even understands everything in them any more," he says.
The Local Problem
Problems can arise when individual developers on a team integrate their local builds with the mainline build.
Ragan refers to this arrangement, where multiple local builds are integrated, as a "mashup." The problem: inconsistent build practices that can lead to errors and slowdowns.
"Often you'll even find individual developers on the same team who are building in different ways," she says.
The result is that a build that is successful on a developer's desktop may break on the mainline build. Aside from the common practice of requiring the offending developer to buy build-break doughnuts for the team, these failures represent a significant schedule and productivity hit.
The term mashup also can refer to combining different technologies into a single application -- and the mashup concept highlights a growing challenge in build management.
Increasingly, development teams combine .NET and Java, for example, or .NET and native code. Frequently, there may be differences between development builds and production builds. These types of problems should be solved only once, rather than for each build.
Ragan says many of these issues can be addressed with build automation practices, starting with the workflow. "You have to understand your process, then gradually automate and build up best practices," she explains. "Rather than working directly with up to hundreds of scripts, you can abstract changes away from the script and to the build-management solution."
The Perils of Branching
Few processes are as painful for dev teams as branching the mainline development stream. It can take several days to manually branch the mainline, effectively preventing check-ins during the transition. And, for some time after the branch, individual developers must check in at least some code, particularly bug fixes, to both the branch and the mainline. Multiple check-ins then mandate multiple builds and tests, which slow the development effort.
Of course, branching is a necessary evil. Development teams are often writing code for currently planned features, fixing bugs in that code and laying the groundwork for future features. In some cases, code should be checked in to both branches; in other cases, it should go to one but not the other.
Dev shops can delay branching until the last possible minute, but at the risk of contaminating the release build with unnecessary code intended for the next release. The choice of the branch point is really more a matter of preference and convenience than a result of any strict rules.
As AccuRev's Poole points out, dev shops are courting risk with branches no matter what. "Multiple check-ins to different branches are highly error-prone," he says.
 |
"Continuous builds give you the confidence you can deliver software at a moment's notice ... You'll build better applications more quickly." |
| Damon Poole, Founder and CTO, AccuRev Inc. |
Tools such as AccuRev SCM attempt to rein in the management overhead and risk of branching by letting developers visualize branches through the use of lightweight streams. Developers create a stream, develop for the stream, then seamlessly merge back into the mainline. This approach makes working with mainline code more natural than with major branch decisions.
Alaska Airlines is moving to adopt continuous builds and code streams. Chris Barnes, a senior systems analyst who administers the AccuRev SCM installations for the company, says that of the 12 internal development teams using AccuRev, two have adopted continuous builds.
"You have to proceed slowly," he explains. "But eventually we'll get everyone there."
Barnes cites the advantage of being increasingly agile in a business that requires a fast response time.
"If our marketing people have identified a revenue opportunity, we have to support it now, not a year from now." Without continuous builds, Barnes says, rapid turnaround simply can't be accomplished.
AccuRev's Poole, who writes a popular blog on agile techniques at damonpoole.blogspot.com, says that continuous builds can be used on large, multi-layer projects. "If you rebuild only the parts that have changed, you can build within minutes, and promoting builds can be both fast and automatic," he says.
Tracking builds and collecting data on success rates and build errors can also help developers improve the build process. For example, Electric Cloud's ElectricCommander software displays build statistics so developers can easily understand and correct errors.
"The only way you can learn from your mistakes is to track and analyze them," explains Electric Cloud's Chief Executive Officer Mike Maciag.
Developers can turn to a rich selection of open source tools for build management and automation. Tools such as Ant (or Nant, the .NET version), Maven and CruiseControl help organize and automate builds. Ant and Nant are build tools, while Maven can be used to manage a project's build, reporting and documentation. CruiseControl is a framework for a continuous build process.
Open source tools should be used with caution. Often, support and new features come infrequently, if at all. These tools can also have less-than-ideal user interfaces, which steepen the learning curve for new team members.
Make the Most of Build Processes
If your projects are small and relatively straightforward, limiting the number of builds can work, but eventually most dev managers will take on a large and complex project that requires robust automation.
SCM software and build-management tools can streamline time-consuming branch management, reduce the number of check-in errors and yield fewer build breaks and bugs during the development lifecycle.
Build automation is a powerful tool for any project. Look for ways to identify the build workflow and automate some or all of that workflow. Gradually automating parts of the workflow can reduce the overall burden of builds, so you can be more agile in your development and build with confidence.