In-Depth

NuGet Inside the Firewall

NuGet extends Visual Studio by simplifying the process of installing and updating third-party libraries and tools. It has limitations in a local private network, however. Here's what you need to know, along with a possible solution.

• By Joe Kuemerle
• 07/09/2013

Full disclosure: I'm currently a temporary contractor for ProGet (since February 2013), but I've been a user of the product long before that, since its first public beta in February 2012.

From the very beginning of modular programming, development teams have been struggling with dependency management. From frameworks to libraries and modules, the pace and quality of modern software development wouldn't be possible without the enormous benefits of building on top of code written by others. The major trade-offs of taking dependencies are managing those dependencies, updating them appropriately and taking breaking changes into account.

Dependency management within organizations often takes an evolutionary route. It often starts with ad hoc distribution systems such as e-mailing files to development teams, then progresses to locating the files in a central location, and then progresses to using a versioned location such as an external sub-repository under version control.

The next step in that evolution arrived with the advent of NuGet, which allows .NET teams to take advantage of the next generation of managing dependencies. NuGet solves the issues of knowing when an update is available (discoverability), knowing if there are breaking changes in the update via semantic versioning, managing sub-dependencies through package dependencies, providing consistent access to previous versions and easing distribution to the consuming developers.

Using NuGet to manage free and open source dependencies has resulted in increased efficiency for development teams. With NuGet.org as a central distribution location for public packages, developers can keep their third-party dependencies up-to-date quickly and easily.

Internal NuGet Servers
The NuGet infrastructure can also provide the same benefits to internal development teams who need to share private or commercially licensed dependencies. By implementing an internal NuGet server in a multi-team environment, developers leverage the discoverability, distribution, semantic versioning, and dependency management of NuGet to increase their efficiency and reduce the friction of dependency management. This idea is gaining traction in the .NET community; for example, Telerik now supports distributing its licensed components from an internal NuGet server.

While NuGet.org is an outstanding resource for distributing free and open source components to development teams, it isn't the appropriate repository for internal or proprietary software.

Using the open source NuGet server is an option that provides basic NuGet implementation within your organization. The basic NuGet server allows development teams to take advantage of the ease of distribution and discoverability of NuGet. However, while there are efficiencies to be gained, there are still some areas where the NuGet server is not perfectly suited for an internal environment and can itself introduce additional friction.

Friction Points
Within an enterprise, the open source NuGet server becomes yet another identity silo. User accounts and permissions are stored in internal tables, which can lead to increased management costs because organizational changes must be duplicated within the NuGet silo. This friction can also lead to additional auditing overhead if the organization must comply with external controls such as Sarbanes-Oxley, Payment Card Industry (PCI), ISO 27000 or other regulatory requirements. In addition, the security of the enterprise infrastructure can be affected if personnel changes aren't duplicated in the NuGet server in a timely manner.

Along with being an identity silo, the open source NuGet server provides only a basic permissions structure, with a single siloed user account being the owner of any particular package; in addition, there are no authorization controls on package distribution. Within a structured environment this can require sharing credentials for access to administrative functions, as well as potential fragmentation into multiple server instances to support separation of privileges.

Because of these limitations, the NuGet open source server may not adequately support an organization's preferred deployment strategy. In many organizations, code isn't shipped directly from a developer's desktop to a production system without undergoing a variety of tests. These can range from fully automated unit and integration tests to manual smoke and regression testing. The NuGet package versioning specification does allow for the appending of a string at the end of the version number to indicate the package is a prerelease package, but has no other method of exposing the quality, stability and appropriateness of a particular package for a deployment environment. Running multiple NuGet servers to accomplish explicit quality segmentation leads to additional environment fragmentation, along with the concurrent increase in overhead and developer friction.

Taking advantage of the large numbers of quality NuGet.org open source libraries enables developers to maintain maximum productivity by building on stable, well-tested and high-quality open source code. One drawback to building on both freely available components and internal libraries is that each developer and build script must maintain references to multiple NuGet servers. This can lead to new developer installations not being able to effectively build code due to missing or incorrect references to the multiple required NuGet server entries, as well as build failures if build machines aren't correctly maintained. Requiring multiple NuGet server references erodes some of the potential gains in discovery and distribution of dependencies via NuGet, as well as potentially introducing instability into the build process if an internal package name overlaps with that of an external package.

Licensing and Liability
As with all intellectual property efforts, software development teams must be aware of the licensing details of the dependencies they reference. Consuming components whose licenses are at odds with the end product's license can result in costly liability lawsuits or patent disputes. While the open nature of the NuGet.org feed provides maximum flexibility by exposing all packages identically, there are real-world consequences to licensing issues. With unfiltered exposure to the entire NuGet.org feed, some development teams may accidentally take dependencies on components whose licenses or underlying technologies aren't compatible with the company's regulatory or legal obligations.

As with any external service, NuGet.org experiences downtime and may encounter service disruptions. This can result in a reduction in developer productivity as essential packages are unavailable in the development environment until the disruption is resolved. If NuGet.org is unavailable in a continuous integration environment, builds will fail and can cause schedule slips and missed expectations. Local caching of packages is supported on the clients, but most continuous integration implementations rely on NuGet's package restore functionality to ensure that each build is created with a fresh download of the dependencies, resulting in downtime at the consuming end for as long as the NuGet.org servers are inaccessible.

Finally, distributing dependencies over NuGet can also be problematic for consuming development teams to utilize efficiently. The majority of NuGet packages are distributed with only the binary dependencies, and don't support step-into debugging by providing symbol packages and source code serving. This can be doubly inefficient for internal libraries, as the required symbol files may not be preserved and the exact version of the source code used to build the binary may be buried many changesets down in a source control system to which the consuming developer has no access. At least with open source libraries, the consuming developer may be able to use search tools and find forums where other people have discovered the same issue and may provide a workaround or fix. With internal components, the developer doesn't have that option; instead, it requires a corporate fact-finding expedition to find the root cause of any package issues encountered.

The ProGet Option
The NuGet platform is a great step forward for managing .NET (and other) dependencies. The open source NuGet server is primarily a tool to support open source component packaging, and as a result it doesn't support features that may be required in a more structured environment. While the server code is open source and can be forked and customized, the amount of effort involved would far outweigh the costs of running a specialized NuGet server designed to support the more structured workflow of many development teams.

One solution I've used to solve the previously listed friction points is ProGet, from Inedo LLC. ProGet is an alternative to the open source NuGet server, designed from the ground up to support the more structured demands of internal software development teams. One of the primary advantages of ProGet is the ease of installation and seamless integration into existing environments. ProGet comes in both free and commercial editions. The free version omits some features useful in more structured environments, while still providing an easy-to-use and nicely configurable NuGet server.

ProGet integrates Lightweight Directory Access Protocol (LDAP) authentication directly into the product. This means it will work seamlessly with Active Directory and use the existing corporate Active Directory identity store to authenticate users. Because there's no local identity store to get out of sync with personnel changes, as soon as a change is made in the corporate Active Directory system, that change is immediately effective in the repositories managed by ProGet.

In addition to integrated authentication, ProGet provides a rich set of fine-grained permissions both for individuals and groups of users. By combining Active Directory management of users and groups with privilege management available in ProGet, it's easy to set up a detailed set of permissions that will enforce any required separation of duties without any extra overhead as users move between roles. Effectively managing identities and permissions provides for better security, as there's no need to share administrative logins. Storing identity data outside the system database enables easier auditing of this critical piece of infrastructure.

Because ProGet supports multiple NuGet feeds on a single server, it's easy to separate packages into different quality levels for use in various environments without requiring additional servers. This provides the ability to express a richer set of stability and quality readiness than the binary stable/pre-release NuGet versioning convention. By providing multiple feeds, developers can easily work with dependencies built for multiple environments within the organization that are semantically versioned as stable for that environment, without the extra friction introduced by taking dependencies on pre-release versions of packages.

By providing multiple feeds with a single server installation, management of ProGet is centralized and more efficient, while still supporting multiple independent development teams, each of which can manage its own custom feed. This allows an organization to make the best use of server resources, take advantage of a single-server system for administration, and still provide the development teams with the most efficient tool for managing dependencies.

By exposing packages from other NuGet feeds directly on a ProGet feed by using a connector, development teams are able to expose a single canonical feed URL for all the developers and build servers for a team. This reduces the amount of overhead required to maintain dependencies and improves build results, as there's only a single-source URL for all packages, no matter how many distinct feeds from which they may be served.

By building on the connector-filtering mechanism within ProGet, teams are able to tighten up the packages exposed on a connector to a feed, and limit the available packages to those in conformity with any licensing or vendor restrictions. By filtering the connector feeds, teams can improve the performance of package management and ensure that developers can easily consume appropriate packages.

ProGet feeds also support caching of packages from connectors. By caching external packages on the ProGet server, teams are protected from connectivity disruptions that can break builds and prevent developers from working when third-party feeds are unavailable. Additionally, by caching packages on systems that are more local to the build systems, organizations are able to achieve faster build times and a reduction in bandwidth consumption for each build.

A prime differentiator of ProGet is built-in symbol and source code serving. Easy hosting of symbol and source code packages on the ProGet server helps reduce infrastructure complexity by providing a single canonical source to enable step-into debugging of other team's packages hosted in the ProGet repository. Enabling step-into debugging of NuGet packages allows consuming developers to work efficiently with components developed by other teams within the organization. By taking advantage of ProGet to include binaries, symbols and source code in a single package, teams can minimize the number of deployment artifacts they ship and also ensure that consumers of their packages only receive the required binaries in the downloaded package. Development teams can leverage the rich set of NuGet packaging actions in Inedo BuildMaster to create an automated intuitive build and deployment process that ensures the correct symbols and source code are distributed with their packages.

ProGet also offers an easy extensibility module with both an API and an SDK that allows organizations to easily customize ProGet to better integrate with existing systems or to quickly add custom functionality. ProGet can be configured and driven from external applications via the API, which supports both SOAP and JSON calling conventions. Many features can be extended or modified with custom functionality written in .NET using the SDK.

Development teams can achieve even greater productivity using the free ProGet client tools. This set of tools works with all NuGet feeds and provides additional features targeted at internal systems developers. The ProGet client tools integrate in Visual Studio and offer all standard NuGet functionality as well as improved dependency management; automated package synchronization to keep all projects in a solution correctly updated; easier access to previous package releases; and the ability to opt out of nonreversible project transformations. The ProGet client tools also support ProGet features such as feed discovery, which will automatically find all feeds hosted on a ProGet server. There's also a free command-line client that's a lighter-weight drop-in replacement for nuget.exe in build scripts.

Because ProGet exposes all NuGet protocols and features, organizations can use it for more than just .NET package management. Native-code Windows developers can distribute their CoApp packages within NuGet and can use the corporate ProGet servers to share their packages within the enterprise.

The NuGet standard is used to support more than just code dependencies. Chocolatey is a machine package manager for Windows that distributes and installs applications and dependencies onto machines. ProGet can also host Chocolatey packages for distribution within an enterprise to simplify installation and maintenance of applications. (A tutorial on using Chocolatey can be found here.)

The ease of distribution provided by the NuGet package conventions has enabled other vendors to build deployment platforms using NuGet as a delivery mechanism. Both the RedGate Deployment Manager and Octopus Deploy are built using NuGet feeds as deployment sources. ProGet feeds plug right into each of these products and provide ease of management with direct package upload and download features, as well as the fine-grained permissions model that's integrated into an organization's existing Active Directory infrastructure.

Resources
Using NuGet inside the firewall is a great way to increase developer productivity as well as build cross-team cohesiveness. NuGet lets you write better code faster by:

• supporting greater discovery
• raising awareness of updates through detailed change tracking and semantic versioning conventions
• sub-dependency management
• integrated tooling

A ProGet server adds seamless management, easier integration, improved discipline and better debugging to the NuGet experience.

Enabling multiple development teams to interoperate and transparently share both internal and external components increases velocity across the organization and builds trust relationships between teams. Using ProGet features will result in an improved developer experience without additional operational overhead. Regulated industries can also achieve better regulatory compliance with minimal additional cost.

Installing ProGet is fast and easy. You can download ProGet and use it either in the free, feature-limited edition permanently or get a free trial license for the full version. You can also take a full installation of ProGet for a test-drive by using the Inedo Test Drive Amazon Machine Image (AMI) and running a preconfigured system in your own Amazon Web Services (AWS) environment. You can use the Inedo Test Drive AMI free for an entire year by registering for a free AWS account and taking advantage of the Free Usage Tier. For a detailed overview of all the features and ways to use ProGet, see the ProGet Documentation center.

ProGet is a compelling enhancement to NuGet and adds a number of valuable features for effective distribution of internal components to software development teams.