Special Reports

The Role of the ESB

ESBs provide a multitude of services for an enterprise committed to optimizing IT in support of business processes.

• By Peter Varhol
• 10/01/2006

Enterprise Service Bus (ESB) sounds as if it should be a grand concept. In the era of the service-oriented architecture (SOA), you hope to find an orchestration of services. This is especially true if the services consist of a combination of different types of applications and code, from Web services to legacy code. It is logical to assume that some mechanism must kick off services, pass messages between them, and restart services when they die.

Yet the widespread use of ESB has not caught on within the industry. Leading vendors in the ESB market, such as Sonic Software and Cape Clear, remain healthy and growing, but not at breakneck speed. And while larger organizations, such as IBM and BEA, have also rolled out ESB offerings, you did not see many enterprises in a big rush to implement one.

Why has the industry been so slow in the uptake of ESB technology? Two reasons come to mind, and neither of them reflects poorly on the future success of the technology. These issues deal with maturity and complexity.

The first roadblock to widespread ESB adoption is the maturity of the SOA within an enterprise. Enterprises that adopted an ESB tend to have advanced plans for SOA and a sophisticated SOA infrastructure. If you are simply experimenting with services, then you are likely to find that an ESB is overkill for meeting your present needs.

The second is the complexity of an ESB implementation. To begin with, messaging is complicated. With the services layered on top and the standards supported, you will find the modern ESB a challenge to understand and configure correctly. As with lots of middleware, ESB implementations are highly dependent on the platforms, infrastructure, applications, and perhaps, even the phase of the moon.

These issues speak to development of the ESB concept that is still necessary. Complexity is vital because ESBs perform difficult and complex tasks, across a wide variety of operating systems, software, and embedded devices. But too much of this complexity still comes through to IT professionals who have to install, configure, and maintain the software. In order for the ESB concept to be widely adopted, more of this complexity has to be abstracted away from the users.

The ESB's Roots
The tenets of the ESB grew out of messaging software. The most notable is the Java Messaging System (JMS). JMS messaging helps you establish asynchronous communication channels between applications and application components that were not designed to work together. It provides a well-understood mechanism for tying together these types of applications. Not surprisingly, you will find that JMS is the basis of many ESB solutions, such as those from Sonic Software and Cape Clear.

Messaging between applications and application components is at the heart of the ESB concept. For example, there are circumstances where it's absolutely vital for a message to be received and processed. In the commerce sector, it's necessary to have a reasonable level of certainty that a purchase, credit confirmation, or stock order is received and processed. And there are circumstances, such as in the health care, defense, or space sectors, where failure to receive and process a message can have catastrophic consequences.

Conceptually, both messaging and Web services deliver data to other components. So, the growth and development of JMS into a Web services and SOA backbone was a natural progression.

But the ESB also has origins in other technologies. The Business Process Execution Language (BPEL) was designed as a way to call services and pass data between them in a high-level orchestration. Because you could not find facilities for reliability of transactions or performance optimization, BPEL provided you with a tool to organize services into workflows, and to pass data between them.

Microsoft also offered early ways of orchestrating Web services, through its BizTalk platform. The BizTalk software allows you to connect diverse software, and then graphically create and modify process logic that uses this software. It also lets information workers monitor running processes, interact with trading partners, and perform other business-oriented tasks.

In an upcoming release of Visual Studio and Windows Vista and .NET 3.0 technologies, Microsoft is offering Windows Workflow Foundation (WWF), a state-based workflow design and development tool. It integrates with Visual Studio and lets you design a workflow and write code (usually as XML Web Services, but it can also be other types of code) to execute the workflow. While it is not by itself an ESB, WWF does help you orchestrate services. WWF has been available for a year now in beta and community form, and Microsoft offers a "Go Live" license for those seeking to deploy workflow applications today.

The Role of the ESB
An ESB incorporates a standards-based abstraction layer on top of enterprise messaging. It adds specific services to messaging, including security, priority-based routing, content-based routing, adapters for different types of code, and a wide variety of other mechanisms.

As for standards, there are many that can potentially be supported by an ESB, including transports (such as, XML and various others, depending on the implementation), adapters, data formats, and security. The variety of accepted international standards supported by ESB is the key aspect that sets apart an ESB from other types of middleware and specific adapters—because this aspect provides you with lots of flexibility and enables an ESB to work with other standards-based products with little or no customization.

Depending on who you ask, an ESB is either an "essential" or "useful, but not required" backbone for an SOA implementation. In theory, it routes, prioritizes, schedules, monitors, and controls the flow of traffic between services and other applications and code components. As the name implies, it is meant to work in an enterprise to provide a bus, or backbone, for services in an SOA environment.

In practice, the ESB passes messages between services and often legacy code in support of one or more workflows. Enterprises use ESBs when they realize that their dependence on individual applications, which have limited or no communications with each other, is preventing them from recognizing greater efficiencies. Reengineering is accomplished by sharing data across applications and establishing workflows that orchestrate your applications and their data.

Of course, you can use other means to achieve data sharing and orchestration. But an ESB provides one-stop shopping for sharing and orchestration, and it is the only single solution that covers a wide range of applications, data types, performance, and layered services, such as security and reliability.

Some enterprises are not yet ready to dive head-first into the ESB pool. Taking on an ESB requires a commitment to use it for connecting and orchestrating multiple enterprise applications, data and data types, and business processes. While you can start with smaller projects to build up the learning curve, you will find the true value of the ESB is implied in its name—a bus for all services across the enterprise.

ESB in the Future
Your ability to connect applications and pass messages flexibly through the use of standards with an ESB offers a wide variety of uses for it within an enterprise, and potentially, between enterprises.

One opportunity for ESB use—and an area of growing enterprise interest—involves the increasing amount of event-based data that is being collected from transactions, radio frequency identification (RFID), or market statistics. To achieve complex event processing (CEP) with this data, you must obtain a human understanding of the low-level events produced by computers and information systems. Your goal is to understand why an end result occurred based on the interaction of these low-level events.

CEP might be used as a complement to the SOA. You can argue that an SOA is entirely event-driven, with applications sending events in the form of simple object access protocol (SOAP) packets to which Web services consume and respond. For services that are high-volume and transaction-oriented, CEP is a complementary set of checks for service to complete. It is a natural extension of SOA to identify, monitor, and regulate events that are designed to produce certain outcome.

In fact, there is a hybrid of CEP and SOA called "event-driven architecture," and it is a specific application of CEP. To create event-driven architecture, you must build a computing architecture specifically for creating, collecting, and analyzing events, and making decisions based on those events. An ESB can drive and control the different parts of this architecture. If your events tend to be few and far between—for example, if calls to your Web services occur infrequently and require significant processing by the services—then you might not need an ESB to control your architecture. But a high-volume SOA can incorporate an event-driven architecture designed to mine intelligence from those transactions. Incorporating these two structures might help enterprises more rapidly evaluate realtime data, and ultimately make better decisions.

Today's ESB might not be a hard and fast fact of life. But there is a need for orchestration of Web services that is beyond the simplicity of BPEL and other orchestration language. And many enterprises would prefer to use a standards-base way of orchestrating Web services.

A product such as Mircosoft's BizTalk might fill this enterprise need. But, Microsoft refuses to use the term ESB, even though BizTalk is evolving into an ESB offering. Can the need for orchestration of Web services in the enterprise SOA also be filled by Microsoft's WWF? Not likely, unless WWF is executed by BizTalk, or another engine that provides the data movement and reliability features in the ESB. BizTalk and WWF offer partial solutions to the problem of sharing data and orchestrating workflows, but they don't offer the range of service or the broad use of accepted standards that the ESB delivers.

The adoption of ESBs within the industry is happening slowly and gradually. But the utilization of ESBs will begin to grow more quickly as enterprises internalize the services approach to applications and business processes. For most enterprises, it is a significant change in thinking about applications and business workflow. Once enterprises shift their beliefs, they will realize the ESB represents the most comprehensive way of executing their vision: Orchestrate reliable and high performing Web services and other code components so that they can meet enterprise business needs.