Special Reports
Will Application Integration Save the Enterprise?
Are you looking for something new to gain advantage over competitors? Application integration could be your vehicle for driving new and innovative information and business processes.
- By Peter Varhol
- 08/01/2004
Most enterprises have fully automated a wide variety of discrete activities that deliver value in specific ways. Whether it is order processing, inventory management, financial tracking, or customer service, there are combinations of commercial and custom applications that fit the bill and improve operations of individual systems. However, we've reached the limits of efficiency improvements in these types of discrete activities. There is only so much improvement that can be had in counting items in inventory, or tracking customer orders.
This limitation is possibly why some have concluded that IT has fundamentally accomplished its task of adding value to the enterprise. By this reasoning, except for replacing older computers and providing for maintenance on applications, further spending on IT is unwarranted. There is no further advantage in investments in new software and technology needed to compete in any business or industry, and additional investment is, in effect, wasted. Nicholas G. Carr expressed this view originally in his article "IT Doesn't Matter" (Harvard Business Review, May 2003).
This point of view, simplistic as it is on the surface, fails to take into account a key proposition. While it may in fact be true that all, or at least the majority, of traditional business processes have been automated successfully, it ignores the fact that there can be substantial increased value from a combination of those processes into an aggregate process, or an entirely new process that takes into account all of the traditional processes.
More to the point, automating traditional business processes is only the first step in improving efficiency, and one that has been pretty much achieved in most organizations. In fact, it is a necessary part of transforming business entirely, moving it from its traditional roots into an entity that achieves entirely new levels of operations and productivity. From the standpoint of continuing a significant rate of return on the investment of new software and systems, application integration is necessary to deliver the next stage in productivity improvement.
To achieve this kind of business process integration, we need to be able to get all of the applications supporting existing business processes working together. This collaboration can be called application integration, or at a less-coordinated way, data integration. In the former, the applications seamlessly share results to produce results that aren't possible with any of the applications by themselves. In the latter, there is more of a hand off of data at set times between different applications, so that one can use the results of another in its own computations.
Application Integration Drivers
The ability of applications to work together in some capacity enables them to perform tasks and make decisions that are difficult or impossible to do manually. For example, when a customer places an order, the order-processing software can ask the inventory management system for an evaluation of how long current inventory will last. If that value drops below a certain watermark, the order-processing system can instruct the inventory system to issue an order to replenish and restock that particular item.
It's unlikely that there would be processes in place to perform these activities manually because companies tend to break up such functions into small and discrete pieces, with different groups responsible for each. In some cases, customer service and order processing may even be outsourced, making it even more unlikely that they can be coordinated without integrating applications.
And this is just a simple example. In fact, there are a great number of ways that separate applications can be complementary to each other. The trick is in finding these integration points, demonstrating the information that can be derived from those integrations, and enabling them.
Finding those integration points and determining how they can provide the enterprise with a business advantage is a challenging and time-consuming process. The opportunities inherent in this process are unique to the individual enterprise and can't be relegated easily to a checklist or other mechanical approach.
Instead, finding those advantages requires a close examination of existing business processes and the applications that support them. With an intimate understanding of the data generated in the execution of those processes, an analyst can then fit the pieces together to make processes work together in a quantum leap in efficiency.
From Analysis to Action
Once the analysts have identified integrated data, however, it becomes primarily a technical issue of getting the applications to produce that data. To the development team, it often seems impossible, or so technically difficult that it's not worth the effort. Most enterprise applications weren't meant to exchange data with others at any level. For the most part, they are self-contained within their function, and use different input and output formats with different and often incompatible platforms and communications protocols. There are no natural, or even reasonable, integration points.
Even if enterprise applications can achieve a measure of integration at the data level, it's likely to be fragile, poorly performing, and frighteningly difficult to maintain. Application A writes a file, which is read by Application B. Whenever application versions change, or even when systems or application servers are upgraded, it's likely to break, requiring significant ongoing work.
This fragile nature of ad hoc interfaces is the primary reason why enterprises resist taking the next step to full application integration. Sure, analysis and identification of the integration points are difficult, but at least the task is complete at some point. Analysis can produce concrete and actionable results. In the case of the integration of applications and technologies, it's a hard task that has to be done over and over again for the life of the applications. And if it fails at some point, it has to be fixed immediately because it's likely to be costing the enterprise in terms of lost revenue or efficiencies.
What is needed, instead, is an automated way of achieving application integration. And it's not sufficient to have a fragile and buggy implementation of data sharing across applications. Integration must support overall business objectives, be seamless in its implementation, be transparent in its maintenance and enhancement, and be flexible in its ability to be changed to meet changing business needs.
Implementing Application Integration
While these characteristics seem like a tall order with little chance for success, in fact application integration can succeed in delivering new information that drives innovative business processes. To achieve the next big return on an investment in software and systems, it's necessary to get to that point.
Application integration is a process that combines both analytical skills and application capabilities. Analytical skills are necessary to understand what information is currently being generated by what applications, and how that application may be combined with other applications to produce new, useful, and cross-functional information. An intimate knowledge of application capabilities delivers the technical skills needed to produce the information identified in the analysis.
The steps needed to build a better business process, or to combine business processes, through application integration might be outlined this way:
- Process analysis – The first step in any integration effort is to identify information that crosses traditional business application lines, yet might be valuable to the enterprise. This step involves delineating each enterprise application, its function, the data it produces, and what that data is used for. As stated earlier, this process is a set of tasks driven largely by business analysts seeking new ways to combine existing processes and data.
- Process modeling – Once the applications and their uses are understood, it's necessary to identify the opportunities for obtaining new information through data or application integration. In this step, analysts use combinations of existing data and processes to create new types of processes. Being able to model business processes enables analysts to create new business processes to fit business requirements.
- Process Automation – The last step is to take the integration models that provide the best potential for value and implement integration that supports those models. These integrations have already been demonstrated through modeling to add significant value to the enterprise, making the implementation the only impediment to achieving this value.
Business processes and data unique to individual enterprises drive analysis and modeling. Technical solutions for automation, on the other hand, can be applied throughout individual enterprises, and across enterprise boundaries, to provide a catalyst for combining data and data processing in new and unique ways.
Automation may seem to be easiest if the applications are custom-developed, rather than commercial off-the-shelf products. However, that's not always the case. Custom applications are usually written for specific purposes, often by separate development teams with little or no coordination between them. And because the applications are to address specific needs, there is rarely any planning for enhancements throughout a life cycle. So custom applications are every bit in need of integration solutions as commercial products.
And commercial products may have defined application interfaces that make integration much easier. Increasingly such applications are already outputting data in an XML format, making it possible to easily achieve a loose type of data integration.
Ways to Integrate Applications
While it may seem like the difficult work is over, the implementation is the step that prevents success at leveraging IT to create new value. Most enterprises decline to place their systems and data at risk to achieve results whose potential benefits aren't clear until they are used as the basis for decision making. Fortunately, there are proven ways to perform application integration, and while the results can vary depending on the quality of the analysis and modeling steps, they can make it possible to share data and establish communication paths between applications.
There are several possible approaches to application integration, or its sibling data integration across applications. They can generally be categorized into three different categories:
Loosely coupled integrations only exchange data among applications, often through a proxy, and may not even know that the other applications exist. It may be only a matter of writing and reading a common file. These integrations are easier to maintain than traditional tight and rigid integrations, and may offer the best general-purpose approach to sharing data between applications.
Asynchronous communication provides for the ability of one application to interrupt another with input. These types of integrations are critical to conducting and protecting business operations when communication goes down. However, they require that those applications be aware of one another, and work together with little loss of efficiency. As a result, this approach can be difficult technically and expensive to implement and maintain.
Coarse-grained communication provides interaction of large chunks of data at infrequent times. It is key to maximizing the efficiency of typically high-cost communication between loosely coupled systems. This approach limits interaction between applications to essential data, and only at specific processing points or times. This approach is used typically when applications are running on incompatible platforms, or when data is in formats that must be translated between applications.
Choosing the appropriate approach to a specific application integration scenario is key to achieving the benefits of integration. For example, with high-volume transactions such as stock trades or call-center activities, asynchronous communications may seem to make sense, but it's likely that this approach may adversely affect performance of the applications involved. Instead, a loosely coupled integration may offer the best way of exchanging data between two applications that have high volumes and strict response requirements.
Implementing Integrations
Once you've selected a strategy for integration, you have to implement a mechanism that follows that approach. There are a number of different ways, for example, to implement loosely coupled integration. Probably the best known is Web services, which form a standard and popular technology for encapsulating specific processing functions within a self-contained component with known calling conventions. Web services provide a mechanism for retrieving data from one application, perhaps processing it in some way, and sending it to another.
Web services accomplish this through a process independent of the applications being integrated, that is, through SOAP transmissions. The applications must be modified to send and receive SOAP packets to the Web service, so that data is transmitted in a common format. The Web service can process the data passing through so that it is useful to other applications.
For example, an order-entry application accepts, stores, and processes discrete orders. At the same time, an inventory application monitors inventory levels, analyzes depletion, and places orders when inventory reaches certain levels. When an order is placed, the application prepares the order for fulfillment. At the same time, it sends information on the order to the Web service. The Web service checks the inventory database, and if the inventory needs replenishing, instructs the inventory application to place an order.
J2EE already has a number of technologies for implementing and working with Web services, such as the Java Web Services Developer Pack and the Java API for XML processing (see Figure 1). Implementations such as the Business Integration Suite from Cape Clear can help build these Web services (see Figure 2), and components are also available in major IDEs such as Borland JBuilder and IBM WebSphere Application Developer.
A second technology for implementing integration is Adapters. Adapters allow IT to reduce the costs of integrating applications and other business resources by delivering native connectivity to leading enterprise-packaged applications, legacy systems, mainframes, databases, and Web technologies. Adapters may also be referred to as connectors, depending on the vendor supplying the technology and the applications being integrated.
Adapters are specific for the applications involved, and may be available for specific commercial products. For example, it's not unusual to see adapters for popular customer resource management (CRM) and enterprise resource planning (ERP) applications to have available adapters that enable these large enterprise applications to effectively share data. An adapter often has to be built and managed as a commercial software product, with updates and modifications whenever one or more of the applications it supports changes.
It's also possible to develop your own adapters for custom applications or for commercial applications without an existing adapter. For example, BEA WebLogic Integration provides an Adapter Development Kit (ADK) for just that purpose (see Figure 3). The ADK is a set of tools for implementing the event and service protocols supported by BEA WebLogic Integration. These tools are organized in a collection of frameworks that support the development, testing, packaging, and distribution of resource adapters for WebLogic Integration. Specifically, the ADK includes frameworks for four purposes: design-time operation, run-time operation, logging, and packaging.
A third approach is messaging using a standards-based message broker. A message broker allows applications to deliver high-reliability, asynchronous messaging for event-enabled data flow inside and outside the enterprise. Messages are discrete packets of data that can be sent serially from one application to another, and are typically processed as they arrive. Messages can also be queued before they are processed, and accepted in turn, or ordered in some way.
Messaging is used in many different types of enterprise applications today. Frequently, it has been used with success in connecting transaction-oriented clients with mainframe processing systems. It enables a high volume of data to arrive from multiple sources and be processed according to predefined rules.
As its name implies, the Java Message Service (JMS) is an implementation supporting Java and the J2EE platform, and makes an excellent messaging system for enterprise application integration. JMS makes no distinctions about message content. It is simply the way to ensure delivery of the data. This functionality makes JMS highly flexible in terms of what it is used for and how it works. JMS implementations from vendors such as Sonic Software can help in building message-based integrations through features like queuing and working in environments that are occasionally disconnected from each other.
JMS can also be extended to support a wide variety of different services. For example, BEA's WebLogic JMS feature implements an optional JMS facility for defining a server-managed pool of message consumers. JMS under WebLogic Server supports distributed connection factories and distributed destinations, providing support for clustering as well as high availability. It offers a publish/subscribe model as well as multicasting for a high measure of control over message receipt. It also provides for message persistence; it can store messages in a persistent storage without the sender having to wait for a successful completion of message disc IO. This feature gives the users a choice between high-delivery reliability and increased performance.
A final option for application integration is data transformation. Data transformation allows you to visually translate between XML and non-XML messages, including legacy formats. This type of implementation can be a part of a Web services architecture, or it can stand on its own in a non-Web environment. The value of this approach is that it enables you to use XML for the definition and storage of different data formats in a lightweight and flexible manner, without the necessity of setting it up as a services-oriented architecture (SOA).
Data transformation offers similar capabilities to Web services, except that the only operation performed is the reformatting of data. There is no opportunity for processing the data in any way. On the positive side, this approach provides capabilities for any-to-any transformation, mapping, and bidirectional translation to allow you to rapidly integrate heterogeneous applications regardless of the format used to represent data.
Application Integration Value
Integration takes careful planning, hard and detail-oriented work, and a measure of luck. At the end of the day, the ability to get applications to communicate with one another at some level must pay off as a competitive advantage to the enterprise, or as a way to increase efficiencies in operations. Achieving this goal requires success at process analysis, modeling, and application integration.
While analysis and modeling involve challenges of their own, the technical side of application integration remains the single biggest technical hurdle. And ad hoc ways of handling integration are likely to end in failure, or at best in a fragile integration that is expensive and time-consuming to maintain.
Instead, enterprises need to apply a systematic methodology to analyzing business processes and data, building new process models, and applying integration solutions. And at the back end of that methodology—integration—enterprises must use proven techniques, along with application tools specifically designed to ease design and coding, to build integrations that are flexible and built to last through multiple application versions. The techniques used may vary depending on the applications themselves and integration goals, but whether using Web services, adapters, messaging, or data transformation, an understanding of the results to be achieved and a road map of how to get there are essential.
Unless you buy into Nicholas Carr's assertion that "IT doesn't matter," then application integration should be on your radarscope. Any enterprise, including your competition, can buy or build the type of software that automates existing processes. Getting the advantage means doing something new. Application integration is the vehicle to new and innovative information and business processes.
About the Author
Peter Varhol is a senior member of the technology staff for Compuware Corporation. Contact Peter at [email protected].