Getting Started
Implement a File Transfer Web Service
Implement a Web service that performs a binary file transfer using Web Service Enhancements 3.0 and MTOM.
- By Martin Kulov
- 05/01/2006
Technology Toolbox: C#, XML, ASP.NET Web Services, SOAP
The Web Service Enhancements (WSE) standard provides a robust infrastructure for securing Web services by using client and/or server X509 certificates, Kerberos, or username tokens.
It does more than this, of course. All messages that WSE transmits conform to WS-Star (WS-*) specifications, a set of standards created by standardization bodies like W3C (http://www.w3.org) and OASIS (http://www.oasis-open.org). This enables you to call Web services independently from the client's operating system and provide integrated best practices in standard specifications. Microsoft's upcoming Windows Communication Framework (WCF) promises to replace WSE in the near future and provide even more capabilities, while preserving interoperability with WSE 3.0.
One of WSE's key features—apart from its security benefits—is the Message Transmission Optimization Mechanism (MTOM), which helps you deliver binary files to the client and back to the server.
These are nice features, but many developers become frustrated or intimidated when they start a WSE project for the first time, finding it difficult to look past a lot of scary XML tags with unknown configuration options (see the sidebar, "HOLs Provide Strong Start"). I'll show you how to take get past these potentially intimidating initial steps by explaining how to leverage and improve on a Web service by relying on the infrastructure provided by WSE 3.0. This improved Web service will be more secure, more scalable, and better optimized, while also conforming to existing standards for security and interoperability.
Combining XML with Web services makes using them far easier and more intuitive. You can read wire message content in plain text, store that content in a file, and even index that content using SQL Server 2005's new XML column type. This looks wonderful until you have to do a seemingly trivial task: download a binary file.
You have two choices at this point. You can either encode the binary file inside the XML message, or you can stream it as a regular file just as you always have. Both approaches have their pros and cons.
The first option is relatively trivial and much easier to implement. You can put the binary file on an IIS Web server and let the IIS handle all the problems such as scalability, chunking, and security. The drawback to this approach is that you cannot integrate it with WS-* specifications. For many circumstances, especially circumstances where performance and scalability are critical, this can be an acceptable trade-off.
Your other option is to encode the file in the XML message. Doing so requires that you adhere to one simple rule: You cannot use special characters in the XML message. When you try to place a byte array in an XML file, the .NET Framework automatically base64 encodes the array, so no special characters remain. Base64 encodes the binary stream with safe-to-use characters only. You can find more information on how the base64 algorithm works at http://en.wikipedia.org/wiki/Base64.
Process the File
The file becomes part of the XML message, which means you can process it just as you would any other XML. For example, you can sign and encrypt it using a predefined policy object (turnkey scenario) in WSE 3.0. The major disadvantage of using this method is that it increases the size of the XML message significantly. A file incorporated in this manner grows by about 33 percent when converted to XML using the base64 transformation. This increase doesn't present much of an issue with small files, but it will matter when you attempt to transfer a 10 MB file. That 3 MB difference can have a significant impact, in many circumstances.
You must consider one other issue, regardless of the increase in size: sending the file in chunks. Assume you want to build a Web service that you use for file sharing. For each download, you must load the file, process it, and send it to the client. If 100 users of your Web service open a 10 MB file simultaneously, this leads to 1GB memory usage on your server, which isn't scalable by any stretch of the imagination. You should keep in mind that building a solidly performing Web service requires that you stream in chunks.
Various standards have tried to incorporate the requirements discussed so far during the evolution of WSE. The first one was SOAP with Attachments (SwA). It sends the XML as a multipart message or MIME (http://en.wikipedia.org/wiki/MIME). The problem with its implementation is that the MIME parts are not described in the SOAP message, so they cannot be composed in a manner that conforms to WS-* specifications.
DIME served as the standard for sending binary data until WSE 3.0 debuted. DIME featured strong performance, not least because it was designed to provide simple binary format. Like SwA, DIME doesn't integrate well with WS-* specifications.
This plethora of standards called out for a new, unified standard to tie them together. Enter MTOM. MTOM is basically an extended SwA specification. It still uses MIME-like parts, but now they are referenced in the SOAP message and perform as though they are part of the entire message. This enables different layers in WSE pipeline such as encryption, policy, encoding, and so on to process the data transparently. MTOM also provides a small performance optimization. If the binary data is too small and the impact of base64 encoding isn't too large, MTOM encodes the data using base64 and inserts the data in the SOAP message. At this time, binary data smaller than 768 bytes is base64 encoded and inserted into the SOAP message (see Additional Resources at the end of this article). Otherwise, MTOM uses MIME parts to send the data. It's important to note that MTOM doesn't require any special implementation in your program to support using it.
All you need to do to use MTOM is to configure it. MTOM also allows you to send chunks of data to the client transparently. By default, all the data is loaded into an XmlDocument object, which results in a big memory footprint. If you want to implement true streaming and allow the sending of packets chunk by chunk, you need to implement IXmlSerializable as shown in the sample project BinaryDataMTOM, rather than loading them into memory first.
Convert an Existing Service
That's about it for the background. Now let's take a look how to convert an existing ASMX Web service built with VS.NET into a next-generation Web service using WSE 3.0.
The sample project for this article includes two solutions: RemoteDiskWSE30Before and RemoteDiskWSE30After. The former contains the original ASMX Web service; the latter contains the project after you convert it to WSE 3.0.
The sample app is a simple file-sharing service. The StoragePath key in the Web.config file specifies which folder you want to share. By default, you find this folder at c:\WSEDemoStorage. The original Web service includes methods for retrieving folders and files using the NTFS hierarchy (see Listing 1). The sample doesn't implement any uploading capabilities. Remember that files in the root directory are not enumerated, only folders and their contents.
Make sure that you create virtual directory RemoteDiskServiceBefore that maps to the corresponding folder in the solution directory before you open the RemoteDiskWSE30Before solution. Also, ensure that the ASPNET account (NETWORK SERVICE in Win2003) includes read access to the virtual directory folder, as well as to the folder in StoragePath key.
Begin by opening the solution and running the RemoteDiskClient project. You should be able to browse the contents of the shared folder if everything is set properly. Next, look in the project's subfolders, select a file, and download it somewhere on your disk.
You might wonder why you need to worry about the virtual directory. After all, you have an ASP.NET development server ready to use from within VS.NET 2005 without setting any virtual directories. In fact, you can use it instead of IIS, but I have found that the development Web server needs to be polished, and I've also encountered some debugging and tracing problems. I have blogged about some problems I have encountered with ASP.NET development server herel.
For the sake of comparison, look at the XML message that is transferred on the wire when downloading small file before you enable WSE with this service. If you have not used TcpTrace utility (http://www.pocketsoap.com/tcptrace/) now is the time to do so. Start TcpTrace and accept the default settings in the dialog box (see Figure 1). Note that you should not close your TcpTrace application until you finish all the examples, or it won't be possible to see tracing results. At this point, every single packet that is sent to port 8080 of your machine will be logged and redirected to port 80 of localhost. This is a simple and efficient way to trace the contents of not only soap messages, but also any HTTP requests you would like to know about.
Once you enable logging, you must redirect the Web service requests to go to port 8080 instead of 80. Open app.config file from RemoteDiskClient project and you can find the URL of the Web service. The client application uses this setting to locate the Web service. Modify the URL to look like this: http://localhost:8080/RemoteDiskServiceBefore/StorageService.asmx.
Use the client application to download a small file. The TcpTrace log shows you what the request and response SOAP messages look like (see Listing 2). It is vital to point out that the file gets base64 encoded and embedded in a SOAP envelope. This is how you serialize the byte[] result of the GetFileData method in ASMX Web services.
Enable WSE 3.0
The next step is to enable WSE 3.0. Two endpoints must speak the same language before you can create successful communication between them. In other words, you need to enable WSE for both the server and client, which allows them to understand each other.
Begin by right-clicking the Web service project, RemoteDiskServiceBefore, in the Solution Explorer. Next, select WSE Settings 3.0 from the context menu. The dialog window that opens is the main management tool for WSE features. In the General tab, check both "Enable this project for Web Services Enhancements" and "Enable Microsoft Web Services Enhancement Soap Protocol Factory." You must check the second option when you enable WSE for an ASPX Web service. Finally, close the dialog window; WSE is enabled for the Web service at this point.
The next step is to enable WSE on the client. From the Solution Explorer, select the project, RemoteDiskClient, then open the WSE dialog window for this project. Select only the "Enable this project for Web Services Enhancements" option. The second option is disabled in this project because it relates only to Web projects, and the client project is a Windows Forms application.
You can also use the standalone tool, "Configuration Tool," from Start\Microsoft WSE 3.0 menu. This standalone tool lets you create a .config file without an existing project, enabling you to use the .config file as a template for this article's example.
The next step is to enable MTOM. Open the WSE Settings dialog for the Web service project again. Go to the Messaging tab, and make sure that in MTOM Settings group Server mode is set to optional. These settings control how the server should respond when it receives a SOAP envelope. The setting "optional" means that the server supports SOAP packets that can be MTOM encoded, but it's not mandatory that the packets get MTOM encoded. If you set the Server Mode to "always," any packet that isn't MTOM encoded causes the server to return HTTP error 415: "Media unsupported." Leave Server Mode "optional" because you want to receive all types of SOAP packets.
Now go to the Messaging tab for the client project. You can set whether packets sent to the server are going to be MTOM encoded from the Client Mode drop-down option. Setting the option to "Off" means that no packet will be MTOM encoded unless you set the RequireMtom property of the proxy class to True.
For this example, leave the Client Mode option on "Off." You will set MTOM mode explicitly in the code responsible for downloading the file using the RequireMtom property.
Update the Web Reference
That's enough preparation. Next, you need to call the WSE-enabled service. Note that a proxy class exists on the client. This class is created when you add your Web reference, and it gives typed access to the Web service. You can access this proxy file by opening the Solution Explorer, clicking on the Show All Files button in the toolbar, and expanding the Web reference under Web References. Find the file, Reference.cs, and take a look at the generated proxy class inside.
This is a key moment in implementing your solution. Enabling WSE on your service added new functionality to your service. You must update the Web reference in the client project, so the proxy class can take advantage of the new changes delivered by the programming model of this service. The net result of updating the reference is that the file containing the proxy class is updated to include a new proxy class that has the same name as the previous proxy class. The only difference is that it has a new suffix, "Wse."
For example, the article's sample code includes a proxy class named StorageService. Updating the reference renames the proxy class to StorageServiceWse. You need to replace all existing references to StorageService in your code with StorageServiceWse. With this, your Web service is WSE enabled.
Trace the WSE-enabled service
Everything should work as intended if you run the client project at this point.
One issue remains. You need to open app.config file again and change the location of the Web service to use port 8080 again. When you updated the Web reference in the previous step, VS.NET restored the changes in the URL. So, change the URL to access the Web service through port 8080 as described earlier, and restart the project. You'll get an exception on the first request to the Web service: "The <wsa:To> header must match the actor URI value of the Web service. You can specify the actor URI value explicitly can be explicitly specified using SoapActorAttribute on the ASMX class."
One aspect of WSE's functionality is to ensure that the SOAP message is delivered to the right person reliably, so it's no longer possible for TcpTrace to redirect the communication between the client and the service implicitly. You must set the intermediate node explicitly:
#if DEBUG
this.wsStorage.Destination.Via = new
Microsoft.Web.Services3.Referral.Via(
new Uri("http://localhost:8080/" +
"RemoteDiskServiceBefore/
StorageService.asmx"));
#endif
You need to restore the URL in app.config to its original value because you've already specified the proxy as an intermediate endpoint. Run the project now. Perform the same steps you did the first time you ran the client, and you will get the WSE SOAP request and response packets in TcpTrace log (see Listing 3). You will probably note that except for the tremendous amount of scary XML attributes, there is nothing new in the SOAP response. The file is still base64 encoded and embedded in the SOAP packet.
Please try to discard the XML attributes that you do not know. Most of them implement WS-Security and WS-Addressing specifications that are outside the scope of this article. Instead, focus your attention to the soap:body element.
The next task is to set the MTOM mode explicitly in your code. Simply set the RequireMtom property of the proxy class before you call GetFileData method and set it back to False after the method returns (see Listing 4). Also, remember that the optimization mechanism checks the message length and processes small messages as base64 encoded data (remember the 768 bytes threshold). Next, try downloading a slightly larger file to see what kind of trace information you get.
This example uses a plain-text file for downloading. Tracing it shows the downloading file is outside the XML packet. This separation of the binary and XML data leads to higher performance, thus still preserving the integrity of WS-* standards.
This solution is only a start, but it provides a good foundation for future projects. For example, you might add features such as uploading files and securing the service. Be sure to look at the BinaryDataMTOM sample and implement real streaming using IXmlSerializable interface. If you intend to implement file uploading, remember that IIS and WSE have separate settings for controlling the maximum size of the uploaded message.