An Extensive Examination of Web Services: Part 10By Scott Mitchell
As discussed in previous installments of this article series, Web services utilize a number of core standards, such as XML, SOAP, and WSDL. The XML and SOAP standards spell out the structure with which the messages sent between a client and a Web service are formatted; WSDL indicates how a Web service can formally define its functionality in a machine-readable syntax. While these core standards are sufficient for implementing Web services, they don't indicate how to accomplish common business tasks, such as authenticating a requestor, encrypting a message, and so forth.
To provide a standardized means for accomplishing everyday Web service scenarios, a number of industry leaders - Microsoft, IBM, BEA, and others - joined forces to craft a number of emerging Web service standards. These standards, called the Web Service Enhancements (WSE, for short), were the topic of discussion in Part 8 of this article series. WSE includes a vast array of emerging standards to accomplish a variety of common tasks, including standards related to security, process flow, privacy, and messaging, among others. Many of these standards are fairly new, and are still being discussed, while others are more mature and have been officially adopted as standards by open standards group, such as OASIS. In Part 8 we also discussed how Microsoft has released a free toolkit - the WSE 2.0 Toolkit - for using the more mature WSE standards. Armed with this toolkit you can easily utilize the supported WSE standards when creating .NET Web services and .NET Web service clients.
In this article we'll examine how to use the WSE Toolkit to efficiently send binary attachments through a Web service call, both from the client to the Web service, and from the Web service back to the client. As we'll discuss in this article, Microsoft's WSE Toolkit allows large attachments to be sent along with a Web service method using the DIME and WS-Attachments standards. We'll examine these standards and why they are more efficient than sending large amounts of binary data in a Web service call through other common means.
Techniques for Sending Large Amounts of Data in a Web Service Call
There may be times where you find yourself creating a Web service that needs to be sent some (potentially) large amount of data from the client consuming the service. For example, a job hunting Web site might offer a Web service for recruiters that provided a means for them to post new jobs. (A Web service interface might be preferred over a Web site interface for recruiters since they may have to post numerous job offers. Through a Web site, a recruiter would have to enter the information by hand; with a Web service, the task could be automated.) In addition to passing details about the job posting, a recruiter might also want to send some large amount of data, such as a picture of the facilities where the job is being hosted, or a PDF that includes a formal description of the job.
There are three general techniques for passing potentially large amounts of data into a Web service:
- As an input parameter to the Web service method - to allow a recruiter to send a PDF containing a formal
description of the job being posted, the Web service method
PostJob()could be augmented to include a
bytearray input parameter, whose contents would be the binary content of the PDF file.
- Using SOAP with Attachments - SOAP with Attachments provides a standardized way to send a MIME-encoded message that includes the SOAP envelope in one MIME part, and an arbitrary number of attachments in the other MIME parts. With this technique, the PDF file could be sent as a single MIME part in a two-part MIME message.
- Using DIME and WS-Attachments - DIME - short for Direct Internet Message Encapsulation - is a standard for sending a message in explicitly-sized chunks. WS-Attachments is a standard that defines how to use DIME to send attachments in a Web service call. For our recruiter example, one chunk could contain the SOAP envelope while another chunk could contain the PDF attachment.
|MTOM - the Next Step in the Evolution of Sending Attachments|
|MTOM - short for Message Transmission Optimization Mechanism - is the next step in the evolution of sending attachments in a Web service. MTOM is very new - a W3C Candidate Recommendation for MTOM was released just a couple of weeks ago (August 26, 2004). While MTOM is not supported in the WSE 2.0 Toolkit, it will be used heavily in Indigo, Microsoft's upcoming messaging system. To learn more about MTOM check out the MTOM specification.|
Why DIME and WS-Attachments is the Preferred Technique (Today)
To understand why DIME and WS-Attachments are the preferred technique for passing large attachments in a Web service setting, let's first look at the shortcomings of sending large data as a method parameter and SOAP with Attachments. Of the two techniques, sending a large amount of data as a Web service method parameter is the least efficient. All of the input parameter values passed into a Web service method from a client, are serialized into XML and sent in the SOAP Body. If you are sending large, binary data through a method's input parameter, you'll likely pass in a byte array. That is, you'd define your method in the Web service like:
While this approach will definitely work, it is horribly efficient when passing large amounts of data. Why? Well, the client must encode the binary data into text using base64 encoding, and then send the contents inside the SOAP Body. That is, the SOAP envelope being sent from the client to the recipient might look something like:
Of course the contents in the
<PDF> element may be hundreds, if not thousands of kilobytes. The reason
this is inefficient is because crafting and parsing XML documents is not trivial, and requires substantial memory and
processor resources. By making the XML document several hundreds of kilobytes or megabytes in size is an easy way to quickly
bring your Web service to its knees.
To improve the situation, we want to move the large data outside of the SOAP Envelope, so it does not hinder the creation and parsing of the XML. By moving this data out of band, it is called an "attachment." That is, ideally we would want to send the message in a format like:
Of course the challenge that now faces us is:
- How to we add the PDF contents to the end of the message so that it can quickly be found. (This process is complicated further since we need to be able to send, perhaps, multiple attachments, and not just one.)
- How do we tie the
<PDF>element in the SOAP Body to the appropriate attachment?
MIME - short for Multipurpose Internet Mail Extensions - is a standardized means for sending
attachments in email. MIME works by dividing the email content into distinct boundaries, separating each boundary with a unique
boundary string. With email, the first MIME boundary contains the email text, while the remaining boundaries contain any
related attachments. The following shows a sample email, with the MIME boundaries made bold:
Date: Sun, 07 Dec 2003 10:45:53 -0800 From: "John Smith"
Working with DIME and WS-Attachments
DIME and WS-Attachments are the preferred technique for sending attachments. Unlike SOAP with Attachments, which uses MIME boundaries to separate the attachments from one another, DIME uses explicitly-sized chunks. Each chunk has a record number and a size associated with it, so the receiver of a DIME-formatted message can quickly move from one attachment to another. With SOAP with Attachments, moving from one attachment to another requires examining the first attachment in its entirety, looking for the MIME boundary; with DIME, the receiver only needs to check the size of the current chunk and then skip ahead however many bytes specified to get to the beginning of the next chunk.
Using DIME to send an attachment in a Web service call is easy, thanks to Microsoft's WSE 2.0 Toolkit. For this article, let's create a Web service that accepts a binary attachment and simply saves it to the Web server's file system. To start, create a Web service application, and configure it to use the WSE 2.0 Toolkit. (Recall that this requires right-clicking on the Project Name, choosing the WSE 2.0 Settings option - see the screenshot to the right - and checking both checkboxes in the General tab.)
Next, in the Web service class create a method called
UploadPDF(). This method will accept a DIME attachment
and save it to the Web server's file system. The code for this method is fairly simple, and shown below. Note that the
method does not take in the attachment as an input parameter of any kind - it simply references the attachment through the
Attachments collection within the code. (For this code be sure to add a
statement at the top of the class file for the namespaces
Microsoft.Web.Services2.Dime, as well as for
System.IO (since we'll be saving the attachment
to the Web server's file system)).
As this example shows, the Web service has access to the set of attachments passed in using DIME via the
RequestSoapContext.Current.Attachments collection. This collection contains a set of
instances, each of which contain a
Stream property. In
UploadPDF(), the attachment size is first
checked to ensure that precisely one attachment was passed in. Next, a
FileStream object is used to save
the contents of the uploaded PDF to the Web service's directory on the file system.
All that remains is to create the client. For this demo I used an ASP.NET Application as the client, although a WinForms application
would work just as well. The client presents the user with a file upload box, allowing them to choose a PDF. Upon submitting
the Web Form, the ASP.NET page creates a
DimeAttachment with the contents of the uploaded file, adds it to
Attachments collection -
RequestSoapContext.Current.Attachments - and invokes the
UploadPDF() method. The complete code, along with a screenshot, is shown below. (For more
information on uploading files with ASP.NET be sure to read Uploading
in ASP.NET, by Tribikram Rath.)
In the code we start by creating an instance of the WSE-enabled proxy class, and then add a
instance to its
Attachments collection. The
DimeAttachment constructor takes in, among other
inputs, a stream to the data to upload. Since we are uploading a file using the file upload control provided by ASP.NET,
we reference this uploaded file's stream using the
DimeAttachment has been added, we simply invoke the
Screenshot of Client in Action...
Closing Comments on DIME with the WSE 2.0 Toolkit
In this article we saw a demo where a client sends an attachment to a Web service, and the Web service saves the attachment on its file system. There are scenarios where you may want to reverse this direction. That is, the client might request a particular file, and the Web service might return it as an attachment using DIME. If you want to send an attachment from the Web service to the requesting client, simply create a
DimeAttachmentin the Web service (as we did in the client example above), and add it to the
ResponseSoapContext.Current.Attachmentscollection (as opposed to
RequestSoapContext.Current.Attachments). Similarly, to receive the attachment from the client, after the call to the Web service method that returns an attachment, the attachment will be available in the
Finally, when sending attachments with DIME it is vitally important that you do not close the stream of the attachment
being sent in the
DimeAttachment. That is, don't do something like:
While it may seem logical to "clean up" after yourself, closing the underlying stream will result in an error when attempting to send the attachment in the Web service request or response. This is because the WSE Toolkit needs to have access to the data that is being attached to the SOAP request or response, in order to craft the appropriate message format. If you close the underlying stream, the WSE Toolkit won't be able to properly access the data it needs. Rest assured, the WSE Toolkit will close the stream for you - so don't do it yourself, or you may end up with a cryptic exception.
In this article we looked at various options for sending large amounts of data in a Web service call. Of the three current options, the best one is DIME with WS-Attachments. Sending attachments with DIME moves the contents of the data outside of the SOAP Envelope, and formats the attachments in a means that can be quickly accessed by the recipient of the message. Microsoft's WSE 2.0 Toolkit provides support for sending and receiving DIME attachments with Web services and Web service clients.