An Extensive Examination of Web Services: Part 10
By Scott Mitchell
An Extensive Examination of Web Services Index
An Extensive Examination of Web Services is a multi-part article series spanning
several months. Listed below are the current "parts," along with a very brief synopsis and the
date published.
Part 1 - Examines the basics of Web services, what Web services are,
and the technologies and standards that serve as the underpinnings of Web services. (October 8th, 2003)
Part 2 - Examines creating Web services using Visual Studio
.NET. Looks underneath the hood of the code created by VS.NET. (October 15th, 2003)
Part 3 - Examines creating a client application that consumes
a Web service. Discusses the purpose and structure of a WSDL document, along with creating and using proxy
classes to consume a Web service. (November 5th, 2003)
Part 4 - Examines the utility of Web services and common
scenarios where Web services make sense. A business-oriented look at Web services. (November 19th, 2003)
Part 5 - Takes an in-depth look at XML serialization, which is the
process of converting a data type, such as an integer, array, or custom class, into its XML representation, and back again.
Every time a message is passed to or from a Web service, XML serialization transpires. (December 17th, 2003)
Part 6 - Looks at sending metadata to a Web method through the use of SOAP headers.
Examines defining and accepting a SOAP header on the Web service end, and looks at sending a populated SOAP header
from the client. (December 31st, 2003)
Part 7 - Examines how the incoming and outgoing messages to a Web
service can be programmatically modified via SOAP Extensions. (January 21st, 2004)
Part 8 - Learn about the Web Service Enhancements (WSE) and Microsoft's
free class library for implementing the WSE standards. (June 30th, 2004)
Part 9 - See how to implement UsernameToken authentication using
the WSE 2.0 Toolkit. (July 14, 2004)
Part 10 - Learn how to send large amounts of data as attachments using
DIME and WS-Attachments. (September 8th, 2004)
Introduction
As discussed in previous installments of this article series, Web services utilize a number of core standards, such as
XML, SOAP, and WSDL. The XML and SOAP standards spell out the structure with which the messages sent between
a client and a Web service are formatted; WSDL indicates how a Web service can formally define its functionality in a
machine-readable syntax. While these core standards are sufficient for implementing Web services, they don't indicate how to
accomplish common business tasks, such as authenticating a requestor, encrypting a message, and so forth.
To provide a standardized means for accomplishing everyday Web service scenarios, a number of industry leaders - Microsoft, IBM,
BEA, and others - joined forces to craft a number of emerging Web service standards. These standards, called the
Web Service Enhancements (WSE, for short), were the topic of discussion in Part 8 of
this article series. WSE includes a vast array of emerging standards to accomplish a variety of common tasks, including standards
related to security, process flow, privacy, and messaging, among others. Many of these standards are fairly new, and are still
being discussed, while others are more mature and have been officially adopted as standards by open standards group, such as
OASIS.
In Part 8 we also discussed how Microsoft has released a free toolkit - the WSE 2.0 Toolkit - for using the more mature WSE
standards. Armed with this toolkit you can easily utilize the supported WSE standards when creating .NET Web services and
.NET Web service clients.
In this article we'll examine how to use the WSE Toolkit to efficiently send binary attachments
through a Web service call, both from the client to the Web service, and from the Web service back to the client. As we'll
discuss in this article, Microsoft's WSE Toolkit allows large attachments to be sent along with a Web service method using
the DIME and WS-Attachments standards. We'll examine these standards and why they are more efficient than sending large amounts
of binary data in a Web service call through other common means.
Techniques for Sending Large Amounts of Data in a Web Service Call
There may be times where you find yourself creating a Web service that needs to be sent some (potentially) large amount of
data from the client consuming the service. For example, a job hunting Web site might offer a Web service for recruiters
that provided a means for them to post new jobs. (A Web service interface might be preferred over a Web site interface for
recruiters since they may have to post numerous job offers. Through a Web site, a recruiter would have to enter the information
by hand; with a Web service, the task could be automated.) In addition to passing details about the job posting, a recruiter
might also want to send some large amount of data, such as a picture of the facilities where the job is being hosted, or
a PDF that includes a formal description of the job.
There are three general techniques for passing potentially large amounts of data into a Web service:
As an input parameter to the Web service method - to allow a recruiter to send a PDF containing a formal
description of the job being posted, the Web service method PostJob() could be augmented to include a
byte array input parameter, whose contents would be the binary content of the PDF file.
Using SOAP with Attachments - SOAP with Attachments provides a standardized way to send a MIME-encoded
message that includes the SOAP envelope in one MIME part, and an arbitrary number of attachments in the other MIME parts.
With this technique, the PDF file could be sent as a single MIME part in a two-part MIME message.
Using DIME and WS-Attachments - DIME - short for Direct Internet Message
Encapsulation - is a standard for sending a message in explicitly-sized chunks. WS-Attachments is a standard
that defines how to use DIME to send attachments in a Web service call. For our recruiter example, one
chunk could contain the SOAP envelope while another chunk could contain the PDF attachment.
The preferred technique today is to use DIME and WS-Attachments for reasons we will examine shortly. As we'll see, the
WSE 2.0 Toolkit makes it easy to send and receive DIME attachments - you just need to write a line or two of code, and
the WSE Toolkit will take care of the rest for you.
MTOM - the Next Step in the Evolution of Sending Attachments
MTOM - short for Message Transmission Optimization Mechanism - is the next step in the evolution
of sending attachments in a Web service. MTOM is very new - a W3C Candidate Recommendation for MTOM was released just a couple
of weeks ago (August 26, 2004). While MTOM is not supported in the WSE 2.0 Toolkit, it will be used heavily in Indigo, Microsoft's
upcoming messaging system. To learn more about MTOM check out the MTOM specification.
Why DIME and WS-Attachments is the Preferred Technique (Today)
To understand why DIME and WS-Attachments are the preferred technique for passing large attachments in a Web service setting,
let's first look at the shortcomings of sending large data as a method parameter and SOAP with Attachments. Of the two
techniques, sending a large amount of data as a Web service method parameter is the least efficient. All of the input
parameter values passed into a Web service method from a client, are serialized into XML and sent in the SOAP Body. If you
are sending large, binary data through a method's input parameter, you'll likely pass in a byte array. That is, you'd define
your method in the Web service like:
_
Public Sub PostJob(title as String, salary as Decimal, ..., PDF() as Byte)
...
End Sub
While this approach will definitely work, it is horribly efficient when passing large amounts of data. Why? Well, the client
must encode the binary data into text using base64 encoding, and then send the contents inside the SOAP Body. That is, the
SOAP envelope being sent from the client to the recipient might look something like:
Of course the contents in the <PDF> element may be hundreds, if not thousands of kilobytes. The reason
this is inefficient is because crafting and parsing XML documents is not trivial, and requires substantial memory and
processor resources. By making the XML document several hundreds of kilobytes or megabytes in size is an easy way to quickly
bring your Web service to its knees.
To improve the situation, we want to move the large data outside of the SOAP Envelope, so it does not hinder the
creation and parsing of the XML. By moving this data out of band, it is called an "attachment."
That is, ideally we would want to send the message in a format like:
<soap:Envelope ...>
<soap:Body>
<PostJob>
<title>Product Tester</title>
<salary>55000.00</salary>
...
<PDF>
PDF DOCUMENT AT END OF ENVELOPE
</PDF>
</PostJob>
</soap:Body>
</soap:Envelope>
QfvW1lJL4ye7abHPruebye7abHPrueiF14Il
W1lJL4oW1lJye7abHPrueL4...==
Of course the challenge that now faces us is:
How to we add the PDF contents to the end of the message so that it can quickly be found. (This process is complicated
further since we need to be able to send, perhaps, multiple attachments, and not just one.)
How do we tie the <PDF> element in the SOAP Body to the appropriate attachment?
There are different ways to solving these two problems. One such way, used by the SOAP with Attachments standard, is to
send the message using the MIME standard, and having the attachments appear in separate MIME boundaries. SOAP with Attachments
has its downsides, the primary one being that while MIME boundaries make it easy for the sender of the attachment to add
an arbitrary number of attachments, it takes considerably more work to pick out the precise attachments from the receiver's standpoint,
since each attachment must be scanned in its entirety, in order to find the MIME boundaries.
Understanding MIME
MIME - short for Multipurpose Internet Mail Extensions - is a standardized means for sending
attachments in email. MIME works by dividing the email content into distinct boundaries, separating each boundary with a unique
boundary string. With email, the first MIME boundary contains the email text, while the remaining boundaries contain any
related attachments. The following shows a sample email, with the MIME boundaries made bold:
Date: Sun, 07 Dec 2003 10:45:53 -0800
From: "John Smith"
Subject: URGENT - NIGERIA MONEY IS YOURS TO BE CLAIMED!!
To: Scott Mitchell
Content-type: multipart/related; boundary="Boundary_(ID_FXnn6OVMs/pgPvsHzyUcwA)"
This is a multi-part message in MIME format.
--Boundary_(ID_IZ1Lhy3/0i8IoHkif3PKpA)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7bit
I am Charles Smith, from the Nation of Nigeria, with URGENT news regarding a bank
account with the sum of 14 MILLION USD. Attached is a PDF file with instructions
on how to claim this vast fortune. ACT TODAY.
--Boundary_(ID_FXnn6OVMs/pgPvsHzyUcwA)
Content-id: <433473118@07122003-0700>
Content-type: x-application/pdf; name=scam.pdf
Content-transfer-encoding: base64
Content-disposition: attachment; filename=scam.pdf
R0lGODlhmQA3APcAAAAAAIAAAACAAICAAAAAgIAAgACAgMDAwMDcwKbK8EAgAGAgAIAgAKAgAMAg
AOAgAABAACBAAEBAAGBAAIBAAKBAAMBAAOBAAABgACBgAEBgAGBgAIBgAK...
Working with DIME and WS-Attachments
DIME and WS-Attachments are the preferred technique for sending attachments. Unlike SOAP with Attachments, which uses
MIME boundaries to separate the attachments from one another, DIME uses explicitly-sized chunks. Each chunk has a record number
and a size associated with it, so the receiver of a DIME-formatted message can quickly move from one attachment to another.
With SOAP with Attachments, moving from one attachment to another requires examining the first attachment in its entirety, looking
for the MIME boundary; with DIME, the receiver only needs to check the size of the current chunk and then skip ahead however
many bytes specified to get to the beginning of the next chunk.
Using DIME to send an attachment in a Web service call is easy, thanks to Microsoft's WSE 2.0 Toolkit. For this article,
let's create a Web service that accepts a binary attachment and simply saves it to the Web server's file system.
To start, create a Web service application, and configure it to use the WSE 2.0 Toolkit. (Recall that this requires right-clicking
on the Project Name, choosing the WSE 2.0 Settings option - see the screenshot to the right - and checking both checkboxes in the General tab.)
Next, in the Web service class create a method called UploadPDF(). This method will accept a DIME attachment
and save it to the Web server's file system. The code for this method is fairly simple, and shown below. Note that the
method does not take in the attachment as an input parameter of any kind - it simply references the attachment through the
Attachments collection within the code. (For this code be sure to add a using or Imports
statement at the top of the class file for the namespaces Microsoft.Web.Services2 and
Microsoft.Web.Services2.Dime, as well as for System.IO (since we'll be saving the attachment
to the Web server's file system)).
_
Public Sub UploadPDF()
'Make sure precisely one attachment was sent
If RequestSoapContext.Current.Attachments.Count <> 1 Then
Throw New ArgumentException("Please upload one file at a time...")
End If
'Save the file to LatestPDF.pdf
Dim fs As FileStream = File.Create(HttpContext.Current.Server.MapPath("LatestPDF.pdf"))
Dim pdfLength As Integer = RequestSoapContext.Current.Attachments(0).Stream.Length
Dim buffer(pdfLength) As Byte
RequestSoapContext.Current.Attachments(0).Stream.Read(buffer, 0, pdfLength)
fs.Write(buffer, 0, pdfLength)
fs.Close()
End Sub
As this example shows, the Web service has access to the set of attachments passed in using DIME via the
RequestSoapContext.Current.Attachments collection. This collection contains a set of DimeAttachment
instances, each of which contain a Stream property. In UploadPDF(), the attachment size is first
checked to ensure that precisely one attachment was passed in. Next, a FileStream object is used to save
the contents of the uploaded PDF to the Web service's directory on the file system.
All that remains is to create the client. For this demo I used an ASP.NET Application as the client, although a WinForms application
would work just as well. The client presents the user with a file upload box, allowing them to choose a PDF. Upon submitting
the Web Form, the ASP.NET page creates a DimeAttachment with the contents of the uploaded file, adds it to
the outgoing Attachments collection - RequestSoapContext.Current.Attachments - and invokes the
Web service's UploadPDF() method. The complete code, along with a screenshot, is shown below. (For more
information on uploading files with ASP.NET be sure to read Uploading
in ASP.NET, by Tribikram Rath.)
Private Sub btnUpload_Click(ByVal sender As System.Object, _
ByVal e As System.EventArgs) Handles btnUpload.Click
'Call the Web service
Dim proxy As New localhost.DIMEDemoWse
'Add the attachment
Dim attachment As _
New DimeAttachment("PDF", TypeFormat.MediaType, _
Me.fileUpload.PostedFile.InputStream)
proxy.RequestSoapContext.Attachments.Add(attachment)
'Call the Web service's UploadPDF() method
proxy.UploadPDF()
...
End Sub
In the code we start by creating an instance of the WSE-enabled proxy class, and then add a DimeAttachment
instance to its Attachments collection. The DimeAttachment constructor takes in, among other
inputs, a stream to the data to upload. Since we are uploading a file using the file upload control provided by ASP.NET,
we reference this uploaded file's stream using the PostedFile class's InputStream property.
Once the DimeAttachment has been added, we simply invoke the UploadPDF() method.
Screenshot of Client in Action...
Closing Comments on DIME with the WSE 2.0 Toolkit
In this article we saw a demo where a client sends an attachment to a Web service, and the Web service saves the attachment
on its file system. There are scenarios where you may want to reverse this direction. That is, the client might
request a particular file, and the Web service might return it as an attachment using DIME. If you want to send an attachment
from the Web service to the requesting client, simply create a DimeAttachment in the Web service (as we did in
the client example above), and add it to the ResponseSoapContext.Current.Attachments collection
(as opposed to RequestSoapContext.Current.Attachments). Similarly, to receive the attachment
from the client, after the call to the Web service method that returns an attachment, the attachment will be available in the
proxyClassInstance.ResponseSoapContext.Attachments collection.
Finally, when sending attachments with DIME it is vitally important that you do not close the stream of the attachment
being sent in the DimeAttachment. That is, don't do something like:
Dim attachment As _
New DimeAttachment("PDF", TypeFormat.MediaType, _
Me.fileUpload.PostedFile.InputStream)
proxy.RequestSoapContext.Attachments.Add(attachment)
'DO NOT DO THIS!
fileUpload.PostedFile.InputStream.Close()
proxy.UploadPDF()
While it may seem logical to "clean up" after yourself, closing the underlying stream will result in an error when attempting
to send the attachment in the Web service request or response. This is because the WSE Toolkit needs to have access to
the data that is being attached to the SOAP request or response, in order to craft the appropriate message format. If you
close the underlying stream, the WSE Toolkit won't be able to properly access the data it needs. Rest assured, the WSE Toolkit
will close the stream for you - so don't do it yourself, or you may end up with a cryptic exception.
Conclusion
In this article we looked at various options for sending large amounts of data in a Web service call. Of the three current
options, the best one is DIME with WS-Attachments. Sending attachments with DIME moves the contents of the data outside
of the SOAP Envelope, and formats the attachments in a means that can be quickly accessed by the recipient of the message.
Microsoft's WSE 2.0 Toolkit provides support for sending and receiving DIME attachments with Web services and Web service
clients.
Beginner's .NET XML Web Services offers nearly eight hours of training
on .NET XML Web services in a video format. This two-disc DVD set, presented by
author Scott
Mitchell, offers a unique opportunity for learning about the fundamentals of Web services.
The 14 lessons begin with an examination of the core Web service standards, and then quickly move
into showing you how to create and consume Web services in Microsoft's .NET Framework. There are in-depth
lessons for each of the core Web service standards - XML, SOAP, and WSDL - along with examples of the
Web Service Enhancements (WSE). Scattered throughout each of
these chapters are extensive demos, depicting how to build, deploy, and access Web services using
Microsoft's Visual Studio .NET. The DVD also contains a thorough examination of a real-world,
end-to-end Web service application.
Scott Mitchell is the editor and founder of 4GuysFromRolla.com, author of the An Extensive Examination
of the Web Services article series, and author of numerous ASP and ASP.NET
books.
DIME and WS-Attachments are the preferred technique for sending attachments. Unlike SOAP with Attachments, which uses
MIME boundaries to separate the attachments from one another, DIME uses explicitly-sized chunks. Each chunk has a record number
and a size associated with it, so the receiver of a DIME-formatted message can quickly move from one attachment to another.
With SOAP with Attachments, moving from one attachment to another requires examining the first attachment in its entirety, looking
for the MIME boundary; with DIME, the receiver only needs to check the size of the current chunk and then skip ahead however
many bytes specified to get to the beginning of the next chunk.
