MSMQ, WCF and IIS: Getting them to play nice - Part 1

A few weeks ago I posted an article describing how my current team built a publish/subscribe message bus using WCF and MSMQ. At that time we had only deployed the application in a single-server test environment. While there were a few tricks to getting that working, once we tried deploying to a multiple server environment we found a whole lot of new challenges. In the end they were all quite solvable, but it seems that not a lot of people have attempted to use the MSMQ bindings for WCF, hosted in IIS 7 WAS, so there isn't a lot of help out on the web. The best source of information I'd found is an MSDN article, Troubleshooting Queued Messaging (which unfortunately I didn't find until after we'd already solved most of our problems). But even that article is a bit lacking, so I thought I'd share some of the things we learned about getting this all working.

The Scenario

The goal here is to set up reliable, asynchronous communication between a client application and a service, which may be on different machines. We will be using MSMQ as a transport mechanism, as it supports reliable queued communication. MSMQ will be deployed on a third server (typically clustered to eliminate a single point of failure). The client application will use WCF's NetMsmqBinding to send messages to a private queue on the MSMQ server. The service will be hosted in IIS 7, and will use Windows Activation Services (WAS) to listen for new messages on the message queue. This listening is done by a Windows Service called SMSvcHost.exe. When a message arrives, it activates the service within an IIS worker process, and the service will process the message. The overall architecture is shown in the following diagram.

[img_assist|nid=4105|title=|desc=|link=none|align=none|width=615|height=310]

The Basics

Let's start simple by setting everything up on a single server, with no security or transactions to complicate things. This first instalment is a bit of a recap of my earlier post, but I'm including it again here as it will be an important foundation for the more complex steps shown in the next instalments.

Install the necessary Windows components

Before writing any code, make sure you're running Windows Vista or Windows Server 2008, and that you've installed the following components (I've taken the names from Vista's "Windows Features" dialog; Windows Server 2008 has slightly different options but all should be there somewhere).

1. Microsoft Message Queue (MSMQ) Server > MSMQ Server Core and MSMQ Active Directory Domain Services Integration (needed for Transport Security in Part 2)
2. Microsoft .NET Framework 3.0 > Windows Communication Foundation Non-HTTP Activation
3. Internet Information Services > World Wide Web Services
4. Windows Process Activation Service
5. Distributed Transaction Controller (DTC) - Always installed with Windows Vista, may need to be added for Windows Server 2008

Of course, you'll also want Visual Studio 2005 or 2008 installed so you can write the necessary code.

Define the contract

As with all WCF applications, a great starting point is to define the service contract. The only real trick when building MSMQ services is to ensure that every operation contract is defined with IsOneWay=true. In my example we'll have just one very simple operation, but you could easily add more or use more complicated data contracts.

[ServiceContract]
public interface IMsmqContract
{
    [OperationContract(IsOneWay = true)]
    void SendMessage(string message);
}

I won't bother with showing any sample client code to call the service, as this is no different from any other WCF client.

Create the Message Queue

Message Queues don't just create themselves, so if you want to build a MSMQ-based application, you'll need to create yourself some queues. The easiest way to do this is from the Computer Management console in Windows Vista, or Server Manager in Windows Server 2008.

In general, message queues can be called whatever you want. However when you are hosting your MSMQ-enabled service in IIS 7 WAS, the queue name must match the URI of your service's .svc file. In this example we'll be hosting the service in an application called MsmqService with an .svc file called MsmqService.svc, so the queue must be called MsmqService/MsmqService.svc. Queues used for WCF services should always be private. While the term "private queue" could imply that the queue cannot be accessed from external machines, this isn't actually true - the only thing that makes a public queue public is that it is published in Active Directory. Since all of our queue paths will be coded into WCF config files, there really isn't any value in publishing the queues to AD.

In this first stage, we won't be using a transactional queue, so make sure you don't click the Transactional checkbox. Transactional queues can add some complexity, but they also provide significantly more reliability so we'll be moving to transactional queues later in the article.

At this time, it's a good idea to configure the security for the queue. You want to make sure that the account running the client is allowed to send messages to the queue, and the account running the service is able to receive messages from the queue. Since the service will be hosted in IIS, by default it will be using the NETWORK SERVICE account.

Configure the Client

Now we know the name of the message queue, we can configure the client to send messages to the correct place. First you need to configure a suitable binding. We'll be using the NetMsmqBinding, which is normally the best option when both the client and service are using WCF. For now we will not be using and security or transactions, so we'll need to specify that in the binding (the exactlyOnce="false" attribute means it's non-transactional).

The endpoint definition is defined in the same way as any WCF client endpoint. One thing to look out for is the address syntax for MSMQ services. Rather than using the format name syntax that you may have used in other MSMQ applications, WCF has a new (and simpler) syntax. The key differences are that all slashes go forwards, and you use "private" instead of "private$". So the address for our locally hosted queue will be net.msmq://localhost/private/MsmqService/MsmqService.svc. Here's the complete config file for the client:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.serviceModel>
    <bindings>
      <netMsmqBinding>
        <binding name="MsmqBindingNonTransactionalNoSecurity" exactlyOnce="false">
          <security mode="None"/>
        </binding>
      </netMsmqBinding>
    </bindings>
    <client>
      <endpoint name="MsmqService"
                address="net.msmq://localhost/private/MsmqService/MsmqService.svc"
                binding="netMsmqBinding" bindingConfiguration="MsmqBindingNonTransactionalNoSecurity"
                contract="MsmqContract.IMsmqContract" />
    </client>
  </system.serviceModel>
</configuration>

Configure the Service

To create the service, start by setting up a new ASP.NET application, hosted in IIS - just as you would for a normal HTTP-based WCF service. This includes creating a .svc file for the service endpoint, and of course a class that implements the service contract. Again, I won't bother showing this code as it's not specific to MSMQ.

You'll also need to modify the service's web.config file to include the configuration details for your WCF service. Not surprisingly, this will look very similar to what we configured on the client.

<system.serviceModel>
<bindings>
<netMsmqBinding>
<binding name="MsmqBindingNonTransactionalNoSecurity" exactlyOnce="false">
  <security mode="None"/>
</binding>
</netMsmqBinding>
</bindings>
<services>
<service name="MsmqService.MsmqService">
<endpoint address="net.msmq://localhost/private/MsmqService/MsmqService.svc"
	binding="netMsmqBinding" bindingConfiguration="MsmqBindingNonTransactionalNoSecurity"
	contract="MsmqContract.IMsmqContract" />
</service>
</services>
</system.serviceModel>

Enable the MSMQ WAS Listener

The last step is to configure IIS 7 to use WAS to listen to the message queue and activate your service when new messages arrive. There are two parts to this: first you need to activate the net.msmq listener for the entire web site, and second you need to enable the protocols needed for your specific application. You can perform both of these steps using either the appcmd.exe tool (located under C:\Windows\System32\Inetsrv) or by editing the C:\Windows\System32\Inetsrv\config\ApplicationHost.config file in a text editor. Let's go with the former, since it's a bit less dangerous.

To enable the net.msmq listener for the entire web site, use the following command. Note that the bindingInformation='localhost' bit is what tells the listener which machine is hosting the queues that it should listen to. This will be important when we want to start listening to remote queues.

appcmd set site "Default Web Site" -+bindings.[protocol='net.msmq',bindingInformation='localhost']

To enable the net.msmq protocol for our specific application, use the following command. Note that you can configure multiple protocols for a single application, should you want it to be activated in more than one way (for example, to allow either MSMQ or HTTP you could say /enabledProtocols:net.msmq,http).

appcmd set app "Default Web Site/MsmqService" /enabledProtocols:net.msmq

Troubleshooting Steps

If all has gone to plan, you should be able to successfully send messages from the client to the service, and have the service process them correctly. However if you're anything like me, this probably won't work first time. Troubleshooting MSMQ issues can be somewhat of an art form, but I've listed a few techniques that I've found to be helpful to resolve issues.

• Check queue permissions. Make sure that you've correctly set the ACLs on your message queue so that the user accounts running the client and service are able to send and receive respectively.
• Check the dead letter queues. In many circumstances, MSMQ will send messages to the Dead Letter Queue (or Transactional Dead Letter Queue) if it couldn't successfully be delivered for any reason. Often the details on the dead letter message will explain why it ended up there (for example, you tried sending a non-transactional message to a transactional queue). If you're using MSMQ across multiple machines, make sure you check the Dead Letter Queues on all servers, as it could end up in different places depending on what caused the delivery failure.
• Enable Journaling. Sometimes it can be hard to tell whether a message never arrived at all, or if it arrived and subsequently got "lost". If you enable the "journal" feature on a message queue, you'll see a record of every message that passed through. However use this feature sparingly, as you can very easily end up with a huge number of journal messages after a few hours of testing.
• Shut down the service listener. When troubleshooting, it can be useful to focus on just the client or just the service. For example, if you aren't sure if the client is sending messages properly, you may want to completely disable the service so you can see if the client's messages are arriving on the queue. To do this, you can shut down the IIS service or application pool, or shut down the Net.Msmq Listener Adapter Service.
• Make sure the MSMQ storage isn't maxed out. MSMQ is designed to be resistant against all sorts of failures, such as temporary network outages. However it seems that if you reach the limit of MSMQ's allocated storage, messages will not be delivered at all. This has happened to us a few times after large amounts of messages ended up in the Dead Letter Queue, or when journaling has been left on for too long. It's easy enough to increase the storage limit, but normally when you reach the limit during testing the best thing to do is purge all of your queues.
• Try pinging the service using the browser. When you are working on WCF services exposed through HTTP, you're probably used to hitting the .svc file in a web browser to check that you can receive the metadata correctly and that there are no configuration problems. Unfortunately there isn't any equivalent way to "browse" to an MSMQ service, so simple configuration errors can be very hard to track down. However if you enable the HTTP protocol for your site, you will be able to hit the .svc file in the browser, even if you haven't configured an HTTP endpoint for your service. If you get the standard WCF service page, that means the service is probably configured correctly.

That's it for the basics. In Part 2 of this article, we'll look at what's required to get this application deployed on multiple servers, and specifically focus on what you'll need to do for the security configuration.