Yesterday I showed a detailed example of a Asynchronous system using MOM. Please read part 1 and part 2 of this series first before continuing below.
The system shown yesterday was Asynchronous since there is no coupling, no conversation or time constraints. The Producer does not know or care what happens to the messages once Published or when that happens. This is a kind of nirvana for distributed systems but sadly it’s just not possible to solve every problem using this pattern.
Today I’ll show how to use MOM technologies to solve a different kind of problem. Specifically I will show how large retailers scale their web properties using these technologies to create web sites that is more resilient to failure, easier to scale and easier to manage.
Imagine you are just hitting buy on some online retailers web page, perhaps they have implemented a 1-click based buying system where the very next page would be a Thank You page showing some details of your order and also recommendations of what other things you might like. It would have some personalized menus and in some cases even a personalized look and feel.
By the time you see this page your purchase is not complete, it is still going on in the background but you have a fast acknowledge back and immediately you are being enticed to spend more money with relevant products.
To achieve this in a PHP or Rails world you would typically have a page that runs top down and do things like generate your CSS page, generate your personalized menu then write some record into a database perhaps for a system like delayed job to process the purchase later on and finally it would do a bunch of SQL queries to find the related items.
This approach is very difficult to scale, all the hard work happens in your front controller, it has to be able to communicate with all the technology you choose in the backend and you end up with a huge monolithic chunk of code that can rapidly become a nightmare. If you need more capacity to render the recommendations you have no choice but to scale up the entire stack.
The better approach is to decouple all of the bits needed to generate a web page, if you take the narrative above you would have small single purpose services that does the following:
- Take a 1-click order request, save it and provide an order number back. Start an Asynchronous process to fulfill the order.
- Generate CSS for the custom look and feel for user X
- Generate Menu for logged in user X
- Generate recommendation for product Y based on browsing history for user X
Here we have 4 possible services that could exist on the network and that do not really relate to each other in any way. They are decoupled, do not share state with each other and can do their work in parallel independently from each other.
Your front controller now would become something that simply Published to the MOM requests for each of the 4 services providing just the information each service needs and then wait for the responses. Once all 4 responses were received the page would be assembled and rendered. If some response does not arrive in time a graceful failure can be done – like render a generic menu, or do not show the recommendations only show the Thank You text.
There are many benefits to this approach, I’ll highlight some I find compelling below:
- You can scale each Service independently based on performance patterns – more order processors as this requires slower ACID writes into databases etc.
- You can use different technology where appropriate. Your payment systems might be .Net while your CSS generation is in Node.JS and recommendations are in Java
- Each system can be thought of as a simple standalone island with its own tech stack, monitoring etc, thinking about and scaling small components is much easier than a monolithic system
- You can separate your payment processing from the rest of your network for PCI compliance by only allowing the middleware to connect into a private subnet where all actual credit information lives
- Individual services can be upgraded or replaced with new ones much easier than in a monolithic system thus making the lives of Operations much better and lowering risk in ongoing maintenance of the system.
- Individual services can be reused – the recommendation engine isn’t just a engine that gets called at the end of a sale but also while browsing through the store, the same service can serve both types of request
This pattern is often known as Request Response or similar terms. You should only use it when absolutely needed as it increases coupling and effectively turn your service into a Synchronous system but it does have it’s uses and advantages as seen above.
Sample Code
I’ll show 2 quick samples of how this conversation flow works in code and expand a bit into the details wrt to the ActiveMQ JMS Broker. The examples will just have the main working part of the code not the bits that would set up connections to the brokers etc, look in part 2 for some of that.
My example will create a service that generates random data using OpenSSL, maybe you have some reason to create a very large number of these and you need to distribute it across many machines so you do not run out of entropy.
As this is basically a Client / Server relationship I will use these terms, first the client part – the part that requests a random number from the server:
stomp.subscribe("/temp-queue/random_replies") stomp.publish("/queue/random_generator", "random", {"reply-to" => "/temp-queue/random_replies"}) Timeout::timeout(2) do msg = stomp.receive puts "Got random number: #{msg.body}" end |
This is pretty simple, the only new thing here is that we are subscribing first to a Temporary Queue that we will receive the responses on and we send the request including this queue name. Below will have some more detail on temp queues and temp topics. The timeout part is important you need this to be able to handle the case where all of the number generators died or if the service is just too overloaded to service the request.
Here is the server part, it gets a request then generates the number and replies to the reply-to destination.
require 'openssl' stomp.subscribe("/queue/random_generator") loop do begin msg = client.receive number = OpenSSL::Random.random_bytes(8).unpack("Q").first stomp.publish(msg.headers["reply-to"], number) rescue puts "Failed to generate random number: #{$!}" end end |
You could start instances of this code on 10 servers and the MOM will load share the requests across the workers thus spreading out the available entropy across 10 machines.
When run this will give you nice big random numbers like 11519368947872272894. The web page in our example would follow a very similar process only it would post the requests to each of the services mentioned and then just wait for all the responses to come in and render them.
Temporary Destination
The big thing here is that we’re using a Temporary Queue for the replies. The behavior of temporary destinations differ from broker to broker and how the Stomp library needs to be used also changes. For ActiveMQ the behavior and special headers etc can be seen in their docs.
When you subscribe to a temporary destination like the client code above internally to ActiveMQ it sets up a queue that has a your connection as an exclusive subscriber. Internally the name would be something else entirely from what you gave it, it would be unique and exclusive to you. Here is an example for a Temporary Queue setup on a remote broker:
/remote-temp-queue/ID:stomp1.us.xx.net-39316-1323647624072-3:3005:1 |
If you were to puts the contents of msg.headers[“reply-to”] in the server code you would see the translated queue name as above. The broker does this transparently for you.
Other processes can write to this unique destination but your connection would be the only one able to consume message from it. Soon as your connection closes or you unsubscribe from it the broker will free the queue, delete any messages on it and anyone else trying to write to it will get an exception.
Temporary queues and this magical translation happens even across a cluster of brokers so you can spread this out geographically and it would work.
Setting up a temporary queue and informing a network of brokers about it is a costly process so you should always try to set up a temporary queue early on in the process life time and reuse it for all the work you have to do.
If you need to correlate responses to their requests then you should use the correlation-id header for that – set it on the request and when constructing the reply read it from the request and set it again on the new reply.
This series continue in part 4.
