Real-life experiences gained on cutting edge BizTalk projects from the City of London.

Tuesday, November 16, 2004

BAM for operational reports

When you start developing with BizTalk you envitably start with orchestrations, pipelines, schemas etc. and ignore some of the other cool features. One such feature is Business Activity Monitoring (BAM).

Ian & I were working on a SWIFT messaging system for an investment bank and were designing the reporting system for tracking the flow of messsages. We considered the built in message body tracking facilities and considered a custom solution based on a custom application database - and then we considered BAM.

We'd both thought of BAM as being a management information tool, i.e. a tool for gathering aggregate data for building OLAP cubes. We hadn't previously thought of it as a tool for operational reports but it turns out that it is a scalable and flexible mechanism for capturing data.

Before I discuss our approach, a quick recap of BAM. BAM is an infrastructure for capturing data into a database. It has been designed (using active and partitioned completed tables) to efficiently capture data and to allow fast queries over the data. BAM has tools for designing the data you want to capture - you define busines activities in terms of milestones and data items - and will create the requisite data tables, views, stored procedures and DTS jobs for partioning and archiving data. Moreover, the event bus service (aka the tracking host) provides a robust, asychronous system for data capture.

In our SWIFT messaging system each message received resulted in 5-6 message interactions with other systems. We wanted a way to capture the flow of the messages through our system and a way of capturing the message bodies (in their native format). We decided that BAM provided the ideal infrastructure.

I created two activities:

  • A flow activity which defined the milestones for the message flow (e.g. received message, sent message to compliance engine, received first response, etc.) and the data items associated with the milestones.
  • a message body activity which defined the single milestone of a message body processed along with the message body as the associated data item.

I then created two .NET classes (FlowTracker and BodyTracker) for invoking the BAM API to create, update and complete the activities. Each class had a set of methods - one for each milestone - with parameters for the data items appropriate for the milestone.

The next job was to create a simple pipeline component for tracking the message bodies. This component was built as a decoder and encoder that could be used as either the first component in a receive pipeline or the last component in a send pipeline. The component simply used the BodyTracker to create and complete a message body activity.

The orchestrations that process our messages use the FlowTracker to create a flow activity on receipt of the activating message and use the milestone methods to update the activity at various key points in the orchestration.

One of the key problems to solve was how to related the flow activity to the multiple message body activities created in the pipeline. This was done by having the tracking pipeline component work in two modes.

In a receive pipeline, the tracking component simply created the body activity using the message interchange ID. Recall, that the interchange ID is constant for all of the messages in the pipeline and is a property in the message context. When the flow activity is created by the FlowTracker constructor it uses the BAM AddRelatedActivity method to make the body activity a child activity of the flow activity. Moreover, the message interchange ID is recorded as a data item of the message received milestone. Similarly, at the other milestones in the flow where messages are received the body activities are related as children of the flow activity and the interchange IDs are recorded as data items in the flow.

When the orchestration came to send a message it would generate a GUID and assign this as the interchange ID of the sent message. The orchestration would also record the flow activity ID as a custom property in the message context. When the tracking component in the send pipeline processed the message it would create the body activity and use the AddRelatedActivity method to make this a child of the parent flow activity.

Finally, if the message was sent successfully, the flow activity would record this as a milestone along with the sent message interchange ID as a milestone.

With this infrastructure in place we had an efficient, robust and very performant mechanism for capturing both our message bodies and message flows.

Our next design decision was how to report this data. We considered a custom ASP.NET solution but decided on using SQL Reporting Services instead. SQL Reporting Services provides RAD facilites for creating reports that can be delivered in multiple formats (e.g. HTML, EXCEL, PDF etc.) and has facilities for caching, paging, resizing etc. that would all need custom development in ASP.NET. It even has facilities to produce scheduled reports that can be emailed to users.

Using SQL Reporting Services we created a top-level report that with multiple search parameters that produced a table of messsages. The operator can use this report to find messages for a particular date/time period, for different message types, from different sources, to different destinations, etc. A drill down link brings up a detailed report that shows the milestones of the flow and the associated data items and further drill down links display the original message bodies.

Using BAM we've created an efficient operational reporting tool. In the furture we'll use the same data to create management information reports and render these with SQL Reporting Services.


Monday, November 15, 2004

Welcome

Hi and welcome to the real BizTalk blog.

This is the blog of Cassium Technologies' BizTalk consultants. Cassium is a consultancy working closely with Microsoft predominately in the City of London. We are engaged in cutting edge projects in several of the major investment banks and many of our projects rely on BizTalk Server 2004.

The idea of this blog is to report on how BizTalk is used out there, in the field and give feedback on real-life experiences on mission critical projects.

Hope you like it and find it useful.