Marcel's SOA & BPM blog

BizTalk and Healthcare

Marcel Fernee — Fri, 30 Oct 2009 20:42:00 GMT

In Belgium BizTalk Server is already the platform of choice when integrating healthcare applications. Slowly, but steadily this is now also happening in the Netherlands. Gerben van Ophuizen has written down his investigations into a nice overview that has also been published in the Software Release Magazine.

You can find his article on-line at:

http://www.ada-ict.nl/Files/SRM_BizTalk_Accelerators_Gerben_Van_Ophuizen.pdf

Persona Non Grata

Marcel Fernee — Wed, 23 Sep 2009 13:25:00 GMT

A while back, I received an invitation from Information Builders to attend their symposium. One of the speaker would be AkzoNobel where I lost a deal to Information Builders. So I was quite intereseted in what he would tell. I registered myself on their web site with full contact and company (Microsoft) information.

To my surprise I got a call last week for Information Builders. They were not amused that I, a competitor, had registered for their event. How would I feel if my competitors would attend an event of mine. Well, actually, that's business-as-usual and I told him so. He didn't like it that I had registered for their event, but he didn't have the guts to forbid me to attend.

Today, when I arrived at the event I discovered that I wasn't registered. Somehow my registration had been deleted Well, the hostess was very polite and welcomed me to the event anyway.

I now also know more about the Information Builders offering and what they consider important. But I also discovered one of the reasons that we lost at AkzoNobel. I already knew that they built a ton of custom code. But it now turned out that Information Builders has taken ownership of the code. That's one way of getting free development for the customer.

BizTalk Performance guide

Marcel Fernee — Thu, 20 Aug 2009 09:06:00 GMT

Microsoft recently release the BizTalk Server Performance Guide. It contains great numbers and tips on how to achieve those performance numbers.

You can find numbers, optimizations, lab configuration, etc. on the brand new BizTalk Server 2009 Performance Optimization Guide. You can find the guide at http://go.microsoft.com/fwlink/?LinkId=160078.

BizTalk Server Operations Guide

Marcel Fernee — Mon, 27 Jul 2009 08:09:00 GMT

"We now have the BizTalk Server 2009 Operations Guide (Ops Guide) live on MSDN and Download Center. We initially started out with the charter to remove duplicate content between the Ops Guide and the BTS core docs, replat the previous version of Ops Guide to include new versions of BizTalk Server/SQL Server/Windows Server, and fix bugs. We are happy to say that we have gone beyond this initial ask and have added some new content including new checklists and information around moving databases.

Thanks to everyone who provided valuable feedback and support to help us achieve this. We would like to periodically update the Ops Guide to include any feedback that you provide. You can continue to send us feedback using the feedback panel in each topic on MSDN and the CHM."

Determining the optimum number of BizTalk Hosts

Marcel Fernee — Tue, 07 Jul 2009 10:54:00 GMT

This post is written to get an overview of the different aspects one should take into consideration when creating multiple BizTalk hosts. There is no one-size-fits-all optimum for each BizTalk project. Generally it is a good practice to have a separate host for Receiving, Sending, Processing and Tracking and then add additional hosts as machine resources allow and tuning and managing require.

Hosts

The BizTalk Host object represents a logical set of zero or more runtime processes in which you can deploy services, pipelines, and other artifacts. The Host object also represents a collection of runtime instances (zero or more) where the deployed items physically run.

After you create a host (a logical container), you can add physical BizTalk servers (host instances) to the host.

Items—such as adapter handlers, receive locations (including pipelines), and orchestrations—contained in BizTalk hosts can perform the following functions:

Receiving. These items do the initial processing of messages after they are picked up in a receive location. When a host contains a receiving item, such as a receive location or pipeline, it acts as a security boundary, and the message decoding and decrypting occurs in a pipeline within the host.
Sending. These items do the final processing of messages before they are sent out to the send port. When a host contains a sending item, such as a send port or pipeline, the host acts as a security boundary, and the message signing and encryption occurs in a pipeline within the host.
Processing. These items process messages based on the instructions in an orchestration.

One BizTalk Host can contain items that receive, send, and process messages. It is recommended that you create different hosts for each function to create security boundaries and facilitate management. In particular, it is recommended that you use different hosts for processing and for receive/send, and that you separate trusted and non-trusted items.

Separating Host Instances by Functionality

In addition to the high availability aspects of the host instance configuration, you should separate sending, receiving, processing, and tracking functionality into multiple hosts. This provides flexibility when configuring the workload in your BizTalk group and is the primary means of distributing processing across a BizTalk group. This also allows you to stop one host without affecting other hosts. For example, you may want to stop sending messages to let them queue up in the MessageBox database, while still allowing the inbound receiving of messages to occur. Also, orchestration and adapter functionality should be separated into different BizTalk Server hosts to minimize resource contention.

Separating host instances by functionality also provides the following benefits:

· Each host instance has its own set of resources such as memory, handles, and threads in the .NET thread pool.

· Multiple BizTalk Hosts will also reduce contention on the MessageBox database host queue tables since each host is assigned its own work queue tables in the MessageBox database.

· Throttling is implemented in BizTalk Server at the host level. This allows you to set different throttling characteristics for each host.

· Security is implemented at the host level; each host runs under a discrete Windows identity. This would allow you, for example, to give Host_A access to FileShare_B, while not allowing any of the other hosts to access the file share.

Redundancy

This host should be run on at least two computers running BizTalk Server (for redundancy in case one fails). Depending on the type of host either fail-over or clustering should be used.

Disadvantages of additional hosts

While there are benefits to creating additional host instances, there are also potential drawbacks if too many host instances are created. Each host instance is a Windows service (BTSNTSvc.exe), which generates additional load against the MessageBox database and consumes computer resources (such as CPU, memory, threads).

Several performance counters are reported per host.

Host consumes memory (at least 200MB (?) per host)

Polling receive adapters will poll for each host that they are enlisted in

SQL connections: Each host instance will poll the db at intervals of 500ms.

Too many hosts will:

· Cause a heavy load on the SQL Server and possibly will prevent processing of messages

· Consume too much memory on the BizTalk servers leading to a throttling situation (reduces performance)

· Too many receive hosts that poll the receive location will degrade the performance

· Too much granularity in performance counters. This has negative impact on the overall view an administrator has.

Tune the Host polling interval

BizTalk Server uses a polling mechanism to receive messages from its host queues in the MessageBox. The MaxReceiveInterval value in the adm_ServiceClass table of the BizTalk Management (BizTalkMgmtDb) database is the maximum value in milliseconds that each BizTalk host instance will wait until it polls the MessageBox. The adm_ServiceClass table contains a record for the following service types:

XLANG/S – for BizTalk orchestration host instances
Messaging InProcess – for in-process host instances
MSMQT – for MSMQT adapter host instances
Messaging Isolated – for out of process host instances, used by the HTTP, SOAP, and certain WCF receive adapter handler

By default, this value is set to 500 milliseconds, which is optimized for throughput rather than low-latency. In certain scenarios, latency can be improved by reducing this value.

Note

Changes to this value impact all instances of the associated service type, therefore, take care to evaluate the impact on all host instances before changing this value.

Note

This value is only used if the MessageBox has no remaining unprocessed messages. If there is a constant backlog of unprocessed messages in the MessageBox, BizTalk Server will attempt to process the messages without waiting on the polling delay. After all messages are processed, BizTalk Server will begin polling using the value specified for MaxReceiveInterval.

Note

In a BizTalk Server environment with a high ratio of host instances to MessageBox database instances, decreasing the value for MaxReceiveInterval may cause excessive CPU utilization on the SQL Server computer that houses the MessageBox database instance. For example, if the MaxReceiveInterval is decreased to a low value (< 100) in a BizTalk Server environment with a single MessageBox and > 50 host instances, CPU utilization on the SQL Server may climb above 50%. This phenomenon can occur because the overhead associated with continually polling host queues is significant. If you reduce MaxReceiveInterval to a value less than 100, you should also evaluate the impact that this has on your SQL Server computer’s CPU utilization.

Quote: “(having many BizTalk hosts) seems like a bad solution. They will most likely see serious performance issues in the SQL layer as each host instance is “polling” for work. I have seen situations where many (maybe around 40) host instances were creating so much chatter with the MsgBox database that even when NO messages were being processed the CPU on the SQL box was at about 60%. Now, the customer did have the MaxReceiveInterval turned way down as well.”

Configuring a Dedicated Tracking Host

BizTalk Server is optimized for throughput, so the main orchestration and messaging engines do not actually move messages directly to the BizTalk tracking or BAM databases, as this would divert these engines from their primary job of executing business processes. Instead, BizTalk Server leaves the messages in the MessageBox database and marks them as requiring a move to the BizTalk Tracking database. A background process (the tracking host) then moves the messages to the BizTalk Tracking and BAM databases.

Advantages of Using a Dedicated Tracking Host

A BizTalk Host that hosts tracking is responsible for moving the DTA and BAM tracking data from the MessageBox database to the BizTalk Tracking (DTA) and BAM Primary Import databases. This movement of tracking data has an impact on the performance of other BizTalk artifacts running in the same host that is hosting tracking. Thus, you should use a dedicated host that does nothing but host tracking.

Using a dedicated tracking host also allows you to stop other BizTalk hosts without interfering with BizTalk Server tracking.

Best practices

Make your plan on how to separate hosts beforehand and how to name them. It is not possible to rename hosts.

Create separate hosts for different functions.

· Receiving

· Sending

· Processing

· Tracking

Add hosts for protocol or application specific requirements.

· Prioritization

· Batching vs. messaging

· Low latency

· Protocols that require clustering

· Protocols that require scale-out

· Separate hosts for applications with known or expects problems
When problematic applications are in a separate host, crashes will not affect applications in other hosts.

· Licensing of adapters
Certain 3^rd party adapters are licensed separate from BizTalk Server. Separating per host can be a cost-effective deployment scheme.

Resources

MSDN: Hosts
http://msdn.microsoft.com/en-us/library/aa578695.aspx

MSDN: Host instances
http://msdn.microsoft.com/en-us/library/aa560673.aspx

Microsoft BizTalk Server Operations Guide
http://msdn.microsoft.com/en-us/library/cc296643.aspxMicrosoft BizTalk Server Performance Optimization Guide
http://msdn.microsoft.com/en-us/library/cc558617.aspx

ESB Toolkit How To Video

Marcel Fernee — Thu, 25 Jun 2009 16:55:00 GMT

Since Microsoft has release the 2.0 version of the ESB Toolkit (formally know as ESB Guidance) it is been much easier to use the toolkit. Now everyone can install it. Without days of frustration :-).

Peter Kelcey has made life for the BizTalk ESB developers even easier. He created several ESB Toolkit 'How to' videos. I strongly advice you to read his blog and take a look at the videos.

http://blogs.msdn.com/pkelcey/archive/2009/06/17/esb-toolkit-how-to-video-4-dynamic-itinerary-resolution.aspx

Upgrading a BizTalk evaluation copy

Marcel Fernee — Thu, 26 Mar 2009 10:09:00 GMT

This week I ran into a rather unusual problem. One of my customers was using the evaluation version of BizTalk for quite a while and recently it expired. Nothing strange here, since this is what evaluation copies are supposed to do. However, this customer was close to going into production with the evaluation machines. They had used evaluation copies of all software (not just BizTalk) to install all of their machines in their DTAP process.

After purchasing the BizTalk licenses, they wanted to enable their BizTalk machines again. However, unlike Windows, BizTalk server is not activated/upgraded or whatever by entering a key. You need to re-install BizTalk.

Luckily we got confirmation from the product team that we could do an in-place upgrade. Just start the BizTalk installation, and your evaluation version will be upgraded to the normal version and everything else remains in place.

SOAP Headers in BizTalk

Marcel Fernee — Mon, 09 Mar 2009 17:52:20 GMT

I recently had to do a proof of concept, where I had to connect BizTalk Server to some internal web services. Normally this isn’t a big deal, but now I ran into a problem: the web services expected a SOAP Header. I will spare you the steps that I took and didn’t work. Below is a working solution, although I have some ideas to make it much more flexible.

Step 1. Import the web service definitions in Visual Studio

Add generated items -> Consume WCF Service -> Metadata Exchange (MEX) endpoint -> path to service

This generates the following files

- An orchestration

- Several xds’s

- 2 binding files: the normal binding file and a custom binding file

Step 2. Use the generated items in the project

Basically this means importing the binding file (the ‘normal’ one) and using the xsd’s and generated port (in the generated orchestration) to do what the orchestration should do.

Step 3. Add SOAP headers

This was the most difficult step, but now I know how to do it, it is actually quite easy and logical.

In the Construct Message shape add a Message Assignment shape to fill the SOAP header.

The SOAP header is called ‘WCF.OutboundCustomHeaders’.

msgRelatieLeesReq(WCF.OutboundCustomHeaders) = "<headers><h:header xmlns:h=\"http://schemas.customername.nl/soap\" xmlns=\"http://schemas.customername.nl/soap\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><account>POC</account><naam>POCESB</naam><wachtwoord>pocesb</wachtwoord><bedrijfsnummer>1</bedrijfsnummer><tussenpersoonnummer>0</tussenpersoonnummer></h:header></headers>";

I also had a namespace problem, so I added all the namespaces from the previous project.

I am sure this can be much ‘nicer’ and more flexible. But this works!

P.S. Thanks to Patrick Wellink for pointing me in the right direction.

Installing WSS for use by BizTalk

Marcel Fernee — Mon, 09 Mar 2009 10:45:00 GMT

After failing to install Windows SharePoint Services 3.0 two times, I decided to take a good look at it and write down the crucial steps for future generations. The problem I had was that during the setup of WSS I could press the OK button until all eternity, but noting would happen. No action and no warnings.

So, what should you do to install WSS 3.0 ?

1. Download the installation files

2. Add the local machine name to the 'local sites' in IE and give 'local sites' the lowest security settings.

This allows scripts and such to run. If you forget this step you can never ever configure SharePoint. You will be presented with a form to fill in and press the OK button. But without this step pressing OK will do absolutely nothing.

3. Run setup and select 'server farm'

4. Now you are ready to create a sharepoint site and document spaces.

That's all.

Workflows, Services, and Models

Marcel Fernee — Wed, 29 Oct 2008 19:54:00 GMT

David Chappel heeft weer een whitepaper voor Microsoft geschreven. Hij legt hierin uit wat er komen gaat in WF 4.0, Dublin en Olso. Er nuttig om te lezen. Ook als je op de PDC bent geweest.

Workflows, Services, and Models

A First Look at WF 4.0, "Dublin" and "Oslo"

http://msdn.microsoft.com/en-us/library/dd200919.aspx

2 andere goede links zijn

De eerste site (modelsremixed) geeft een korte introductie in wat wij met models willen bereiken. De filmpjes zijn zeker de moeite waard. En de tweede link spreekt voor zich. Dit is de plek waar je bij Microsoft informatie over Oslo kunt lezen.

Oslo

Marcel Fernee — Wed, 10 Sep 2008 09:23:00 GMT

In eWeek staan nu 2 artikelen over Oslo die zeker de moeite waard zijn om eens door te nemen.

The Origins of Microsoft's Oslo Software Modeling Platform

Microsoft's Distributed Destination: Oslo

De titels van de artikelen maken wel duidelijk waar het over gaat. Geniet er van.

BizTalk Server 2006 R3 = BizTalk Server 2009

Marcel Fernee — Mon, 08 Sep 2008 11:26:00 GMT

Ik had het eigenlijk vrijdag al mogen melden. Maar hier is het dan toch: de naam van BizTalk Server 2006 R3 is veranderd in BizTalk Server 2009. Onze vrienden in de VS vonden toch dat er naast de stap om de '2008' servers te ondersteunen er zo veel meer is gedaan dat een nieuwe naam terecht is. En eigenlijk ben ik het daar wel mee eens.

De BizTalk roadmap op Microsoft.com is aangepast zodat er nu ook over BTS 2009 wordt gesproken. De details over de naamsverandering, maar belangrijker wat er dan in zit, is te vinden in een Q&A met Oliver Sharp.

In het algemeen kunnen we zeggen dat de beloofde items allemaal in BTS 2009 komen. En wat minstens zo goed is, is wat er daarna gaat komen. Maar meer hierover tijdens de PDC

Daily build en XSLT debuggen in R3

Marcel Fernee — Tue, 05 Aug 2008 06:18:00 GMT

Dat BizTalk Server 2006 R3 ook Visual Studio 2008 zal gaan ondersteunen was natuurlijk al bekend. Maar ik ben er vorgie week achter gekomen dat dit nog een aantal extra voordelen biedt. BizTalk gaat namenlijk voor het eerst 'echt' Visual Studio ondersteunen. BizTalk wordt een zogenaamde 'first class citizen' van het Visual Studio project systeem. Dus nu kunnen we BizTalk eindelijk mee laten draaien in de daily build!

BizTalk kan straks gewoon meedraaien met alle andere artififacts in de Application Lifecycle Management capabilities van Visual Studio.

Dit is al erg handig. Maar er is meer :-) BizTalk schemas, mappings en pipelines kun je nu makkelijker mee laten draaien in de unit tests. Je krijgt in R3 een 'code behind' bij al deze artifacts. En je kunt straks je mappings debuggen. En dat is erg lekker. Ik vond het zoeken naar fouten in mijn mappings nou niet erg makkelijk. Maar straks kun je gewoon je XSLT deuggen. Breakpoints zetten en kijken wat ie doet

Wow

Marcel Fernee — Tue, 29 Jul 2008 19:04:00 GMT

Wow. Dat is toch even wat ik kwijt moet. Ik zit nu in Seattle op TechReady en ik hoor en zie veel mooie dingen. Natuurlijk gaat mijn aandacht uit naar BizTalk Server 2006 R3 en Oslo. Daar zal ik in een volgende post meer over vertellen. Maar dat zijn niet de enige mooie dingen die ik hier zie. Het is allemaal nog hush-hush en we mogen niet niets aan klanten vertellen, maar wat is Red Dog toch goed. En Zurich! En ...

Anyway, ik begin er toch over om iedereen over te halen om toch vooral naar de PDC (http://www.microsoftpdc.com/) in Los Angeles te gaan. Dan wordt 'alles' wereldkundig gemaakt.En ik snap ook wel dat mijn amerikaanse collega's dat graag op zo'n evenement doen. Dan kunnen ze namelijk alles zelf vertellen en goed de verschillende brokken functionaliteit aan elkaar relateren. Want die relatie is er zeker. De verschillende projecten zijn geen losse flodders om een antwoord te hebben op de een of andere concurrent. Wij hebben echt een geweldige Software + Services visie. Ik wou dat alles vandaag al beschikbaar was.

Writing Customer data from BizTalk 2006 R2 to SAP

Marcel Fernee — Wed, 07 May 2008 09:37:00 GMT

Writing data to SAP consists of 2 difficult steps. The first step is to get all the prerequisites from SAP installed on your BizTalk development machine. And the second is to understand the SAP data structures. You absolutely need a local SAP expert for this. For our proof of concept we were lucky. We were allowed to ignore SAP errors about missing underlying data structures. It was enough proof that we could receive the SAP customer ID and client ID, even though SAP did not accept our customer record.

You can choose to install the WCF LOB adapter for SAP before you install the actual prerequisites from SAP. When you do, it installs without problems. But when you start Visual Studio and start “Add Generated Items – Consume Adapter Service” you won’t see the sapBinding. You will only see the siebelBinding and the oracleBinding. You may also get a different error: if you see the sapBinding and fill in the required data, you press ‘Connect’ and poof: Visual Studio is gone. In order to get the sapBinding working you need to obtain the RFC SDK Unicode libraries from SAP and put them in your windows\system32 directory. For a detailed description refer to the installation manual “InstallationGuide.htm” included in the AdapterPack.

Once we got everything correctly installed, we could finally begin with the real interface to SAP.

1. First create a new BizTalk project in Visual Studio. Then right click on the project and select “Add Generated Items” -> “Consume Adapter Service”.

2. Then the LOB Adapter configuration dialog shows up.

- Select “sapBinding”from the “Select a binding”

- Bring up the SAP Adapter configuration by pressing the button “Configure”. At first the username and password for SAP are set.

Then at the URI properties the SAP connection properties are set.

In our case the following were set:

Application Server
System number
Client
Language

Then at the Binding properties ensure that the EnableBusinessObjects for Bapi is set to “True”. After clicking “Connect” on the “Consume Adapter Service” the adapter connects to SAP and gets all its metadata which is available. This includes BAPIs, IDOCs and RFCs.

3. Select BAPI and search in category using “Customer”, select the search result “http://Microsoft.LobServices.Sap/2007/03/BAPIOBJ/KNA1”.

After pressing the OK button BizTalk extracts some metadata from SAP and creates a schema for the BAPI and a BizTalk binding file for creating the messaging port for the BizTalk solution.
4. Create a simple BizTalk Orchestration which gets the xml message of the new Account to be created. The received message is sent to SAP and the response from SAP is used in updating CRM (MS Customer Dynamics). Note that the message context properties (OPEN, REUSE, CLOSE, and ABORT) should be used to enable transaction on a SAP system. For detail explanation see reference “Performing BAPI transactions by using BizTalk server”.

Example:

The following example shows that the “BAPICommitMessage” corresponding to BAPI_TRANSACTION_COMMIT for committing the transaction on the SAP system uses the context property “CLOSE”

BAPICommitMessage(Microsoft.Adapters.SAP.BiztalkPropertySchema.ConnectionState) = "CLOSE";

5. Ensure that the assembly is strong-named, in the solution explorer right-click the solution and then click Build Solution, after the solution successfully builds right-click the solution and then click “Deploy Solution”.

6. In the BizTalk Server Administration console, right-click the deployed application and then select Import àBindings. In the dialog box that opens navigate to the folder that contains the binding file created (Refer to step 3) and then click “Open”. This creates a preconfigured WCF Solicit-Response Send port with a WCF custom binding (sapBinding).

7. Modify some of the settings by right-clicking the WCF-Custom send port and selecting “Properties”. In the port properties dialog box do the following:

a. Click configure

b. Specify the connection URI (client number, language, server name, system number)

c. Click the “Credentials” tab and specify a user name and password to connect to the SAP system. Click “Ok”

d. Click the “General” tab; modify the action mapping of the SOAP action header to map an operation in the BizTalk Orchestration to an action in the LOB adapter (the selected BAPI call in the SAP adapter)

<BtsActionMapping> <Operation Name="BAPIMessage" Action="http://Microsoft.LobServices.Sap/2007/03/Rfc/BAPI_CUSTOMER_CREATEFROMDAT1" /> <Operation Name="BAPICommitMessage" Action="http://Microsoft.LobServices.Sap/2007/03/Rfc/BAPI_TRANSACTION_COMMIT" /></BtsActionMapping>

The above is an example of how operations “BAPIMessage” and “BAPICommitMessage” are mapped to actions “http://Microsoft.LobServices.Sap/2007/03/Rfc/BAPI_CUSTOMER_CREATEFROMDAT1” and "http://Microsoft.LobServices.Sap/2007/03/Rfc/BAPI_TRANSACTION_COMMIT" in the LOB adapter.

The name of the operations “BAPIMessage” and “BAPICommitMessage” should correspond with the names of operations in the BizTalk Orchestration.
For detail explanation see reference “Specifying SOAP action”.

e. Create the remaining Receive and send-Ports and bind the Orchestration.

Authors

Marcel Fernee: marcel.fernee@microsoft.com
Henry Osagiede: henry.osagiede@atosorigin.com
Mark Wijngaarden: mark.wijngaarden@atosorigin.com
Mariёtte Mak: mariette.mak@atosorigin.com

Resources

BizTalk WCF LOB adapters

Obtaining the RFC SDK Unicode Libraries from SAP Service Marketplace
http://blogs.msdn.com/adapters/archive/2007/10/07/obtaining-the-rfc-sdk-unicode-libraries-from-sap-service-marketplace.aspx

Performing BAPI Transactions by Using BizTalk Server
http://msdn2.microsoft.com/en-us/library/cc185462.aspx

Specifying SOAP action
http://msdn2.microsoft.com/en-us/library/cc185151.aspx

Using the new BizTalk 2006 R2 SAP LOB Adapter
http://www.cubido.at/Blog/tabid/176/EntryID/128/Default.aspx

SAP Transaction
http://msdn2.microsoft.com/en-us/cc196386.aspx#sap