Intalio Benchmarks

Intalio have published benchmarks for their BPEL engine. Benchmarks are a double-edged thing: you desperately want them to be able to compare different systems together, but experience teaches you that published benchmarks should never be given much credence. Their primary purpose is to benefit vendors' marketing agendas, rather than providing objective data upon which customers can inform purchasing decisions.

So where do the Intalio's one lie. It's difficult to say. 3.5 million BPEL transactions on a basic two-way server with a separate mysql database. That's about the sum of infomation. What is this BPEL doing? If it's simply echoing 'Hello World' or adding two numbers together it's a pretty meaningless benchmark.

The performance of a BPEL process is a sum of three things:

• Transports - SOA means you don't get to dictate which ones your
  customers use, you work with them.
• SOAP Stack - Use the fastest (but SOAP parsing is a small portion
  of a real-world BPEL transaction).
• BPEL processing - Obviously performance / speed is interesting but
  it is not really the most important issue here. For most enterprises these are the critical issues:
• Reliability: This is the key issue. Can the process survive
  server failure. How transactionally integrated are the transports and the BPEL engine? Messages sent or received
  should never be lost or duplicated.
• Availability: Can your system survive server failure.
  
• Scalability: Can your system scale as load increases.
  

These last properties (the RAS of RASP) mean you need effective BPEL clustering.

There are two basic scenarios we should keep in mind when
people discuss benchmarking BPEL:

Unreliable Applications

Use an unreliable transport (e.g. raw HTTP) with a BPEL engine that
doesn't save state to a persistent store. You can produce benchmarks for
this, but this is really not the sweet-spot for BPEL - if a BPEL is
being used to model a Business Process then normally reliability is key.

Reliable Applications

Use a reliable transport (WS-RM / JMS) with a BPEL engine that provides
the highest degree of reliability. But what does "the highest degree of
reliability" mean? Different products open and commercial have differing degrees of reliability. Establishing a benchmark for this as some sort of
conformance test suite would be very useful to the industry.

So assuming you can determine the level of reliability you are happy with from a set of products (this is a big if), what is a useful measure for reliable applications? Well here are three suggestions:

1. Measure how well a BPEL engine can support large volumes of processes. BPEL Processes can live anything from seconds, to weeks and even years. Can your system support millions of live process instances?
   
   
2. Does your BPEL cluster scale? Can you add more servers to handle
   additional load and maintain response times? Or does your cluster degrade as it handles replication of state.
   
3. Can your BPEL server support multiple workloads with different load / transport characteristics? Does it do so in a fair manner, or can one process starve resources from other BPEL processes?