Notable Notions

Motivational Foundations

2009-09-15T09:48:00.000-07:00

Just saw an interesting presentation about motivation, science & business: Dan Pink on the surprising science of motivation.

Probably the coolest takeaway are the three core foundations of successful motivation:

1. Mastery
2. Autonomy
3. Purpose

Mastery would be the drive to improve and become the best at what one does. Autonomy would be the feeling of empowerment and control over what one does. Purpose would be the feeling of being a part of something bigger.

I've seen all of these things work well in the past but the genius is putting them all together to drive motivation across the whole team. Dan Pink says that incentives don't work as expected and I believe him. From my experience the most motivated teams have everyone feeling engaged, improving their skills, and helping the company kick some ass.

Thanks Dan for clarify what I've observed in a way that I can now take it an purposefully work at implementing it.

Geez, TED is a cool site.

Bought some memory for my camera

2008-12-19T16:16:00.000-08:00

I just bought some memory for my camera. It was SmartMedia which is actually pretty hard to find these days.

I bought it from MemorySuppliers.com which seems to have some good deals on memory of all kinds. If you are looking for such things then check them out at memorysuppliers.com.

Error Handling and Event Logging

2008-03-26T17:43:00.001-07:00

As part of my continuing effort to make this blog so esoteric that nobody ever reads it I'm going to post some thoughts on Error Handling and Event Logging. I felt it was a good time to post this as I've been discussing it a bunch with the development team lately and from those discussions we've worked out couple of methodologies that I thought were worth sharing. I'm going to say up front that I recognize that its very likely that this has all be discussed and written about in numerous programming books so it's probably not super original stuff. Nonetheless, I'm going to take a moment to describe how we are approaching the problem anyway.

The "Caller Responsibility" Error Handling Practice

Simply put, "He who starts the ball rolling catches any error thrown back."

The idea behind this best practice is that the process/method/procedure/whatever that started the workflow which ultimately generated an error is best suited to understand the context of the error in terms of the greater operation of the application. For example, if we have an application with two independent services that form a workflow then we cannot give either service responsibility for handling an error since neither understands the other service (they are "independent"). Instead, the process that governs the workflow which calls each service understands how errors in either are handled specifically.

This concept is important in SOA applications where abstraction between services is key to layering functional blocks. Each service only reports the error back, cleans up any internal state inconsistencies that may result, and then trusts the caller to react to the error in an appropriate way. I've found this approach helpful when breaking up work within teams by assigning distinct services to groups of individuals. Since error handling is distinct it's quite easy for different teams to plug into a bigger picture of error handling by trusting someone else to handle it.

The next thing I wanted to write about today was…

The "Reporter On Scene" Event Logging Practice

The idea here is that don't even bother logging exceptions/events if you are not going to capture enough information to do anything useful later on when you are trying to figure out what the heck happened. It turns out that newspaper reporters have pondered and solved this problem for us by using reporting on the "Six W's". They are: Who?, What?, Where?, When?, How?, and Why?. We can use the answers to provide a complete picture of any event to the poor sucker reading the log at 3am when something breaks.

What exactly do we mean by all of this? Let's use an exception event as an example:

Who

"Who" is the component reporting the error. Usually, this is the website and class name (or page name).

What

"What" are exception details such as error description and stack trace.

Where

"Where" is method and line number in the source code.

When

"When" is the time that the exception occurred (not the time it was reported).

How

"How" is additional context provided by the caller (see "Caller Responsibility" above) to explain the workflow that lead to the exception.

Why

"Why" is probably the most difficult to automate but helps describe why the error happened. Often this is accomplished by providing additional supporting information such as the values of related variables and objects.

When the event log provides details that address each of these questions it is immensely useful for analyzing failure after the fact. Give it a try!

Managing an Agile team

2007-10-25T15:06:00.001-07:00

Wow, it's been a while. Hopefully, I can get back into the swing of posting now that things have calmed down a little at the new job.

Speaking of the new job...

One of the biggest challenges I face daily is finding a practical methodology for executing on 'Agile' projects. I'm not talking about whether I choose SCRUM or Extreme or some other developer-focused methodology but rather a link between the 'project' side of software development and the 'code' side. Most of my reading and experience with Agile leans heavily toward the developers and writing code. However, there is much more to a software project than writing code. How does a Project Manager manage an agile project? How does he monitor progess and stay on budget?

I mean to dive into this over the next few months and I'll be marking my findings here.

The Siren Call of Web Services

2007-05-07T17:48:00.000-07:00

Most of the solutions that I've worked on over the last few years have implemented a Service Oriented Architecture (SOA) and generally use web services to implement the services. Usually, the web services have worked out great but occasionally they have failed spectacularly. Why have they failed? Read on to find out.

The "Siren call"
The great thing about web services is that they are really easy to create and (i.e. web service project template) and just as easy to consume (i.e. web reference) due to tight integration in Visual Studio. They are so easy to use that even new developers can be productive in a short period of time.
Web Services also make traversing application or network boundaries easy since constructing a SOAP call is easy (thanks to Visual Studio) and HTTP over port 80 will get you past most firewalls.

The "Danger"
The thing to keep in mind is that web services are slow. The ease of development and simple usability mean that we pay a price in performance.
Any call to a web service needs to be translated into SOAP, sent over the wire as wordy HTTP, translated from SOAP, service logic executed, translated back into SOAP, sent back as wordy HTTP, and finally translated back from SOAP. This may be a simplification of the actual process but you get the idea. There is a lot of activity around even invoking a simple "hello world" web service.

The "Fate"
At best, a slow web service means that your speedy application needs to wait while the service is invoked. Most of the time this is the only problem you'll have and it can be managed in a variety of ways (i.e. caching, asynchronous calls, etc.).
At worst, a slow web service could take down your beautiful solution and duplicate or lose important data.

How could a web service do so much damage?

It generally comes down to threads. Specifically, not having any available threads to handle another service request. I'm not claiming to know exactly how the threading works but every call to a web service requires threads from IIS and .NET. These technologies maintain a pool of threads that service requests and then wait for the next one.
Normally, a request can be serviced quite quickly and the thread is freed quickly. However, under high load a thread might not free up in time to service the next request so another thread is needed to handle it. As requests hit the server the threads are allocated and freed as quickly as possible but under high load requests can 'pile up' until eventually there are no threads left to service the next request. When this happens you will begin to see longer delays in service response as the caller waits for the service to answer while the service waits for a thread to free up. Around this time you will likely start to see "request timed out" errors.
If the duress continues and requests continue to arrive then the server will ultimately give up and start throwing IIS and/or .NET exceptions such as "There were not enough free threads in the ThreadPool object to complete the operation."
It is at these times when I have noticed unexpected conditions to arise that can create unpredictable results. If the caller gives up on the service (timeout expires) you would expect the service to perish but there are certain boundary conditions by which a service can continue to execute even though the client has given up on it. The service may not fully complete executing but even a few lines can cause problems. We have seen duplicated rows inserted into a database due to this problem when the client retries the service thinking the service has timed out but the service continued to execute.

The "Safety Net"
The first thing to do when working with web services is have a good understanding of the maximum load that service will have to bear. Plan for that load and test the crap out of it before going live. Hit that service as hard as you can for as long as you can before you trust it with your data. I've seen very innocuous web services cause absolute chaos under extreme duress so don't assume that it will 'just work'.

The second thing is to actively work to reduce the execution time of that service to the smallest time possible. Threads are only used during execution so if the service executes quickly then it will free that thread up quickly.

Lastly, think about farming out your web servers. There are a multitude of configuration settings for IIS and .NET to tune threading but all of these settings will ultimately fail under extreme load. It might just die later but it will still die eventually if the traffic is extreme enough. The best way to give your application more threads to work with is to use multiple servers. Usually, these would be physical web servers but more commonly farms of virtual servers are being used quite effectively.

I hope you excuse my crude metaphor to hapless sailors but perhaps my little brain dump helps you plan your web services better in the future and avoid some of the pain I've experienced.

Avoid Deadlocks using the SQL Adapter and BizTalk Server 2006

2007-02-21T18:07:00.001-08:00

As applications scale it is often necessary to add additional servers in order to take advantage of the power of parallel processing. For BizTalk solutions this means that there can be two or more BizTalk hosts running identical solutions and all sharing the same resources. BizTalk does a good job of managing multiple hosts internally (i.e. maintaining unique identifiers) but sharing external resources (i.e. a database) needs to be handled more carefully. This article describes a strategy for maximizing efficiency and minimizing conflict between multiple BizTalk Hosts communicating with a single database.

The primary reason why you would want to use the following approach is to avoid errors like this:
"Transaction (Process ID 56) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. Error Number : -2147467259."

The approach described in this article follows the pattern where two (or more) BTS servers are polling a single database using identical SQL Adapter port configurations but this approach also applies to a single BTS server where deadlocks are an issue under load.

The following goals were used to define how this strategy should behave:

No duplicate records
Maximize use of all BTS instances
No SQL locking conflicts
Acquire data as quickly as possible

The first three requirements can be handled by a carefully crafted stored procedure (see section 1.3) while the fourth is controlled by BizTalk. In the receive port properties set Poll While Data Found to True. This will make BizTalk keep reading the database until there are no more records to process. If you do not set this option then the solution will not scale to handle larger volumes.

The Aquisition Stored Procedure

Due to the fine-grain control that stored procedures provide it is strongly recommended that they be used over a direct SQL query. Especially since a direct query will not allow you to manage SQL locking which can be problematic with multiple BTS hosts competing for the same resource. The basic steps of the acquisition procedure are (complete source code available in section 1.5):

Set the Transaction Isolation Level
Acquire unique identifier of the record
Mark the record as processed
Return the contents of the record

Set Transaction Isolation Level

By default BizTalk wraps all calls made with the SQL Adapter in a transaction with an isolation level of SERIALIZABLE. This level is much more aggressive with resource locking than the default level of READ COMMITTED. As a result it is important to specify the Transaction Isolation Level for the transaction within the procedure. In the sample provided the transaction level of READ COMMITTED is.

In order to maximize concurrency while eliminating conflict it is often necessary to further refine resource locking by the use SQL locking. Locking hints provide a fine degree of control over how SQL handles the visibility/availability of records to other transactions.

It’s worth noting that there is more than one way to approach the way that SQL locking is handled and that a REPEATABLE READ solution with some modifications to the locking hints could also theoretically work. Although, since the sample solution discussed in this document has been implemented and tested in a real-world application so it is the recommended solution.

Acquire Unique Identifier

The first step is to get the unique identifier of the record to process. Generally, this is a select statement with any WHERE or JOIN expressions necessary to return a single row’s identifier where the processed flag is not set (see the next step details). In the sample provided TOP and ROWCOUNT are also used to insure a single row result. It is important to note that the acquisition procedure will only acquire and process a single row. It is left up to the BTS receive port to poll the database and control execution of the acquisition procedure to consume multiple rows.

Two SQL locking hints are used to lock the row (UPDLOCK) and read past (READPAST) any existing locks. The row is locked at this point (as early as possible) to prevent another SQL instance from also acquiring it for the lifetime of the transaction. The read past hint is used to skip past any existing locks (e.g. created by other instances of this procedure) without stopping and waiting for the lock to clear which is the default behavior.

It is recommended that the number of JOINS be minimized so as to reduce the chance of locking a row in a table required by other processes. One possible solution is to populate a ‘Staging’ table and then consume rows from there.

Mark As Processed

Using the identifier acquired in the previous step, a processed flag is set to indicate that this record has been consumed by BizTalk and should not be included in future acquisition attempts. The query used to acquire a unique record identifier specifically avoids rows that have been processed so once this is set it will not be consumed again.

In the sample provided, another field called ProcessedOn is set at the same time to date stamp the time that the record was consumed by BizTalk. This field is not strictly required but can be handy for auditing and testing purposes.

Get Message Contents

The final step of the acquisition procedure is to return the contents of the row to the SQL Adapter. The sample provided uses SELECT * but this is only for demonstration purposes and is not recommended for real-world scenarios.

Since the stored procedure is designed to facilitate the BTS SQL Adapter it is important to include the FOR XML AUTO clause in the SELECT statement. This will return the results of the query in a form (XML) that BizTalk can consume.

Finally, when the transaction ends all locks are released.

SQL Locking Hints

There are a variety of specialized locking hints available in SQL Server 2000. Below is a summary of hints that are useful for sharing a single resource by multiple hosts (descriptions taken from SQL 2000 Books Online go there for more information).

READPAST

Specifies that locked rows are skipped (read past). For example, assume table T1 contains a single integer column with the values of 1, 2, 3, 4, 5. If transaction A changes the value of 3 to 8 but has not yet committed, a SELECT * FROM T1 (READPAST) yields values 1, 2, 4, 5. READPAST applies only to transactions operating at READ COMMITTED isolation and reads past only row-level locks. This lock hint is used primarily to implement a work queue on a SQL Server table.

ROWLOCK

Specifies that a shared row lock is taken when a single shared page or table lock is normally taken.

UPDLOCK

Specifies that update locks instead of shared locks are taken while reading the table, and that they are held until the end-of-statement or end-of-transaction.

READ UNCOMMITTED

Implements dirty read, or isolation level 0 locking, which means that no shared locks are issued and no exclusive locks are honored. When this option is set, it is possible to read uncommitted or dirty data; values in the data can be changed and rows can appear or disappear in the data set before the end of the transaction. This option has the same effect as setting NOLOCK on all tables in all SELECT statements in a transaction. This is the least restrictive of the four isolation levels.

REPEATABLE READ

Locks are placed on all data that is used in a query, preventing other users from updating the data, but new phantom rows can be inserted into the data set by another user and are included in later reads in the current transaction. Because concurrency is lower than the default isolation level, use this option only when necessary.

SERIALIZABLE

Places a range lock on the data set, preventing other users from updating or inserting rows into the data set until the transaction is complete. This is the most restrictive of the four isolation levels. Because concurrency is lower, use this option only when necessary. This option has the same effect as setting HOLDLOCK on all tables in all SELECT statements in a transaction.

Sample Stored Procedure

/*
This stored procedure efficiently consumes a single row for multiple BizTalk without deadlocks.
Receive Ports using the SQL Adapter.

Author: Adam.Bowron@Visiphor.com
Created: October 25, 2004
*/
CREATE Procedure Shared_BTS_Select
AS

BEGIN TRANSACTION

--Explicitly set the Transaction Isolation Level since the procedure relies on it
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
DECLARE @Record_ID integer
SET NOCOUNT ON
--Acquire a single row and lock it so that another concurrent query will ignore it.
--READPAST is used to skip locked rows without waiting for them to release.
--UPDLOCK is used to lock the row for the lifetime of the transaction
SET ROWCOUNT 1
SELECT
@Record_ID = Source_Table_ID FROM
Source_Table WITH (READPAST, UPDLOCK) WHERE
Processed = 0
ORDER BY
Last_Update_Dt ASC
SET ROWCOUNT 0
--Mark the row as processed so that it is ignored by future queries
UPDATE
Source_Table
SET
Processed = 1,
ProcessedOn = getdate()
WHERE
Source_Table_ID = @Record_ID
--Return the contents of the row in XML form to the SQL Adapter.
SELECT
*
FROM
Source_Table
WHERE
Source_Table_ID = @Record_ID FOR XML AUTO

COMMIT TRANSACTION
GO

References

SQL 2000 Books Online (transaction/locking hints are copied verbatim)

"Best Before" Phase

2007-02-01T10:32:00.000-08:00

There are many software development methodologies and all of them have variations on phases with a project. There are names like "Envisioning", "Planning", "Stabilization", etc. All good names with valid, serious, meanings...

At this point, I'd like to officially introduce another phase that lays on top of all the stodgy official phases. I call it the Best Before phase. This is generally near the end of the project when you start to notice that things in the fridge expire on or after your project is due to be completed. There's a funny emotional reaction to seeing that the expiry date on the carton of milk is a few days after your project is scheduled to complete. The reaction ranges from relief ("Thank god it's almost over") to outright fear ("I'll never get it done in time"). Either way, you know you're in the Best Before phase.

"I don't know phases"
- Doctor Evil

One-Way Web Services with BizTalk Orchestrations

2007-01-26T15:38:00.000-08:00

Today I want to pass on a nifty trick that gives you the ability to invoke 'fire and forget' one-way web services within a BizTalk orchestration.

My problem was that I wanted the port to handle delivery (including retries) and not have the orchestration wait until the result of the send was successful or not. Normally, doing this with ports is generally quite easy since you just use a one-way send port. However, with web services there is actually a trick to getting your web service to behave as a one-way service.

If you follow the standard procedure for hooking up a web service in an Orchestration you will get a request/response logical port shape. You get this even if the webmethod is a 'void' method that does not return anything. You can try creating your own port type but that causes other problems with web services since you'll get errors like AssemblyName context property was not valid unless you add special code to set some context properties.

The best solution is to inform BizTalk that the web service is 'one-way' so that the configured port type reflects this pattern. The way to do this is quite easy. All you need to do is add a parameter to your web method called SoapDocumentMethod with a specifier indicating the message pattern.

Sample code looks like this:

[WebMethod]
[SoapDocumentMethod(OneWay=true)]
public void OneWayService ()
{
}

When you add a web reference to your Orchestration project BizTalk now configures the port type to be one-way and the port shape reflects this. It's easy if you know the trick!

I'd like to thanks Matt Milner for posting the solution on the microsoft.public.biztalk.orchestration newsgroup.

Virtual PC Optimizations

2006-10-19T17:42:00.000-07:00

Tags: Virtual PC, Virtual Server
A coworker recently sent me a link to an excellent document describing the difference between VPC and Virtual Server and how to squeeze the most performance from it. I'm not going to reproduce the entire document here for brevity but I do want to describe the settings made within the virtual system to improve performance (because I know I'll want to do this on every new VPC I start working on). You can grab the whole document here: http://blogs.msdn.com/nolansax/attachment/700992.ashx.

Configure performance settings on the Microsoft Windows XP or Microsoft Windows Server 2003 virtual machine

Start Virtual PC 2004, and then start the Windows XP or Windows Server 2003 virtual machine that you want to configure.
Click Start, click Run, type sysdm.cpl, and then click OK.
On the Advanced tab, click Settings under Performance.
On the Advanced tab, click Programs under Processor scheduling, and then click OK.
** Note Before you modify this setting, you must verify that the host is not running critical services in the background.
On the Hardware tab, click Device Manager.
Expand Disk Drives, right-click the drive that you want, and then click Properties.
On the Policies tab, click to clear the Enable write-caching on the disk check box, and then click OK.
On the File menu, click Exit to close Device Manager.
Click Start, click Run, type desk.cpl, and then click OK.
On the Settings tab, click Advanced.
On the Troubleshoot tab, move the Hardware acceleration slider to Full, and then click OK two times.
Click Start, click Run, type regedit.exe, and then click OK.
Locate and then click the following registry subkey:
My Computer\HKEY_CURRENT_USER\Control Panel\Desktop
In the right pane, right-click MenuShowDelay, click Modify, type 400 in the Value data box, and then click OK.
Quit Registry Editor.

Dropdown Engineering

2006-10-16T11:46:00.000-07:00

Tags: BizTalk, Dropdown Engineering, Software Development

In my work I do a lot of development within products or frameworks. The best example is BizTalk but others are K2.net or BIE (Bryante Integration Server). Creating business solutions within these frameworks requires a different approach to software development. How is it different? Well...

The first thing you need to do as a developer is embrace the features. Only by doing this do you fully leverage the power of the framework Why even use the product if you intend to hand roll everything? I often see new BizTalk developers concocting long and involved code in Orchestration Expression Shapes (think: code snippets). Naturally, they are complaining the whole time about how the shape dialog doesn't have intellisense or other useful coding features. I try to gently remind them that they are missing the point; expression shapes shouldn't really need more than a few lines of code. Anything more and you are likely doing something that could be done better in another way. (If the built-in features of BTS don't do what you want then at least write a helper class to encapsulate your valuable business logic). The point is that the framework gives you the flexibility to write your own code but that doesn't mean you should (or need to).

The second thing you need comes from experience with the framework (usually after 2-3 projects) and that is to know the pitfalls. Every framework has strengths and weaknesses. Part of what makes a superior Dropdown Engineer is understanding this and knowing where to invest work to shore up the weaknesses. This may be one of the single biggest advantages of using experienced engineers.

I sometimes lament the loss of creative freedom when using product frameworks but I take solace in the fact that they allow me to develop solutions faster, more reliably, and with greater feature-sets than I would be able to do on my own. In essence they (the good ones, anyway) allow me to concentrate on business value without investing heavily in the foundational work. That's a good thing, right?

Microsoft SOA and BPM Conference: Day 4

2006-10-12T14:08:00.000-07:00

Tags: Windows Communication Foundation, BizTalk, Microsoft, SOA

Last week was the conference, this week it's back to work and wow am I ever paying for spending the week away from the office. Nonetheless, the blogging must go on...

What is Federation?
During an interesting discussion about integration design patterns by an architect from Software Architects, inc. something interesting came up. He asked the audience to define 'Federation' and a couple of answers flew back but most people had trouble defining it. He provided what I believe is a very succinct definition that I will be happy to borrow and use from now on. Thanks, Mr. Architect! :)

Federation is the unification of self governing entities.

The new WCF Adapter
With the release of BizTalk 2006 R2 (tentatively scheduled for mid 2007) there will be direct support for the new Windows Communication Framework (part of .NET 3.0). From the looks of things this adapter will rapidly become our most commonly used adapter. This to me is yet more proof that WCF will have a dramatic impact on how we build applications.

WCF adapter features

Transactional message send/receive

Uses ws-* headers for routing/processing

Numerous security scenarios

In-proc hosting of non-http (netTCP, netNamedPipes, netMSMQ) endpoints

Business Rules
A very interesting seminar was held discussing business rules within your enterprise. Where to store them and how to express them. There was lots of good stuff with the usual justifications for separating business logic from code. I thought there were a couple of points in particular that were worth recording.

Deciding factors on where to keep rules

Volatility

Ownership

Reasons to manage Business Rules separately

Accessibility

Flexibility

Manageability

Web Services Software Factory
On a final note, there was a great seminar presented by the Patterns and Practices team which showed us some of the great work they are doing around standardizing Web Services application development. I strongly encourage everyone to go and check it out at the Web Service Software Factory Community page.

Benefits of using a software factory

Quality

Consistency

Performance

That's it for the conference reporting. It was a great educational experience and I feel privileged to have attended. Hopefully, next year we have the opportunity to go again!

Microsoft SOA and BPM Conference: Day 3

2006-10-05T14:33:00.000-07:00

Tags: BizTalk, Microsoft, SOA

Here we are in the home stretch of the conference and the gems keep on popping up.

A replacement for the SQL Adapter?
I asked my annual question about whether they will be re-writing the SQL adapter and the short anser is "no" but... a surprise during the adapter presentation was the revelation that there will be a new Oracle adapter in R2. It turns out that the new Oracle adapter is a complete rewrite based on ADO.NET. Furthermore, it was suggested that since it's based on ADO.NET that it could also include support for SQL Server. It may be the case that we see a new 'database' adapter that supports Oracle and SQL server. Let's keep our fingers crossed!

Large Messages
A common challenge with BizTalk solutions is handling large messages. A large message is defined as anything over 1-2 megabytes. A persuasive demo showed us how dramatic the performance hit can be if you do not handle large messages carefully. The general approach is to avoid pumping huge messages into the message box. Instead, store the large part of the message in a file or database and then tag the message with a reference to the storage location. This would happen in the incoming pipeline. When messages leave BizTalk an outgoing pipeline reconstructs the message by retrieving the large bit and joining back to the message.
A very impressive solution was demonstrated that actually examined the incoming message for large elements, pulled them out (and replaced them with a reference id), stored them in the database, and then reconstructed the message on the way out. The performance improvement of this approach was huge. Something like a 60% improvement.
The presenter has a blog that's a good place to seek out more information on large message handling. It can be found here: http://blogs.digineer.com/blogs/tabraham/default.aspx

Microsoft SOA and BPM Conference: Day 2

2006-10-04T16:53:00.000-07:00

Today's presentations were more technical than yesterday and hit on a number of interesting topics.

My favorite presentations were about Windows Communication Foundation (WCF). First off, WCF looks like the answer to many of our solutions. Being able to implement a variety of communication transports with run time configuration is awesome. It's also going to be great to bring the WS-* protocols into the mainstream with direct support for BizTalk ports.

I picked up a few best practices that we'll have to be sure to follow:

Don’t plan on migrating to WCF. Plan on engineering new applications. The value of migrating is limited since existing applications ‘work’ and moving web services projects to WCF web services doesn’t really gain much.
When validating messages avoid ‘schema validation’ as it can be costly. Validating Data Contracts is preferable.
Protocols

BasicHTTP is for all HTTP and web services type communication except for if you need WS-*
WSHTTP is for WS-* web services communications
NetTCP is for fast, efficient communication (and is best for machine-to-machine within your .NET solution)
NetNamedPipes (I think it’s called) is best for service calls within the same machine

Add Fault Operations to the WSDL contracts

Lastly, I picked up a couple of answers to my standing questions around SOAP Faults and NULL parameter validation. On the SOAP Fault front the good news is that the WCF WSDL tools will support SOAP Faults which is a huge improvement over the current tools. With NULL parameter validation the answer seems to be to use a new group of .NET 2.0 data types within the System.Nullable namespace. These are effectively a bunch of value types that can be NULL which should bypass the fun behavior of .NET where it assigns default values for NULL parameters.

Microsoft SOA and BPM Conference: Day 1

2006-10-04T09:09:00.000-07:00

Another year passes and here I am at the BPM conference in Redmond again.

Last year the conference was very focused on the impending release of BTS 2006 and the impact of the recently unveiled WinFX (since renamed to .Net 3.0). This year BTS seems to be taking a lesser role and is being presented more as a supporting technology for the - currently fashionable - Services Oriented Architecture (SOA). In fact, if the first day is any indicator SOA is the central theme of the conference with technologies used to support Services Oriented (SO) architectures presented as precisely that, technologies to enable SOA.

Why is SOA the focus of the conference this year?

SOA is certainly nothing new. As my colleague pointed out, 'distributed computing' has been around for many years and was an early attempt to distribute 'services'. Dave Chappelle explained in his keynote that the reason SOA is now so hot is because the communication protocol to communicate services has finally been decided. The Simple Object Access Protocol (SOAP) allows diverse computing environments to communicate in a consistent way which effectively lowers previously near-impenetrable barriers. Chappelle also pointed out that the last time this happened was with TCP/IP when it assumed the mantel of dominant networking protocol and effectively enabled the Internet to be realized by opening up all of the diverse networks and allowing them to communicate with each other. So, it seems that SOA is the focus this year because the technology has reached a critical mass.

What benefit does SOA provide?

Normally, when I explain the benefits to a customer I describe three main benefits but in a couple of the presentations yesterday I came to realize a nagging suspicion about one of the ‘benefits’ that has moved me to officially remove it from my list of SOA benefits. Here are the benefits:
1. Abstraction
2. Accessibility
3. Reusability
For a while, I’ve suspected that my argument for reusability was a bit weak and after hearing others question it yesterday I’ve come to the conclusion that it is very weak. I can think of very few real-world examples where a service has been re-used to the point where it can reasonably be described as a benefit. Usually, a service is too specialized for a business case to be shared by other processes. Other times, the owner of that service has a vested interest (i.e. data ownership) in preventing access by other parties. In fact, the only example that immediately comes to mind of successfully reusing services is with simple ones like GetActiveDirectoryProperty which is a simple service that returns a property value given an account and property name. I think this was successfully reused because the business case was very precise and the AD values were not particularly proprietary.

Anyway, more on the conference to follow…

SOAPException class as WSDL SOAP Fault?

2006-09-15T11:50:00.000-07:00

Recently I embarked on a seemingly simple mission. I wanted to explictly define a SOAP Fault to return a .NET SoapException. This turns out to be not so simple. In fact, after a few hours of experimentation, searching, and consultation I have still to come up with a decent answer. The search continues...

.NET and WSDL SOAP Faults

2006-09-15T11:48:00.000-07:00

Recently I was working with a customer that wanted to explicitly define the SOAP Fault related to a failed web method call. I thought this would be simple since tools like WSDL.exe would provide the mechanism to handle the faults. Boy, was I wrong. It turns out that none of the 'mainstream' .NET WSDL tools handle SOAP Faults at all. If you generate a service from a WSDL file using "WSDL /server" it clears out all of the Fault references. So much for automation.

A guide to good naming

2006-08-29T13:57:00.000-07:00

A question that I often get is "what should I name it?"

The best advice I can give is to take off your 'developer' hat and put on your 'business' hat. The most meaningful names reflect the purpose of the artifact. I have found that most often the purpose is best defined in business terms.

For example, we have a component that notifies a user of a new task. We could call this componet "SendEmail" or "SMTPOut" but it is much more meaningful to call it something like "SendTaskNotification".

It's almost never appropriate to name things based on the technology used. My favourite example of this is to name your XmlDocument variable "xmldoc". Could this name be any less descriptive? Surely, if we are going through all the trouble to instantiate/use this variable it must have some purpose. When naming we need to highlight this purpose and ideally put it in terms of the task at hand without focusing on the technical. It would be much more descriptive to call this variable "PurchaseOrder" or something like that (if that it what it was being used for). It is obvious from the code that it's an XmlDocument so we don't need to name it that.

A big win for good naming is the massive reduction in the need to write comments to describe what the variable is for. Good name swill take you a long way to self-commenting code.

BizTalk 2006 Documenter has been released

2006-08-22T17:32:00.000-07:00

Recently, the super handy UK SDC BizTalk 2006 Documenter utility (think NDoc for BTS) was released. I can tell you that after making the mistake (once) of retrofitting a 2006 project onto a 2004 machine so that I could run the documenter (the old version didn't support 2006) I'm really happy to see this update.

You can grab it here along with some other handy-looking BizTalk tools: BizTalk server 2006 Scripts: Home

Developing Business Process and Integration Solutions by Using Microsoft BizTalk Server 2006

2006-07-28T17:07:00.000-07:00

I found out this week that I passed the BizTalk 2006 beta exam. I suppose this makes me a certified BizTalk Engineer. Woohoo!

The test was quite long (almost 90 questions) and was broken into the following sections:

1. Plan a BizTalk Application
2. Develop and Debug an Integration Application
3. Develop and Debug a Business Process Application
4. Implement Business Rules
5. Enable Business Activity Monitoring
6. Deploy a BizTalk Application

Those of you who know me will not be surprised in any way that the deployment section was my worst. It's hard to delegate deployment to someone else on a test! :)

Consultant's Rules of Engagement

2006-06-23T09:47:00.000-07:00

I was chatting with a colleague a while back and we were discussing ways to make our consultants feel more comfortable when they are dropped into a customer site. The biggest factor in their comfort-level seems to be how prepared they feel they are.

This totally makes sense to me as I realize now that the nature of the engagement is always one of the first things that I start investigating when I know I will soon be working with a customer. Ideally, the investigation yields enough details that I can begin establishing some expectation that focus on preparation appropriately. So what sort of expectations should I be looking for to maximize my preparedness?

That's where the Rules of Engagement come in. Here at Visiphor Consulting Services (VCS) we often use a military metaphor for our engagements. We are 'air-dropped' in, we are 'marines' who build the objective and move on, or we 'establish a beach head'. So, given the military context the concept of defining Rules of Engagement for our consultants seems natural. I know this term is not 'new' to consulting (as a quick Google search shows) but it very applicable to our consulting work.

Consultant's Rules of Engagement

Objectives
1. What is the vision or 'big picture' of the engagement?
2. What are the shared goals that constitute a successful conclusion?

Expectations
1. What are the customer's expectations?
2. What are our expectations?

Technical
1. What are the skills required to achieve the objectives?
2. What are the technologies involved?

Diplomatic

What is the political climate?

Who is the project sponsor?

What are the profiles of key players in the engagement?

A consultant provided with these Rules of Engagement should be armed with enough information to maximize their potential for success.

Team Foundation Server Administration Tool

2006-06-16T15:24:00.000-07:00

This tool looks very helpful for configuring security within Team System. There are three different applications that need to by 'synchronized' (Team Foundation Server, SQL Reporting Services, and SharePoint) and this tool helps by administering all three simultaneously.

Team Foundation Server Administration Tool

Team System Software Topology

2006-06-16T15:02:00.000-07:00

There are two important divisions in the Team System software universe:

Client
Server

The server product is called Team Foundation Server and provides the backbone of the Team System. An example of what it provides would be Source Control.

The client products are different flavours of Visual Studio 2005 based on roles
within the project team.

Team System Explorer

This tool is bundled along with Team Foundation Server and is
intended for non-techies to access things like work items.

Visual Studio Team Suite

The uber-product that contains everything for all roles

Visual Studio for Architects

Intended for Architects but really just seems to be a weaker version
of Team Suite (which is what all the Architects will want).

Visual Studio for Developers

Intended for Developers with a decent range of features for
developing software within Team System.

Visual Studio for Testers

Intended for Testers with some interesting testing features.

Visual Studio for Database Professionals

Intended for Database Developers with some special features within
the IDE specific to databases. Still no T-SQL intellisense,
though

There you have it! We are hoping to get a hold of Team Suite for our evaluation but we may have to settle for Developer since that's all we can get a hold of. The DVD trial ordering site on ms.com says Team Suite is back-ordered and there is no download offered.

Investigating Team System - Licensing FAQ

2006-06-06T17:05:00.000-07:00

We've been looking at this new beast called Team System for some time now and the time has come to jump in and start working with it. Due to the complexity and 'newness' of Team System we've decided to evaluate it by using it to develop an internal application. Also, since we are planning on using the 6-month demonstration version of Team Foundation Server we thought it was best not to undertake a customer engagement and run the danger of our demo expiring before we delivered!

But, perhaps I'm jumping ahead of myself. My goal with this blog over the next few weeks is to record the trials and tribulations of our Team System evaluation. Perhaps it will be useful to readers but more likely it will be a record of our discoveries for posterity.

There seem to be a couple of big challenges when starting a Team System evaluation. These are (1) Licensing, and (2) Application Topology.

Team System Licensing FAQ
Well, you know this is a bit complicated when MS has to release a 'white paper' describing how Team System (TS) licensing works! Here's a licensing FAQ based on what I've translated.

Q: Where do I get information regarding Team System Licensing?
A: Check out the Microsoft® Visual Studio® 2005Team System Licensing White Paper.

Q: I have MSDN Universal, surely I get Team System with it?
A: Short answer: 'not all of it'. Long answer: You do get Microsoft® Visual Studio® 2005 Team Edition for Software Developers and Microsoft® Visual Studio® 2005 Team Foundation Server Workgroup Edition. Unfortunately, these two products are really only suitable for small development shops and leverage only part of the Team System feature set. You need MSDN Premium to access more of Team System.

Q: Can I upgrade MSDN Universal to MSDN Premium in order to score Team System?
A: Short answer: "yes". Long answer: There are some options for upgrading your current MSDN subscription. Check them out here. Note: It looks like the best promotion ends on June 30th, 2006.

Q: What do I need to buy to use Team System?
A: You need one or more client licenses (Visual Studio), a Team Foundation Server License, sufficient CALs to support your user base (one for each user), and a non-MSDN Windows 2003 Server license. The client licenses come in a variety of flavours based on the 'role' of the team member. The best one is Team Suite since it covers all of the others but you can also get a Developer, Tester, Architect, or Database version.

That covers the main questions I had. Hopefully this helps!

I'll talk about Team System Application Topology in a future post.

When is InfoPath the correct technology?

2006-05-31T16:00:00.000-07:00

Lately, we've (the very talented Developer Group at my company) have been discussing when to use InfoPath instead of something more traditional like ASP.net. Here is the synthesized result of our conclusions based on past experience. Hopefully, it aids you in making the correct choice in your projects.

InfoPath is suitable for:

Simple to medium ‘form’ applications
Applications based on data-entry. i.e. validate/submit
Applications based on xml data
Applications that follow a pattern supported by the out-of-the-box GUI. This reduces custom-code considerably.
Applications requiring offline data entry/saving
Scenarios benefiting from a consistent Application environment – No cross-browser Issues
“No installation” simple applications – for simple data entry forms

InfoPath is not suitable for:

Applications based on data-management. i.e. download/upload/validate of existing data sets
Complex forms where significant work is needed to optimize performance by working around InfoPath’s foibles
Applications that have no need for the concept of ‘client-side’ data entry. i.e. reports
Time-sensitive form initialization – complex forms can be slow to load
Applications with a large user base. i.e. licensing issues – purchase 1000 copies of Office Premium
Non-Windows environments – Office Premium must be installed
"No installation” complex applications – complex forms require security certificates or installers