Generic Dev Team Recommendation

A few months back, a friend and I were hired to lead a development team that was to be built from the ground up, to serve as the development arm for an established company that was rebuilding its development group. When we were originally hired, plan A of senior management was to staff the development leadership locally, and offshore the bulk of the implementation team. However, our own experiences in the past gave us the opinion that for this particular situation, the company would be best served by having a single co-located development team.

To be clear, everyone's situations are different, and what I present may not apply to your situation. I have a lot of extremely talented friends and colleagues in this business that live many time zones away, and I've worked on a number of successful projects with them...despite us being widely distributed.

But specifically, why is it so hard? (We are programmers, and communication is often not our strongest skill.) And how does being distributed impact the business? The goal of this opinion paper was to answer those questions, viewing the situation from both a high level (annual operational costs) and low level (a single feature).

---

ABC Development Team Organization Recommendation

We're currently facing an important decision in determining the ABC development team structure and organization. The goal is to provide ABC with a long-term development capability that is productive, high quality, and cost-efficient. We are currently considering from the following choices:

• Create an offshore development center, either independently or in partnership with existing relationships.
• Consolidate the development team in one location at the Minneapolis MN location.

Based on the situation and our experience, we recommend a single location for the development team. At a high level, this is because the cost of communication issues associated with distributed teams usually outweigh their advantages.

The Communication Issues Explained

• It's difficult to have quality design discussions with distributed teams. For example, ad-hoc whiteboarding is an essential process when communicating complex design concepts. In distributed teams, this is not feasible. Some workaround technologies exist, such as Microsoft LiveMeeting, but they are inadequate replacements because of their technical limitations.
• Limited non-verbal communication. Remote teams are restricted to the lowest quality forms of communication - written and voice. This leaves out much more effective styles of communication, such as gestures, body language, eye contact, touch, facial expressions, and posture.
• Limited window of communication. Speaking of India specifically in this case, the work day overlap is a couple hours. By regular work day standards (8-5), there is no overlap at all, but teams in this situation accomodate by adjusting their work schedules to work either early or late to get a small window where they can talk.
• Slow communication turnaround. Because of the limited window of communication, it takes additional time for issues and tasks to go through their feedback and approval cycle. A typical example is the following scenario: The project manager gives the developer a 2 hour task on a Monday morning, which is the end of that developer's day. On Tuesday morning, the developer has her detailed design questions answered for the task. Wednesday morning, the technical lead is able to review the initial draft of the development and finds some feedback items. By Thursday, the team can mark the issue as resolved and move on to the next task. Contrast this common four-day flow with a co-located team, where the task is resolved in one day.

The Benefits of Co-location

Co-located teams make software faster and more efficiently

Due to the differences in communication quality and efficiency, software with a moderate degree of complexity can be built faster and more efficiently when the team is co-located. It can be built faster due to the quicker feedback loop, as is discussed in the earlier slow communication turnaround scenario. In an ideal situation, developers working around the globe can create software more quickly. This scenario is rarely a reality though, because building anything but the simplest software requires signficant back and forth collaboration. The same communication turnaround scenario also explains how it can be built more efficiently when the teams are co-located. The following diagram illustrates how this happens in real life using a sample four-hour development task. 

In the above example, the distributed team gets started on Tuesday morning at 8AM, and finishes Friday at 10:30AM - an elapsed time of four calendar days.  The co-located team also gets started Tuesday at 8, but finishes on Wednesday at 4:30PM - an elapsed time of two calendar days.   The amount of work is the same in both scenarios, with one small exception.  The co-located developer discusses questions she has with the business analyst in person rather than over email.  As will be discussed later, this illustrates how you can develop software more efficiently with co-located teams, without doing anything differently other than choose how you locate people.  In this example, almost all of the elapsed time differences happen because of the timezone challenge. 

As a result, the co-located team finishes this task in nearly half the time as that same time in a distributed situation.  In reality, people will have more than one task assigned at a time, so this inefficiency can be mitigated to a degree by multitasking (assigning more than one development task at a time).  But multitasking has its own challenges, as context-switching itself drops efficiency levels.  There are also times when a person simply has only one thing they can work on at a time, due to project priority or constraints.

In a best case scenario for distributed development, people are always working on something and never waiting, and each task simply takes longer to complete compared to a co-located team.  In a worst case scenario, people spend 100% of their time on this task, and it costs the company four days of time and money to implement the feature as compared to two days.

The Cost Factor

There isn't much debate in saying putting people together in a room produces higher quality software compared with separating them. So why do most companies choose a distributed model? Resource availability and cost are the two most common reasons. As resource availability isn't a factor in this case, we're left with the potential benefit of cost in leveraging an off-shore model.

Can we make software that meets the goals of ABC in a more cost-effective way by leveraging an off-shore distributed model? Calculating this is difficult if close to impossible. For example, we can't do what-if scenarios for things like "how many customers will we lose if we build mediocre software?", or "how much is the business impacted if we can't accommodate change in the time required?". However, we can compare most operating costs quite easily. We can also roughly compare development and maintenance staffing costs by making some assumptions about how much additional waste is incurred in the process due to the communication issues.

For example, here are some predicted costs for a distributed team working in timezones with a differential greater than a working day where most development is done in one location and most leadership tasks done in the other. 

Contrasting the same task implemented by a distributed with a co-located team, what would be a 12-minute informal chat in a co-located team must be a one hour prepared call in a distributed team. Extra time is required to set up the means to communicate with remote collaborators.  For a verbal meeting, this means preparing an online voice or video session; for written correspondence, this means organizing questions or thoughts into complete and concise statements.  Hallway chats and quick brainstorming sessions are simply not feasible.  The closest tool we have for impromptu discussions when working remotely is instant messaging.  Again though, written communication is a weak method compared with other richer forms of communication, where things like body language, inflection, and eye contact can drastically affect the message.

---

Cost differential at a higher level

The earlier examples show the development process for a single typical task.  Another way to contrast costs between co-located and distributed teams is by their annual operational costs.  The following tables illustrate those cost differences:

	ABC-Controlled ODC		Subcontract ODC Resources		Local Resources
Major Categories of Expense Differences	Startup Cost	Yearly Cost	Startup Cost	Yearly Cost	Startup Cost	Yearly Cost
Variable Consulting Costs*		325,000		415,000		450,000
Additional Building Lease		10,000
Additional Office Equipment and Services	20,000	10,000
Telecommunication	5,000	1,000
Additional Administration		1,000
Additional Server Hardware and Licenses	60,000	15,000	60,000	15,000
	$85,000	$362,000	$60,000	$430,000		$450,000
		$447,000		$490,000		$450,000

* The Variable Consulting Costs category only shows the yearly burn rate for development. It does not factor in software quality or the amount of features built in that timeframe, which will be higher in a co-located environment.

The consulting cost for distributed offshore resources are lower than US-based resources.  In the example of using a ABC-controlled Offshore Development Center, the costs end up being slightly lower due to the additional equipment and maintenance.  When partnering for ODC resources, the additional equipment costs actually push the cost higher than operating with a single US-based team.

---

The Variable Consulting Costs in this table are derived from the following information. 

	Annual Cost
Resource/Role	ABC-Controlled ODC	Subcontract ODC Resources	Local Resources
Project Manager	40,000	60,000
India Admin	25,000
US Admin	50,000
Developer	30,000	50,000	65,000
Developer	30,000	50,000	65,000
Developer	30,000	50,000	65,000
Developer	30,000	50,000	65,000
Developer	30,000	50,000	65,000
Jr. QA	25,000	45,000	50,000
Jr. DBA	35,000	60,000	75,000
	$325,000	$415,000	$450,000

The leadership roles are excluded from this table, as they are fixed in the sense that they would be necessary in any case.

Co-located teams make higher quality software

All other things equal, a software team in a single location will produce software that conforms more closely to the intended features, and has fewer defects. To quote Alistair Cockburn's research on the subject:

Osmotic communication makes the cost of communications low and the feedback rate high, so that errors are corrected extremely quickly and knowledge is disseminated quickly. People learn the project priorities and who holds what information. They pick up new programming, design, testing, and tool handling tricks. They catch and correct small errors before they grow into larger ones.

By having a better understanding of the requirements, it can be inferred that the resulting software will also be more fault-tolerant and maintainable. More fault-tolerant because the well-known high risk areas can be engineered more defensively. More maintainable because fewer mistakes go into the design and construction.  Cheaper to maintain because business changes cost less to implement.  Attempts have been made to quantify the long-term effects of quality on a project, but these metrics are still too subjective and unreliable to be used as an argument in this discussion.  

Summary

We've depicted how the reduced cost of a less expensive developer can be offset by inefficient communication.  We have also shown that it is likely to take considerably longer to complete a project when using a distributed team.  However these examples do not consider software quality, which has lasting effects on cost - from maintenance to scalability, and often with greater consequences than initial cost.

Given our conclusion that the project will cost roughly the same with a co-located team, and have a higher productivity rate with increased quality, it is our recommendation to use a co-located team for this project.