Daring

Developer

This is the second article in a series about exponential engineering (EE) : A daring methodology for extreme productivity.

Part 1 : Building the Team

Part 2 : Use and Abuse Reuse

Part 3 :  Building In-house Frameworks

Part 4: Domain Specific Design Patterns

In the first article of the series I talked about the attributes to look for in hiring developers and some nuggets of wisdom to enable the team to be productive. In this article I am moving onto the topic of reuse.

In EE there are three steps to "reuse" in order to reap the productivity benefits fully  :

• Find or design technologies that make developers more productive individually.
  
  
• Implement your own generic software artifacts that reduce the quantity of and simplify the code.
  
  
• Find or create design patterns that fit the specific problem domain.
  
  

In this article I'll introduce the concept of reuse and then explore the first step of reuse : how to make hard choices in finding or designing reusable technologies . In the following two articles I will talk about the latter two steps.

Why reuse?

If you're reading this then I don't have to convince you that productivity is something desirable to reduce the time and costs of a software project.

By far the most effective way to increase productivity is to decrease the work . "But how can I decrease the work?" you ask. Understand that the quantity of "work" is not proportional to the quantity of function points or deliverables because there is always some degree of commonality between function points in a large project. Taking advantage of this inherent commonality is the trick. As an architect or engineer you have to find the commonalities and see them as opportunities for reuse. This nugget of wisdom comes to mind:

• " The best line of code is the line of code I don't have to write"
  
  

You're saving yourself time, every time, you reuse a line of code instead of writing it again with a slight difference. The savings really add up over the course of a project and you can get an exponential increase in productivity if reuse is done well. However, nothing comes for free : to get a good return you always need a good investment. In EE there is a big, but good investment in "reuse" during the first stage of the project. 

It may seem to management and the customer like progress is very slow during this investment stage because they are measuring your progress by tangible business deliverables not by the intangible software deliverables which must precede the tangible ones. Investing in extreme reuse will pay off provided everyone is faithful enough to pay for the investment and patient  enough to wait for the benefits.

What is reuse?

There are many forms of reuse. In the material sense reuse can result from using libraries, frameworks, components, design patterns and languages just to name a few. There are at least five material types of reuse that are observable in a project for example :

• Analysis: e.g. Use-Case scenario patterns.
  
  
• Design: e.g. Object-oriented design patterns.
  
  
• Source code: e.g. In-house frameworks, components, open source.
  
  
• Binary code: e.g. 3rd party libraries, runtime environments.
  
  
• Tools: IDE, plug-ins, compiler, project management application.
  

Focusing on executable software artifacts there are many levels of reuse such as :

• Executable Specifications.
  
  
• Framework/Library.
  
  
• Software Component.
  
  
• Class/Abstract data type.
  
  
• Function/Method call.
  

Finally it can be helpful to categorise reusable artifacts into to broad camps: the vertical and the horizontal. "Horizontal" means it is a general purpose need that most projects have. "Vertical" means the need is very specific to one problem domain (but still generic within the confines of that domain). Most products are horizontal because there is obviously a bigger market for that. Vertical products also exist but tend to be more expensive due to the restricted market and more complex due to the vendor's attempt to generalise the product for a broader market. A good example of vertical reuse is a business design pattern and a good example of horizontal reuse is a user interface framework such as Google Web Toolkit(GWT).

• Commercial 3rd party (no source code, indirect support and maintenance).
  
  
• Open source product (access to source, community based support may be limited)
  
  
• In-house (full control of source, supported by the team).
  
  
• In-module (full control of source, implementor and reuser are usually the same person).
  

Obviously full control is better but we also can't afford to write everything ourselves. An appropriate balance must be reached on a case by case basis. The question then becomes how to choose between a short list of competing technologies for a given need and how to know when to and when not to roll your own?

For general needs such as database and user interface your only realistic answer is 3rd party products.  Usually, the best choice is to find the simplest solution that meets your foreseeable needs regardless of whether its commercial or open source and also keep a look out for powerful features that can add value to your product. In my most recent project I used a combination of commercial products for database and reporting, and open source products for user interface and inter-process communication. In each case I chose the technologies based on their ability to increase productivity in the project. (see An Agile Development Platform for an example of a complimentary set of simple technologies that has been proven in a commercial production environment)

The most important nugget of wisdom to take home is this : "Avoid highly complex technologies". Many tools and frameworks try to be all things to all men and fail at being good at one. Often a simpler technology that does all you need and most of what you want is far more productive than one that does all you want. The reason is that the complexity (and therefore lines of code and bug counts) rise as the technology is made bigger and more generic. This is one case where generality is a bad thing because it sacrifices simplicity. As a bonus often the simpler technologies are cheaper - but don't make price a criteria because it has an insignificant effect on the project costs compared to productivity.

For more specific needs there is a multitude of choices. This diagram can help you identify where to focus you search for each need :

The more general the need the more likely you'll find a 3rd party solution and the more specific the need the more likely you won't. If you can find a 3rd party solution that fits your needs well, great! Otherwise you have to consider fully or partially implementing your own solution. If your need is very specific you won't find a reusable solution out there - you'll have to roll your own.

Use me but don't abuse me

In many cases your needs are similar enough to an existing solution that you think  "I should be able to reuse this" but also dissimilar enough that it would cause problems for the project as well. In these cases there is a middle ground between using the technology wholesale (abuse), and rolling your own custom solution.

If its an open source project consider if its worthwhile making the changes you need and contributing it back to the project. There are a couple of dangers to watch out for here :

• The core contributers of the project are heading in a different architectural direction to you.
  
  
• The core contributers are not co-operative enough to support your efforts.
  

If its a commercial product (or an unfriendly open source project) then you have two strategies to consider. You can choose to do either one or both :

• Wrap the product in a thin API layer. This can do many things for the project such as insulating it from change in the product and simplifying or improving a bad  API.
  
  
• Extend the product to add the missing functionality you need.
  

Finally, if all else fails then you fall back to rolling your own solution. In some cases you may be able to find a related open source project with an Apache or MIT license or even a public domain license that allows you to use a modified version of their source code without having to open source your own project. This will give you a head start and save time.

In the following article I will delve into rolling your own framework and how to make it pay off big time.

© 2009 Keith Foster. "The Daring Developer" was created for people who share an insane excitement about advanced software engineering.