Daring

Developer

This is the first article in a series about exponential engineering (EE) : A daring methodology for extreme productivity.

Part 1 : Building the Team

Part 2 : Use and Abuse Reuse

Part 3 :  Building In-house Frameworks

Part 4: Domain Specific Design Patterns

The impossible project

In my 2007/8 role as an architect for a software startup, I was given the task of creating a point of sale (POS) system for a major retail chain. By normal corporate standards a team of 6-7 developers, BAs, architects and QAs would be hired for a term of 18-24 months. I was daring enough to promise the job will be done in 12 months with a team of three developers (a budget average of 36 developer man months). Although this was seemingly an impossible goal by normal standards I had secret weapon up my sleeve.

To make matters worse, one month into the project we were hit with a surprise retreat of funding which meant I had half the budgeted resources : one junior and one senior developer. Despite the lack of resources we completed a working release (albeit with some rough edges) in 9.5 months (team hired to first release).

After delivering the first product we were hit with very late scope creep : The customer had not involved all their people in the initial requirements phase so now some key people were being asked what they needed for the first time!  This increased the function points by 25% as the features were of high complexity (many beyond what any commercial POS system offers).  Due to the complexity of feature interaction between these already complex features the project workload increased by 50% beyond the original plan. The project was now estimated at 54 man months.  The expanded product was delivered in 18.5 months (team hired to pilot release at the first store).

It took one and a half months for the customer do a stocktake, enter additional product data and finally print labels so during this time we added self diagnostics features to the product and implemented a live monitoring and support infrastructure. A total of 20 months elapsed to go live. In the end we wrote 84,000 lines of code (51,000 Groovy + 33,000 Java).

In summary a 108-168 man month project was planned for 54 man months and delivered in 37 man months with half the resources. This result could never have been achieved by working harder and longer alone, rather it was due to three deliberate mandates of the project from the beginning :

• Agile project management
  
  
• Pragmatic programming
  
  
• Exponential Engineering
  
  

Agile methods and pragmatism in development are topics with an abundance of material so they're best left to other sites. What I want to explore in this series of articles is the technologies and techniques that can empower a small team to become incredibly productive.

Its all about productivity

The way I measure productivity is by business features deployed not by how much "work" has been done. There is a good article about this called "Its not about lines of code".  There are many factors that improve an individual developer's productivity including :

• Knowledge of a specific technology (the lower the learning curve the more time saved)
  
  
• Experience with a specific development paradigm (e.g. model-view-controller for UI development)
  
  
• Discipline to follow good low-level coding techniques (e.g. generalisation-specialisation using classes)
  
  

(a) and (b) only reduce the time for a developer to "come up to speed" initially which has no impact on long-term development and maintenance costs. (c) might actually decrease the developer's productivity during the initial stage but on the upside it can reduce the costs of development and maintenance by say, 20-30% over the long term.

These are all good things but they have a limited impact on overall productivity. (a) and (b) are nice to haves but not necessary. (c) is in fact essential for EE - but its not going to give us the extreme 2-3 fold increase in productivity that we're striving for.

What is Exponential Engineering ?

• Logarithmic reduction of the time to deliver new functionality reaching a limit of constant low-cost.
  
  
• Logarithmic reduction of the on-going costs of maintaining software reaching a limit of almost zero cost.
  
  

The more this logarithmic decrease is a achieved the more time is freed for the development team to focus their energies on innovation. This is a win-win situation for all involved : The business gets a better product sooner and the developers get rewarded with less stress and overtime, more time to work on the exciting parts of the product and opportunities for new and stimulating challenges. Sounds great, but how?

The answer is reuse and I will be talking in a lot more in depth about it in the next article : Part 2 : Use and Abuse Reuse

The software engineer vs the technologist

Extreme reuse and productivity does not happen easily.  Just to enable the potential requires quality software engineers focused on productivity. To make EE work you need a special kind of developer.  It is far far better to have disciplined software engineers with limited experience than experienced technology specialists who write code they way most corporations do : inefficiently.

Its funny when you realise that most companies recruit people based on specific knowledge and experience - I like to call it the "pattern matching technique" when agents decide who to promote to a prospective employer largely based on the number of acronyms in common between a resume and the job description! I guess this explains part of the reason why corporate IT is so inefficient.

I like to illustrate the range of "IT professionals" with this diagram which - I know - is a blatant over-simplification of the uniqueness of people but never-the-less, is quite revealing about the core problem with mainstream development :

           

The vast majority of developers are heavily inclined to climb the Y-axis. They are hands-on with a "material" technology such as a popular framework, they collect "knowledge" about specific tools, become very "fast" operating within that particular environment and gain a lot of "experience" with the quirks of a related set of technologies. All this leads to a specialist who's head is stuck in a single paradigm and they often find it difficult to "unlearn" what they know and have repeated for so many years.

This "specialist syndrome" is reinforced by the IT industry as most corporations prefer to structure their teams vertically (i.e. distinct technical positions) and therefore recruit technical specialists and reward further technical specialisation.

On the X-axis are the really important attributes that are critical for quality software engineering and saving time and costs in the medium-term. Unfortunately these attributes are rarely found, partly because the IT industry "system" discourages them and partly because, in the case of "studious" and "disciplined", it is human nature to be lazy and neglect making the effort.

To make matters worse there tends to be an inversely proportional relationship between the remaining two attributes:

• Strong material people are rarely conceptual thinkers and the conceptual types often have a longer learning curve for material things.
  
  
• Fast working people tend to be undisciplined and disciplined people tend to be slower because they're perfectionists.
  

The good news is that there are talented software engineers (SEs) out there who love to learn continuously, are disciplined in their work and have an aptitude for the conceptual. The challenge of recruitment is to find the software engineers who are also "average technologists" - what I call real software engineers :

• Real SEs don't need to be super knowledgeable and experienced : if they are talented and studious they'll come up to speed quickly.
  
  
• Real SEs, however, do need to have at least an average aptitude for understanding material technologies. Then, since they're good with concepts they'll learn specific technologies well enough to do a quality job - they don't have to be a technology expert.
  
  
• Real SEs will generally be slower at tangible (human observable) things such as typing, coding, even speaking. - but the productivity gains from successfully realising EE will outweigh any negatives a hundred fold.
  
  
• Real SEs are often introverted and may lack "normal" social skills. As a recruiter you have to ask your self this question: " Do you want to hire a buddy you like to socialise with or someone who has this weird passion to solve the most difficult parts of the job you need solved?".
  

        

Real software engineers are usually frustrated working in the typical corporate IT department which means they will be easier to recruit away with the promise of a creative engineering culture.

Don't forget the team

The other side of the productivity coin is teamwork. Many development projects fail due to dysfunctional teamwork despite the talent of the members. There are many good techniques to improve a team's harmony and combined productivity through reducing overheads and removing hinderances but I don't want to go into all those. I just want to mention a few ideas that I think are key to productive teamwork with the EE style :

1. Develop a common verbal and written jargon language for the project in the very beginning - A  consistent semantic terminology between team members (including BAs) removes one of the major obstacles to efficient communication.
   
   
2. Mold the roles to fit the strengths of the person (The "strength finder" books are particularly good to help members of the team to discover their talents)
   
   
3. Build a culture of fun and excellence
   
   
4. Strike a smart balance between laws and freedom (e.g. enforce certain design patterns to be reused but give freedom for developers to challenge and therefore improve the patterns)
   
   
5. Design "Islands of Freedom" into the project that allow developers to work independently with minimal interference interaction from others. This is achieved via architecture techniques such as software layers with well defined interfaces and testing techniques such as drivers & stubs that break the dependencies between development tasks.
   
   
6. Develop a willingness and sense of responsibility in each team member to sacrifice "What should be done" for "What has to be done" to make the deadlines.
   
   

In the follow up article will discuss about the core power of exponential engineering : Reuse

Part 2 : Use and Abuse Reuse