Learning from Mistakes

I attended a brown bag conference call hosted by the Industrial Research Institute on the topic of "Learn from New Product Failures". Our speaker was Jim Hlavacek, one of the authors of the paper by that name published in Research-Technology-Management. They have had experience in reviewing failed projects at manufacturing companies based on techniques used in the medical profession. In many teaching hospitals, when there is an adverse patient outcome to treatment, the clinical team undergoes a Mortality and Morbidity Conference, where the course of treatment is reviewed by an objective team and mistakes identified.

Hlavacek reported experiences at companies like Intuit, Toyota, and 3M where a regular process of review is built into the engineering/marketing culture.

Under Hlavacek's approch the Failed Product Review (FPR) would consist of the following:

Background

Name of the failed venture

Dates project began and was terminated or shelved

New Venture leader and cross-functional team members

Objective and qualified principal investigators

Inputs

Face-to-face interviews with people who were particpants on the project

Face-to-face interviews with OEM and end-use customers who were involved

Face-to-face interviews with distributors/dealers and/or key suppliers

Obtain all e-mails, business plans, documents, trials, and project presentations

Methodology

Develop timelines and milestones of critical events or decisions

Doucment the unfavorable outcomes with data

Develop fishbone diagrams for the project and processes

Develop root-cause analysis of the fishbone diagrams

Recommendations

What went well for the project

What went wrong for the project
Lessions learned and corrective actions

This approach to learning from mistakes reminded me of some similar approaches I am familiar with. Those of you with a military background may have participated in an After Action Review. This is used primarily during training exercises to understand what happened, evaluate everyones performance, and discuss what could be done better. From the USArmy manual comes a similar outline for an AAR:

Introduction and rules

Review of objectives and intent

Training objectives

Commanders mission/intent

OPFOR commander's mission/intent

Relevant doctrine, tactics, techniques, and procedures

Summary of recent events (what happened)

Discussion of key issues

Chronological order of events

Battlefield operating system

Key events/themes/issues

Discussion of optional issues

Soldier/Leader skills

Tasks to sustain/improve

Statistics

Others

Discussion of force protection (safety)

Closing Comments

The final example is from my experience as a Certified Scrum Master. Under SCRUM a small team produces a product release using a succession of short time boxed miniprojects called Sprints. Sprints typically take anywhere from 2-4 weeks for the team to produce a functional version of the product. After every Sprint the team gets together for a Restrospective Meeting. In this meeting the team discusses the following:

What worked well last Sprint that we should continue doing?

The practices that worked well during the previous Sprint should be identified and continued in the coming Sprint.

What didn’t work well last Sprint that we should stop doing?

The team or customers should identify practices that worked against the team during the last Sprint and focus on stopping those things during the next Sprint.

What should we start doing?

The team identifies practices that should be implemented during the coming Sprint that will help them work better together.

Out of this discussion comes a list of actions that the Scrum Master captures and and is responsible for implementing during the next Sprint.

So lets compare/contrast these three approaches:

1. The FPR is conducted when things go wrong. The AAR and Retrospective occur for all outcomes.
2. Both the FPR and AAR require the participation of one or more objective reviewers. The Retrospective depends on the team and, sometimes, invited guests.
3. The FPR and AAR both use detailed analysis to find root causes. The Retrospectice is more adhoc.
4. The AAR assumes that particpants will change behavior based on issues being surfaced, the FPR has a deliverable of lessons learned but no clear followup for change, and the Retrospective has a set of actions with the Scrum Master responsible for seeing they are implemented immediately.

In my opinion, the key to making any of these techniques work is to have a culture of trust surounding the proceedings where everyone understands that people make mistakes and that for the majority of participants mistakes that are surfaced will not be used to punish them. I say majority, because if the same individual is constantly exposed as making repeated mistakes and not correcting behavior then it may result in termination.

Over my career, I have participated in and led a lot of project/product reviews. Almost always, people are defensive and guarded about what happened. To set the right tone in establishing a review program executives should lead by example, being willing to have their actions reviewed and also demonstrating with their subordinates that mistakes they make are not used to influence annual appraisals.

Pampered Pooch 2

I am a sucker for a "59 Minute Scrum" hosted by Bob Galen. I had participated in one last November as documented in this Blog. Last night the IIBA hosted Bob and I was again a member of a six person team producing a brochure for the Pampered Pooch Day Care. I wanted to get another feel for the team dynamics during the sprints and to compare the results with the last session.

Here are my observations:

1. While the aRTP session was comprised mainly of programmers and the IIBA was comprised of business analysts (duh), there was little difference in the results of producing a brochure. I guess if Bob ran a session at the Pet Care Services Association the outcome might be different.

2. Before we started the Day 2 Sprint, Bob pulled the four Scrum Masters aside and told two of them to go back and emphasize the quality and completeness of the brochure, and told the remain two Scrum Masters to tell there teams to push for as much content as possible in the time remaining. The results were telling. The two teams pushing for Quantity delivered 13 and 8 user stories respectively, and the two focused on Quality delivered 6 and 4 user stories. So teams will listen to the direction of the Scrum Master. Ultimately to have a released product both the Quantity and Quality need to be good enough. So is it better to get lots of 60% quality content in early sprints and then tighten it up all at once towards the end of the iteration? OR do you push for 80% quality content and achieve less content per sprint. A real trade off that the team needs to decide based on coupling/cohesion of the user stories. If the stories have few dependencies then push for the higher quality per sprint. With lots of dependencies you need the total content present to debug and refactor.

Enterprise 2.0 Conference

I had some local commitments so I could not fly up to Boston again (See earlier post on Johnny goes to Harvard) and thought I would miss the Enterprise 2.0 Conference Bummer. Little did I know that these people are trying to practice what they preach. I joined the Twitter stream #e2conf and am getting real-time tweets from all over the conference. Also there is a blog where I can read material from most of the presenters and see the opinions from other attendees.

Finally there is a e2TV video stream of the general sessions and vendor demos.

This morning I participated in the Launchpad contest. Four vendors who were finalists got a chance to do a quick demo for the audience and then the audience voted for a winner via SMS.

The four finalists were:

Bantam Networks - they have a enterprise project workspace that allow team mates to communicate, share info and manage relationships.

Brainpark - they also create a social network for projects. The difference seems to be an engine that suggests to the user people who have skills that might help, docs with info that might help, feeds/links with info that might help.

Manymoon - another social network for developers. This one has a good integration with Google apps and with external participants.

YouCalc - this is a different one... an analytics environment that can pull info from a lot of different sources and present graphic analysis. Also it is a product based on "wikinomics". The apps are created by the user community. If you create an app it must be made publicly available for others to use modify. The data that is analyzed remains private.

So the voting took place (I thought Manymoon was really good) and the winner was YouCalc with 53% of the vote.

As I am finishing up this post I am listening to a demo of Lotus Live from IBM. Last night they won the big Buyers Choice Award for best product of the show. This was a vote by attendees.

This type of virtual conference experience still lacks the level of deal making / networking that can happen in a f2f environment but you can't beat the price (FREE) and not having to sit on an airplane and then that ride in from Logan airport (Ugh).

It's Raining, It's Pouring

I attended a Webinar today where IBM discussed their Cloud Computing initiative including their "Cloudburst" offering. David Dworkin of the Tivoli business unit took the audience through

justifications for going to cloud which included a survey conducted last year of companies that had implemented a cloud application. The top three reasons for going to cloud for these companies was:

1. Innovation
2. Time to Profitability
3. Reduced costs

IBM is recommending that companies first move to internal clouds that reside safely inside the corporate firewall but consolidate various departmental applications. They claim such a move will have following benefits:

• Can reduce IT Labor costs by 50%
• Can improve capital utilization by 75%
• Reduce provisioning cycle times from weeks to minutes
• Can reduce end user IT support costs by 40%

In my opinion it seems Amazon EC2 is more SMB start-ups with quick roll out and low up front costs while IBM is aiming at Fortune 500 with large IT budgets under pressure.

During the webinar the host asked the audience (I did not see how many were attending) a couple of survey questions...

Which best describes your organization's level of adoption of Cloud Computing services? None, but not evaluating. 33.8% None, but currently evaluating one or more services. 27.4% Currently getting ready to trial a Cloud Computing service. 9.6% Limited trial adoption of one Cloud Computing service. 14.5% Currently running one or more crucial set of business tasks through the Cloud. 14.5%

David thought this was a little surprising compared with survey results IBM sponsored last year. He speculated that companies may be using clouds without being aware of it.

What are the biggest reasons your organization has yet to migrate any services off to the Cloud? Concerns over security 38.0% Need to "own" and manage the data center 19.0% Regulatory obstacles 12.0% Management does not see the potential for quick ROI 13.0% No skepticism, just looking for the right solution 42.0%

Of course remember that this population had already self selected to having enough of an interest in cloud to invest time in attending the webinar so these answers do not represent the general market.

Web2.0 revisited

On June 4th I presented my Web2.0 briefing to 21 participants representing 17 companies at an event hosted by Matrix Resources. Matrix offers briefings to its customers as a complimentary service and from the comments I heard as people were settling in for my talk it seemed to be a much appreciated program.

During the event I captured some informal statistics from the audience on specific Web2.0 usage patterns.

As a percentage of participants how many:

Use Wikipedia?- 100% as reader. 0% as author.

Author a blog? - 0%

Use Digg? - 10%

Have a LinkedIn Account? - 76%

Have a Facebook Account? - 76%

Have used Craigslist? - 76%

Participate in Second Life? - 0%

Use Twitter? - 24%

Use Ajax to develop apps? - 15%

So what does this mean? Like most statistics with small samples... not much. But I like to ask people and see if any trends are appearing that are different then the official surveys.

Also during the briefing we had a lot of discussion about how companies developing web2.0 apps are realizing revenue. The table below is my analysis of some of the more popular companies I mentioned in the talk.

I have added this table to my Web2.0 presentation page 32. In general I found that the strategy for most of these companies is to give the functionality away for free and rapidly grow a large user base. As the application matures and more users are locked in they obtain revenue streams through a combination of advertising and premium services.

Johnny goes to Harvard

Last weekend I attended the Deep Agile 2009 conference at Harvard University. This two day conference was put on by Agile Bazzaar, an ACM Chapter dedicated to the improvement of all things agile. Approximately 90 attendees participated in the conference and I thought that this was a very well run production. Kudos to the Agile Bazzaar volunteers and especially Nancy Van Schooenderwoert who chaired the program. 

The links above give the highlights from the events and I only want to add my own observations:

Jack Ganssle represented the non-agile development community and gave several presentations on his approach to embedded systems development. He favors object oriented development and follows Bertrand Meyer's Design by Contract process.

James Grenning was one of the original signers of the Agile Manefesto  He says he attended that event for the skiing but I suspect he had more involvement... He is a strong advocate of Test Driven Development, Pair Programming, and SCRUM. He showed all these elements to the audience but in some cases they were the regular versions without a significant embedded twist.

Russell Hill is a development manager at Key Technologies. He is an advocate of Test Driven Development for embedded and thoughout his presentations talked about his experiences at Key building a reusable set of boards for the various products Key manufactures. This system includes hard real-time behavior developed for an FPGA connected to a Motorolla micro processor handeling the UI and control logic. He brought a valuable perspective to the conference on what is achievable for embedded development using agile.

Here are my notes from a panel session:

In practice how can a HW based system be delivered incrementaly?

James - I don't do a lot of HW dev BUT "is it working?" is a good test.

Russel - Our HW is based on FPGA and has some flexibility. But boards would be developed incrementaly.

Jack - There is a lot of religion in agile.... HW tends to be late and broken.... and we don't anticipate that for SW

Do you expect to modify an embedded system on a 2-3 week sprint cycle?

James - would like visable progress without neccessarily being deliverable

Russel  - Key Technology can turn around a HW change very rapidly (2 days!)

When an embedded system is used for a consumer product who is the product owner?

Russel -  We have thousands of customers, Marketing dept, Field Serices represents the interests of the customer. Nancy - any problems with that. Russel - Sometimes. Only in last couple years has marketing been strong participant.

James - this is an organizational problem (of getting participation)

How to reconcile "HW requires long lead times" vs "Agile is incremental" ?

Russel - Board was not available for a couple years. SW was written to spec and was able to deliver 2 weeks after HW release.

Jack - Up front commitment to HW features and to the schedule is different for embedded.

James - At the beginning of an agile project one needs a vision of both HW and SW.

How do you handle scheule constraint.

James - in agile nothing is negotiable until it is late.

Russel - when we are late our internal customers now about it real soon. Visibility to decision makers is important.

Jack - This is not unique to agile. Quality, Schedule, Features pick any two. Jack thinks features should be unconstrained while keeping Quality and Schedule fixed.

Any experience including HW engineers in the SCRUM.

Russel - intersted but not active daily 

James - integrate early and often. Some teams use HW engineers as customer. One client in Finland does overnight board turns.

For an embedded system can a product backlog include User Stories that target either HW or SW? If so, what does a HW User Story look like.

Russel - We have not done that.

James - if you try it write a paper.

Jack - I think it would be very difficult.

I am concerned about the lack of upfront design. What if a User Story requires a big change?

Russel - we just experienced this with a project that required a substantial redesign. The requirement was given a year ago on an eight year project.

In classic agile the key to development is decomposition of the epic story into slices. How do you decompose in embedded?

Russel - Frankly we struggle getting our HW engineers to think agile but it is getting better. We had one example where a story to eliminate a spurious image required both sensor, hw platform, and sw.

What role if any does architecture have in agile?

James - we work on it every day. Every time a sprint is completed there is an architecture that supports SW completed to that date.

Russel - Some of our architecture is harder to change. 

Jack - Architecture and Design in embedded world is a lot less maliable. Up front design is very important.

Nancy - Some companies have a culture of detail which is driven by politics.

My question was the one about HW User Stories.... I am still looking for a way to weave the EE participation into a high tempo devlivery using agile approaches. I will take James up on his suggestion to write a paper when I have it all figured out.

mobile check-in

As a road warrior I am always interested in anything that will improve my experience in the airport. I recently saw where Delta Airlines and the TSA have teamed up to trial a paperless boarding pass at the Memphis airport.  The traveller can download an electronic boarding pass that is displayed on the device. The TSA has a 2D bar code scanner that verifies the boarding pass and Delta uses its current gate scanner. So as a time saver, the time at security and at the gate is not reduced, however I save the time in line at a kiosk to pick up a boarding pass. 

Getting the TSA and Delta personnel familiar with the mobile device is a good step to what I hope will be the next big change... Near Field Communication. Instead of using an optical scan the NFC enabled device can be waived over the reader and convey equivalent information as the 2D Bar code. So why is that an improvement? 

1. In an NFC enabled device the eBoarding Pass is kept as encrypted content on a separate chip that is more secure then the device general memory. Hackers can attack a mobile device in several ways and could steal or corrupt the information. The NFC devices resist unauthorized access.

2. NFC will be used primarily for electronic replacement of credit/debit cards. Using the tap and go ISO 14443 based interaction will become second nature to users of the device.