There is an interesting thread going on over at LinkedIn Answers where Venkat asks whether testing is a cost or an investment.
Some of the key thoughts that crossed me when I read the trail are:
- Be it a product or a services setting, software testing is an integral part of the development lifecycle. I would also go ahead and state that in cases (specially in services industry) where client does not pay explicitly for testing, testing should be treated as an internal investment that will help organization satisfy and in some cases, retain clients. Testing is directly related to an organizations reputation (quality of delivery).
- So, software testing is an important investment from organization’s perspective. For clients, it is a cost that should be budgeted when a project is planned.
- From Cost of Quality perspective, software testing is an appraisal cost which cannot be completely done away with. It can however be controlled by having good resources on project and making them accountable for unit level checks and basic integration testing. It also calls for streamlined requirements management and design practices.
- That brings me to another realization that the outcome of a tester depends on quality of developers/designers on the project. I have seen very good testers failing to perform if the development team is average. On the other hand, average testers have performed well on projects where the team was more mature and accountable for unit level checks.
- Lastly, defect trends should be closely monitored to improve upon the process - also to raise timely escalations on quality (raising alarm bells) so that client’s expectations can be managed early in the lifecycle.
These are some immediate thoughts to jot down - any thoughts to take this further ?
I would love to hear. Thanks again Venkat, for raising this all important question.
We all know that Quality on a project is everybody’s business and right processes have to exist to ensure that everyone thinks about quality - be it review mechanism or having a set of metrics. Around the same time that I was thinking of documenting my learnings on this topic, I came across an article “Quality Doesn’t Just Happen“over at CIO.com which also documents lessons on project quality (included below are the ones which resonate well with my own learnings) :
“Lesson #1: Don’t reward for shipping on schedule. Anyone can ship garbage. Base rewards on quality metrics
Lesson #2: Don’t reward heroes for their Herculean effort late in the project to fix problems that could have—and should have—been fixed by the same people much earlier in the lifecycle.
Lesson #5: It’s easy to ignore documents that are sent by e-mail for approval. No response does not equal approval: No response means, “I didn’t have time to read it.”
Lesson #6: Don’t start coding until the requirements are stable and understood, or else budget time for subsequent rework.
Lesson #7: Code isn’t “complete” until it works. Good unit testing is part of the development effort, not an optional item to be jettisoned when the schedule is tight.
Lesson #10: Buggy software takes longer to ship. “
Any additions that you can suggest to the list above? Help me while I think of a few more!
Definition of a bug by James Bach and Michael Bolton (via Techtarget.com) -
“A bug is something that bugs somebody who matters.”
Who matters? I think testers matter here, since they work for client. They are clients advocate. Project Manager matters because he works closely with client to fulfil their business need. He has to think for and understand client’s perspective. Clients matter, because the application fulfils their business need. Finally, end users matter, since they use the system. Just my $ 0.2!
Load Testing is one of the most difficult phases of development cycle - since it deals not only with the testing tools but also with the overall test infrastructure.
I suggest reading “The Ten Commandments of Load Testing” over at “My Load Test” weblog listing some of the very basic considerations on load testing.
The ones I particularly liked are:
Thou shalt have testable requirements. - Requirements specific to load testing is one of the most common pitfalls where a tester does not have any idea about the clients expectations. It helps to gather the client requirements and note them down so that they can be tested against.
Thou shalt test for the worst case. - Load Tests are generally planned for the average scenarios and not for peak ones. E.g A travel booking site may have 20 concurrent users on an average day and 100 concurrent users during vacations.
Thou shalt monitor your test environment infrastructure. - Load Testing should always be done after the application has stabilized and it should be done from a live environment. There could be many implications as far as deployment environment, hardware etc. are concerned and these factors should be carefully thought of.
Read the full post here for more details.
Michael Bolton at DevelopSense blog says that a significant question for software testers is:
“Is there a problem here?”
Great question for testers. In a thought exchange, one of our clients said “I am a firm believer in the fact that however long we test, there-is-always-one-more-bug. As a testers, it is very crucial for us to know where to draw a line. Obviously, when we complete a test cycle, there should be no show stopper but there-is-always-one-more-bug lurking somewhere in the application. It is the same “there-is-always-one-more-bug” theory that pushes a tester to think differently, to think beyond what an analyst or designer thought and challenge the system.”
I believe that combining the quest of finding a bug (”Is there a problem here?”) with “there-is-always-one-more-bug theory” can work wonders for software testers in coming out with findings no one in the project team would have ever thought of. I believe it is these findings that differentiate great testers from average ones.
A lot of software projects are confronted with quality issues sometime in the project lifecycle. Worst is the case when escalation comes from the customer.
My experience so far reveals that when a customer raises a quality escalation, typical approach of a project manager is to initiate rigorous checks and not release the application until each and every bug is identified and resolved. Luc Richard, in his article “The Separation of Church and State” (membership required) at GanttHead.com calls this “Project Mangler” approach. The downside of this approach is that while clients want quality, they would definitely not compromise on the timeline and features required. In the event of a quality related escalation, customer gets extra defensive and seek continuous feedbacks. Features (quantity) have to be delivered, timelines have to be met and quality has to be consistent. How can this be prevented in project planning phase?
Luc Richards suggests that we treat quality and quantity separately by virtue of the types of releases we provide to the clients.
Quantity can be dealt with in major/minor releases – where we provide tested features. That means Release 1.1 would have some more features added in Release 1.0 and so on. Major/Minor releases essentially deal with delivery of tested functionalities in incremental fashion.
Quality can be dealt with in maintenance releases – or “patches” which deal with bug fixes and providing intermediate releases. So, release 1.1 can have multiple maintenance releases in form of 1.1.0, 1.1.1 etc. (This does not mean that Major/Minor releases would not deal with quality - however as we know zero-defect release is a myth and defects coming from client can be dealt with by planning maintenance releases in the project plan)
The challenge is in planning the major/minor releases and the maintenance releases in the project planning phase. While quantity and quality are separated, planning for both happens together. The plan should also be revisited as the project progresses to add more maintenance releases as required during the course of the project. Planning can also specify what elements each of these releases would comprise of.
This strategy clubbed with right expectation management and communication can prove to be very effective. When each release is done, client expectations have to be set. Following parameters can be considered when making a release.
- Release Details (Major/Minor release, Maintenance Release etc.)
- Release Coverage (What all functionalities, features, issues, change requests are incorporated in this release - what all is not included)
- Known Issues in the release (open issues that are being worked upon)
- Dependencies/Deviations/Constraints
- Expectations (what client can expect and what is the purpose of release. E.g. an in-development release provided to the client for status update, a major release provided for UAT etc.)
Addressing the above parameters during a release sets the clients expectations and they know what to expect in the release at hand. I have experienced that a lot of post-delivery escalations can be effectively avoided with the strategy above.
Any ideas to take this one step further?
Testing can control quality, but cannot assure it. Quality Assurance, as we know, is much more than just applying tests on the projects - its about choosing, customizing and applying a methodology that is best suited to the project considering the project scope, complexity, technical risks and many other parameters involved. When we do QA, we think processes.
We associate testing with quality control - but when we do testing, are we really empowered to “control” the quality? After we report bugs, is it in our control to get them fixed? My experience states that the answer is “No”. Even when we do testing, we are not really contolling the quality - we are just reporting the state of application. This belief of mine found a strong support when I read similar views by Elisabeth Hendrickson who thinks of testing as an information service. Here is the snippet from her post Questions and Answers -
” My new insight was the classic: Testing can tell us about the absence of quality, but cannot ensure it. I still think that’s true, but that insight doesn’t guide what or how I should test. It fails to inspire me. It’s accurate, but not helpful.
My next realization was that Testing is an information service. Testers provide information that decision makers can use to mitigate risk and make better decisions about software projects. This insight explains why testers should not be the gatekeepers. We provide information, not judgment. We identify and explain risks. We act as advisors. We don’t make the news, we report it. And we shouldn’t ever accept the role of quality police.”
She hits the point when she states,
I now see testing as an information service that answers questions for project stakeholders. Or, bumper-sticker style: Project stakeholders have questions; testers have answers.
Excellent!
As we know, building quality into the software involves cost - I came across a detailed and useful article on “Quality Cost” at LogiGear Newsletter.
Author Rob Pirozzi classifies quality costs into four types: Prevention costs, Appraisal costs, Internal failure costs, and External failure costs. For software practitioners and quality professionals, it is very important to know these four categories of quality cost and decide where to invest most of the efforts to see that the other costs are balanced.
Rob also describes the activities that we typically do for each of these quality cost types. According to Rob,
Prevention costs represent everything a company spends to prevent software errors, documentation errors, and other product-related errors.
Appraisal costs include the money spent on the actual testing activity. Any and all activities associated with searching for errors in the software and associated product materials falls into this category.
Internal failure costs are the costs of coping with errors discovered during development and testing.
External failure costs are the costs of coping with errors discovered after the product is released.
Thanks Rob for a detailed article and analysis done.
How do we do that ? Barbara on Business Analyst Blog has an answer - “Write Better Requirements“.
This post throws up a very fundamental question : “If you don’t have excellent requirements, how do you evaluate the software’s ability to address the business problem?”
Poor requirement definition by the stakeholders and poor understanding (and hence documentation) of requirements by the development team are two major reasons why testers take so much time in testing (and also why so many software projects fail).
Everyone has suggestions about how to improve your testing—implement a testing process or methodology, utilize IEEE standards, work towards CMMI compliance, etc. No one mentions that improving requirements will improve testing!
Without testable and verifiable requirements, testing is always a difficult and time consuming job.
Earlier in by career, when I handled a lot of processes (based on industry best practices), I thought of best practices as a solution to day to day software issues from poor requirements to testing. I used to “go by books” when it came to implementing these best practices.
Gradually, as I advanced in my area of work, I realized that software projects “can” go wrong even after implementing available best practices for each stage of software lifecycle. Best practices are nothing but lessons people have learnt over a period of time and can be a useful reference - but not an off-the-shelf solution.
Wikipedia defines best practice as “a term which refers to the best possible way of doing something; it is commonly used in the fields of business management, software engineering, and medicine, and increasingly in government.”
Each software project is unique, in terms of client requirements and in terms of project requirements. A lot of things in software projects are contextual. By definition, best practice is a “possible” best way of doing something.
So, while its important to keep best practices as reference, more importance has to be given to the project requirements. Before we go ahead and implement a best practice, asking following questions might help:
- Will the implementation of this best practice really help us in gaining efficiency and addressing anticipated issues without compromising on project timeline, in context of project at hand?
- If yes, will this best practice be implemented as it is? (Though a rare chance!)
- If no, what parts/portions of a best practice can be implemented in this project to ensure that the anticipated issues are addressed and stakeholder / project requirements are fulfilled.
Answer to these questions might lead to effective customization of a given best practice. It might also lead to something new altogether.
The point is - best practices are processes that have been successful for some people in a given context. Hence they have to be related to the current project context and benefits have to be closely evaluated.
Also read: James Bach’s blog where he says No to Best Practices.
Managing Projects, Process Improvement, Software Testing | Tanmay September 26, 2007 | 
Comments (3) 
