Agile vs Quality
If you would be asked or granted to fly to the Moon, utilizing a new carrier rocket – one developed and tested by a Team following the Waterfall Model-Approach, the other one by an Agile-Team – which one would you choose for your trip?
I want to reflect upon the both Development-Methods from a point of view of an experienced QA-Engineer, focusing on the aspect of Quality.
Disclaimer: The views, perceptions and opinions expressed in this article are my own and do not necessarily reflect the views / opinions / positions of other protagonists involved, or mobile.de as a whole. Furthermore, I want to state that I feel lucky to be a part of this technical odyssey, working in a great environment with brilliant people.
The Agile way of doing things had promised a Revolution and found a lot of enthusiasts, born Evangelists and faithful disciples.
A new, simple and lean method of developing Software, getting rid of Waste, getting things done quickly, being able to readapt swiftly to changes – sounds very promising indeed!
The Agile way of doing things had pledged relief and salvation for all those who dislike following Guidelines, writing Documentation and working in a disciplined, structured, predefined way.
The main focus is lying on Velocity, and therefore the Test-Automation had been declared as a universal remedy.
Some of our Evangelists/Councils had even claimed/believed, that a catch-all Test-Automation would make the QA-Engineers obsolete – but this myth had been dissented by reality.
We had practiced Scrum, Lean, Kanban and more, striving to achieve an optimal throughput, eliminating Waste, getting more automated, speedier and more effective.
At the same time, the Product-Quality had become secondary to the Release-Velocity.
For me as a QA-Engineer with a strong focus on Quality, the turn to an Agile-Approach had been coupled with pain. Firstly, the QA-Team had been dissipated and declared obsolete. The new magic cool thing was a Cross-functional-Team, utilizing an embedded QA. The new dogma proclaimed (apparently resting upon the Agile Manifesto) – no Documentation (declared as Waste), no defined Processes, Structures, Guidelines etc.
The experts of our Agile-turn had identified the QA-Engineers as a Delivery-Process-Bottleneck – apparently the Testing-Phase took too long, the Quality-Level demanded too high. And this Problem should be rigorously eliminated.
The so called Waterfall -Model-approach had been declared as the Mother of all Evils and was banned.
The explorative manual Testing had been announced as inferior, time-consuming and inaccurate - the DevLeads wanted to see more Test-Automation.
Ideally, everybody should automate – the Product Owners (PO) as well. Concordion & fixtures had been introduced to motivate POs to create a living Documentation in a given-when-then-style.
Trainings had been arranged, resources obliged exclusively just for Test-Automation.
The idea with the POs writing Tests had been futile though – it bore no fruits, the POs weren’t enthusiastic about the matter.
Albeit, many new automated UNIT/Frontend-Tests had been issued by Developers and QA.
High Test-Coverage had become a desired aim, which was a good thing.
Eliminating Waste-Initiative proclaimed the Product-Documentation, Code-Documentation and Process-Guidelines as irrelevant and dispensable – as lastly, the information required (for Devs and QA) should be extractable from the Backlog-Items filled by a Product Owner.
The Wind of Freedom had been blowing thru the sacred Developer Halls, the chains of strangling discipline had been dissolved.
At the same time, we were striving for a Team-Throughput-Optimization, following the Kanban-ideology. The idea – regulating Team-throughput by limiting/controlling the number of tasks in Development/QA. The idea was good, as it had introduced a rule (!) – if the WiP (Work-in-Progress) limit had been reached/breached and an Issue/Task-Ticket could not be moved further (on Kanban-Board), the Team-members affected had to deal with other tasks or help colleagues out.
The main focus had been put on Velocity though – Release-Velocity.
Argumentation: - in our cruel, rapidly changing IT-World, the competition is fierce. Steadily & swiftly bringing out new features & introducing new products is the only way to win the race.
Alas, Product-Quality is secondary. Apparently, a Customer getting/holding “something in his hands” (asap), even imperfect, is more important than getting an unflawed product – as long as we are faster than the competition. We can inject Quality after the Product had been delivered, we are good at fixing things – MVP was the new Credo.
Now this was a hard blow for a QA-Engineer being used to make decisions based primarily on the aspects of Quality, focusing on the Customer-Satisfaction, defending and caring for the Company-Reputation. But the times, they are changing…Nothing stands still, everything flows.
Adaption of the new way to do things had been a most fascinating experience.
I feel lucky to be given the opportunity to accompany this evolutionary step!
Frankly, the turn to an Agile approach had introduced numerous advantages and made a life of QA Engineers easier. One of the most valuable changes – no more “Pizza”-Releases (contributions tested till a late hour and traditionally enriched by a Pizza-supper at the Office).
Prior to Agile turn, we were rolling out Software every 3-4 weeks. I must admit, testing hundreds of Changes, Bugfixes, Refactorings which such a Release contained, strained a lot. And despite of Documentation available, discovering a major Bug so late had been fatal. And of course, impossible to fix in time for the Rollout. Which again, killed the motivation and produced strain.
Let me state here, I do not miss these days from the technological point-of-view.
Albeit, having a Reference to test against, a disciplined mutual Process and the desire to gain and keep a high level of Quality were fun as well. We had a mutual aim, a defined route, a cause – gatekeeping the Platform’s Quality, defending Company’s Reputation and boosting the Customer-Experience and satisfaction.
But these times had passed, evolving new Market/Business-philosophies, priorities and goals.
Development had to adjust and recalibrate its Processes to serve the new needs.
As far as the Total Test-Automation as an ultimate weapon concerns – I am not a friend of this doctrine. Definitely, stable and meaningful Automated Tests are a powerful ally during a Development-Process, but mainly as a Regression-Test-Run assuring nothing vital gets broken. However, automating every possible Use Case just for the Automation-Coverage sake is a waste of time, IMO. Moreover, in a frequently changing environment, new Technologies, ongoing Product-Changes, Refactorings etc, the costs and effort for the Automated-Tests-Maintenance are considerable and have to be counted as well. Especially the Frontend-Tests need a lot of care and they require frequent adjustments.
For my part, I prefer to test in an explorative way. Why? I’ll explain.
When I test explorative, I see and feel the designated Workflow under Test much better. I can notice things (Site-Layout, Feature-Workflow, UX-Issues) which Automated Tests cannot reveal. Good Software is more than just a flawlessly functioning Application. A user/customer should enjoy the experience and should have a will to return. When I test explorative, I become the user/customer and try to see and experience things from this unburdened perspective. This is essential for being a good QA-Engineer. Explorative Testing relies on Intuition and experience and may appear esoteric to a strictly logical mindset.
Other hot Topics are the definition of done, Throughput-Measurement, Product-Lifecycle.
Strangely, such an apparently straightforward-topic as definition of done had been causing irritation across the Teams. For some, done meant simply Development done (no QA-Approval yet), for another ready for Rollout (pass QA-Approval, but not rolled out yet), and for some it meant developed, tested, rolled out in Production-Environment and reassured everything is OK.
Therefore, if one Team proclaimed to be done, it was possibly just a halfway for others.
In an ideal Agile world, QA should be actually accompanying Development and keep pace with it. Ideally, when Development is done, QA should be done as well – but I had rarely seen this happen.
Concerning Team-Throughput-Measurement I can say, this is an imprecise Science. Why? Because it only measures the Time a Ticket spends on a Kanban-Board.
(NOTE: I love Kanban-Board!)
Alas, there are more pieces involved, but not tracked – time/effort/resources for Bugfixing after the Rollout, reduction of Technical debt, handling “loose ends” etc.
A new Feature should be built/implemented asap, two Development-Teams do their best to deliver a best Product feasible in a reasonable Time frame.
The first Team states to be ready for Rollout after 7 days of Development, the second one needed 3 days more. Apparently, thumbs high for the first Team?
According to the Team-Throughput-Measurement, yes – they were faster, obviously more efficient. The second Team needed three days more, QA wanted to have some more time to test (explorative), moreover the Team wanted to implement a better logging as well.
Reality was: - a Bug (in Production) had been revealed at the implementation of Team 1. They had to fix the issue, retest and redeploy it. It took three days…
Well, and what about the harm to Company’s reputation, bad User Experience, CS-hassle which took place due to a Production-Issue?
Is this really considered, measured and cared for while eagerly striving for a quicker Release-cycle?
IMO, the fact that you are good and quick at fixing Bugs is not an excuse to release an inferior Product in first place - this is the sweet Path to the Dark Side.
Very tempting, but dangerous and destructive.
Therefore I say:
Release-Velocity should NOT be more desirable/important than Product-Quality!
My Lessons learned
- Do not start developing, if the Requirements are fuzzy/unclear/obfuscated.
When needed, adjusting an application having clear requirements is cheaper and easier than tearing down the fundaments and re-writing the whole thing anew.
Identifying the implementation’s complexity, respecting the Software Architecture, various Stakeholder-Requirements, Technologies involved, Hosting, Testing and Rollout-Topics are Basics and should be clarified asap.
Yes, Changes are Welcome in Agile, but a stable and healthy Fundament is crucial!
- Think about Security!!! (Application, APIs, Databases, Environments)
- Issue Automated Tests only for cases worth Automating (e.g. crucial Flows, major Features, Security/Pen-Tests), do NOT automate every single Application-Use case and/or Function
- Have the courage to question things unclear, suspicious or unspecified – perhaps forgotten/unconsidered?
- Not every new “bleeding edge Technology” is cool and must be used, not every new library-Version is automatically better/faultless (from my own experience: I saw how a newer Library-Version fixes one Issue, but introduces a new one due to incompatibility or omitted methods/functions in a new version)
- Rolling out hastily before the end of the work day or weekend is prone to ill-fate (Murphy’s Law)
The List can be prolonged with more pearls of wisdom, albeit the best wisdom evolves from your own experience.
I favor a Mix of Worlds, or the best of both Worlds (Waterfall and Agile) as a QA-Process-to-go.
For me as QA Engineer it is crucial to have a sufficient Reference to test against. I can’t prove and assure the Application under Test works correctly, if only fuzzy Requirements are given.
It is a task of a Product Owner, to know the Product he is responsible for. It is crucial for the Product Owner to know and understand the sometimes complex (business/technical) dependencies the implementation requires. It should be clear, that if a flaw at a fundament of an implementation, an architectural Bug is discovered at a late phase of Development, the Price of Correction will be very high.
Thus, the Product Owner should invest time and effort to clarify possible dependencies asap.
As for the Development-Team, the engineers should try to identify the technical, environmental, cross-cutting dependencies (Process-Flow-Chart is helpful!) before diving into Programming.
The QA-in-charge should have an idea on how/what/where to test and should take care of an appropriate Test-Environment in time.
In order to identify potential Issues timely, I suggest to do short Demo-Sessions (30min) with a Product Owner once a week. Besides of the benefit on Development-Status Quo, this exercise is mind refreshing and sharpens the Team’s focus.
It is not enough, to write some common descriptive sentences into a Backlog-Ticket, call it Product-Requirement and let it loose on a Development-Team. The fruit of such approach can be devastating, vague instructions can deliver a surprising result.
On a technical level, Developers and QA (+ SiteOps) should inspect and discuss the technical concerns – Technology used, Infrastructure/Administration/Maintenance-Matters, Performance, Scalability, Security topics, possible third-party involvement and dependencies.
Because the Product-Responsibility does NOT end with its Rollout to a Production-System, it has to be cared off, refactored, adjusted, improved.
To summarize, my 2 Cents on both Approaches from a point of view of a QA Engineer.
Top 5 good things about Waterfall Approach:
- Structured, organized Process
- Proper Documentation (Product/Technical, Code-Documentation)
- Clear Strategy, Transparency of Roles and Responsibilities
- QA-concerns are respected, often a dedicated independent QA-Team
- Quality is more important than Velocity
Top 5 good things about Agile Approach:
- Flexible, adaptive to Change at any Project-Stage
- Rapid Delivery, Issues are fixed in a timely manner
- Hunger for new Technologies and Experiments, learning and trying new things out
- Communication is quick, open, efficient and non-bureaucratic
- Less stressful despite of the high pace
Ideally, I would merge the best of the two worlds and have a proper Quality Product developed in a timely manner, having a proper Product-Documentation and regular Product-Demos, the good Agile spirit and an organized, mutual, agreed way of doing things.
Have a good flight to the Moon, Qapla’ and make it so!