Writing and analysing solution requirements is one of those things where it’s generally simple and straight forward enough to get, well… very wrong. To help solve this, we wanted to see how AI technologies (specifically Natural Language Processing) could make it easier to create and manage good quality requirements. Our ultimate goal was to enable our clients to develop better solutions, faster and with less risk.
Earlier this year and with great excitement, we partnered with US based company Logapps to leverage their AI based requirements analysis technology MARINE (Machine Assisted Requirements Inspection and Evaluation). While the technology is rapidly evolving, there is one thing that has struck us that we should have really guessed from the beginning.
Like any automated process, there is still a garbage in : garbage out problem that is probably quite hard to solve. As we started running live and historical project requirements through the tool, it became apparent just how bad the state of affairs really was.
In one case, some 2/3rds of the requirements were marked as ambiguous or vague. With an understanding of the problem space, you could probably have guessed what those requirements really meant. But can you always assume that will be the case? What if your assumptions differ to mine?
It did have me thinking though; In today’s age of agile, iterative delivery methods, do requirements still matter? Do the iterative methods and philosophies that are now common place, really offset the cost of a poor requirement?
In a waterfall style delivery method, the cost of a poor requirement increases proportionally with each phase of the delivery. Caught at the thin end of the wedge, a poor requirement can be fixed with a few taps of the keyboard. Wait until final testing and the cost can be significant.
So how does agile change that? Well in theory, a poor requirement should only have at most, one sprint cycle to propagate before being caught. In practice it’s probably much worse, but let’s stick with that for the moment.
Let us assume the internal costs of a developer are around $500 to $800 a day, and an external one around $1200 to $1500 per day. For simplicity sake we will blend that down to $1000 a day. The cost of a Scrum master on a per developer level is perhaps 10% of that. Add in devops time, additional testing, product owners, other support staff and costs, and let’s make it a round, back of the envelope, $1500 per day.
If a poor requirement requires the developer to rework the solution in another sprint, then on a two-week sprint cycle, that poor requirement just cost the business $15,000. In one of our real-world examples, approximately 90 requirements were marked as ambiguous. If each one of those required a sprints worth of rework, then the total cost would be $1.35m. The time spent would be roughly equivalent to a six-month project delay for an eight-person dev team.
Am I being a little extreme? Maybe, these are back of the envelope numbers after all. They’re also assuming an error is actually discovered within that single two-week sprint. If for whatever reason, the outcome of the poor requirement was left to fester, the impacts could be compounded in much the same way as any other delivery method.
And, we haven’t yet taken into account any lost opportunity cost! What business value could that development team have generated with each lost two-week sprint? what about over six months? Assuming the business case stacks up, you’re probably looking at an ROI multiplier on your developer time of between three and ten. I’m sure you can do the maths…
So yes, even in an agile world, your requirements and user stories do matter and should be treated with respect. If practiced properly, the agile methods should at least give plenty of opportunity to catch and tweak a poor requirement before solutions are delivered. This may leave some teams to take that assumption for granted.
It’s not hard to write a good set of requirements. It just takes a little consistency, structure, and thought. That seemingly simple line of text though belies the hidden cost of a more flippant approach. Whatever you do, just don’t assume your chosen delivery method alone will help you pick up the pieces. Those sorts of assumptions are what brought us here in the first place…