Second-generation Agile Methodology: Dan North's BDD Tales

Interview with Dan North

Writing an introduction to an interview with Dan North is not an easy task, being one of the very first names that come to anyone’s mind when talking about Agile and, as the saying goes, being one of those people who need no introduction.

Originator of BDD aka Behavior-Driven Development, Dan is a frequent conference speaker and a star of the software development world by all means. Dan will also be in Milan on May 15th-16th for his 2-day workshop “Software Faster”.

Among many other interesting things, in this interview we find out why the principles of the Agile Manifesto are aging well (like a good wine we might say!) and what we can do to make the most out of them in an ever changing world, almost 20 years since they were written for the first time.

Avanscoperta: Hi Dan, it’s a pleasure talking to you! Your twitter handle is @tastapod. What’s the hidden meaning behind it? 🙂

Dan: Ah now, that’s a long-kept secret! I can tell you it is a made-up word, and that it has a definition. I can’t tell you the definition though. (Ok, I can tell you the definition, and why it was made up, but you’ll have to buy me a drink first.)

Avanscoperta: Ha! We’ll write a follow up of this interview then… 🙂 Let’s kick off with a classic: tell us something about your first encounter with the tech world.

Dan: I remember my first encounter with a computer. It was 1981 so I was about 10 years old. My friend’s older brother had a Sinclair ZX81 with a whopping 1kb of memory. (You could plug a RAM pack in the back to get 16kb but it would reset the computer if you jogged the table.)

He was playing 3D Monster Maze (which you can still play!) and it was the coolest thing I had ever seen. My first computer was a Sinclair ZX Spectrum a couple of years later. I used to type in games from code listings in magazines.

Avanscoperta: You’ve always been extremely active in the tech and software development community as a whole. What’s the best piece of advice you’d give to anyone who’d like to start being more involved into it?

Dan: In terms of public speaking, people like stories and they want to hear yours, probably more than you realise. A lot of people I encounter in tech—and there is evidence that this applies more to women—don’t think they have anything interesting to say. Then I discover they’ve been doing something genuinely innovative but they thought everyone did that! I encourage people to share their ideas and discoveries, at work, in meetup groups, online, or at conferences.

When I work with people who are underrepresented in tech, particularly women and people of colour, I encourage them to get out and speak. They don’t know who will be in the audience, and a young person seeing an older version of themselves on stage at a conference is more likely to be inspired by that than by yet another middle-aged white guy.

Avanscoperta: Change is always a constant element in life, and the software development industry is no exception. How is the tech industry responding to it? What are the next challenges you can see coming in the following years, and how are we getting ready for them?

Dan: Predicting the future is always dangerous! The obvious things coming along in the near term are cloud-native computing, machine-assisted computing (the term “AI” is already far too overloaded), wearables and connected devices, and a huge increase in connectedness and tech sophistication in developing countries.

I can’t say how we are preparing for them because it won’t be me doing it! The best way to adapt to something is to grow up with it. There are already people coming through who have only ever seen touchscreens and who have grown up with a quad-core supercomputer in their pocket.
Soon there will be people who have only ever seen applications running in the cloud, who think owning your own servers is as quaint as running your own electricity generator or water pump. These are the true mobile natives and cloud natives, and they see things differently from me or you.

I hope that in my working lifetime we come to think of using a keyboard to interact with a computer is as quaint as we think of punch cards. I still haven’t seen the thing that will replace a keyboard for day-to-day programming, but then I might be thinking too small, and something is coming along that will replace programming altogether! As William Gibson says: “The future is already here, it’s just not very evenly distributed.”

Avanscoperta: Over 10 years after its creation, let’s look back at how and why BDD (Behaviour-Driven Development) has helped the community improve and what impact it’s had.

Dan: I am as surprised as anyone that it took off as an idea. I developed it as a coaching technique in 2003-4 while I was at ThoughtWorks, which makes it 15 years old this year!
I think the power of BDD is its focus on collaboration across roles and throughout the development cycle. BDD itself grew as a collaboration, first with a handful of ThoughtWorks colleagues, then with a wider group as I started talking about it at conferences, which is why I think it has endured and evolved.

I made a conscious choice not to lock it down or create a certification programme around it. I was more curious about where people like Chris Matts, Liz Keogh, Gojko Adzic, Olav Maassen and others were taking it, and it evolved in directions I would never have thought of. It influenced Chris and Olav’s Real Options, Gojko’s Specification by Example, Liz’s Capability Red, and of course gave rise to tools like RSpec, JBehave and Cucumber. I remember hearing about behaviour-driven infrastructure and thinking “Wow! Why didn’t I think of that?”

It has its own conference, the BDD and Testing eXchange (currently undergoing a name change), where people come together to share different and innovative ways they have been applying BDD, often outside of software development. I like that there is as much a focus on the human beings involved in software development as in the processes and tools themselves.

Avanscoperta: One can be confused by the flourishing of very similar acronyms in the industry… What’s the relationship between BDD, TDD, and DDD? Apparently all the same but they’re not, isn’t it?

Dan: BDD, Behaviour-Driven Development, is derivative. I describe it as a second-generation agile method. It started as an attempt to describe Test-Driven Development, TDD, without using the word “test”. I figured that TDD was about design rather than testing, and I found developers were tying themselves in knots trying to understand what all these “tests” were for. They seemed to find it easier when I described them as code examples and executable specs instead, and dropped the testing language altogether.

At the time I was very influenced by Domain-Driven Design, or DDD, which says if you focus on understanding and modelling the domain of a problem, the programming part seems to take care of itself.

DDD looks at the names, roles and behaviours within a domain, and says these names should be pervasive: everywhere you look, in classes and methods, functions, packages or namespaces, you should find a consistent domain language being used throughout. If your domain has Customers and Orders, so does your code. If your domain has Hospitals and Patients, so does your database schema, and so on.

I saw BDD as defining a domain language for software behaviour. Its vocabulary of stories, features, scenarios, executable specs and so on, were a great way to describe the behaviour of computer systems, and led naturally into automation and tooling to verify behaviour.

Avanscoperta: Could you explain what do you mean by Deliberate Discovery, and how this is useful in software development?

Dan: Software development, in general, is a process of discovery, experimentation and elaboration. We often don’t know the answer at the start, and the act of building the system teaches us more about the problem, which in turn informs the solutions we build. This is known as a complex adaptive system. This insight is at the heart of DDD, which is why the primary activity is modelling—i.e. learning about—the domain, rather than programming.

Most software planning and estimation methods are reductive: they assume you can break down a problem into ever smaller pieces until the pieces are so small you can code them up one at a time. Most planning focuses on this breaking down activity, usually with some estimation theatre.
People say they find planning useful for discovery, but I observed that this discovery is accidental at best, and suggested it may be more useful to make discovery the objective rather than planning. I call this approach Deliberate Discovery.

Conventional planning, even on agile projects, assumes we know the answer and we just need to break the work down and schedule it. Deliberate Discovery assumes that we don’t know the answer, and that this ignorance is our biggest impediment, so we should do everything we can to reduce our ignorance.
Whatever we learn will guide the next step in the solution, which in turn will produce the next learning, and so on. We treat it as an emergent system: we can choose where we steer but we can’t see over the next hill.

As a concrete example, agile planning suggests putting the “most valuable” features at the head of the queue. There are even formulae like Weighted Shortest Job First, where the priority is a function of estimated value and effort. If we are planning a release with a whole lot of features then the order we build them doesn’t really matter in value terms: they are all going out together in any case.   Deliberate Discovery says to prioritise the work with the greatest uncertainty first, because then when we get surprises —which is inevitable in uncertainty—they happen earlier and we have more time to respond to them.

Avanscoperta: How is Deliberate Discovery related to Vasco Duarte’s No Estimates approach (if at all)?

Dan: As I said, estimation in software delivery is broken because it is reductive, it assumes each feature in a release is an independent entity that can be estimated in isolation. However this doesn’t mean you can’t estimate at a macro level using techniques like Blink Estimation.

I’ve said before that I think “No Estimates” is an unhelpful name, because it repels exactly the people we want to be engaging. The proponents of the No Estimates movement say the name is immaterial and the important thing is to be having the conversation, but words have power, and in my experience senior managers see the “No Estimates” label and assume we are either amateurs or attempting to avoid accountability, so they look no further.
You can’t have a conversation with someone if they don’t turn up.

Avanscoperta: You’re the author of a two-day workshop titled “Software Faster”. Could you tell us something about it we won’t find in the course description? 🙂

Dan: Sure, let me tell you where the class came from. The year was 2010 and I was about 20 years into my programming career. I had just spent eight years working at ThoughtWorks introducing agile methods into large, complex organisations, developing BDD, and working on some of the techniques and methods that became known as Continuous Delivery. I was publishing articles and speaking at conferences.
In short I thought I was pretty good at this software development thing. And then I joined a team in a trading firm which challenged everything I thought I knew about software delivery!

This tiny team—I was the third member and we eventually grew to seven—was delivering complex, state-of-the-art trading software faster than anything I had ever seen, iterating on new ideas on an almost daily basis, in a complex and rapidly changing context, with enough care and quality that they were allowed to trade real money on live exchanges in a very risk-aware company!

They didn’t have projects or backlogs or backlog grooming; they didn’t have features or story points or estimates; they didn’t have Scrum Masters or agile coaches; they didn’t have sprints or iterations or timeboxes; they didn’t have business analysts or testers or Product Owners; they didn’t have retrospectives or Three Amigos sessions or acceptance criteria; they didn’t have build servers or UAT environments or test coverage metrics.

What they did have was a close relationship with the traders—in fact we all sat together—and a desire to learn about the kind of trading they were doing and how software could help that. And they had a keen interest in the profit-and-loss figures of the trading desk and how their software was impacting this.

I stayed with this team for about a year, and the Software Faster class evolved from wanting to share my experiences working in this dynamic, fast-moving, refreshing, exciting, genuinely agile way.

Avanscoperta: Who shall attend this workshop, and what are the main takeaways for an attendee?

Dan: This is not a class for beginners. There is plenty of good training out there for them. This is for people who have several years experience of software delivery, and who have tried agile methods and maybe started to outgrow them, asking themselves what lies beyond sprints and story points and test coverage.
It is not just for developers. I’ve had testers, analysts, delivery managers, technical leads and architects in the class and they have all said they found it valuable.

The main takeaway is that there are no absolutes and no best practices, so high-performing teams are able to make educated trade-offs based on their context. The class is about patterns and techniques that challenge existing thinking. When should you not test? When should you not automate? Why is software estimation fundamentally broken? When is it OK to put half-baked software into production? How can you confidently delete code?

Avanscoperta: You state “Software Faster brings software delivery principles into the 21st century.” Can you expand a bit more on this?

Dan: Agile methods like Scrum and XP were developed in the last century, in the 1990s. They operate on the order of months or weeks, and have ceremonies that are designed to create alignment by managing throughput—getting everyone moving at the same cadence.
When they were designed this was a huge accelerator: you went from multi-year releases to planning every few weeks and releasing every few months or so, but now those same methods act as a throttle, slowing down throughput by introducing unnecessary ceremony.

Fast-forward a couple of decades and the technologies we have now mean we can prototype multiple product ideas in software in a single morning! In that context why would you want a multi-week planning horizon, or a backlog of features stretching out for months, when you can get near real-time feedback on new product ideas from customers sitting right next to you?

Software Faster takes the same principles that gave rise to Scrum, XP and the rest, and applies those principles to the modern world of software delivery, shortening the release feedback cycle from months to weeks or even days, and the feature feedback cycle from weeks to hours.

I believe the underlying principles of the Agile Manifesto are still largely true today, but the methods haven’t moved with the times. This class is about challenging those methods and seeing what else is possible.

Avanscoperta: What kind of entry requirements are needed to fully benefit from this workshop?

Dan: As I said, the main thing is that you have been involved in software delivery for long enough that you want to know what else is out there. The class is fun, interactive, and high-energy. I invite people to challenge the teaching and to get into some good discussions, because that is where the best learning happens.

Avanscoperta: You’re a frequent conference speaker and must have attended and took part in a lot of events… What’s the funniest thing that happened to you on stage?

Dan: I don’t know about funniest, but one of the most fun things was having someone doing a graphic recording of a talk, kind of a large-scale sketchnote, while I was speaking. They did a great job of capturing the core messages of the talk, and it was fascinating as a speaker to have someone interpreting my talk as I gave it.

Avanscoperta: What do you like doing in your free time?

Dan: I recently became a father so “free time” is something of a rarity! I play drums and percussion at my church so I like to get involved in that, and spending time with my wife and baby and our two little dogs.

Avanscoperta: Wow – congrats for the new one!
What’s your favourite song and how does it relate to your work life?

Dan: That’s a tough one. I have lots of favourite songs so it would be unfair to pick just one. If I were to pick a theme song for Software Faster it would probably be David Bowie’s “Changes” 🙂

Avanscoperta: Thank you so much for your time Dan, see you soon!

Dan: Thanks, I’m looking forward to it!

Pics credits: https://www.enkiquotes.com/william-gibson-quotes.html, Skills Matter, Ed Telling, Chris Liverani, Igor Rodrigues, Markos Mant, Aurélien Dockwiller.

More on Avanscoperta

Check out our upcoming workshops: Avanscoperta Workshops and Training courses.

Get our updates, a hand-picked selection of articles, events and videos straight into your inbox... Once a week!
Subscribe to Avanscoperta's Newsletter (available in Italian and English).