Why learning journals should be a part of your experimentation program
Capturing lessons learnt and applying improvements is key to getting the most from iterative programs, yet for some reason, these aspects are often missed in experimentation practices. As experimentation matures as a discipline, we can no longer rely on team members memories to piece together experience and insights from prior tests. We need a more structured knowledge management system.
"Those who cannot remember the past are condemned to repeat it," said Spanish philosopher George Santayana.
Now that plenty of organisations have a few years of experimentation behind them, many are struggling with how they recorded (or didn't record) their past experiments, which means they're failing to note insights that are often crucial to the progress of their experimentation program.
This frequently manifests when there's a change to the team, and new team members follow the previous line of thinking, running the same tests to come to the same conclusions, because they don't have the cumulative insights built out of a range of experiments.
Because experimentation as a discipline is still evolving, those who were early adopters are now finding gaps in their initial processes. According to the Deloitte Digital 2022 Experimentation Maturity Index, more organisations run experimentation programs for teams at different levels, which has revealed how much people don't know if they haven't been there from the start.
These organisations have started to realise they haven't been logging or recording past experiments correctly, or they've used a tool that isn't scalable to the growth of the practice. An Excel spreadsheet might suffice when you were running a handful of experiments a month, but as soon as velocity and volume increases, Excel's lack of indexing capability and content management functionality can be more cumbersome than helpful.
As part of this lessons learnt, we want to make sure newcomers set up the right practices from the beginning.
A learning journal is an important tool to advance experimentation, but what is it? At its simplest level, a learning journal is a tool that helps you capture lessons learnt. A good journal is:
Consider who it is for, and how easily they can access insights as your experimentation program expands; ensure there is program literacy across your organisation and the lessons captured are transferable across the organisation, not just for one department.
One of Australia's leading airlines uses a wiki as a journalling tool, whereas we use a tool called Airtable as a way of building a database. According to our research, mature companies are using technology such as Optimizely, which has a guided program management tool that allows you to journal.
It's important to note that beginners tend to opt for tools like Google Optimize, which may be attractive as a low barrier to entry but lacks the program management capabilities. Poor program management corresponds with a deficit of knowledge management and points to a need for learning journals.
An experiment should be about understanding the right experiences that customers want, in order to create the outcomes that both the customer and the organisation want. What we do in design is augment the experience to deliver the best customer experience. An experiment is a data layer to validate the best possible customer experiences. When used properly, a learning journal democratises the knowledge about customer behaviour, and enables different teams from various departments to feed insights back into the organisation to advance business outcomes.
The experimentation team need to be wary of developing a learning journal or creating a database for themselves; the insights should not just be about incremental gains on the website or a digital asset. A learning journal should be presented as an insights tool for the wider business, that offers actionable data and is applicable to many areas of the business.
An operations manager would take a different action or insight compared to the experimentation squad, compared to the marketing team. It's through this practice that learning journals - and experimentation itself - become valuable assets to the organisation.
Capturing lessons learnt helps build on what you already know, so you don't waste time, money and effort reinventing the wheel.
This article was originally written by Nimi Yassini, Partner at Deloitte Digital.
Interested to learn more?