Transforming test management for a retail giant

Justin Watts and his engineering team at Loblaw Digital – the software arm of Loblaw, Canada’s largest retailer – couldn’t find a test management solution that would address their complex needs – until they discovered they could integrate Jira with Adaptavist Test Management for Jira (TM4J).

Justin, Director of Productivity Engineering, had been disappointed with the firm’s test management tools for a while. When Loblaw Digital grew from six employees to over 250 in just six years, it was time to make a change.

Engineering needed a solution that would support the unique role Loblaw Digital plays within its parent organization and meet the team’s three main goals: reduce toil, increase productivity, and get stuff done. After trying several tools to no avail, they finally found one that hit the mark.

In close collaboration with Adaptavist, an Atlassian Solutions Partner, Loblaw Digital customized an end-to-end, scalable testing solution that has reduced testing time, increased productivity, improved communication, and empowered the company to deliver more effectively on their mission to help customers “Live Life Well.”

These are just a few of the variables that made Loblaw Digital’s testing procedures so complex. Even their role within the larger organization is more involved than the typical in-house agency relationship. On top of building digital properties for over 20 brands, Loblaw Digital also owns the profit and loss statement for all of those properties. “Because we maintain the platforms we build, we value testability, reusability, and scalability,” Justin explains. “We own the technical debt, so it’s in our own best interest to care just as much about the journey as the end result.”

Since Loblaw is in the retail space, the company also has to plan for the end-to-end customer experience, from the user placing the order, to Loblaw completing it, to the customer potentially picking it up in person. These considerations made Justin’s wish list for a test management system even longer.

After extensive conversation, the engineering team narrowed their criteria to five main requirements for their new solution:

1. World-class manual and automatic testing capabilities
2. Native UI and API interfaces
3. Integration with one of their existing platforms, like Jira
4. The ability to track hundreds of permutations
5. Clear, shareable test results for both technical and non-technical stakeholders

It was a tall order, but the team had faith that they could find the right partners with the right tools to make their dream a reality.

This revelation not only made test results faster and easier for the engineering team to read, but also made them more understandable for non-technical stakeholders. Adaptavist was so impressed with the reports that they added this feature to TM4J.

Loblaw Digital has also leveraged TM4J and Atlassian to increase productivity by creating a Slack app, affectionately named Jeanie, that automates the process of running tests and reporting results. Justin’s team benefits from not only the automation itself, but also the resulting artifacts. “TM4J is really awesome because it’s generating an artifact in Jira, so anyone can go back and see what tests were run, when they were done, and view tons of fancy reports,” he says. “Atlassian is helping us be good neighbors and give feedback to everyone else while letting us operate the way we want to without having to click around multiple UIs.”

Another challenge Loblaw Digital solved with TM4J and Jira was tracing defects. Justin reports that one of his favorite things about using Jira for tests is the ability to bring together issues, stories, and tests into a “traceability matrix.” “What’s awesome about that is we can generate a report for every test we run and every story it relates to. That way, we can make sure bugs don’t accidentally slip through,” he says. “It also warns you if there’s something that’s unaddressed. This is great for stakeholders and keeps us honest.”

While the results have been undeniable so far, Justin is even more excited for what’s ahead. The team is now running around 200,000 tests a day, which is speeding up the delivery of new features and shortening regression times by 250%. What’s more, they believe they can make the process five times faster.

“We were worried we would pick a tool that would only help us for part of the journey,” admits Justin. “But we have transitioned from Waterfall, to Scrum, to Kanban effortlessly without changing tools. We’ve been able to grow with the product and not feel like it’s in our way.”

Justin has so much confidence in the solution’s scalability that he didn’t flinch when the parent company took notice and wanted to adopt Loblaw Digital’s tools into the larger enterprise. “Our usage of Atlassian has made them envious,” Justin chuckles. “We have a 2000-user license because anyone who collaborates with us in the enterprise wants to use it.”

Empowering employees to provide more value to the business has trickled down into more value to customers. With the support of Atlassian and Adaptavist, Loblaw is proud to be delivering on their promise to help customers “Live Life Well” faster and better than ever before.