Measuring the UX of an asset health monitoring SaaS

Mixed methods research

Analytics

Project Overview & goal

Our goal was to measure the user experience of an asset health monitoring SaaS and report back to upper management on the product's performance. My secondary goal was to generate a culture of continuous research within the team.

Role & duration

Sole UX researcher. Collaborated with the PO, one Subject Matter Expert (SME). 1 month

The problem

Despite plans to conduct UX research, the teams continually postponed it due to time constraints. They sporadically interviewed a few users but lacked a consistent research process. UX was not considered a performance indicator, and there was no usage analytics tracking due to the app’s lack of analytics tool integration.

Hypothesis

Relying solely on user comments for UX analysis is insufficient. By introducing other quantitative research methods like usage analysis and surveys, I aimed to uncover insights the teams were unaware of.
Conducting research collaboratively would enable product teams to continue these efforts independently.
Regular usage reports to upper management could drive the need for ongoing research, ultimately leading to a better understanding of the user experience and more effective performance indicators.

Results

By the end of the quarter, the SaaS received a comprehensive UX report that included analytics and user satisfaction analysis. This led to a better understanding of different user segments by roles and locations.
As a result, the SaaS received budget approval to continue operations.
Product team has since continue to conduct research and report on user experience to leadership.

👍

$2M

Budget approved for product ops

👍

UX quarterly reports created in the last year

The process

Understand

Goal Clarification: Defined the goals of the product and what the team wanted to discover.
Initial Assessment: Mapped out the different user roles using the app with the help of the PO and SMEs.

Investigate and Analyze

User Experience: Conducted a mixed-method approach to understand user behaviours and attitudes.
Log Analysis: Analyzed user activity data from dashboards.
Surveys and Interviews: Collected quantitative and qualitative data through surveys and user interviews.

Report

Collaborative Research: Conducted research together with the product team to ensure they could sustain the efforts.
Reporting and Recommendations: Provided actionable insights and recommended regular usage reporting.

Key Insights

Current User Experience

User and Roles

After collecting all data from interviews and generative research. We were able to clearly identify different user groups, how they worked together, their motivations, and pain points.

User satisfaction

Key finding: Low satisfaction at Blairmore office

Identified the Blairmore office as having the lowest satisfaction score (4.0), primarily due to issues with ease of use and customer support.

Problems at Blairmore office

Lack of documentation on how to sort the new updates
Due to recent updates historical data was deleted, preventing users from performing deeper analyses.

"The system has been great, but the more recent updates (…) have been introduced with very little training or online content to help users such as myself with little to no computer experience." – User at Blairmore

For viewers, the most common problems were trouble finding the right dashboards and a lack of training.

"Trying to find specifics for units in our fleet seems impossible. I would recommend having someone who is familiar with this program show us how to properly use it." – Heavy duty mechanic

Advanced viewers had issues with not being able to save dashboards.

"I would like the ability to have and save custom dashboards." Maintenance Engineer

Solutions and Recommendations

Immediate improvements:

Enhanced training materials and documentation.
Improved communication regarding updates and changes.
Provided better support for data analytics and dashboard management.

Other improvements:

Established a regular cadence for user feedback collection and analysis.
Recommended integrating usage analytics tools for continuous monitoring.
Suggested upper management request periodic usage and satisfaction reports to maintain focus on UX.

Deliverables

User Journey Maps for all Roles

UX Quarterly Report

Satisfaction and Usage Dashboard

Conclusion

Through this project, we provided a comprehensive understanding of the SaaA user experience and demonstrated the value of continuous user research. This initiative lead to tangible improvements in user satisfaction and operational efficiency, and secured ongoing funding for SaaS. The collaborative research approach empowered the product teams to sustain these efforts independently, ensuring a user-centric focus moving forward