Video SDK – User Testing + Competitive Analysis

Client Project | HFID 710 – Managing a User Centered Design Team

The Class

In Fall 2017, I took HFID 710 – Managing a User Centered Design Team. The class focused on teaching students about integrating Agile business frameworks and traditional UX processes. Students were placed into teams of four, and partway through the semester each team was assigned a client project.

The Client

The client is an online video hosting platform and online video player solution for clients around the world. They offer a range of services, including cloud hosting, video marketing, and a live streaming platform. Our team was assigned to the company’s software development kit (SDK), which gives their customers a video player and library framework which they can then customize (visually and structurally) to suit their needs.

Over the course of the semester, our team designed and conducted a usability study and completed a competitive analysis, both of which we presented and delivered to the client.

Competitive Analysis

Overview

We completed a comprehensive comparison with similar aspects of Netflix’s application interfaces and the client’s download and offline video functions. Netflix was selected by the client as the most applicable competitor, based on prior data and market research.

Comparisons are drawn from the following aspects of the client’s interface:

View video listings
Select a video (buy/rent)
Check videos ready to be downloaded
Download a video and track downloading progress
Watch a downloaded video offline

Process

Based on the several key functions to be tested, we mapped the functions to Netflix’s mobile video application, evaluated pros and cons of each function interface within certain user context, and drew comparisons accordingly.
We also incorporated general experience comparison questions into the guerilla testing and interviews to create a holistic evaluation.
By focusing on several key aspects of Brightcove’s mobile video application, we drafted specific potential improvements and design recommendations.

Guerilla Usability Testing

Overview

Given that a portion of our study would include a competitive analysis against Netflix, and given the small number of participants we were expected to interview (8-10), we felt that in-person guerilla testing would yield insightful, actionable results.

We discussed participant criteria with the client, and determined that anyone who has prior experience with mobile video watching would qualify. Once participants were selected, we started with a few background questions to understand their prior experience, observed and recorded participants as they performed the tasks, and followed up with brief wrap-up questions. All procedures were approved by the Institutional Review Board (IRB).

The Process

Identify the target user population. Based on the user data given to us by the client, and due to the tight time limitations, we chose to guerilla test Bentley University students, between the ages of 19-26, who have experience watching videos on a mobile device.

Location and equipment. Recruit and study were conducted in Bentley’s Student Center. A Mac Laptop was used to record audio only, and those recordings were made available exclusively to research team members.

Provide a task scenario. Users were directed to find, download, and watch a video offline via the client app on an Android device. They were also asked to articulate their thoughts and think aloud as they walked through these steps. If participants became stuck, minimal assistance was given.

Triangulating Results

Competitive analysis and user testing on the client video app were conducted independently. Results generated from the two tasks are designed to complement and confirm one another in order to provide a holistic evaluation of the SDK and video watching experience.
Several key aspects that need to be improved upon within the client app are consistent between both competitive analysis and user testing. These include: efficiently downloading a video, checking and understanding download progress, and successfully watching downloaded videos offline.
While user testing generated findings by observing users interacting with the client app and interviewing users directly, competitive analysis generated findings by comparing the client app against Netflix, the primary client-identified competitor. By comparing results from the two studies, we generated recommendations for implementation and future analysis.