Building Lumosity's First Mobile Cognitive Assessment
My team built our first NeuroCognitive Performance Test, based on common cognitive assessments, for iPad. It is an essential component of our ongoing efforts to establish a valid measure of cognition and it is central to our efforts to examine and understand product efficacy. Eventually, we would love to see an NCPT tool adapted to clinical and research environments.
Background
Cognitive assessments have been traditionally administered with pen and paper since the 1930s. Even today, a licensed clinician sits in a room with a patient and administers a series of pen and paper tasks while taking notes by hand and timing the patient with a stopwatch.
Lumos Labs created their first web-based version in 2009, which we now call the NCPT or the Neurocognitive Performance Test, when the idea to do large, online, remote cognitive training studies was born. We used it as a tool to show the efficacy of Lumosity training in the Crosswords study published in 2015. We have provided it as a free tool for research partners and collaborators to help advance research in human cognition. Over a million of our users have taken the NCPT over the years, and we are also able to provide that data for our collaborators.
Problem
Why a mobile assessment?
We need to begin efficacy studies with Lumosity Mobile to understand how the product is currently working and how we can improve.
On the consumer side, we learned from the web NCPT that subscribers who complete the NCPT are 20-30% more likely to renew and we believe we can achieve the same success for mobile.
We’d also like to better support our research partnerships, who have been requesting a mobile assessment. A mobile assessment would be faster, less expensive than traditional lab-based assessments and would enable our partners to reach a wider audience.
Goal
Phase 1: Release a set of cognitive assessments on iPad for a small subset of users in order to conduct validation studies (test-retest reliability, concordance to web NCPT, concordance to pen+paper assessments).
Phase 2: Deliver a comprehensive set of cognitive assessments that cover key cognitive functions on iPad for use in a Lumosity mobile efficacy test.
Long-term: We hope to offer mobile NCPT to partners for use in clinical and research studies at a greater scale than ever before. We also would like to use it in product to help users measure their performance gains after playing Lumosity.
Approach
We selected a subset of assessments to port and design for Unity, supporting iPad first with plans to port later to iOS and Android. Before we could conduct an efficacy study or release a consumer version of the app, we first needed to release an alpha version to the App Store in order to collect enough data to perform validation studies and follow up with optimizations as needed.
My Role
Our small team of 5 comprised of a product manager, two Unity engineers, a research scientist, and myself as Senior Product Designer. My core responsibilities included but were not limited to:
Research
Brand identity
User experience
Task flows
Information Architecture
Wireframing
Prototyping
Usability testing
Interaction design
Visual/UI design
Game design
Motion design
Illustration/iconography
Copywriting
I worked very closely with the product manager and research scientist on all parts of the product’s development and partnered with Unity engineers during implementation.
My Design Process
Research
Every Lumosity game or assessment starts with research. With the help of our very own in-house research scientist, we dissected paper testing and paper test methods to better understand the pain points of paper testing and how we might ease those pain points through a touchscreen interface.
We also reviewed our web based NCPT assessments and identified their successes and and the areas in which we could improve.
PRototyping & User Testing
I started with wireframes and prototypes to test both internally and externally with some of our research partners. I went through multiple feedback loops to drive the final designs and interactions. Finally, I created the tutorials for each assessment to help ensure that the tasks are clear and intuitive.
BRand & Visual Design
I defined the style guide including both branding and UI. I was careful to define styles that fit within the ecosystem of the Lumosity brand while creating a new identity and visual style that is more tailored for our audience. The result is a design that falls on the expert side of our visual style spectrum while also maintaining bright qualities. The expert side emphasizes an authoritative presentation of accurate scientific content while the bright side clarifies and focuses user interactions.
Unique assets I illustrated for the Object Recognition subtest
DEVELOPMENT
I handed off these designs to our Unity engineers and we worked very closely to ensure that the motion and interactions were useful, responsive and fast.
Tutorial Storyboards & Flows
App Designs
Outcomes
In total, our team designed and built 10 new neurocognitive assessments that have been adapted for tablet touchscreen interfaces and created in Unity. The alpha was released to the iPad App Store and shared with a subset of our users. Despite the limited release and lack of incentive, a large majority of our users who downloaded mobile NCPT completed the assessment in one sitting.
Our roadmap and priorities shifted shortly after its release in 2016 and the company decided to pause further development of the mobile NCPT. Meanwhile, the science team continued validation studies to identify areas of improvement. Here’s what we’ve learned:
We found that the mobile NCPT test-retest reliability was rated good-to-excellent for most assessments and in some cases more reliable than our web NCPT.
The Trails A & B assessments scored the lowest in test-retest reliability and will require some optimization.
We found that the mobile NCPT displayed good concordance with traditional assessments — evidence that we’re measuring the same aspects of cognitive performance.
In 2017, the company revisited the potential of the mobile NCPT and is undergoing further development and integration into Lumosity for its full consumer release for all platforms in 2020.
Lessons Learned
Usability and our judgement conflicted with established assessment rules.
Optimizing usability while adhering to assessment rules and requirements was challenging at times. The established rules can impact usability, particularly when delivering error feedback for incorrect responses. For example, the Trail Making A & B have specific rules and requirements for when an individual recovers from an error. Reproducing this error feedback and recovery on a touch screen interface is not as intuitive and has led to confusion about what to do after inputting an incorrect response. Coincidentally, this was our least reliable assessment and required some further design iterations to improve its reliability and validity.
We may need to consider how touch screen interfaces can introduce biases and impact validity the NCPT.
External studies are underway to identify potential issues in administering cognitive tests with touch screen tablet devices. For instance, an individual’s test score/results can be contaminated by extraneous factors related to the test itself, the procedure, the platform, the test environment, all of which can’t be negated without an administrator. Existing biases can also impact validity and performance. These things must be taken into consideration if we want to seek FDA approval and provide a reliable assessment product for both our consumers and our research partners.