A typical interaction

The first screen shows that Watson has high confidence that hypokinesia is a finding of Parkinson's disease.  The student then has the option to agree, disagree, or view evidence that supports this assertion. 

The next screen shows that the student has disagreed, and as a result, Watson's confidence is reduced from high to medium.

Problem statements and evidence

Below is the full text of a scenario.  Watson uses natural language processing to extract the factors that it believes are most relevant to the solution.  These details are underlined, and while Watson uses all the information in the scenario as input, the ones it identifies as significant have the greatest influence on its diagram of the solution and final answer.

Each inference that Watson asserts is based on evidence from a corpus of medical texts it has ingested.  The engine is not perfect, however.  Watson might misunderstand a phrase or the text itself may be incorrect.  In this view, the student can see that Watson has 36 pieces of evidence that support an assertion it has made.  They can view each citation and Watson's confidence that it is corroborative.  They can also verify or refute its relevancy, thereby supporting or eliminating its impact on the final answer.

Getting started

Inspired by the lean startup movement, my team aimed to build an MVP to put in front of users as quickly as possible.  The project was managed using Scrum Methodology and began with a two week "Sprint 0" discovery phase that included:

How to role-play your users

Once we were confident that the main visualization was a clear and elegant representation of Watson's decision making process, we began imagining ways for users to interact with it. 

Based on our initial discussions with medical students, as well as our own internal brainstorming sessions, I superimposed thought bubbles on top of the areas that could generate valuable interactions.  For each bubble, I imagined what a user might be thinking, what action their thought would trigger, and how that would provide value back to them.

Next, I converted the thought bubbles and their corresponding actions into epics that were added to the backlog.  Each epics included the object on the graph that the action would be initiated on as well as an estimated availability based on the design and engineering effort associated with it.  

User testing at Cleveland Clinic

Once we had enough of the system designed, I flew to Cleveland to conduct in-person user testing.  Each one-on-one session included an interview, hands-on testing, and an open discussion that allowed us gather additional ideas from the subjects.  Given that different parts of the product were in different stages of design and development, the tests included multiple fidelities, from fully implemented code to digital prototypes to paper prototypes. 

In order to create a setting that would ensure the subjects were relaxed and focused, I was the only person in the room with them.  Members of the WatsonPaths team observes the users’ screen via web conference and the room itself via webcam.  I did not want to juggle multiple tools during the sessions and used only a rainbow spreadsheet (below) to track whether or not each user demonstrated 41 different quantitative and qualitative conditions focused on the usefulness and usability of WatsonPaths.

 In a rainbow spreadsheet, each observation is assigned a row and each subject is assigned a column.  As a session progresses, I mark whether or not the condition being assessed has been fulfilled and add a check or 'x' for every row.  If a row is filled with checks, I can be reasonably certain that the design's intent has been achieved.  If a row includes mostly 'x's, I've identified an opportunity for improvement.  

For each row that had a high failure rate, I created a new story in the backlog.  A story prompts teams to perform work, and the ultimate outcome of user testing should be new stories that address the issues that were uncovered.  The Cleveland Clinic user tests resulted in 10 new stories. Below is an example subset.

Videos

Results