Unmoderated Remote User Testing, Task Analysis
I was part of a team of four usability experts from Pratt Institute School of Information and Library Science. We conducted a series of nine Unmoderated / Asynchronous Remote User Tests to determine how specific tasks were being accomplished by users on the MoMA website.
A kick-off meeting with museum staff shaped the way the testing was carried out to ensure museum goals were being met. We were provided with already established information, such as personas groups and previous surveys and testing methods. The objective was to determine potential improvements for MoMA.org’s event discoverability through researching the user’s journey from search to ticket purchase.
We observed users’ abilities in searching, filtering and purchasing a ticket to their target event(s). Tasks were designed and users were tested on a remote user testing platform UserTesting (www.usertesting.com). The team and I were able to determine if MoMA’s website could clearly, effectively and accurately offer the valued information to users on their smartphones. After careful analysis of testers’ rate in completing tasks, feedback on the difficulty of the task, and screen recordings the findings and results were presented in a report.
In total, 12 users were screened to meet the requirements, while the test required a mobile device, the range of expertise with the device and operating system were left open, since art museum visitors’ technical expertise varies between novice to expert. Three tasks were developed to be performed on the MoMA.org website on the remote users mobile phones and were recorded by the software. The participants completed the tasks on their own, and as administrators, and the team then reviewed the videos of their experiences.
We employed the Unmoderated/ Asynchronous Remote User Test available with a subscription on UserTesting.com. *Note there are two type of remote testing available
Moderated remote tests are conducted synchronously with a (human) moderator leading the study through video chat and screensharing software (i.e., WebEx, Skype).
Unmoderated remote tests are conducted asynchronously with specific software that leads participants through the study (i.e., UserTesting, TryMyUI).
Testers were provided with the following premise:
You are planning for things to do this upcoming weekend in New York City. They were then asked to perform the following tasks on their mobile device:
You want to attend a lecture/ panel. Use the MoMA website to find a lecture you can attend.
Purchase a ticket to a lecture of your choice.
Once users completed the testing, they answered the following questions:
Certian limitations of unmoderated/asynchronous remote user testing must be considered in any study that uses this method. Some specific instances of limitations found in this test:
Scrolling Through Long Videos, and Slow/Fast TalkersThe rate of speed of speech varies per each tester. Researchers can use the option to speed up the play rate of the video on UserTesting.com, but that means it will skip portions that may be crucial in that they hold critical feedback.
Language/Slang LimitationsAs moderators, we limited the feedback to only include English speakers, but that doesn’t necessarily limit the proficiency and/or articulation in the speech of the testers. No issues reported in this case.
Gender DemographicsNarrowing a target market is proposed by the researcher, but defined by the system/algorithm of UserTesting.com. Essentially, defining the pool of testers is not exactly in the researcher’s hands, so target market matching is highly dependent on the software. In this case, we determined an age range, but had to edit it to the confines of the age range provided by UserTesting.com. We skipped defining target market by gender, because UserTesting.com only allows identification by a male/female binary.
By administering a Unmoderated/Asynchronous Remote User Test on UserTesting.com we were able to determine several instances where users were impacted while navigating MoMA.org. Since we tasked testers to find and buy tickets for a particular event, we looked at the instances in which they encountered difficulties performing and completing related tasks.
TASK DIFFICULTY RATINGS
1 (Very difficult) — 5 (Very easy)
AVERAGE 4.25 | MODE 4
TASK 1 Use the MoMA website to find a lecture you can attend.
AVERAGE 4 | MODE 5
TASK 2 Purchase a ticket to a lecture of your choice.
TASK 1: You want to attend a lecture/panel. Use the MoMA website to find a lecture you can attend.
Task Analysis: Some users mentioned they used the search tool to find a lecture, because the link to the Events page is not noticeable. It should be noted that ‘Exhibitions/Events’ is the current navigation label that produces the drop-down menu. In the drop-down, users can link to the Events page. This navigation label is seemingly crowded—this is either due to the minimal space on mobile screens, the stark contrast that is produced between the black font and the white background, or both.
Once at the Events page, users experienced difficulty using the events search filters, due to the limited, yet unsorted range and category selectors. The difficulty in finding the Event page is amplified when users attempt to find a lecture. The search filters that MoMA provides are not obvious to users. One user sums up the frustration: “Filter is very helpful, but not obvious.” In this case, documentation can help users learn to use the tool. The sub-list of categories can also use organization that reflects a system that attendees can relate to while making searches.
First, MoMA can consider splitting up Exhibitions/Events into Exhibitions and Events. On the other hand, they can simplify the navigation menu label to just ‘Events,’ since a drop-down menu with sub-options populates, when the user selects that label. Through more testing, MoMA can determine what label easily guides users in their events searches.
The search filter for events categories is automatically set to ‘Everything.’ This label can be updated to ‘Choose Event Type’ or ‘Select Event Type’ to clue users on what steps they can take to narrow searches. Further, the icon can be replaced by an arrow facing downward, which is a more intuitive icon than the one currently in place. Users also reported that the search filters are not immediately obvious. Adding a color that contrasts with the white background and grey buttons can be beneficial in guiding users.
Task 1: You want to attend a lecture/panel. Use the MoMA website to find a lecture you can attend.
Task Analysis: Selecting ‘Tomorrow’ or a similar time period of events to view displays events for that time period and onward. The search results are not confined to the particular date selected, causing confusion. Some users reported feelings of frustration; they explained that they were not able to only look at events for one day at a time. They would like be able to choose a date range that provides results specific to those days only.
Events search results should be limited to the specific day or date range that the user has selected. Currently, the results display the date chosen as well as the events on days that follow that date.
Task 2: Purchase a ticket to a lecture of your choice.
Task Analysis: Task Analysis: Free events do not state if ticket purchase is necessary. Instead, the site proceeds the user to a page where they purchase a general day ticket for museum. Users were observed in their attempts to purchase admission tickets, when they are actually looking for event tickets. It is unclear of the process a visitor should take when purchasing a ticket to an event.
When users seek information attending an event at MoMA, they must first navigate directly into pages for individual events to find ticketing information omitted from the primary listing of events. They are unsure if the event requires a separate ticket, and, if so, how to purchase it. This confusion comes mostly from free events, which aren’t labeled with a clear statement that tickets, apart from general museum entrance, are not required. When a user attempts to purchase a ticket to a paid event, they seek a general admission ticket with confidence, but are still left feeling unsure on how to purchase a separate ticket for a particular event. Ticketed events provide a ‘Buy Tickets’ button, but the page that populates is extremely different, in regards to branding, and some shows are sold out by the time you get to that page.
On the event page tell users if the event requires a separate ticket in addition to the museum admission. More information can also be provided, by using text tags like, “Sold Out” and “Free Event”. The ‘Buy Tickets’ button requires more support for users to fully comprehend the actions the link provides.
MoMA showcases the most thought-provoking modern and contemporary art in the country to millions of visitors each year. The museum offers events to the public and members, which creates opportunity of discussion within the art community. Many users are accessing content on mobile devices and locally utilizing the calendar to attend events at the museum.
Upon careful analysis of user feedback by the usability experts, the team offered three suggestions. In order to help MoMA give their users the best possible experience on mobile devices, certain features of the interface should be updated. The team wants to make it easier for users to locate and search for events. By updating labels, filters and event specific details on ticket purchases, users will have a better experience. They will be more likely to purchase tickets and attend events.