THE FOLLOW-SOLVE-LISTEN TEST (FSLT): A SAMPLE TASK FOR MEASURING COGNITIVE WORKLOAD

The Follow-Solve-Listen Test (FSLT) is a three-part test, with each part requiring about 45 seconds of response time from the subject. The task is available for general use and may be of interest to researchers who use a variety of different sensors, not just eye tracking.  It can be downloaded at http://www.eyetracking.com/tests/follow-solve-listen

Screen images of the three FSLT subtests

Screen images for the three subtests are shown here. The first subtest of the FSLT requires the subject to follow one of several dots moving randomly around the left half of the screen. The target dot is initially colored red and then turns yellow like all the other dots, and all dots move slowly around the display. At the conclusion of the test, all dots drift to the bottom of the display and are identified by letter (A, B, C, etc.). The subject says the letter that corresponds to the dot he was following.

The second subtest adds another component to the task. On the left side of the display, dots move as before. On the right side are numbers which are to be added or subtracted. The numbers appear one at a time, and the subject needs to make the indicated mental calculations. At the same time, the subject must keep track of the dot he is following. At the end of the task, the numbers disappear, the dots drift to the bottom, and the subject makes two responses: the letter corresponding to the dot he followed and the result of his mental calculations.

The third subtest retains both of the components of the second subtest and adds a third. A wheel is positioned in the center of the screen and turns clockwise or counterclockwise during the test. Its direction changes randomly every few seconds. At 10-second intervals, a tone sounds, and the subject needs to say whether the wheel was moving in a clockwise or counterclockwise direction when the tone sounded. At the same time, of course, the subject needs to track the dot and solve the arithmetic problem. Final subject responses are the letter corresponding to the dot and the result of the mental calculations.

Most individuals find the FSLT to be difficult but not impossible. Even if individuals cannot perform correctly, they nonetheless almost always try, which means that they are still exerting mental effort. The entire test takes only about 5 minutes and yields three distinct subtest estimates of an individual’s cognitive workload.

The FSLT has been used successfully with 9 different eye trackers that produce suitably accurate pupil information. On each hardware system, we tested between 30 and 50 subjects, for a total of more than 300 individuals. Whenever feasible, we tested an individual twice on two different eye trackers (using a second version of the FSLT with different dot patterns and numbers). Eye trackers were usually provided to us by the manufacturers. Subjects were recruited from the local community and were paid for their participation. Our subjects were males and females from a range of ethnicities, ages, and general abilities. We excluded only individuals with known eye problems, hard contact lenses, and/or bifocals.

How did we analyze the data? We began with a traditional repeated-measures analysis of variance with one factor (task with three levels). This test allows us to ask the primary question: Were overall differences in group performance observed across the three levels of the FSLT? For all systems, the ICA performed as expected across the three task levels: Follow was significantly lower than Follow+Solve which in turn was significantly lower than Follow+Solve+Listen. All statistical tests of means for each eye tracker were significant beyond α=0.05 on repeated measures analyses of variance F-tests and on all pairwise Sidak comparisons. Further, when all data were combined into a single analysis with one within-subject factor of task (3 tasks) and one between-subject factor of tracker (9 trackers), the same results were obtained: the task factor was significant (multivariate F(2,326)=165.91, p<.0.001) and the tracker factor was not significant (multivariate F(8,327)<1.0). The linear trend was significant as were all pairwise comparisons between tasks.

What do these statistical results tell us? First, no matter which system is being used (head mounted or remote) and no matter what the sampling rate of the eye tracker (30 Hz up to 500 Hz), the ICA produces similar estimates of mental workload for the FSLT. The metric is not dependent upon the hardware used to collect the data (although it does require sufficient precision from the eye tracker).

Second, these results provide useful individual estimates of cognitive workload for each of our subjects. Most subjects produced the pattern we expected (low, medium, high) but some did not. Reviewing data and videos of the task for these individuals, we usually found that they got confused on one of the later tasks and either quit trying to calculate the numbers or quit trying to follow the dot. It was quite evident from their pattern of eye data what was going on and when they essentially gave up on one component of the task. The result was a lower than expected estimate of mental activity from the ICA for a specific subtest, but the result was consistent with their behavior.

And third, these results also provide quantitative task load estimates for our three subtasks. Using data from our 336 subjects over all eye trackers, we now have stable estimates of the task loads for the components of the FSLT. The average Index values across all systems were 0.289, 0.339, and 0.360 for Follow, Follow+Solve, and Follow+Solve+Listen respectively. Although the differences among the three seem small, we nonetheless observed statistically significant differences among them.