Evaluation

Evaluation is an essential part of the development of new visualizations or visual analytics systems. Different methods can be used, e.g. eye tracking, interaction protocols, think aloud studies, perspective studies, or crowd sourcing experiments. Experiments can be conducted as lab studies, online studies, or field studies.

Eye tracking has become a widely used method to analyse user behaviour in marketing, neuroscience, human–computer interaction, visualization research, reading, scene perception and visual search. Apart from measuring completion times and accuracy rates during the performance of visual tasks in classical controlled user experiments, eye-tracking-based evaluations provide additional information on how visual attention is distributed and changes for a presented stimulus. Eye tracking devices record gaze points of a participant as raw data. Afterward, these gaze points can be aggregated into fixations and saccades, the two major categories of eye movement. In dynamic stimuli, smooth
pursuit is an additional data type used for analysis. Additionally, areas of interest (AOIs) can be defined to measure the distribution of attention between specific regions on a stimulus.

For the analysis of eye tracking data a large number of visualizations are available. Some examples include:

• Parallel Scanpath visualization [Raschke et al. 2012, Raschke et al. 2014]
• ISeeCube [Kurzhals et al. 2014]
• Gaze Stripes [Kurzhals et al. 2016]
• Radial Transition Graphs [Blascheck et al. 2012, Blascheck et al. 2017]
• Word-scale visualizations [Beck et al. 2015, Beck et al. 2016, Beck et al. 2017]

Perception studies have a long tradition in visualization [Cleveland and McGill 1984, Cleveland and McGill 1985, Cleveland and McGill 1986]. To find out how a visualization is perceived or to test different visual variables for their suitability for different data types, such experiments are useful. In most cases, one tries to measure the barely perceptible difference with a staircase method. But it can also be tried to find out the minimum threshold value when a difference is no longer perceptible.

Crowdsourcing experiments are online experiments and have the advantage in its capability to recruit a large number of participants within a short time period with relatively less amount of money. However, researchers loose control over potential confounds and external validity. Furthermore, it is hard to measure exact and trustworthy task completion times when it is unclear how or if participants were distracted from the task. Despite the trade-offs, online experiments are useful evaluation methods that can complement in-lab evaluations. It is particularly useful when researchers want to compare several visualization candidates through an experiment design (e.g., between-subjects study) with relatively low cost.