Header

UZH-Logo

Maintenance Infos

Scalable visualization of large datasets


Steiner, David. Scalable visualization of large datasets. 2018, University of Zurich, Faculty of Economics.

Abstract

An exponential growth of datasets from different fields of science creates the need for scalable visualization systems to display and explore the data interactively. Such datasets include laser scans of architecture or cultural heritage, which can consist of many hundred millions or even billions of points. Other examples include high-resolution X-ray microtomographies of objects that need to be closely examined in a non-destructive manner, in fields like biology, medicine, or anthropology. The resulting volumetric models can capture details in the micrometer range and also often consist of many billions of points.

Visualizing such datasets at interactive frame rates poses a major challenge to the underlying rendering system, as it often means to process gigabytes of data within a time frame of only a few milliseconds. Consequently, there are high demands regarding the system's throughput and latency. These are often met via scaling the system, i.e., allowing it to accomodate more workload.

Strategies for scaling can include making better use of the available resources, e.g., reducing bandwidth requirements and computational costs. A specific example is our volume visualization system that we extended to allow interactive filtering of volume models (e.g., for feature detection or denoising) in the tensor-compressed domain. These filter operations can be performed significantly faster than with comparable approaches, due to reduced computational and bandwidth costs.

More significantly, a visualization system can be scaled by utilizing additional resources within a machine, or additional machines. Especially the latter creates further challenges, such as additional communication and synchronization overheads as well as load imbalances. For the development of scalable visualization systems, overcoming such load imbalances is critical, especially when facing the unpredictable load often created by user interaction. Similarly, the amount of available resources might fluctuate, if a machine is not dedicated to only a single task, e.g., in the context of virtualization.

We consequently developed a scalable and flexible rendering task partitioning method and associated node affinity model which allow fine-grained implicit dynamic load balancing via a task pulling mechanism. Our method often outperforms traditional load balancing approaches in terms of performance and scalability, especially in the context of unpredictable load and varying compute resources.

Furthermore, we conducted a study in which we in detail examined the scalability of various load balancing methods provided by the Equalizer parallel rendering framework, which our visualization systems are based on. Finally, we also extended the set of utilities provided by the framework, providing diverse features for alleviating tasks like systematically, reproduceably, and automatically evaluating the performance of scalable visualization systems, the collection of data, and using optimized I/O.

Abstract

An exponential growth of datasets from different fields of science creates the need for scalable visualization systems to display and explore the data interactively. Such datasets include laser scans of architecture or cultural heritage, which can consist of many hundred millions or even billions of points. Other examples include high-resolution X-ray microtomographies of objects that need to be closely examined in a non-destructive manner, in fields like biology, medicine, or anthropology. The resulting volumetric models can capture details in the micrometer range and also often consist of many billions of points.

Visualizing such datasets at interactive frame rates poses a major challenge to the underlying rendering system, as it often means to process gigabytes of data within a time frame of only a few milliseconds. Consequently, there are high demands regarding the system's throughput and latency. These are often met via scaling the system, i.e., allowing it to accomodate more workload.

Strategies for scaling can include making better use of the available resources, e.g., reducing bandwidth requirements and computational costs. A specific example is our volume visualization system that we extended to allow interactive filtering of volume models (e.g., for feature detection or denoising) in the tensor-compressed domain. These filter operations can be performed significantly faster than with comparable approaches, due to reduced computational and bandwidth costs.

More significantly, a visualization system can be scaled by utilizing additional resources within a machine, or additional machines. Especially the latter creates further challenges, such as additional communication and synchronization overheads as well as load imbalances. For the development of scalable visualization systems, overcoming such load imbalances is critical, especially when facing the unpredictable load often created by user interaction. Similarly, the amount of available resources might fluctuate, if a machine is not dedicated to only a single task, e.g., in the context of virtualization.

We consequently developed a scalable and flexible rendering task partitioning method and associated node affinity model which allow fine-grained implicit dynamic load balancing via a task pulling mechanism. Our method often outperforms traditional load balancing approaches in terms of performance and scalability, especially in the context of unpredictable load and varying compute resources.

Furthermore, we conducted a study in which we in detail examined the scalability of various load balancing methods provided by the Equalizer parallel rendering framework, which our visualization systems are based on. Finally, we also extended the set of utilities provided by the framework, providing diverse features for alleviating tasks like systematically, reproduceably, and automatically evaluating the performance of scalable visualization systems, the collection of data, and using optimized I/O.

Statistics

Downloads

35 downloads since deposited on 25 Jan 2019
28 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Dissertation (monographical)
Referees:Pajarola Renato, Staadt Oliver
Communities & Collections:03 Faculty of Economics > Department of Informatics
Dewey Decimal Classification:000 Computer science, knowledge & systems
Language:English
Place of Publication:Zürich
Date:2018
Deposited On:25 Jan 2019 11:27
Last Modified:26 Feb 2020 04:17
Number of Pages:113
OA Status:Green
Other Identification Number:merlin-id:17264

Download

Green Open Access

Download PDF  'Scalable visualization of large datasets'.
Preview
Content: Published Version
Filetype: PDF
Size: 34MB