Efficient and portable distribution modeling for large-scale scientific data processing with data-parallel primitives

Hao Yi Yang, Zhi Rong Lin, Ko Chih Wang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)

Abstract

The use of distribution-based data representation to handle large-scale scientific datasets is a promising approach. Distribution-based approaches often transform a scientific dataset into many distributions, each of which is calculated from a small number of samples. Most of the proposed parallel algorithms focus on modeling single distributions from many input samples efficiently, but these may not fit the large-scale scientific data processing scenario because they cannot utilize computing resources effectively. Histograms and the Gaussian Mixture Model (GMM) are the most popular distribution representations used to model scientific datasets. Therefore, we propose the use of multi-set histogram and GMM modeling algorithms for the scenario of large-scale scientific data processing. Our algorithms are developed by data-parallel primitives to achieve portability across different hardware architectures. We evaluate the performance of the proposed algorithms in detail and demonstrate use cases for scientific data processing.

Original languageEnglish
Article number285
JournalAlgorithms
Volume14
Issue number10
DOIs
Publication statusPublished - 2021 Oct

Keywords

  • Data-parallel primitive
  • Distribution-based approach
  • Large-scale data processing
  • Parallel algorithm
  • Scientific dataset

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Numerical Analysis
  • Computational Theory and Mathematics
  • Computational Mathematics

Fingerprint

Dive into the research topics of 'Efficient and portable distribution modeling for large-scale scientific data processing with data-parallel primitives'. Together they form a unique fingerprint.

Cite this