View Full Version : Sifting Through a Trillion Electrons

06-27-12, 08:30 AM
https://www.nersc.gov/assets/NewsImages/2012/Trillion-Particles/_resampled/SetWidth230-VPIC1.jpg (https://www.nersc.gov/news-publications/news/science-news/2012/sifting-through-a-trillion-electrons/)What happens when your scientific instrument generates more data than you can decipher? Linda Vu writes that Berkeley researchers are designing new strategies for extracting interesting data from massive scientific datasets.

In a recent case, a team ran a state-of-the-art plasma physics code called VPIC on NERSC‚??s Cray XE6 ‚??Hopper‚?? supercomputer that generated a 3D magnetic reconnection dataset of a trillion particles.

Although our VPIC runs typically generate two types of data‚??grid and particle‚??we never did a whole lot with the particle data because it was really hard to extract information from a trillion particle dataset, and there was no way to sift out the useful information,‚?? said Homa Karimabadi, head of the space physics group at UCSD.

Using their new tools, the researchers wrote each 32 TB file to disk in about 20 minutes, at a sustained rate of 27 gigabytes per second. By applying an enhanced version of the FastQuery tool, the team indexed this massive dataset in about 10 minutes, then queried the dataset in three seconds for interesting features to visualize.

Read the Full Story (https://www.nersc.gov/news-publications/news/science-news/2012/sifting-through-a-trillion-electrons/).

http://insidehpc.com/?ak_action=api_record_view&id=30088&type=feedRelated posts:

Japanese Super Figures Pi to 2.5 Trillion Digits (http://insidehpc.com/2009/08/20/japanese-super-figures-pi-to-25-trillion-digits/)
Sci vis package VisIt tested on 2 trillion point datasets (http://insidehpc.com/2009/06/12/sci-vis-package-visit-performs-well-in-scale-tests/)
TACC Ranger Testing Technicolor Physics (http://insidehpc.com/2011/02/15/tacc-ranger-testing-technicolor-physics/)

http://feeds.feedburner.com/~ff/InsideHPC?d=yIl2AUoC8zA (http://feeds.feedburner.com/~ff/InsideHPC?a=HcL8pallz9M:_tl4UQ11SAw:yIl2AUoC8zA) http://feeds.feedburner.com/~ff/InsideHPC?i=HcL8pallz9M:_tl4UQ11SAw:F7zBnMyn0Lo (http://feeds.feedburner.com/~ff/InsideHPC?a=HcL8pallz9M:_tl4UQ11SAw:F7zBnMyn0Lo) http://feeds.feedburner.com/~ff/InsideHPC?i=HcL8pallz9M:_tl4UQ11SAw:V_sGLiPBpWU (http://feeds.feedburner.com/~ff/InsideHPC?a=HcL8pallz9M:_tl4UQ11SAw:V_sGLiPBpWU)

More... (http://feedproxy.google.com/~r/InsideHPC/~3/HcL8pallz9M/)

06-27-12, 02:02 PM
Yup, presenting huge amount of data in a meaningful way is a *very* difficult problem.