Standard data compression throws away the tiny details scientists need most. A new method keeps them while still shrinking files dramatically.
Researchers at the Department of Energy's SLAC National Accelerator Laboratory built the AI data compression tool to handle a coming flood of scientific data current storage systems can't keep up with. They published the work in Nature Machine Intelligence.
The Problem: Too Much Data, Too Little Storage
SLAC's Linac Coherent Light Source, an X-ray laser used to photograph atoms and molecules, will eventually fire up to a million pulses per second. Some instruments drawing on its full capability generate nearly a terabyte of data every second.
"There is going to be such a flood of data that there's really no way to handle it in the way we've done before," said Joshua Turner, the study's principal investigator and a lead scientist at SLAC.
Conventional compression makes that flood manageable, but it costs something real. Tiny speckles in X-ray images often carry the actual scientific signal, information about how a material's structure changes over time.
"Those speckles often reflect the underlying arrangement, disorder, or dynamics of a material," said Yuan Ni, the study's lead author. "If we lose them, we would lose unique scientific insights."
How the AI Data Compression Method Actually Works
The team's approach doesn't compress a dataset all at once. It first separates features by scale using a mathematical technique called wavelet analysis.
A neural network then compresses each scale of feature separately. Fine details get their own dedicated treatment instead of getting smoothed away by a single, generalized compression pass.
The results are significant. Depending on the data and the fidelity needed, the method typically achieves 10 to 100-fold reductions in file size, according to Ni.
The team tested it across a genuinely wide range of data. Materials science measurements, solar magnetic field readings, and ordinary photographs all worked.
The neural network adapted to each type on its own. It learned which features actually mattered for that specific kind of data.
A Bonus: Decompressing Only What You Need
Traditional compression forces researchers to decompress an entire file just to look at one small piece of it. That can take minutes, hours, or sometimes days.
"This method can decompress only the region of interest rather than the entire dataset," said Zhantao Chen, an assistant professor at the University of Texas at Austin who worked on the project as a SLAC research associate. "So it's much more efficient."
The researchers are clear this isn't meant to replace existing compression tools entirely. It's designed to work alongside them, adding a new option rather than displacing what already exists.
"Rather than replacing existing compression methods," Ni said, "our work provides an additional AI-based approach."
The project drew on contributors from UC Davis and Carnegie Mellon University as well. Training the neural networks required Perlmutter, a major DOE computing resource at Lawrence Berkeley National Laboratory.
What This Means for Miami
Miami's research institutions generate the same kind of data-heavy experiments this method targets. The University of Miami runs major medical imaging programs. FIU runs materials and environmental science labs.
Genomics, medical scans and climate sensor data all face the same tradeoff. Storage cost versus scientific detail.
This is also a useful, concrete example of AI solving a real infrastructure problem. Not just generating text or images.
For Miami institutions weighing where AI investment actually pays off, this is an easy case to make. Data compression that cuts storage costs without losing the details researchers need beats most AI pilot projects on value alone.
