Huawei & Peking U's DiJiang matches LLaMA2-7B with 1/50th the training cost

2 mins

Jude Foster

Published by: Jude Foster

08 April 2024, 10:50AM GMT+00:00

In Brief

Huawei & Peking U develop DiJiang, rivaling LLaMA2-7B at lower cost.

DiJiang uses Frequency Domain Kernelization, slashing AI training costs.

Model makes training AI up to 50 times cheaper and 10 times faster.

Technique reimagines data processing for efficiency without losing quality.

DiJiang's innovation could broaden AI's reach in daily tech applications.

Researchers from Huawei Noah’s Ark Lab and Peking University have created a groundbreaking model called DiJiang. It delivers the same punch as the heavyweight LLaMA2-7B but is 50 times cheaper to train. Detailed in their study, "DiJiang: Efficient Large Language Models through Compact Kernelization," this breakthrough uses a smart technique called Frequency Domain Kernelization to cut down drastically on the computing power needed to train large language models.

The Transformer architecture is at the heart of AI's language understanding, speech recognition, and more. But its thirst for larger, more complex models means sky-high computing costs. Enter DiJiang, a game-changer that introduces a simpler, more cost-effective way to do the heavy lifting. By reimagining how data is crunched, DiJiang significantly slashes the training expenses associated with these AI giants.

Digging into the nitty-gritty, the research team explored how to streamline data processing without losing accuracy or efficiency. They struck gold with a technique that changes how information is sampled and processed. Using something called the Discrete Cosine Transform, DiJiang cleverly rearranges the data to lighten the computational load. This innovation not only makes training models way cheaper, but also speeds up their "thinking" process, making them up to ten times faster than their predecessors.

DiJiang's success could revolutionize the AI field, making powerful models more accessible to everyone. Its efficiency and affordability mean we could soon see more advanced AI applications in everyday life, from better language tools to smarter technology interactions. This collaboration between Huawei Noah’s Ark Lab and Peking University is a testament to the power of teamwork in pushing the boundaries of what's possible with AI.

Bell notification
Blue mail
Blured bell
Blue Mail
Mail plane
Mail plane
Mail icon
Mail icon
Mail icon

Join our newsletter

Stay in the know on the latest alpha, news and product updates.