Bittensor has successfully trained a 72-billion parameter large language model named Covenant-72B on its decentralized network, marking the largest such decentralized training effort to date. The model, which matches the performance of Meta’s LLaMA-2-70B, was created using globally distributed compute power from over 70 providers without centralized infrastructure. This achievement signals a potential shift toward decentralized AI development and reduced reliance on major tech companies.
The decentralized artificial intelligence network Bittensor has completed training a 72-billion parameter large language model, named Covenant-72B. This represents the largest decentralized LLM pre-training event to date according to network data.
The model was trained on Bittensor’s Subnet 3, known as Templar, using compute power from more than 70 global nodes. The process operated without central infrastructure and allowed permissionless participation from contributors.
The training mechanism involved participants providing GPU compute in exchange for the network’s native token, TAO. Validators assessed output quality, and rewards were distributed based on contribution value, enabling dynamic node participation.
On performance benchmarks, Covenant-72B achieved a score of 67.1 on the MMLU benchmark. This performance is comparable to Meta’s LLaMA-2-70B model, as reported in the training results.
The model was trained on approximately 1.1 trillion tokens. The project’s weights and checkpoints have been released publicly, enabling further research and innovation.
This milestone addresses a key industry question about whether decentralized systems can compete with centralized labs. Implications include reduced dependence on big tech and lower barriers to AI development.
The achievement has impacted the crypto AI sector, with the token TAO seeing substantial price increases following the announcement. Interest in decentralized AI among investors and institutions has surged concurrently.
The project now stands as a leading endeavor at the intersection of blockchain and artificial intelligence. However, challenges remain in scaling distributed computing to larger model sizes and ensuring consistent quality.
