MIT introduces Self-Distillation Fine-Tuning to reduce catastrophic forgetting; it uses student-teacher demonstrations and needs 2.5x compute.

Knowledge Distillation (KD) has been established as an effective technique for reducing the resource requirements of models when tackling computer vision tasks. Prior work has studied how to distill ...

Anthropic accused three Chinese AI firms of engaging in concerted "distillation attack" campaigns. U.S. companies like Anthropic and OpenAI are concerned with ceding a competitive advantage to such ...

OpenAI has accused DeepSeek of malpractice in developing the next version of its artificial intelligence model — even before any official launch. “DeepSeek’s next model (whatever its form) should be ...

In mathematics, proofs can be written down and shared. In cryptography, when people are trying to avoid revealing their secrets, proofs are not always so simple—but a new result significantly closes ...

Abstract: Knowledge distillation is a popular technique for transferring the knowledge of a teacher model to a smaller and more efficient student model. However, previous work often used certain ...

This repository showcases a complete pipeline for high-quality Image Sharpening using Knowledge Distillation (KD). A pretrained Restormer model acts as the high-capacity teacher, while a lightweight ...

The original version of this story appeared in Quanta Magazine. The Chinese AI company DeepSeek released a chatbot earlier this year called R1, which drew a huge amount of attention. Most of it ...