IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i22p4589-d1276927.html
   My bibliography  Save this article

Model Compression Algorithm via Reinforcement Learning and Knowledge Distillation

Author

Listed:
  • Botao Liu

    (School of Computer Science, Yangtze University, Jingzhou 434025, China)

  • Bing-Bing Hu

    (School of Computer Science, Yangtze University, Jingzhou 434025, China)

  • Ming Zhao

    (School of Computer Science, Yangtze University, Jingzhou 434025, China)

  • Sheng-Lung Peng

    (Department of Creative Technologies and Product Design, National Taipei University of Business, Taipei 10051, Taiwan)

  • Jou-Ming Chang

    (Institute of Information and Decision Sciences, National Taipei University of Business, Taipei 10051, Taiwan)

Abstract

Traditional model compression techniques are dependent on handcrafted features and require domain experts, with a tradeoff between model size, speed, and accuracy. This study proposes a new approach toward resolving model compression problems. Our approach combines reinforcement-learning-based automated pruning and knowledge distillation to improve the pruning of unimportant network layers and the efficiency of the compression process. We introduce a new state quantity that controls the size of the reward and an attention mechanism that reinforces useful features and attenuates useless features to enhance the effects of other features. The experimental results show that the proposed model is superior to other advanced pruning methods in terms of the computation time and accuracy on CIFAR-100 and ImageNet dataset, where the accuracy is approximately 3% higher than that of similar methods with shorter computation times.

Suggested Citation

  • Botao Liu & Bing-Bing Hu & Ming Zhao & Sheng-Lung Peng & Jou-Ming Chang, 2023. "Model Compression Algorithm via Reinforcement Learning and Knowledge Distillation," Mathematics, MDPI, vol. 11(22), pages 1-12, November.
    • Handle: RePEc:gam:jmathe:v:11:y:2023:i:22:p:4589-:d:1276927
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/22/4589/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/11/22/4589/
    Download Restriction: no
    ---><---

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:22:p:4589-:d:1276927. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.