Distillation from Heterogeneous Models for Top-K Recommendation

This repository provides the source code of "Distillation from Heterogeneous Models for Top-K Recommendation" accepted in TheWebConf (WWW2023) as a research paper.

1. Overview

We present HetComp Framework that effectively compresses the valuable but difficult ensemble knowledge of heterogeneous models, generating a lightweight model with high recommendation performance.

2. Main Results

Training curves of w/o KD, DCD, and HetComp. Testing recall per 10 epochs. After convergence, we plot the last value.

2-a. Benchmark setup

2-b. Generalization setup

We found that the sampling processes for top-ranked unobserved items are unnecessary, and removing the processes gave considerable performance improvements for the ranking matching KD methods (i.e., RRD, MTD, CL-DRD, and DCD). For this reason, we remove the sampling process for all ranking matching methods in our experiments.

3. Requirements

3-a. Dataset

• A-music dataset can be downloaded from: http://jmcauley.ucsd.edu/data/amazon/
• CiteULike dataset can be downloaded from: https://github.com/js05212/citeulike-t/blob/master/users.dat
• Foursquare dataset can be downloaded from: https://github.com/allenjack/SAE-NAD

3-b. Software

• Python version: 3.6.10
• Pytorch version: 1.10.1

3-c. Else.

• The target teacher models and their trajectories need to be located in Teachers directory.
• Due to its large size, we provide pretrained teacher trajectories through another file-sharing system: https://drive.google.com/file/d/1IYNJBbhzi2ETcKzruYOKFsJ__3RwyPJM/view?usp=share_link

4. Follow-up study

Thank you for your attention to our work. We would like to introduce our follow-up study, "Continual Collaborative Distillation for Recommender System," presented at KDD '24. In this study, we investigated a systematic approach to applying HetComp to real-world data streams, where new users, items, and interactions are continuously added.

For more details, please refer to the following links: pdf / code