HDRec

Paper

• HDRec: Hierarchical Distillation for Enhanced LLM-based Recommendation Systems.
• Lingyan Zhang, Wanyu Ling, Shuwen Daizhou, and Li Kuang

Architecture

Instruction

Step. 1 hierarchical distillation before pretraining

(a) Install llama 3（download model weights and tokenizer）

    get the License from [the site](https://llama.meta.com/llama-downloads/)
    >> cd llama 
    >> ./download.sh (License required)
    >> pip install -e .

(b) Test llama 3 environment (under ./llama )

    >> torchrun --nproc_per_node 1 example_chat_completion.py \
      --ckpt_dir llama-3-8b-instruct/ \
      --tokenizer_path tokenizer.model \
      --max_seq_len 512 --max_batch_size 6

(c) Hierarchical distillation ({dataset}: beauty, sports, and toys.) (under ./HDRec )

    >> torchrun --nproc_per_node 1 data/{dataset}/distillation_{dataset}.py \
      --ckpt_dir llama/llama-3-8b-instruct/ \
      --tokenizer_path llama/tokenizer.model \
      --max_seq_len 512 --max_batch_size 6

HDRec involves multiple stages of distillation, and to obtain the final distilled information, you need to run through the entire hierarchical distillation process.

distillation_{dataset}.py for user_preference and item_attribution, distillation_{dataset}2.py for user_personality, distillation_{dataset}3.py for item_audience, distillation_{dataset}4.py for review_summary, distillation_{dataset}5.py for groundtruth_explanation.

Step. 2 train and test HDRec

(a) Install requirement

    >> pip install -r requirements.txt

(b) Pre-training ({dataset}: beauty, sports, and toys.) (under ./HDRec )

    >> python pretrain.py --data_dir ./data/{dataset}/ --cuda --batch_size 64 --checkpoint ./checkpoint/{dataset}/

(c) Recommendation inference

    >> python seq.py --data_dir ./data/{dataset}/ --cuda --batch_size 32 --checkpoint ./checkpoint/{dataset}/
    >> python topn.py --data_dir ./data/{dataset}/ --cuda --batch_size 32 --checkpoint ./checkpoint/{dataset}/
    >> python exp.py --data_dir ./data/{dataset}/ --cuda --batch_size 32 --checkpoint ./checkpoint/{dataset}/

Others

• All experiments, including rationale distillation, can be conducted on a single Nvidia V100 GPU (32GB memory). Reduce the batch size if CUDA out of memory.
• If you have any questions, please feel free to contact me at [email protected].

Code Reference

• RDRec
• llama 3