[SUBMISSION] December 2024 - Module 2: Preference Alignment

[jucanbe]

December 2024 Student Submission

Module Completed

• Module 1: Instruction Tuning
• Module 2: Preference Alignment
• Module 3: Parameter-efficient Fine-tuning
• Module 4: Evaluation
• Module 5: Vision-language Models
• Module 6: Synthetic Datasets
• Module 7: Inference
• Module 8: Deployment

Changes Made

Describe what you've done in this PR:

1. What concepts did you learn?
   How to do DPO and ORPO, and how to work with the datasets in each case
2. What changes or additions did you make?
   Working with a different dataset in the ORPO case, and testing the models in each case
3. Any challenges you faced?

Notebooks Added/Modified

List any notebooks you've added or modified:

• Added new example in module_name/student_examples/my_example.ipynb
• Modified existing notebook with additional examples
• Added documentation or comments

Checklist

• I have read the module materials
• My code runs without errors
• I have pushed models and datasets to the huggingface hub
• My PR is based on the december-2024 branch

Questions or Discussion Points

Add any questions you have or points you'd like to discuss:
1.
2.

Additional Notes

Any other information that might be helpful for reviewers:

[burtenshaw]

Nice work @jucanbe !

Would you like to take part in peer review? If so, mention me on a PR from another student, review it, and I'll get a student to review yours.