Building DoRA Support for Embedding Layers in PEFT

Introduction

Contributing to open source can be a daunting experience, especially if you're new to the community or tackling a challenging feature. However, it's also one of the most rewarding experiences a developer can have. It’s an opportunity to collaborate with brilliant minds, learn from others, and make a tangible impact on widely-used software.

In this blog post, I’ll take you through my journey of contributing to the PEFT repository in Hugging Face by implementing DoRA support for embedding layers, and I hope to inspire others to take the plunge into open-source contributions.

The Beginning: Spotting the Opportunity

It all started with an issue titled “DoRA support for Embedding.” The original post requested support for DoRA (Weight-Decomposed Low-Rank Adaptation) with embedding layers to integrate seamlessly with the LoRA (Low-Rank Adaptation) framework.

The request was motivated by the need to fine-tune a model with additional tokens. Benjamin, the maintainer of the repository, responded, acknowledging that DoRA was not yet supported for embeddings and inviting community contributions.

Taking on the Challenge

As someone who was interested in the DoRA paper, I found this issue particularly intriguing. I took on the challenge of implementing the embedding module, knowing that it would be a significant contribution to the community. However, it wasn’t easy. My first commit took some time as I had to understand the inner workings of the repository, understand how LoRA adapters were injected into a module, and grasp how DoRA adapter weights worked for Linear and Convolutional layers (as they were already implemented).

After much research and effort, I pushed my first commit, marking the start of the pull request.

A Word of Encouragement

The response from Benjamin was encouraging. He remarked that the code looked good and suggested that while some refactoring might be needed in the future, we should keep the current implementation as is. He also pointed out that there wasn’t any existing reference for applying DoRA to embedding layers, which made this contribution particularly exciting.

This feedback was incredibly motivating. It felt amazing to be at the forefront of developing a new feature that could potentially set a new standard.

He proposed the next step: testing the new DoRA embedding layer against existing examples and comparing the results to LoRA embeddings.

A Thrilling Suggestion

The excitement didn’t stop there. Sayak, another prominent figure in the open-source community, suggested requesting a review from the DoRA author, similar to what had been done when adding Conv DoRA support. The idea of having my work reviewed by the author of DoRA was both thrilling and intimidating. It opened up the possibility of making mistakes, but it also presented an incredible learning opportunity.

The Learning Curve: Writing Tests and Finding Bugs

As the PR progressed, Benjamin reviewed my code and suggested that I write tests to ensure everything was functioning correctly. This was a critical step, as it led to the discovery of a major bug—one that I had completely overlooked. While writing the tests, I realized that I had was forward propagating the inputs incorrectly through the DoRA embedding layer.

It was a face-palm moment, but it also underscored the importance of always writing test cases.

An Unforgettable Moment: Feedback from the DoRA Author

Then came a moment that I will always cherish. Shih-Yang, the first authors of DoRA, joined the conversation and reviewed my implementation. He suggested a potential optimization for inference speed, noting that since there was no dropout on the input for embedding layers, we could skip one forward call. It was a small but significant detail that could improve performance.

Being part of this discussion felt surreal. I was surrounded by brilliant minds who were far more experienced than me, yet here I was, contributing to a conversation that would directly impact the codebase. It was a humbling experience, and I was grateful for the opportunity.

Conclusion: The Joy of Contributing to Open Source

This journey taught me a lot about coding, collaboration, and the open-source community. It reinforced the idea that contributing to open source isn’t just about writing code; it’s about engaging with others, learning, and growing as a developer.

If you’re considering contributing to open source, I encourage you to take the leap. Start with something small, engage with the community, and don’t be afraid to make mistakes. The experience is invaluable, and the satisfaction of seeing your code in a widely-used library is unmatched.

I’m excited to see how the community will use the new DoRA embedding layer, and I’m already looking forward to my next contribution. Happy coding!