Extending the Library

Integrating new Transformer models

Currently, not all model types included in HuggingFace’s transformers support adapters yet. However, it is possible to the existing adapter implementation to new models. For a detailed guide, refer to Adding Adapters to a Model.

Loading custom module weights

adapter-transformers provides support for saving and loading adapter and prediction head modules from the local file system or the Hub out of the box. However, countless additional module integrations into language models are thinkable. To provide a basis for such new custom model plugins, adapter-transformers integrates a basic mechanism to save and load custom weights.

All adapter and head module weights are extracted, saved and loaded by implementations of the WeightsLoader class, the two preincluded being AdapterLoader and PredictionHeadLoader. To add basic saving and loading functionalities to your custom module weights, you can implement a new subclass of WeightsLoader. The two required abstract methods to be implemented are:

• filter_func(self, name: str) -> Callable[[str], bool]: The callable returned by this method is used to extract the module weights to be saved or loaded based on their names.

• rename_func(self, old_name: str, new_name: str) -> Callable[[str], str]: The callable returned by this method is used to optionally rename the module weights after loading.

For more advanced functionalities, you may also want to override the save() and load() method.

Using the custom loader class, weights can now be saved with:

loader = MyCustomWeightsLoader(model)
loader.save("path/to/save/dir", "custom_weights_name")

You can also upload these weights to the Hub and then load them from there together with an adapter:

model.load_adapter(
    "adapter_name",
    custom_weights_loaders=[MyCustomWeightsLoader]
)