AdapterHub Documentation

Warning

This is the documentation of the legacy adapter-transformers library, which has been replaced by the new Adapters library, found here: https://docs.adapterhub.ml.

This documentation is kept for archival purposes, and the library will not be updated in the future. Please use the new library for all active projects.

AdapterHub is a framework simplifying the integration, training and usage of adapters and other efficient fine-tuning methods for Transformer-based language models. For a full list of currently implemented methods, see the table in our repository.

The framework consists of two main components:

• adapter-transformers, an extension of Huggingface’s Transformers library that adds adapter components to transformer models

• The Hub, a central repository collecting pre-trained adapter modules

Currently, we support the PyTorch versions of all models as listed on the Model Overview page.

Getting Started

• Installation
  • Using pip
  • From repository
• Quickstart
  • Introduction
  • Quick Tour: Using a pre-trained adapter for inference
  • Quick Tour: Adapter training
• Adapter Training
  • Train a Task Adapter
  • Train a Language Adapter
  • Train AdapterFusion
  • AdapterTrainer

Adapter Methods

• Overview and Configuration
  • Table of Adapter Methods
  • Configuration
• Adapter Methods
  • Bottleneck Adapters
  • Language Adapters - Invertible Adapters
  • Prefix Tuning
  • Compacter
  • LoRA
  • (IA)^3
• Method Combinations
  • Mix-and-Match Adapters
  • UniPELT

Advanced

• Adapter Activation and Composition
  • Adapter Activation
  • Composition Blocks - Overview
  • Stack
  • Fuse
  • Split
  • BatchSplit
  • Parallel
  • Nesting composition blocks
• Prediction Heads
  • AdapterModel classes
  • Model classes with static heads (HuggingFace Transformers)
  • Automatic conversion
  • Custom Heads
• Embeddings
  • Adding and Deleting Embeddings
  • Saving and Loading Embeddings
• Extending the Library
  • Integrating new Transformer models
  • Loading custom module weights
• Transitioning from Earlier Versions
  • Transitioning to v3
  • Transitioning to v2

Loading and Sharing

• Loading Pre-Trained Adapters
  • Finding pre-trained adapters
  • Using pre-trained adapters in your code
  • How adapter resolving works
• Contributing Adapters to the Hub
  • Getting started
  • Add your pre-trained adapter
  • Add a new adapter architecture
  • Add a new task or subtask
• Integration with HuggingFace’s Model Hub
  • Downloading from the Hub
  • Uploading to the Hub

Supported Models

• Model Overview
• ALBERT
• Auto Classes
• BART
• Bidirectional Encoder representation from Image Transformers (BEiT)
• BERT
• BertGeneration
• CLIP
• DeBERTa
• DeBERTa-v2
• DistilBERT
• Encoder Decoder Models
• OpenAI GPT2
• EleutherAI GPT-J-6B
• MBart
• RoBERTa
• T5
• Vision Transformer (ViT)
• XLM-RoBERTa

Adapter-Related Classes

• Adapter Configuration
  • Single (bottleneck) adapters
  • Prefix Tuning
  • LoRAConfig
  • IA3Config
  • Combined configurations
  • Adapter Fusion
  • Adapter Setup
• Model Adapters Config
• Adapter Modules
• AdapterLayer
• Model Mixins
  • InvertibleAdaptersMixin
  • ModelAdaptersMixin
  • ModelWithHeadsAdaptersMixin
  • ModelWithFlexibleHeadsAdaptersMixin
  • PushAdapterToHubMixin
• Adapter Training
• Adapter Utilities

Contributing

• Contributing to AdapterHub
• Adding Adapter Methods
• Adding Adapters to a Model

Citation

@inproceedings{pfeiffer2020AdapterHub,
   title={AdapterHub: A Framework for Adapting Transformers},
   author={Jonas Pfeiffer and
            Andreas R\"uckl\'{e} and
            Clifton Poth and
            Aishwarya Kamath and
            Ivan Vuli\'{c} and
            Sebastian Ruder and
            Kyunghyun Cho and
            Iryna Gurevych},
   booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations},
   year={2020},
   address = "Online",
   publisher = "Association for Computational Linguistics",
   url = "https://www.aclweb.org/anthology/2020.emnlp-demos.7",
   pages = "46--54",
}

Indices and tables

• Index

• Module Index