Skip to content

Commit

Permalink
DOC Improve target modules description (#1290)
Browse files Browse the repository at this point in the history
For LoRA and IA³, it is allowed to not specify a target module, in which
case the correct layers are derived from the model architecture. This
was not documented so far.
  • Loading branch information
BenjaminBossan authored Dec 21, 2023
1 parent 1c9679a commit 993836f
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 6 deletions.
13 changes: 10 additions & 3 deletions src/peft/tuners/ia3/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,10 @@ class IA3Config(PeftConfig):
Args:
target_modules (`Union[List[str],str]`):
The names of the modules to apply (IA)^3 to.
The names of the modules to apply (IA)³ to. If this is specified, only the modules with the specified names
will be replaced. If this is not specified, modules will be chosen according to the model architecture. If
the architecture is not known, an error will be raised -- in this case, you should specify the target
modules manually.
feedforward_modules (`Union[List[str],str]`):
The names of the modules to be treated as feedforward modules, as in the original paper. These modules will
have (IA)^3 vectors multiplied to the input, instead of the output. feedforward_modules must be a name or a
Expand All @@ -44,8 +47,12 @@ class IA3Config(PeftConfig):
target_modules: Optional[Union[List[str], str]] = field(
default=None,
metadata={
"help": "List of module names or regex expression of the module names to replace with ia3."
"For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$' "
"help": (
"List of module names or regex expression of the module names to replace with (IA)³."
"For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'. "
"If not specified, modules will be chosen according to the model architecture, If the architecture is "
"not known, an error will be raised -- in this case, you shoud specify the target modules manually."
),
},
)
feedforward_modules: Optional[Union[List[str], str]] = field(
Expand Down
13 changes: 10 additions & 3 deletions src/peft/tuners/lora/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,10 @@ class LoraConfig(PeftConfig):
Args:
r (`int`): Lora attention dimension.
target_modules (`Union[List[str],str]`): The names of the modules to apply Lora to.
target_modules (`Optional[Union[List[str], str]]`): The names of the modules to apply LoRA to. If this is
specified, only the modules with the specified names will be replaced. If this is not specified, modules
will be chosen according to the model architecture. If the architecture is not known, an error will be
raised -- in this case, you should specify the target modules manually.
lora_alpha (`int`): The alpha parameter for Lora scaling.
lora_dropout (`float`): The dropout probability for Lora layers.
fan_in_fan_out (`bool`): Set this to True if the layer to replace stores weight like (fan_in, fan_out).
Expand Down Expand Up @@ -82,8 +85,12 @@ class LoraConfig(PeftConfig):
target_modules: Optional[Union[List[str], str]] = field(
default=None,
metadata={
"help": "List of module names or regex expression of the module names to replace with Lora."
"For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$' "
"help": (
"List of module names or regex expression of the module names to replace with LoRA."
"For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'. "
"If not specified, modules will be chosen according to the model architecture, If the architecture is "
"not known, an error will be raised -- in this case, you shoud specify the target modules manually."
),
},
)
lora_alpha: int = field(default=8, metadata={"help": "Lora alpha"})
Expand Down

0 comments on commit 993836f

Please sign in to comment.