gy77/add freeze hook #1387

gy-7 · 2023-10-10T09:27:14Z

Motivation

Motivation:

Freeze some parameters of the model when training the model.

Goal:

Specify the epoch to freeze the specified network layer.
Available for all downstream repositories.

Modification

Add FreezeHook and FreezeHook unit tests.

Use cases

Network layers matching freeze_layers are freeze before freeze_iter/freeze_epoch starts.
Network layers matching unfreeze_layers are freeze before unfreeze_iter/unfreeze_epoch starts.
freeze_layers/unfreeze_layers matches network layers via regular expression
The index of iter/epoch starts at 0, with epoch=0 for the first epoch.
unfreeze_iter, unfreeze_epoch and unfreeze_layers are optional. If freeze_epoch/freeze_iter is not None, unfreeze_layers must not be None.
Only one of freeze_iter and freeze_epoch can be set, as well as unfreeze_iter and unfreeze_epoch.

ImageClassifier(
    (backbone):ResNet(
        ...
        (layer1):Sequential(...)
        (layer2):Sequential(...)
        (layer3):Sequential(...)
        (layer4):Sequential(...)
    )
    (neck):GlobalAveragePooling2d(...)
    (head):Linear(...)
)

Freeze the parameters of backbone before the start of 1st training epoch.

custom_hooks = [
...
dict(
    type="FreezeHook",
    freeze_layers="backbone.*",
    freeze_epoch=0)
]

Freeze the layer1 and layer2 parameters in the backbone before the start of 10th training epoch.

custom_hooks = [
...
dict(
    type="FreezeHook",
    freeze_layers="backbone.layer1.*|backbone.layer2.*",
    freeze_epoch=10)
]

Freeze the parameters of backbone before the start of 1st training epoch. Unfreeze the parameters of the the backbone before the start of 10th training epoch.

custom_hooks = [
 ...
 dict(
     type="FreezeHook",
     freeze_layers="backbone.*",
     freeze_epoch=0,
     unfreeze_layers="backbone.*",
     unfreeze_epoch=9)
]

The verbose parameter is used to determine whether to print the requires_grad variable for each model layer.

custom_hooks = [
 ...
 dict(
     type="FreezeHook",
     freeze_layers="backbone.*",
     freeze_epoch=1,
     verbose=True)
]

mmengine - INFO - backbone.conv1.weight requires_grad: True
mmengine - INFO - backbone.bn1.weight requires_grad: True
...
mmengine - INFO - head.light_head.weight requires_grad: True
mmengine - INFO - head.light_head.bias requires_grad: True

mmengine/hooks/freeze_hook.py

HAOCHENYE · 2023-10-11T07:31:10Z

mmengine/hooks/freeze_hook.py

+        unfreeze_epoch (int): The epoch number to start unfreezing layers.
+        unfreeze_layers (tuple[str]): Model layers containing the keyword in
+            unfreeze_layers will unfreeze the gradient.
+        log_grad (bool): Whether to log the requires_grad of each layer.


Suggested change

log_grad (bool): Whether to log the requires_grad of each layer.

verbose (bool): Whether to log the requires_grad of each layer.

HAOCHENYE · 2023-10-11T07:33:13Z

mmengine/hooks/freeze_hook.py

+
+    Args:
+        freeze_epoch (int): The epoch number to start freezing layers.
+        freeze_layers (tuple[str]): Model layers containing the keyword in


Suggest making freeze_layers the first argument, and it should be a regex expression

mmengine/hooks/freeze_hook.py

HAOCHENYE · 2023-10-11T07:38:25Z

mmengine/hooks/freeze_hook.py

+        self.unfreeze_layers = unfreeze_layers
+        self.log_grad = log_grad
+
+    def modify_layers_grad(self, model, layers, requires_grad):


Suggested change

def modify_layers_grad(self, model, layers, requires_grad):

def _modify_layers_grad(self, model, layers, requires_grad):

Please update the type hint

HAOCHENYE · 2023-10-11T07:38:33Z

mmengine/hooks/freeze_hook.py

+                    v.requires_grad = requires_grad
+                    break
+
+    def log_model_grad(self, model, log_grad=False):


Suggested change

def log_model_grad(self, model, log_grad=False):

def _log_model_grad(self, model, log_grad=False):

mmengine/hooks/freeze_hook.py

HAOCHENYE · 2023-10-30T12:00:40Z

mmengine/hooks/freeze_hook.py

+
+    def __init__(
+        self,
+        freeze_layers: Union[Sequence[str], str],


Since it has been a regex expression, it is not necessary to make it a tuple of str ('exp1|exp2|exp3' is enough)

HAOCHENYE · 2023-10-30T12:04:08Z

mmengine/hooks/freeze_hook.py

+                      (tuple, list)) and not isinstance(freeze_layers[0], str):
+            raise TypeError(
+                '`freeze_layers` must be a tuple or list of string')
+        if not isinstance(freeze_iter, (int, type(None))):


Suggested change

if not isinstance(freeze_iter, (int, type(None))):

if not isinstance(freeze_iter) and freeze_iter is not None:

HAOCHENYE · 2023-10-30T12:04:57Z

mmengine/hooks/freeze_hook.py

+        if not isinstance(verbose, bool):
+            raise TypeError('`verbose`  must be a boolean')
+        # check arguments value
+        if freeze_iter and freeze_iter < 0:


Suggested change

if freeze_iter and freeze_iter < 0:

if freeze_iter is not None and freeze_iter < 0:

HAOCHENYE · 2023-10-30T12:05:09Z

mmengine/hooks/freeze_hook.py

+        if freeze_iter and freeze_iter < 0:
+            raise ValueError(
+                '`freeze_iter` must be greater than or equal to 0')
+        if freeze_epoch and freeze_epoch < 0:


Suggested change

if freeze_epoch and freeze_epoch < 0:

if freeze_epoch is not None and freeze_epoch < 0:

Merge this check into:

if (freeze_iter is None) ^ (freeze_epoch is None): raise ValueError(...) if freeze_iter is not None and freeze_iter < 0: raise ValueError(...) if freeze_epoch is not None and freeze_epoch < 0: raise ValueError(...)

HAOCHENYE · 2023-10-30T12:18:13Z

mmengine/hooks/freeze_hook.py

+        """Modify the `requires_grad` of the specified layers.
+
+        Args:
+            model (BaseModel): a BaseModel of mmengine.


Suggested change

model (BaseModel): a BaseModel of mmengine.

model (BaseModel): A BaseModel of mmengine.

HAOCHENYE · 2023-10-30T12:18:24Z

mmengine/hooks/freeze_hook.py

+
+    def _modify_layers_grad(self, model: BaseModel, layers: Sequence[str],
+                            requires_grad: bool):
+        """Modify the `requires_grad` of the specified layers.


Suggested change

"""Modify the `requires_grad` of the specified layers.

"""Modify the ``requires_grad`` of the specified layers.

HAOCHENYE · 2023-10-30T12:22:25Z

mmengine/hooks/freeze_hook.py

+            print_log(
+                f'{k} requires_grad: {v.requires_grad}', logger='current')
+
+    def _main(self,


Suggested change

def _main(self,

def _freeze(self,

HAOCHENYE · 2023-10-30T12:25:19Z

mmengine/hooks/freeze_hook.py

+        if self.freeze_iter is not None:
+            self._main(runner, runner.iter, self.freeze_iter,
+                       self.unfreeze_iter)


Suggested change

if self.freeze_iter is not None:

self._main(runner, runner.iter, self.freeze_iter,

self.unfreeze_iter)

if self.freeze_iter is not None and runner.iter in (self.freeze_iter, self.unfreeze_iter):

self._freeze(runner.model)

codecov · 2024-12-27T02:10:49Z

Codecov Report

Attention: Patch coverage is 83.00000% with 17 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@e43bbb5). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
mmengine/hooks/freeze_hook.py	82.82%	8 Missing and 9 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1387   +/-   ##
=======================================
  Coverage        ?   70.43%           
=======================================
  Files           ?      154           
  Lines           ?    14368           
  Branches        ?     2999           
=======================================
  Hits            ?    10120           
  Misses          ?     3773           
  Partials        ?      475

Flag	Coverage Δ
unittests	`70.43% <83.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gy-7 added 2 commits October 10, 2023 05:23

add freeze_hook

458fdff

add freeze_hook

7b2e068

gy-7 requested review from zhouzaida and HAOCHENYE as code owners October 10, 2023 09:27

HAOCHENYE reviewed Oct 11, 2023

View reviewed changes

gy-7 and others added 2 commits October 27, 2023 15:55

Merge branch 'open-mmlab:main' into gy77/add_freeze_hook

8e20853

update freezehook

9c0f7e4

HAOCHENYE reviewed Oct 30, 2023

View reviewed changes

gy-7 added 2 commits October 31, 2023 02:56

update freeze hook

d6edcad

fix freeze_layer_names

ab9d677

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gy77/add freeze hook #1387

gy77/add freeze hook #1387

gy-7 commented Oct 10, 2023 •

edited

Loading

HAOCHENYE Oct 11, 2023

HAOCHENYE Oct 11, 2023

HAOCHENYE Oct 11, 2023

HAOCHENYE Oct 11, 2023

HAOCHENYE Oct 11, 2023

HAOCHENYE Oct 30, 2023

HAOCHENYE Oct 30, 2023

HAOCHENYE Oct 30, 2023

HAOCHENYE Oct 30, 2023

HAOCHENYE Oct 30, 2023

HAOCHENYE Oct 30, 2023

HAOCHENYE Oct 30, 2023

HAOCHENYE Oct 30, 2023

HAOCHENYE Oct 30, 2023

codecov bot commented Dec 27, 2024

	log_grad (bool): Whether to log the requires_grad of each layer.
	verbose (bool): Whether to log the requires_grad of each layer.

	def modify_layers_grad(self, model, layers, requires_grad):
	def _modify_layers_grad(self, model, layers, requires_grad):

	def log_model_grad(self, model, log_grad=False):
	def _log_model_grad(self, model, log_grad=False):

	if not isinstance(freeze_iter, (int, type(None))):
	if not isinstance(freeze_iter) and freeze_iter is not None:

	if freeze_iter and freeze_iter < 0:
	if freeze_iter is not None and freeze_iter < 0:

	if freeze_epoch and freeze_epoch < 0:
	if freeze_epoch is not None and freeze_epoch < 0:

	model (BaseModel): a BaseModel of mmengine.
	model (BaseModel): A BaseModel of mmengine.

	"""Modify the `requires_grad` of the specified layers.
	"""Modify the ``requires_grad`` of the specified layers.

gy77/add freeze hook #1387

Are you sure you want to change the base?

gy77/add freeze hook #1387

Conversation

gy-7 commented Oct 10, 2023 • edited Loading

Motivation

Modification

Use cases

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 27, 2024

Codecov Report

gy-7 commented Oct 10, 2023 •

edited

Loading