Weight norm code uses del module._parameters[name] for that. Also requested on forum: https://discuss.pytorch.org/t/how-to-unregister-a-parameter-from-a-module/36424 and many other: https://discuss.pytorch.org/t/exclude-parameters-from-model/77238/6 https://discuss.pytorch.org/t/how-to-...