🚀 Feature Same issue with #4927 The current clip_gradient uses clip_grad_norm; can we add clip_grad_value? https://github.com/PyTorchLightning/pytorch-lightning/blob/f2e99d617f05ec65fded81ccc6d0d59807c47573/pytorch_lightning/plugins/nati...
For example: https://github.com/PyTorchLightning/pytorch-lightning/blob/c7451b3ccf742b0e8971332caf2e041ceabd9fe8/pytorch_lightning/plugins/precision/fully_sharded_native_amp.py#L28-L46 What happens to the trainer flags for gradient clip value or gradient clip algorithm? How does someone know if...
This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!stale bot added the won't fix label Dec 31, 2020 stale...
Current environment #- PyTorch Lightning Version (e.g., 2.4.0): #- PyTorch Version (e.g., 2.4): #- Python version (e.g., 3.12): #- OS (e.g., Linux): #- CUDA/cuDNN version: #- GPU models and configuration: #- How you installed Lightning(`conda`, `pip`, source): More i...
🚀 Feature See code here: https://github.com/pseeth/autoclip Motivation a simple method for automatically and adaptively choosing a gradient clipping threshold, based on the history of gradient norms observed during training. Experimental...
[](https://console.paperspace.com/github/gradient-ai/PyTorch-Lightning?machine=Free-GPU)Collection of Jupyter notebook tutorials from the [PyTorch Lightning documentation](https://pytorch-lightning.readthedocs.io/)....
Thanks in advance for your help. What version are you seeing the problem on? v2.4 How to reproduce the bug No response Error messages and logs # Error messages and logs here please Environment Current environment #- PyTorch Lightning Version (e.g., 2.4.0): ...
🚀 Feature Implement Image-Gradients for PT Lightning. Motivation Recently I was working on a vanilla PT implementation of the DenseDepth paper. They happen to use a DepthLoss as one of their loss functions. Incidentally, DepthLoss is bas...
Description & Motivation When training different model sizes on a different number of devices or different hardware, the batch size needs to be carefully tuned in order to achieve maximum GPU utilization without incurring Out Of Memory E...
Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes. - Add stronger typing to gradient accumulation scheduler callback (#3558) · Lightning-AI/pytorch-lightning@c61e1e6