Figure 6: The impact of using an early checkpoint of the reference model in pruning based on Perplexity and EL2N metrics. Motivated by several works that have found that there is a signal in early training checkpoints (Paul et al., 2023; Agarwal et al., 2022; Siddiqui et al., 2022),...