Dr Shiwei Liu

Adriana Fernandez-Lopez, Honglie Chen, Pingchuan Ma, Lu Yin, Qiao Xiao, Stavros Petridis,  Shiwei Liu,  Maja Pantic.  MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization. Interspeech, 2024.

Qiao Xiao, Pingchuan Ma, Adriana Fernandez-Lopez, Boqian Wu, Lu Yin, Stavros Petridis, Mykola Pechenizkiy, Maja Pantic, Decebal Constantin Mocanu, Shiwei Liu. Dynamic Data Pruning for Automatic Speech Recognition. Interspeech, 2024.

Zhang, Zhenyu, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. "Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding." arXiv preprint arXiv:2403.04797 (2024).

Yin, Lu, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. "Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity." The Forty-first International Conference on
Machine Learning (ICML), 2024.

Lu Yin, Ajay Jaiswal, Shiwei Liu, Souvik Kundu, and Zhangyang Wang. "Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs Difficult Downstream Tasks in LLMs." The Forty-first International Conference on Machine Learning (ICML), 2024.

Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, Rongrong Ji. "CaM: Cache Merging for Memory-efficient LLMs Inference." The Forty-first International Conference on Machine Learning (ICML), 2024.

Jie Ji, Gen Li, Lu Yin, Minghai Qin, Geng Yuan, Linke Guo, Shiwei Liu, Xiaolong Ma. Advancing Dynamic Sparse Training by Exploring Optimization Opportunities. The Forty-first International Conference on Machine Learning (ICML), 2024.

Zhangheng Li, Shiwei Liu, Tianlong Chen, Ajay Kumar Jaiswal, Zhenyu Zhang, Dilin Wang, Raghuraman Krishnamoorthi, Shiyu Chang, Zhangyang Wang. Sparse Cocktail: Co-Training Many Sparsity Patterns and Ratios at Once. The Forty-first International Conference on Machine
Learning (ICML), 2024.

Zhang, Yuxin, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. "Dynamic sparse no training: Training-free fine-tuning for sparse llms."  In The Twelfth International Conference on Learning Representations. 2024.

Li, Gen, Lu Yin, Jie Ji, Wei Niu, Minghai Qin, Bin Ren, Linke Guo, Shiwei Liu, and Xiaolong Ma. "NeurRev: Train Better Sparse Neural Network Practically via Neuron Revitalization." In The Twelfth International Conference on Learning Representations. 2024.

Yang, Enneng, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. "Adamerging: Adaptive model merging for multi-task learning." In The Twelfth International Conference on Learning Representations. 2024.

Pham, Hoang, Shiwei Liu, Lichuan Xiang, Dung Le, Hongkai Wen, and Long Tran-Thanh. "Towards Data-Agnostic Pruning At Initialization: What Makes a Good Sparse Mask?." Advances in Neural Information Processing Systems 36 (2023).

Hoang, Duc, Souvik Kundu, Shiwei Liu, and Zhangyang Wang. "Don’t just prune by magnitude! Your mask topology is a secret weapon." Advances in Neural Information Processing Systems 36 (2023): 65056-65068.

Yin, Lu, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, and Shiwei Liu. "Dynamic sparsity is channel-level sparsity learner." Advances in Neural Information Processing Systems 36 (2023).

Jaiswal, Ajay, Shiwei Liu, Tianlong Chen, and Zhangyang Wang. "The emergence of essential sparsity in large pre-trained models: The weights that matter." Advances in Neural Information Processing Systems 36 (2023).

Huang, Tianjin, Lu Yin, Zhenyu Zhang, Li Shen, Meng Fang, Mykola Pechenizkiy, Zhangyang Wang, and Shiwei Liu. "Are large kernels better teachers than transformers for convnets?." In International Conference on Machine Learning, pp. 14023-14038. PMLR, 2023.

Jaiswal, Ajay Kumar, Shiwei Liu, Tianlong Chen, Ying Ding, and Zhangyang Wang. "Instant soup: Cheap pruning ensembles in a single pass can draw lottery tickets from large models." In International Conference on Machine Learning, pp. 14691-14701. PMLR, 2023.

Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, Ajay Jaiswal, and Zhangyang Wang. "Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!." ICLR 2023.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkäinen, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang. "More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity."  ICLR 2023.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. "The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training." ICLR 2022.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. "Sparse training via boosting pruning plasticity with neuroregeneration." Advances in Neural Information Processing Systems 34 (2021): 9908-9922.

Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. "Do we actually need dense over-parameterization? in-time over-parameterization in sparse training." In International Conference on Machine Learning, pp. 6989-7000. PMLR, 2021.