Accelerate documentation
Contoh Zoo
Memulai
Tutorial
IkhtisarTambahkan Accelerate ke kode AndaProses eksekusiPelatihan TPUMeluncurkan skrip AccelerateMeluncurkan pelatihan terdistribusi dari Jupyter Notebooks
Panduan cara
Accelerate
Mulai di Sini!Perkiraan memori modelKuantisasi modelPelacak eksperimenProfilerCheckpointingPemecahan masalahContoh Zoo
Pelatihan
Akselerasi gradienLocal SGDPelatihan presisi rendah (FP8)DeepSpeedMenggunakan beberapa model dengan DeepSpeedHook Komunikasi DDPFully Sharded Data ParallelMegatron-LMAmazon SageMakerGPU Apple M1Intel CPUIntel GaudiKompilasi
Inferensi
Konsep dan dasar-dasar
Mekanisme internal AccelerateMemuat model besar ke dalam memoriMembandingkan kinerja di antara konfigurasi terdistribusiMenjalankan dan menunda tugasSinkronisasi gradienFSDP vs DeepSpeedFSDP1 vs FSDP2Paralelisme konteksParalelisme urutanMetode pelatihan presisi rendahPelatihan pada TPU
Referensi
Contoh Zoo
Di bawah ini berisi daftar tutorial dan skrip yang tidak lengkap yang menampilkan Accelerate.
Contoh Resmi Accelerate:
Contoh Dasar
Contoh-contoh ini menampilkan fitur dasar Accelerate dan merupakan titik awal yang sangat baik
- Contoh NLP sederhana
- Contoh NLP terdistribusi sederhana dalam Jupyter Notebook
- Contoh computer vision paling dasar
- Contoh computer vision terdistribusi sederhana dalam Jupyter Notebook
- Menggunakan Accelerate di Kaggle
Contoh Spesifik Fitur
Contoh-contoh ini menampilkan fitur-fitur spesifik yang ditawarkan oleh framework Accelerate
- Akumulasi gradien yang sadar memori secara otomatis
- Menyimpan status checkpoint
- Validasi silang
- DeepSpeed
- Fully Sharded Data Parallelism
- Akumulasi gradien
- Pencari ukuran batch yang sadar memori
- Perhitungan Metrik
- Menggunakan Pelacak
- Menggunakan Megatron-LM
Contoh Lengkap
Contoh-contoh ini menampilkan setiap fitur dalam Accelerate sekaligus yang telah ditunjukkan dalam “Contoh Spesifik Fitur”
- Contoh NLP lengkap
- Contoh visi komputer lengkap
- Contoh visi yang sangat lengkap dan dapat diperluas, menampilkan SLURM, hydra, dan penggunaan framework yang sangat fleksibel
- Contoh penyesuaian halus model bahasa kausal
- Contoh penyesuaian halus model bahasa berpenutup
- Contoh pra-pelatihan ucapan
- Contoh fine-tuning terjemahan
- Contoh penyesuaian halus klasifikasi teks
- Contoh penyesuaian halus segmentasi semantik
- Contoh penyesuaian halus untuk penjawab pertanyaan
- Contoh penyesuaian akhir penjawab pertanyaan pencarian berkas
- Contoh penyesuaian akhir untuk menjawab pertanyaan pilihan ganda
- Contoh penyesuaian halus pengenalan entitas bernama
- Contoh penyesuaian halus klasifikasi gambar
- Contoh fine-tuning ringkasan
- Contoh end-to-end tentang cara menggunakan integrasi AWS SageMaker dari Accelerate
- Contoh Megatron-LM untuk berbagai tugas NLP
Contoh Integrasi
Ini adalah tutorial dari pustaka yang terintegrasi dengan Accelerate:
Tidak menemukan integrasi Anda di sini? Buat PR untuk menyertakannya!
Amphion
- Melatih Model Text-to-Speech dengan Amphion
- Melatih Model Konversi Suara Bernyanyi dengan Amphion
- Melatih Vocoder dengan Amphion
Catalyst
DALLE2-pytorch
Diffusers
fastai
- Pelatihan terdistribusi dari Jupyter Notebooks dengan fastai
- Contoh pelatihan terdistribusi dasar dengan fastai
GradsFlow
imagen-pytorch
Kornia
PyTorch Dipercepat
PyTorch3D
Stable-Dreamfusion
Tez
trlx
Comfy-UI
Dalam Sains
Di bawah ini berisi daftar tidak lengkap dari makalah yang memanfaatkan Accelerate.
Tidak menemukan makalah Anda di sini? Buat PR untuk menyertakannya!
- Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, Omer Levy: “Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation”, 2023; arXiv:2305.01569.
- Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, Ee-Peng Lim: “Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models”, 2023; arXiv:2305.04091.
- Arthur Câmara, Claudia Hauff: “Moving Stuff Around: A study on efficiency of moving documents into memory for Neural IR models”, 2022; arXiv:2205.08343.
- Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang: “High-throughput Generative Inference of Large Language Models with a Single GPU”, 2023; arXiv:2303.06865.
- Peter Melchior, Yan Liang, ChangHoon Hahn, Andy Goulding: “Autoencoding Galaxy Spectra I: Architecture”, 2022; arXiv:2211.07890.
- Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, Diyi Yang: “A Cheaper and Better Diffusion Language Model with Soft-Masked Noise”, 2023; arXiv:2304.04746.
- Ayaan Haque, Matthew Tancik, Alexei A. Efros, Aleksander Holynski, Angjoo Kanazawa: “Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions”, 2023; arXiv:2303.12789.
- Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, Andrea Vedaldi: “RealFusion: 360° Reconstruction of Any Object from a Single Image”, 2023; arXiv:2302.10663.
- Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, Hongsheng Li: “Better Aligning Text-to-Image Models with Human Preference”, 2023; arXiv:2303.14420.
- Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang: “HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace”, 2023; arXiv:2303.17580.
- Yue Yang, Wenlin Yao, Hongming Zhang, Xiaoyang Wang, Dong Yu, Jianshu Chen: “Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination”, 2022; arXiv:2210.12261.
- Sheng-Yen Chou, Pin-Yu Chen, Tsung-Yi Ho: “How to Backdoor Diffusion Models?”, 2022; arXiv:2212.05400.
- Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, Seungryong Kim: “Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation”, 2023; arXiv:2303.07937.
- Atau Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, Daniel Cohen-Or: “Localizing Object-level Shape Variations with Text-to-Image Diffusion Models”, 2023; arXiv:2303.11306.
- Dídac Surís, Sachit Menon, Carl Vondrick: “ViperGPT: Visual Inference via Python Execution for Reasoning”, 2023; arXiv:2303.08128.
- Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, Qifeng Chen: “FateZero: Fusing Attentions for Zero-shot Text-based Video Editing”, 2023; arXiv:2303.09535.
- Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, Yejin Choi: “NaturalProver: Grounded Mathematical Proof Generation with Language Models”, 2022; arXiv:2205.12910.
- Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, Daniel Cohen-Or: “TEXTure: Text-Guided Texturing of 3D Shapes”, 2023; arXiv:2302.01721.
- Puijin Cheng, Li Lin, Yijin Huang, Huaqing He, Wenhan Luo, Xiaoying Tang: “Learning Enhancement From Degradation: A Diffusion Model For Fundus Image Enhancement”, 2023; arXiv:2303.04603.
- Shun Shao, Yftah Ziser, Shay Cohen: “Erasure of Unaligned Attributes from Neural Representations”, 2023; arXiv:2302.02997.
- Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim, Minjoon Seo: “In-Context Instruction Learning”, 2023; arXiv:2302.14691.
- Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, Anima Anandkumar: “Prismer: A Vision-Language Model with An Ensemble of Experts”, 2023; arXiv:2303.02506.
- Haoyu Chen, Zhihua Wang, Yang Yang, Qilin Sun, Kede Ma: “Learning a Deep Color Difference Metric for Photographic Images”, 2023; arXiv:2303.14964.
- Van-Hoang Le, Hongyu Zhang: “Log Parsing with Prompt-based Few-shot Learning”, 2023; arXiv:2302.07435.
- Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi, Kentaro Inui: “Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?”, 2023; arXiv:2302.07866.
- Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, Prithviraj Ammanabrolu: “Behavior Cloned Transformers are Neurosymbolic Reasoners”, 2022; arXiv:2210.07382.
- Martin Wessel, Tomáš Horych, Terry Ruas, Akiko Aizawa, Bela Gipp, Timo Spinde: “Introducing MBIB — the first Media Bias Identification Benchmark Task and Dataset Collection”, 2023; arXiv:2304.13148. DOI: [https://dx.doi.org/10.1145/3539618.3591882 10.1145/3539618.3591882].
- Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, Daniel Cohen-Or: “Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models”, 2023; arXiv:2301.13826.
- Marcio Fonseca, Yftah Ziser, Shay B. Cohen: “Factorizing Content and Budget Decisions in Abstractive Summarization of Long Documents”, 2022; arXiv:2205.12486.
- Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, Daniel Cohen-Or: “TEXTure: Text-Guided Texturing of 3D Shapes”, 2023; arXiv:2302.01721.
- Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, Yulia Tsvetkov: “On the Blind Spots of Model-Based Evaluation Metrics for Text Generation”, 2022; arXiv:2212.10020.
- Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham: “In-Context Retrieval-Augmented Language Models”, 2023; arXiv:2302.00083.
- Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P. Xing, Hao Zhang: “MPCFormer: fast, performant and private Transformer inference with MPC”, 2022; arXiv:2211.01452.
- Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, Jianfeng Gao: “GODEL: Large-Scale Pre-Training for Goal-Directed Dialog”, 2022; arXiv:2206.11309.
- Egil Rønningstad, Erik Velldal, Lilja Øvrelid: “Entity-Level Sentiment Analysis (ELSA): An exploratory task survey”, 2023, Proceedings of the 29th International Conference on Computational Linguistics, 2022, halaman 6773-6783; arXiv:2304.14241.
- Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, Sergey Levine: “Offline RL for Natural Language Generation with Implicit Language Q Learning”, 2022; arXiv:2206.11871.
- Zhiruo Wang, Shuyan Zhou, Daniel Fried, Graham Neubig: “Execution-Based Evaluation for Open-Domain Code Generation”, 2022; arXiv:2212.10481.
- Minh-Long Luu, Zeyi Huang, Eric P. Xing, Yong Jae Lee, Haohan Wang: “Expeditious Saliency-guided Mix-up through Random Gradient Thresholding”, 2022; arXiv:2212.04875.
- Jun Hao Liew, Hanshu Yan, Daquan Zhou, Jiashi Feng: “MagicMix: Semantic Mixing with Diffusion Models”, 2022; arXiv:2210.16056.
- Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao: “LiST: Lite Prompted Self-training Makes Parameter-Efficient Few-shot Learners”, 2021; arXiv:2110.06274.