LLoCO: Learning Long Contexts Offline
Paper
• 2404.07979 • Published
• 22
Long Context
-
-
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
Paper • 2402.13753 • Published • 116 -
LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration
Paper • 2402.11550 • Published • 19 -
LongAlign: A Recipe for Long Context Alignment of Large Language Models
Paper • 2401.18058 • Published • 24 -
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
Paper • 2404.07143 • Published • 111 -
Long-Context Language Modeling with Parallel Context Encoding
Paper • 2402.16617 • Published • 2 -
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Paper • 2406.10149 • Published • 52 -
RULER: What's the Real Context Size of Your Long-Context Language Models?
Paper • 2404.06654 • Published • 39 -
Extending Llama-3's Context Ten-Fold Overnight
Paper • 2404.19553 • Published • 34 -
Make Your LLM Fully Utilize the Context
Paper • 2404.16811 • Published • 55 -
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
Paper • 2404.08801 • Published • 66 -
LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
Paper • 2406.00605 • Published • 2 -
Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
Paper • 2402.02244 • Published • 1 -
Resonance RoPE: Improving Context Length Generalization of Large Language Models
Paper • 2403.00071 • Published • 24 -
Extending LLMs' Context Window with 100 Samples
Paper • 2401.07004 • Published • 16 -
Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
Paper • 2406.11230 • Published • 33 -
Long Code Arena: a Set of Benchmarks for Long-Context Code Models
Paper • 2406.11612 • Published • 25 -
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Paper • 2406.16008 • Published • 6 -
LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs
Paper • 2406.15319 • Published • 64 -
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
Paper • 2406.16747 • Published • 19 -
Can Few-shot Work in Long-Context? Recycling the Context to Generate Demonstrations
Paper • 2406.13632 • Published • 5 -
LongIns: A Challenging Long-context Instruction-based Exam for LLMs
Paper • 2406.17588 • Published • 23 -
Training-Free Long-Context Scaling of Large Language Models
Paper • 2402.17463 • Published • 24 -
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
Paper • 2406.17419 • Published • 17 -
Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models
Paper • 2405.17915 • Published • 2 -
Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
Paper • 2407.01370 • Published • 89 -
Human-like Episodic Memory for Infinite Context LLMs
Paper • 2407.09450 • Published • 62 -
NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context Window?
Paper • 2407.11963 • Published • 44 -
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
Paper • 2407.14057 • Published • 46 -
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
Paper • 2407.14482 • Published • 26 -
Writing in the Margins: Better Inference Pattern for Long Context Retrieval
Paper • 2408.14906 • Published • 144 -
LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA
Paper • 2409.02897 • Published • 48 -
LongRecipe: Recipe for Efficient Long Context Generalization in Large Languge Models
Paper • 2409.00509 • Published • 42 -
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Paper • 2409.16191 • Published • 41 -
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval
Paper • 2409.10516 • Published • 43 -
Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models
Paper • 2409.04774 • Published -
L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?
Paper • 2410.02115 • Published • 10 -
Minimum Tuning to Unlock Long Output from LLMs with High Quality Data as the Key
Paper • 2410.10210 • Published • 6 -
LongReward: Improving Long-context Large Language Models with AI Feedback
Paper • 2410.21252 • Published • 19 -
Why Does the Effective Context Length of LLMs Fall Short?
Paper • 2410.18745 • Published • 17 -
Language Models can Self-Lengthen to Generate Long Texts
Paper • 2410.23933 • Published • 18 -
Large Language Models Can Self-Improve in Long-context Reasoning
Paper • 2411.08147 • Published • 65 -
Star Attention: Efficient LLM Inference over Long Sequences
Paper • 2411.17116 • Published • 53 -
LongKey: Keyphrase Extraction for Long Documents
Paper • 2411.17863 • Published • 12 -
Reverse Thinking Makes LLMs Stronger Reasoners
Paper • 2411.19865 • Published • 23 -
Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?
Paper • 2411.05000 • Published • 22 -
How to Train Long-Context Language Models (Effectively)
Paper • 2410.02660 • Published • 2 -
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
Paper • 2411.13476 • Published • 16 -
LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
Paper • 2408.07055 • Published • 69 -
Differential Transformer
Paper • 2410.05258 • Published • 181 -
LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
Paper • 2502.13922 • Published • 27 -
LongRoPE2: Near-Lossless LLM Context Window Scaling
Paper • 2502.20082 • Published • 36 -
A Controllable Examination for Long-Context Language Models
Paper • 2506.02921 • Published • 33