Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance as Transformers due to limitations in parallelization and scalability. The Receptance Weighted Key Value (RWKV) model architecture combines the efficient parallelizable training of Transformers with the efficient inference of RNNs. Our experiments reveal that RWKV performs on par with similarly sized Transformers, suggesting that future work can leverage this architecture to create more efficient models. This work presents a significant step towards reconciling the trade-offs between computational efficiency and model performance in sequence processing tasks.
