Attention neural networks with sparse attention mechanisms
Номер патента: EP4121908A1
Опубликовано: 25-01-2023
Автор(ы): Amr Ahmed, Guru GURUGANESH, Joshua Timothy Ainslie, Kumar Avinava Dubey, Manzil Zaheer, Philip PHAM, Santiago Ontañón
Принадлежит: Google LLC
Опубликовано: 25-01-2023
Автор(ы): Amr Ahmed, Guru GURUGANESH, Joshua Timothy Ainslie, Kumar Avinava Dubey, Manzil Zaheer, Philip PHAM, Santiago Ontañón
Принадлежит: Google LLC
Реферат: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing network inputs using an attention neural network that has one or more sparse attention sub-layers. Each sparse attention sub-layer is configured to apply a sparse attention mechanism that attends differently for input positions that are in a first proper subset of the input positions in the input to the sub-layer than for positions that are not in the first proper subset.
Sparse attention neural networks
Номер патента: EP4040339A1. Автор: Lukasz Mieczyslaw Kaiser,Wojciech Gajewski,Afroz Mohiuddin,Aakanksha Chowdhery,Henryk Michalewski,Jonni Miikka KANERVA,Sebastian Dariusz Jaszczur. Владелец: Google LLC. Дата публикации: 2022-08-10.