Center for Advanced Intelligence Project

Date and Time:
March 28, 2025: 11:00 – 12:00 (JST)
Venue: Online and Open Space at the RIKEN AIP Nihonbashi office
Open Space is available to AIP researchers only

Title:
Denoising Objectives and Bidirectional Attention in 2025: Why Training a Modern Encoder Matters

Speaker:
Benjamin Clavié (Answer.AI)

Abstract:
While the recent NLP landscape has largely focused on massive, decoder-only, causal models, encoder-based models like BERT remain the backbone of many real-world applications, from information retrieval to classification tasks. Surprisingly, smaller encoders often match or exceed the performance of much larger decoders in these scenarios. Yet, until recently, many encoder applications relied on architectures dating back several years, untouched by recent advances in transformer architectures. In this talk, we will first provide some background on encoder models, notably shedding light on the various language modeling training objectives, and their relationship to downstream uses. We will then discuss ModernBERT, our effort to establish a modern, reliable baseline encoder incorporating recent architectural improvements while maintaining simplicity. We will outline the key architectural choices and highlight intriguing emergent behaviors observed with instruction tuning. Finally, we’ll briefly discuss the exciting developments in encoder models that have followed the release of ModernBERT, and highlight the potential of upcoming research directions combining various approaches to language modeling.

Bio:
Benjamin Clavié is an applied researcher at Answer.AI in Tokyo, Japan, specializing in natural language processing and information retrieval, with a focus on compact, specialized language models. He is also passionate about broadening access to AI by creating software that bridges the gap between cutting-edge research and everyday uses. Recently, Ben co-led the ModernBERT project, contributing significantly to renewed interest in streamlined, encoder-only models. He earned his MSc in Artificial Intelligence from the University of Edinburgh, UK.