Caglar Gulcehre

Professor and Lead of CLAIRE lab @ EPFL

Research Consultant at Google DeepMind

Ex: Staff Research Scientist @ Google DeepMind

Google Scholar: Click here!

Twitter: caglarml@

Github: github.com/caglar ***Not up to date!***

Email: ca9lar At Gmail

Location: Lausanne, Switzerland

Important note for students: I receive a lot of emails from students, and it is impossible for me to reply to all of them. I will post it here when our process is in place to remedy it. For now, instead of cold-emailing me directly for PhD or MSc positions, please check this page. I am not planning to hire PhD students for 2024-2025 but hiring postdocs.

Home: Welcome

Bio

I am currently a professor at EPFL and leading the CLAIRE research lab. I was a staff research scientist in Google DeepMind working on the intersection of Reinforcement Learning, Foundation Models, Novel Archtiectures, safety + Alignment and Natural Language Understanding. I have led or co-led several projects during my time at DeepMind ranging from next generation of sequence modeling architectures, alignment and safety to offline RL.

I am interested in building agents that can learn from a feedback signal (often weak, sparse, and noisy in the real world) while utilizing unlabeled data available in the environment. I am interested in improving our understanding of the existing algorithms and developing new ones to enable real-world applications with positive social impact. I am particularly fascinated by the scientific applications of machine learning algorithms. I enjoy working in a multi/cross-disciplinary team and am often inspired by neuroscience, biology, and cognitive sciences when working on algorithmic solutions.

I finished my Ph.D. under the supervision of Yoshua Bengio at MILA.

I defended my thesis "Learning and time: on using memory and curricula for language understanding" in 2018 with Christopher Manning as my external examiner. Currently, the research topics that I am working on include but are not limited to reinforcement learning, offline RL, large-scale deep architectures (or foundational models. as they call it these days), and representation learning (including self-supervised learning, new architectures, causal representations, etc.) I have served as an area chair and reviewer to significant machine learning conferences such as ICML, NeurIPS, ICLR, and journals like Nature and JMLR. I have published at numerous influential conferences and journals such as Nature, JMLR, NeurIPS, ICML, ICLR, ACL, EMNLP, etc... My work has received the best paper award at the Nonconvex Optimization workshop at NeurIPS and an honorable mention for best paper at ICML 2019. I have co-organized the Science and Engineering of Deep Learning workshops and three other workshops at NeurIPS, ICML, and ICLR.

Learn More

Home: Welcome

Recent Updates

Organizing the Next Generation of Sequence Models workshop at ICML 2024.
Our paper Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers is on arxiv!
Gave a lecture on transformers and foundation models at EEML 2024.
Published our work on Griffin an efficient, high-performant state space model architecture for foundation models.
Published our Reinforced Self-Training work for an efficient approach to the alignment of LLMs.
Gave a talk at the EPFL IC department on the evolution of LLM architecture.
Gave a talk at ICRC in Geneva on the open-source State of Art Foundation models November 2023).
Gave a talk UN in Geneva workshop on foundation models about the risks and potentials of foundation models for humanitarian operations (October 2023.)
Gave a talk at EEML 2023 in Albania on the History of Large Language models.
Our paper ReST about Reinforcement Learning from Human Feedback is on Arxiv!
We developed a new sequence modeling paradigm called LRU (Linear Recurrent Units), and our paper was published at ICML 2023.
Our paper "On integrating a language model into neural machine translation" got the best research paper award at Interspeech 2022.
asdasdOur paper "An Empirical Study of Implicit Regularization in Deep Offline RL" is on arXiv.
We are organizing the ML Evaluation Standards workshop at ICLR 2022.
We presented our paper "StarCraft II Unplugged: Large Scale Offline Reinforcement Learning" at the Deep RL workshop at NeurIPS 2021.
Our paper, Active Offline Policy Selection, has been accepted to NeurIPS 2021.
I have presented Intro to RL (part 1 slides) and Offline RL lectures (part 2 slides) at DeepLearn 2021 Summer School.
We have released DeepMind Lab and Bsuite datasets for Offline RL Under RL Unplugged.
Our paper On Instrumental Variable Regression for Deep Offline Policy Evaluation is on arXiv.
Our paper, Regularized behavior value estimation on a single-step policy improvement method, is on arXiv.
Our paper Addressing Extrapolation Error in Deep Offline Reinforcement Learning was Oral at the Offline RL Workshop at NeurIPS 2020.
We released the hard-eight task suite used in the "Making Efficient Use of Demonstrations" paper.

Selected Publication

Regularized Behavior Value Estimation

Authors

Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, Nando de Freitas

Abstract

Offline reinforcement learning restricts the learning process to rely only on logged data without access to an environment. While this enables real-world applications, it also poses unique challenges. One important challenge is dealing with errors caused by overestimating values for state-action pairs not well-covered by the training data. Due to bootstrapping, these errors get amplified during training and can lead to divergence, thereby crippling learning. To overcome this challenge, we introduce Regularized Behavior Value Estimation (R-BVE). Unlike most approaches, which use policy improvement during training, R-BVE estimates the value of the behavior policy during training and only performs policy improvement at deployment time.

Further, R-BVE uses a ranking regularisation term that favors actions in the dataset that lead to successful outcomes. We provide ample empirical evidence of R-BVE's effectiveness, including state-of-the-art performance on the RL Unplugged ATARI dataset. We also test R-BVE on new datasets, from suite and a challenging DeepMind Lab task, and show that R-BVE outperforms other state-of-the-art discrete control offline RL methods.

Read Paper

View All