Most relevant real-world problems are wicked.¹ One consequence is that solutions are “good” or “bad”, not “true” or “false”. This severely limits the applicability of learning paradigms that rely on verifiable solutions: verifiers (or ground-truth reward functions) usually just don't exist.

Reinforcement learning from human feedback (RLHF) has been a promising way around this: if we can't procedurally define a reward function, we learn it from human judgements about model behavior, called feedback.² Then we optimize it with standard RL algorithms.³ But current RLHF methods fall short. They are overly restrictive about which signals can be integrated,⁴ and they produce brittle reward models that do not capture the intended behavior, even though they are trained on human data.⁵

I don't think that's sufficient reason to give up on RLHF – I firmly believe that to solve real-world problems, we need to learn from whatever the world tells us about our solutions, in all its nuance and diversity.⁶ In that spirit, my current research focuses on:

• Jointly learning from multiple feedback types. Feedback (human or not) comes in many forms, and jointly it constrains the reward more effectively than any single type on its own.
• Learning distributional reward models. This is key to avoid brittle (and thus hackable) reward estimates. Representing uncertainty about preferences is useful: it allows us to quantify whether we should ask again, represent genuine disagreement, and transfer more robustly to unseen settings.
• Actively constructing informative queries (while taking cost into account). Feedback types differ in informativeness and cost. I aim to develop methods that can trade these factors off flexibly and at scale.