Selected Publications
Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning
We investigate large multimodal models and their limitations such as hallucinations and lack of explainability. We then show that multimodal in-context learning can reduce some of these flaws.
Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
We introduce rewarded soup, a new strategy to trade-off between multiple rewards when fine-tuning foundation models with RLHF; we first learn one network for each reward, and then linearly interpolate their weights.
Diverse Weight Averaging for Out-of-Distribution Generalization
To improve out-of-distribution generalization on DomainBed, we average diverse weights obtained from different training runs; this strategy is motivated by an extension of the bias-variance theory to weight averaging.
Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization
We introduce and motivate a new regularization that enforces invariance in the domain-level gradient variances across the different training domains in order to improve out-of-distribution generalization.
MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks
We introduce a new generalized framework for learning multi-input multi-output subnetworks and study how to best mix the inputs. We obtain sota on CIFAR and Tiny ImageNet by better leveraging the expressiveness of large networks.