Metadata-Guided Adaptation of Vision Foundation Models for Breast MRI

Background

Deep learning models for breast MRI are usually trained by supervised fine-tuning on task labels. In our setting, these labels are expensive: the reference standard per examination side is derived from BI-RADS assessment, histopathology from biopsy, or verified imaging follow-up. Label-efficient methods are therefore of direct practical interest.

A recent method from Meta FAIR, FINO (Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have), proposes an alternative. Instead of fine-tuning a foundation model with task labels, it adapts the representation using acquisition metadata as weak supervision: metadata factors judged informative are encouraged in the representation, factors judged spurious are suppressed via gradient reversal, on top of a self-supervised DINO/iBOT objective. Only afterwards is a small probe trained on the frozen backbone using task labels. The authors report that this outperforms fully supervised fine-tuning across microscopy, satellite, wildlife and chest X-ray data, with especially large gains when few labels are available.

Breast MRI is a natural test case for this idea, because DICOM headers supply metadata at no additional cost (scanner and vendor, field strength, coil, acquisition protocol and sequence parameters, examination year, patient age, and contrast timing), and because the domain shifts these factors induce are a known problem for generalization.

Goal

Evaluate whether metadata-guided representation adaptation transfers to breast MRI, and compare it against the supervised baselines already established in our group.

The central research questions:

1. Does FINO-style adaptation improve malignancy classification over supervised fine-tuning and over a frozen foundation model baseline?
2. Which MRI metadata factors behave as informative and which as spurious, and does the entanglement problem described in the paper (correlated metadata factors that resist a binary informative/spurious assignment) appear here as well?
3. How does the benefit scale with the amount of labelled data? Our existing evaluation protocol already sweeps label fractions, so the comparison is directly available.

Data

• Public challenge dataset from 5 centers across Europe, each providing 31 to roughly 300 examinations from 31–100 patients, containing a high-resolution DCE pre-contrast sequence, the first post-contrast sequence and the corresponding subtraction image, as well as a T2-weighted image.
• One label per examination side (the most severe label is used): malignant, benign, no finding.

Requirements

• Solid Python and PyTorch; comfortable reading and adapting a research codebase.
• Basic familiarity with self-supervised learning; prior exposure to medical imaging or DICOM is helpful but can be learned during the project.
• Access to GPU resources is provided. Note that the original work is compute-intensive; part of the project is to find a setting (backbone size, resolution, schedule) that is feasible within our budget, and to report that trade-off honestly rather than to match the original scale.

What we offer

• A welcoming team that supports you from day one
• Close supervision with regular feedback and discussions
• Modern GPU resources for your experiments
• Flexible working hours and remote work options
• An open, collaborative atmosphere where questions and ideas are welcome
• Opportunities to present your work and contribute to publications