Real whole-slide data and a modern foundation model, where the method stopped helping.
After the synthetic result I was optimistic. The obvious next step was real whole-slide data with a modern embedding model, and I expected the adapter to recover part of the cross-center drop there, as it had in every stage before. Unfortunately, it did not. This stage is where I could no longer make the approach work, and where I spent a long time trying to understand why. It is also the setting the method was built for. It worked in the simpler, controlled stages, but the realistic whole-slide case is where it was supposed to pay off, and this is where it did not generalize.
I moved to real-world whole-slide images. The embedding model was UNI-2, a state-of-the-art pathology foundation model. Where TCGA-2k had used a curated set of about 200 tiles per center, I now used every tile from each slide, with no selection. The training setup aggregates those tiles into a slide-level prediction with attention-based multiple-instance learning (ABMIL), which is the standard way to classify whole slides.
I was hopeful, but the initial results were disappointing. I tried several classification tasks: tissue subtyping (breast versus colon, BRCA versus COAD), three DCIS endpoints (ER status, HER2 status, and grade), and colorectal microsatellite-instability (MSI) status. I could not make the adapter recover the cross-center drop reliably across them. Often it did worse than nothing: aligning the tile embeddings frequently degraded performance, because a transform that deforms the embedding geometry tends to destroy useful information, which is the expected outcome when there is no genuine shift to correct.
One target broke this pattern. On DCIS grade, unlike the others, there seemed to be a genuine cross-center drop that adaptation was able to partially recover. That was enough to pursue it on its own, which is what Section 5 describes.
It is possible that ABMIL’s extra layer of complexity made the problem harder. To change just one thing at a time, I stepped back to tile-level classification without ABMIL, the same tile-level setup that had worked on TCGA-2k. Joeri saw gains there in his thesis, and I reproduced them in my own pipeline, so those results hold. Now on uncurated data, with far more tiles per slide, most of the cross-center drop had itself largely disappeared, so there was little left for the adapter to recover.
The choice of embedding model was deliberate. UNI-2 is a modern model, and I want the method to help on a model people actually use. If it only worked on an older, weaker embedding model, it would be solving an artificial problem, one created by using outdated machinery. That would not be an interesting result.
To check this was not a UNI-2 quirk, I went back to Phikon on the non-curated tiles. The drop was smaller than in the curated TCGA-2k setup, and adaptation did not recover it consistently. The curation and the model had probably both contributed to the thesis result.
I could not pin the failure on a single cause, so what follows is merely suggestive. The gap between the hopeful TCGA-2k result and this more realistic setting has, I think, three main sources.
The curation. The curated TCGA-2k tiles were cleaner and easier to match than a full slide’s worth. Widening to all tiles took that advantage away, and the matching procedure had less structure to work with.
The size of the shift. On UNI-2 the cross-center gap is already small. For the tasks above, there was often little left to recover before the adapter even started.
The diagnostics. I ran a cluster diagnostic on 20 center pairs. In all 20, the UNI-2 embeddings clustered by biology, by tissue content, rather than by center. The measured cross-center drop was small, and it looked compositional. It came from differences in what tiles a center happened to contain, such as artefacts or fat, rather than from a center-wide nuisance shift the adapter could undo.
Put together, the earlier success leaned on a favorable setup, and in a realistic one the little shift that remained was mostly not the kind the adapter could remove.
I invested a lot of time in this stage, working through many possibilities. Everything had to stay label-free, since in deployment there are no labels on the new data. The common thread was to give the adapter more structure to respect and less freedom to misbehave.
LoRA. I constrained the adapter to the identity plus a low-rank correction, with the rank controlling how much it could change. A low rank is a simplicity bias: fewer degrees of freedom, a gentler correction, and less room to overfit the distribution match.
Cluster-conditional MMD. Rather than match the whole Y cloud to the whole X cloud at once, I first clustered the embeddings by their content, without labels, then trained the adapter to minimize the MMD within each cluster instead of globally. The hope was that the adapter would remove center effects while keeping the distinct tissue types apart, instead of aligning the clouds by folding classes together.
Bandwidth. The MMD kernel has a scale, and if it sits far from the size of the residual shift, the MMD barely responds. Joeri’s automatic bandwidth selection (Section 2) was built to handle exactly this, and I relied on it. I also tried fixed and multi-bandwidth choices, to make sure the objective stayed sensitive to whatever center structure was left.
None of it changed the outcome. When LoRA did nothing on the real data, I went back and ran it in the synthetic setting as a sanity check. There it behaved exactly as expected: fine at modest ranks, and it broke down when I dropped the rank too far. The tool worked. On the real data the picture was messier. Sometimes there was a little performance that could have been recovered, but no approach consistently delivered such a recovery.
At least one avenue remains that I have not yet tried. Throughout our ABMIL experiments, the adapter matched tile embeddings. One option would be to match at the slide level instead: each slide produces a single aggregated embedding from ABMIL’s attention, which is the level where the prediction is made. That gives one embedding per slide rather than thousands of tiles, so the match would rest on far fewer samples and be harder to train, but it would act where the shift reaches the output.
‹ Previous: Synthetic histopathology · Up: FOMO-Shift · Next: DCIS grading ›