Systema turns AIVC scores into a harder test

Why do virtual cell models look so good and still feel wrong?

Systema shows AIVC scores often reflect generic stress, not specific biology.

I’ve been following virtual cell work for a while now, and honestly, the whole thing has felt a bit too tidy. You feed in a perturbation, the model predicts the response, and the leaderboard numbers look impressive. But then you try to use those scores to answer a real question, like whether a model actually understands a drug, a gene knockdown, or a chromosomal instability signal, and the confidence starts to wobble. I kept running into the same annoyance: the benchmark said “great,” while the biology said “slow down.”

That mismatch is exactly why the Systema paper hit me so hard. It doesn’t just complain about noisy evaluation. It says the usual setup may be rewarding models for catching broad perturbed-vs-control shifts, not perturbation-specific mechanisms. That is a very different claim, and it changes how I think about every shiny AIVC score I’ve seen lately.

I’m using the Zhihu article at zhuanlan.zhihu.com/p/2065024050586563909 as the trigger here, but the key idea comes from the 2025 Nature Biotechnology paper on Systema. The paper’s argument is simple enough to state and annoying enough to matter: if your evaluation reference point is wrong, your leaderboard is flattering the wrong model behavior.

1) The benchmark was grading the wrong thing

“High prediction scores are mainly driven by systematic variation rather than perturbation-specific effects.”

What this actually means is that a model can look smart by learning the shared transcriptional shift that happens whenever cells are perturbed, without learning what makes one perturbation different from another.

I’ve seen this pattern in a lot of ML systems: if the task definition is loose, the model happily solves the easier version of the problem. In AIVC, that easier version is “is this cell perturbed or not?” instead of “what did this specific perturbation do?” The Systema paper, as summarized in the source article, argues that many current metrics reward that easier distinction.

The big shift is the evaluation reference point. Standard practice compares perturbed cells to a control centroid. Systema instead compares against the perturbed centroid. That sounds like a small statistical tweak, but it changes the question from “did the model notice a perturbation happened?” to “did the model capture the unique effect of this perturbation?”

How to apply it:

• Don’t trust a single leaderboard score without checking the evaluation setup.
• Ask whether the metric compares perturbed samples to control only, or whether it tests perturbation-specific structure.
• When you read a model paper, look for what the model is being rewarded for, not just the final AUC or correlation.

When I re-read AIVC papers with that lens, a lot of “strong performance” starts to look like stress detection dressed up as mechanistic prediction. That’s not useless, but it’s not the same thing.

2) Control-centroid evaluation makes easy wins look like insight

“Systema shifts the evaluation reference point from control centroid to perturbed centroid.”

What this actually means is that the usual benchmark anchor can hide how much of the prediction is just a generic response to disturbance.

In plain English, if you always measure a perturbed cell against the control average, any model that learns broad stress programs can score well. Cells often react in fairly consistent ways to being pushed around. They change transcription, activate stress pathways, and move away from baseline in ways that are not specific to the exact perturbation. That broad movement is real biology, but it is not the whole story.

I ran into this same issue while working through model outputs that looked “directionally correct” but were biologically bland. The model could tell me something changed. Great. But could it tell me what changed in a way that mattered? That’s where the evaluation started to fall apart. A control-centered metric can make a model look like it has discovered mechanism when it has mostly learned perturbation presence.

Systema’s point is uncomfortable because it suggests that some of the field’s favorite numbers are inflated by a kind of benchmark shortcut. If the control centroid is too forgiving, then the model is being graded on distance from baseline rather than fidelity to the perturbation’s identity.

How to apply it:

• Separate “perturbation detection” from “perturbation specificity” in your own experiments.
• Report both baseline-shift performance and perturbation-matched performance if you can.
• Be suspicious of broad claims like “the model predicts unseen perturbations well” unless the evaluation explicitly tests that claim.

Once I started thinking this way, I stopped treating a high score as evidence of understanding. It became evidence of one narrow skill: the model can spot that the cell is off baseline.

3) Unseen perturbations are much harder than the charts suggest

“Predicting unseen perturbation effects is much harder than standard metrics imply.”

What this actually means is that generalization across perturbations may be far weaker than the leaderboard implies, especially when the benchmark is built around broad response patterns.

The source text points to a concrete example from the Systema paper: fine-tuned scGPT reaches an AUC of 0.73 for CIN, while other models land near random. That matters because it shows the gap between benchmark comfort and real predictive difficulty. Even a respected model can struggle when the task is genuinely about unseen perturbation behavior rather than generic response classification.

I like this part of the argument because it cuts through the usual benchmark bravado. A lot of AIVC discussion sounds like we’re one scaling step away from perfect virtual perturbation simulation. Systema says, not so fast. If your evaluation is too close to the training distribution, or too generous in what it counts as success, then the model may be learning a shortcut that collapses when the perturbation is actually new.

That’s been my experience too. Models often look best when the test set is only mildly different from the training set. Once you ask for a truly unseen perturbation, the performance drops and the biology gets messy again. Which is fine, by the way. Biology is messy. The problem is pretending the mess disappeared because the benchmark got prettier.

How to apply it:

• Separate seen, held-out, and truly unseen perturbations in your evaluation.
• Check whether your train/test split leaks perturbation family information.
• Prefer benchmarks that force the model to extrapolate, not just interpolate.

If I were reviewing an AIVC paper today, I would ask one blunt question: what exactly was unseen, and how hard was the extrapolation really?

4) Leaderboards can reward generic stress biology

“Leaderboard scores may largely reflect generic perturbation effects such as cellular stress.”

What this actually means is that a model can get credit for learning a common response program that shows up across many perturbations, while still missing the perturbation-specific mechanism.

This is the part that annoys me the most, because it explains why so many results feel impressive and vague at the same time. If a model captures stress, cell-cycle disruption, or other common states induced by perturbation, it will often look broadly useful. And to be fair, that can be useful. But if the goal is virtual cell modeling as a tool for mechanism, target selection, or drug discovery, then generic stress detection is not enough.

The Systema framing makes that distinction explicit. It suggests that a high score may mean the model is good at recognizing that the cell has been perturbed, not at understanding the perturbation itself. That’s a subtle difference on paper and a huge difference in practice.

I’ve been burned by this in adjacent ML work. A model that learns the easiest confounder often looks excellent until you move to a setting where the confounder no longer helps. Then the actual task shows up, and the score drops off a cliff. This is why I’m increasingly allergic to benchmark claims that don’t tell me what signal the model is really using.

How to apply it:

• Test for confounders like stress response, batch, and cell-state shifts.
• Ask whether a model still performs when generic perturbation signatures are controlled for.
• Use evaluation designs that isolate perturbation identity from shared response programs.

If a model mostly tells me “this cell is upset,” I’m not calling that virtual biology. I’m calling that a useful but incomplete detector.

5) The real fix is better evaluation, not louder claims

“The challenge is deeper than current metrics suggest.”

What this actually means is that the field probably needs to stop celebrating scores that collapse multiple biological questions into one number.

I’m not saying the whole AIVC effort is broken. I’m saying the evaluation stack is too forgiving, and Systema is a reminder that a flattering benchmark can delay the hard work of building models that actually generalize. If the model is supposed to predict perturbation response, then the test should punish generic shortcuts and reward perturbation-specific structure.

That means better splits, better references, and better reporting. It also means being honest about what a model can and cannot do today. A lot of the hype around virtual cells comes from the idea that we can simulate biology the way we simulate physics. I don’t buy that yet. Biology is full of context, state dependence, and hidden coupling. A model that gets the average direction right is useful. A model that gets the mechanism right is much rarer.

How to apply it:

• Report multiple metrics, not just the best one.
• Explain the biological meaning of each metric in plain language.
• Design benchmarks that distinguish generic response from perturbation-specific prediction.

The practical takeaway is pretty simple: if your benchmark cannot tell the difference between “the cell is stressed” and “this perturbation has a distinct effect,” then the benchmark is too weak for the claims being made.

6) What I would do before trusting an AIVC model

“AIVC models should be judged on perturbation-specific generalization, not just control separation.”

What this actually means is that I would treat every impressive score as a starting point, not a conclusion.

Here’s the checklist I’d use now. First, I’d ask what the model is learning: perturbation identity, general stress, or both. Second, I’d look at whether the test set contains truly unseen perturbations or just lightly shifted ones. Third, I’d check whether the evaluation compares against control only, because that can make the task too easy. Fourth, I’d want to see whether the model still works after common confounders are controlled out.

I know that sounds fussy. It is fussy. But this is exactly the kind of fussiness biology ML needs if it wants to be more than leaderboard theater.

How to apply it:

• Read the benchmark protocol before reading the headline result.
• Write down the exact biological question the model is supposed to answer.
• Reject any claim that mixes “perturbation detection” with “mechanistic prediction” unless the paper proves both.

That’s the main reason I found Systema useful. It doesn’t just lower the temperature on AIVC hype. It gives me a cleaner way to ask whether a model is actually learning something specific, or just being rewarded for noticing that cells are unhappy.

The template you can copy

# AIVC evaluation checklist inspired by Systema

## 1. State the biological question
- What perturbation is being predicted?
- Is the goal detection, specificity, or mechanism?

## 2. Define the reference point
- Control centroid only
- Perturbed centroid
- Both, reported separately

## 3. Split the data honestly
- Seen perturbations
- Held-out perturbations
- Truly unseen perturbations
- No leakage across perturbation families

## 4. Check for shortcut signals
- Generic stress response
- Cell-cycle shifts
- Batch effects
- Shared state changes

## 5. Report multiple metrics
- AUC / correlation
- Perturbation-specific error
- Performance on unseen perturbations
- Sensitivity to confounder removal

## 6. Write the claim carefully
Bad: "The model predicts perturbation response well."
Better: "The model separates perturbed from control states, but perturbation-specific generalization remains limited."
Best: "Under a perturbed-centroid evaluation, the model captures X but fails on unseen perturbations Y and Z."

## 7. Review question
If the model only learned generic stress biology, would this benchmark still look strong?

The source article that triggered this breakdown is on Zhihu at https://zhuanlan.zhihu.com/p/2065024050586563909. My interpretation is original, but the central evaluation critique comes from the Systema paper discussed there, published in Nature Biotechnology in 2025.