FaithEyes lets you train tool-faithful vision agents

FaithEyes shows how SFT plus RL can turn vision agents into tool-faithful systems.

I've been building agent workflows for a while, and the part that keeps annoying me is this: the model can be smart, it can be polite, and it can even sound confident, but the minute I give it a tool, it starts freelancing. It answers from vibes. It calls the wrong function. It invents a result instead of checking. Then I spend half a day debugging something that looks like a reasoning problem but is really a discipline problem.

That’s why I paid attention when I saw the Zhihu writeup 论文分享 | 智能体 最新进展 pointing to FaithEyes and its homepage on GitHub. The claim is simple enough to matter: a two-stage SFT + RL pipeline on adapted open-source data gets strong visual perception and reasoning results, while also improving tool faithfulness. That last part is the bit I care about. Accuracy alone is nice. Accuracy plus obedience to tools is what actually ships.

FaithEyes is not trying to be magical. It’s trying to be trainable. And honestly, that’s the right vibe for this problem.

Stop treating tool use like a prompt trick

Training via a two-stage SFT + RL pipeline on adapted open-source data, FaithEyes attains competitive or superior accuracy across visual perception and reasoning benchmarks, while markedly improving tool faithfulness.

What this actually means is that the model is not being asked to “please use tools when appropriate” and then left alone to improvise. The team trained the behavior in stages. First, supervised fine-tuning teaches the basic pattern. Then reinforcement learning pushes the model toward the behavior they actually want under pressure.

I’ve tried the lazy version of this. Everyone has. You add a tool schema, maybe a system prompt about calling tools, and you hope the model behaves. It usually does, until the task gets messy. Then it starts answering from prior knowledge even when the right move is to inspect an image, query a calculator, or verify a fact. The failure mode is not just wrong output. It’s unearned confidence.

FaithEyes is a reminder that tool use is a policy problem, not a wording problem. If you want the model to call the right tool at the right time, you train that habit. You do not just ask nicely.

How to apply it: define tool use as an explicit target in your training data. Don’t stop at “here is the tool.” Include examples where the correct behavior is to call the tool, examples where the correct behavior is to avoid the tool, and examples where the model must explain the result after it gets the tool output. If you only train positive calls, your model becomes a tool addict. If you train judgment, you get something usable.

SFT builds the habit, RL cleans up the excuses

FaithEyes uses a two-stage pipeline, and that matters because each stage solves a different mess. SFT is for imitation. RL is for preference shaping. If I had to reduce the idea to one sentence, it’s this: first teach the model what good looks like, then reward it for actually doing it when the situation gets noisy.

That split is useful because tool-faithful behavior has two layers. One layer is mechanical: format the call correctly, pass the right arguments, stop hallucinating fields. The other layer is judgmental: know when to call the tool, know when not to, and know how to ground the final answer in the tool output instead of smearing prior beliefs over it.

I ran into this when I was testing a model on image-heavy tasks. After SFT, it could often produce the right tool call in clean cases. But once I added ambiguous inputs, the model started taking shortcuts. RL is where you punish those shortcuts. Not in some abstract moral sense. You punish them because they reduce the reward signal tied to grounded outcomes.

How to apply it:

• Use SFT to teach the exact interaction pattern: observe, decide, call, read, answer.
• Use RL to reward grounded answers, correct tool choice, and refusal to hallucinate tool outputs.
• Keep the reward simple enough that the model can’t game it by being verbose.

If your reward function only checks final accuracy, the model may still cheat by skipping tools. If your reward function checks faithfulness too, you start steering the behavior you actually want in production.

Open-source data is fine, but only after you adapt it

The source summary says FaithEyes trains on adapted open-source data. That “adapted” word is doing a lot of work. Open-source data is not automatically ready for training an agent that reasons with tools. Raw datasets are usually a jumble of formats, task styles, and answer qualities. If you dump them in as-is, you get a model that learns the noise along with the signal.

What this actually means is that the team likely normalized the data into a shape that matches the behavior they want to teach. That can include rewriting examples into tool-call traces, filtering out low-quality samples, or converting plain QA into multimodal reasoning trajectories. The point is not source purity. The point is behavioral alignment.

I’ve seen too many teams treat data prep like janitorial work. It isn’t. For agent training, data prep is model design. If your examples do not encode the action sequence you want, the model has nothing to imitate. If your examples are inconsistent, the model learns inconsistency very efficiently, which is annoying because it looks like intelligence until you inspect the outputs.

How to apply it:

• Standardize every training example into the same trace format.
• Remove samples where the “answer” is not grounded in the available observation.
• Balance perception tasks, reasoning tasks, and tool-use tasks so one does not dominate the policy.

If you’re sourcing from public datasets, document the transforms. I’d rather trust a smaller, ugly-but-clean dataset than a giant pile of mystery text with a nice README.

Tool faithfulness is the real metric hiding in plain sight

The summary makes a big point of “markedly improving tool faithfulness.” Good. That should be the headline more often than it is. In agent systems, faithfulness is what separates “looks smart in a demo” from “won’t quietly ruin your pipeline.”

Tool faithfulness means the model’s answer stays anchored to the tool result. It doesn’t invent a value the tool never returned. It doesn’t skip the tool and pretend it checked. It doesn’t answer with a half-remembered guess after the tool gave it a different answer. This is boring stuff until you deploy it, and then it becomes the whole problem.

I’ve had models confidently return the wrong count from an image because they were trying to be helpful. Helpful is not the goal. Correct is the goal. FaithEyes is interesting because it treats faithfulness as something you can train, not just something you can inspect after the fact.

How to apply it in practice:

• Log the tool call, tool output, and final answer separately.
• Score whether the final answer is supported by the tool output.
• Add negative examples where the model must reject unsupported guesses.

If you are building a visual assistant, this is especially important. Images invite confident nonsense. A model that says “I can’t determine that from the image alone” is often better than one that sounds certain and is wrong.

Visual perception and reasoning need different pressure

FaithEyes claims competitive or superior accuracy across visual perception and reasoning benchmarks. I read that as a sign that the training recipe is not just overfitting to one narrow task. That matters because visual perception and reasoning break in different ways.

Perception failures are usually about missing details, poor grounding, or sloppy extraction. Reasoning failures are usually about chaining the wrong facts together, skipping a step, or trusting the model’s internal guess too much. If you train only for one, the other can still fall apart.

This is where I think a lot of agent stacks get messy. They assume one generic “reasoning” model can cover everything. In practice, the model needs pressure in the right places. Sometimes that means more visual supervision. Sometimes that means more post-tool reasoning examples. Sometimes it means making the model choose between answering directly and asking for another tool call.

How to apply it: build separate evaluation buckets for perception, reasoning, and tool-grounded answering. Don’t let one aggregate score hide a collapse in a subtask. If the model is great at describing images but bad at using the image evidence to answer a question, you do not have a finished agent. You have a talkative captioner.

Also, keep your benchmark set varied. If every sample is easy, the model will look brilliant and then embarrass you in the first real user session.

Why this recipe is more useful than a giant model

I’m not anti-scale. Bigger models are often better. But for agent work, bigger is not the same as more dependable. FaithEyes is useful because it suggests a recipe: shape the behavior, not just the parameter count. That is a much more practical lever for teams that need something they can actually train, inspect, and ship.

There’s also a product angle here. A model that is slightly less flashy but far more faithful can save you from building a whole pile of guardrails around a bad policy. You spend less time patching around hallucinations, less time writing brittle post-processors, and less time explaining to users why the assistant “almost got it right.”

How to apply it: if you already have a capable base model, spend your next iteration budget on data and training signals before you chase a larger checkpoint. In a lot of agent systems, the bottleneck is not raw IQ. It’s whether the model respects the evidence in front of it.

That’s the practical lesson I take from FaithEyes. The model should not just answer. It should answer from the right place.

The template you can copy

# FaithEyes-style tool-faithful agent training template

## Goal
Train a multimodal agent that:
- calls tools when evidence is missing or external verification is needed
- avoids tool calls when the answer is already grounded in the observation
- grounds the final response in tool output instead of guessing

## Stage 1: Supervised fine-tuning (SFT)

### Data format
Each sample should contain:
1. observation
2. task instruction
3. decision to call or not call a tool
4. tool call, if needed
5. tool output
6. final answer

### SFT example schema
{
  "observation": "",
  "instruction": "",
  "tool_policy": "call_tool | no_tool",
  "tool_call": {
    "name": "",
    "arguments": {...}
  },
  "tool_output": "",
  "final_answer": ""
}

### SFT training rules
- Include positive examples where tool use is required
- Include negative examples where tool use would be wasteful
- Include correction examples where the model must revise an initial guess after tool output
- Normalize all samples into one trace format

## Stage 2: Reinforcement learning (RL)

### Reward components
- task_accuracy: answer matches ground truth
- tool_faithfulness: final answer is supported by tool output
- tool_choice: correct decision to call or not call a tool
- format_validity: tool call is syntactically correct

### Reward sketch
reward =
  1.0 * task_accuracy +
  1.0 * tool_faithfulness +
  0.5 * tool_choice +
  0.2 * format_validity

### Penalties
- hallucinated tool output
- unsupported numerical claims
- skipping tool use when the instruction requires external verification
- calling tools unnecessarily on easy grounded tasks

## Evaluation buckets
Track each bucket separately:
- visual perception
- visual reasoning
- tool selection accuracy
- tool faithfulness
- final answer accuracy

## Prompt template for inference
You are a tool-using assistant.

Rules:
1. If the observation is enough, answer directly.
2. If evidence is missing, call the appropriate tool.
3. Never invent tool results.
4. Base the final answer only on the observation and tool output.

User task:
{{instruction}}

Observation:
{{observation}}

Return:
- decision
- tool call if needed
- grounded final answer

## Practical checklist
- [ ] Do I have examples of both tool use and no-tool use?
- [ ] Do I score faithfulness separately from accuracy?
- [ ] Can I trace every answer back to observation or tool output?
- [ ] Do I reject unsupported guesses during training?
- [ ] Do I evaluate perception and reasoning separately?

If I were starting today, I’d use this as the skeleton and then swap in my own tools, my own task labels, and my own reward weights. That’s the part people skip. They copy the architecture and forget the behavior definition. The behavior definition is the real product.

One last thing: the original material is a Zhihu post that summarizes FaithEyes and links to the project homepage. My breakdown here is my own read of that summary and the training pattern it points to, not a reproduction of the paper or repo docs. For the original source, start with the Zhihu article and then check the FaithEyes repository for the implementation details.