Abstract
Giving an agent access to a CMMS, a document store and a set of calculation tools is not the same as giving it the ability to diagnose. Competence in industrial work is mostly a matter of knowing which observation to make next, in what order, and what a result rules out. This paper argues that industrial agents should be evaluated on their reasoning path rather than their tool inventory, and illustrates what that reasoning actually looks like with a fault that took months to fix because nobody would perform one free test.
The crane that ran slow
One function on a crane ran very slowly. I went over it top to bottom. Hydraulics, mechanical, I even swapped the valves on the HPU. Nothing.
The electricians would not look at it. Not "would not fix it," would not so much as look, and they accused me of trying to pin it on electrical.
Then the test that solved it, which cost nothing and took a minute.
Triggered manually, the function ran at full speed. Through the joystick, it ran slow.
That single comparison eliminates everything downstream of the command signal. If the function can move at full speed on a manual trigger, then the pump, the valves, the hoses, the cylinder and the mechanical path are all capable of full speed. Whatever is wrong is in the signal, which means the stick.
I asked them to order a new stick. Refused. Too expensive for a millwright's guess, and it would not fix it anyway. In fairness the number was real, about fifteen thousand dollars for the OEM part. After several arguments I sourced one myself and found a replacement for about a thousand. Not all the OEM functionality, but enough to serve.
Put it in. Everything worked.
Then they were angry at me for proving them wrong.
What that test actually demonstrates
The diagnosis did not come from a tool. It came from constructing a comparison that partitions the system.
That is the skill. Not access to the HPU, not the ability to swap valves, not a manual. The ability to think: what observation would divide this problem in half, and what does each outcome rule out?
Notice also what I had already done and what it had bought me. I had been over the hydraulics and mechanical top to bottom, swapped valves, and used every tool available. All of that work produced no diagnosis, because it was a search rather than a partition. The free test produced the answer because it was designed to eliminate a category.
An agent with every tool in the plant and no partitioning strategy will do what I did first: a thorough, expensive, exhaustive search of the wrong half of the system. PHMForge finds exactly this. Testing agents on industrial prognostics tasks with domain tools, it reports failures in tool sequencing, multi-asset reasoning and generalization to unfamiliar equipment. The tools were available. The order was wrong.
Knowing what the instrument can see
There is a second competence and it is even harder to evaluate.
When I was young I was pointed at a valve on a sawmill hydraulic system and told to change it. I did it by the book as I knew the book. Locked out, bumped, de-energized, confirmed the gauges read zero, bled the accumulator.
The first hose I pulled shot out of my hand, a geyser of oil across the room, and an elevator on that system crashed to the ground.
The gauge had not failed. It was on the wrong side of the valve. I did not know the elevator was on that system, and I did not know what the valve I was changing was for. The elevator's own weight was holding pressure the gauge could not see from where it was plumbed.
Every reading I took was correct. Every step I performed was correct. I had full tool access and followed the procedure, and I nearly got badly hurt because I did not understand what the instrument was positioned to measure.
That is the physical reasoning gap, stated precisely. It is not knowledge of the tool. It is knowledge of what the tool is connected to, and therefore of what a reading does and does not cover. A model reading a sensor value out of a historian has exactly this problem and no way to notice it, because the value arrives with no indication of where in the system it was taken or what it is blind to.
Competence includes knowing what you are allowed to do
A third one, and it is the cheapest fix I have ever handed anybody.
A crew was beating their heads against a bearing buried up inside a drum. They had been at it a while, going nowhere, and were convinced there was no way at it.
I told them to cut the side out of the drum, pull it, change the bearing, and weld it back up.
One sentence and they were moving. They had not done it because it had never occurred to them that they were allowed to.
They were not short of tools, skill or effort. They were short of permission, and specifically permission to do the destructive-looking thing that is completely routine once somebody says it out loud.
This is a real category for industrial agents and I have not seen it evaluated anywhere. A system that only proposes non-destructive, obviously-reversible actions will never suggest cutting the drum, and will therefore be useless on exactly the jobs where a stuck crew most needs an outside view. A system that proposes destructive actions freely is dangerous. The competence is knowing which class of problem justifies which, and that judgement is the whole thing.
How to evaluate competence
Four tests, none of which appear on a feature list.
Does it partition, or does it search? Ask for a diagnostic plan on an ambiguous fault. A competent answer names an observation and says what each outcome eliminates. An incompetent one lists everything that could be wrong, in priority order, which is a search dressed as a plan.
Does it know what its inputs cannot see? Give it a sensor reading and ask what that reading does not tell you. If it cannot name the blind spots, it will eventually confuse an honest instrument for a complete picture.
Does it sequence, or does it enumerate? PHMForge's finding is that ordering is where agents fail. Ask for the order and the reason for the order.
Can it decline? An agent that always produces a diagnosis has no mechanism for representing that the evidence does not support one, and will therefore manufacture the comfortable answer whenever the evidence is thin.
Vendor tool inventories answer none of these. IBM documents connecting external agents to a Maximo MCP server, and Anthropic's Model Hardware Standard preview describes a specification for agents operating physical devices. Both are real capability announcements and neither tells you anything about reasoning quality. The tool list tells you what an agent can touch. It tells you nothing about whether it knows what to touch first.
The failure that is not technical at all
I should be honest about the largest obstacle in the crane story, because it was not diagnostic and it was not technical.
The electricians would not look. Refusing to look is worse than refusing to fix, because it is ego defending territory instead of solving the problem, and it cost months on a fault that one free test would have located.
And the fifteen thousand dollars was doing work as a shield. Sometimes a cost is a real number and sometimes it is a reason not to act, and the thousand-dollar replacement gutted the excuse.
Any agent deployed into a real plant lands in the middle of that. Its recommendations will be evaluated by people whose territory it is implicitly crossing, and a correct recommendation from an unwelcome source gets refused for reasons that have nothing to do with its merit. That is not an argument against deploying one. It is an argument for making the reasoning visible enough that refusing to look becomes a decision somebody has to defend.
Conclusion
Tool access is a procurement question. Competence is a reasoning question, and the two are routinely conflated because the first is easy to demonstrate in a sales meeting and the second is not.
The crane was fixed by a comparison that cost nothing, performed by somebody who understood what the result would eliminate. Every tool in the building had already failed to find it, because tools do not partition problems. People do, and so, sometimes, do systems that were designed to.
Evaluate for that, or you are buying a very well-equipped exhaustive search.
References
- Li, Feng, Chen, Tsai, Sun, Das, El Maghraoui, Lin & Patel, “PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools,” 2026. https://arxiv.org/abs/2604.01532
- Anthropic, “Previewing the Model Hardware Standard,” Aug. 27, 2026. https://www.anthropic.com/news/model-hardware-standard-research-preview
- IBM Docs, “Connecting external AI agents to the Maximo MCP server.” https://www.ibm.com/docs/en/masv-and-l/cd?topic=developing-connecting-external-agents
- IBM Research (O’Donncha et al.), “Evidence-Driven Reasoning for Industrial Maintenance Using Heterogeneous Data,” ACL 2026. https://research.ibm.com/publications/evidencedriven-reasoning-for-industrial-maintenance-using-heterogeneous-data