Why an AI reviewer is not enough to keep a trading bot in check

The reviewer says the order looks fine. The log shows the request timed out. The broker may have it anyway. If the agent retries, you may be recovering from a failure—or creating a duplicate side effect. At that point, the strategy is not the thing eating your afternoon.

The problem is not that the agent made a bad prediction. It is that the system cannot tell you what already happened. Did the first request reach the broker before the timeout? Did a restart clear the local state while the broker still has the request? Is the retry new, or is it the second copy of the same action? A chat transcript and the reviewer answer do not resolve any of that.

That is where an AI reviewer runs out of jurisdiction. It can read the strategy, flag missing assumptions, and generate counterexamples. It cannot turn an ambiguous request into a recorded fact, or make a limit check return the same answer every time from the same inputs.

A reviewer is still useful

Reviewers are good at the work that needs interpretation. Ask one whether the proposed action matches the stated strategy. Ask it to look for a missing cost, a stale assumption, a contradiction in the rationale, or the case that makes the plan fall apart. Those are useful questions precisely because the answer may need explanation.

The problem begins when that same kind of model call becomes the last word before an external side effect. A reviewer can change its answer when the prompt, context, or model changes. It can receive incomplete state. It can time out. Even when its reasoning is excellent, it does not replace a record of what the system sent or what came back.

When a request times out, the log shows failure but the broker may have already acted. The system cannot tell you what already happened. If the agent retries, you may be recovering from a failure—or creating a duplicate side effect. Check the broker with the original client order ID before assuming nothing occurred.

Five questions that need a fixed rule

Some questions need an explicit record, a rule, and a repeatable answer:

• Is the requested quantity within the limit you set?
• Would this request exceed the exposure limit you set?
• Is the process paused, even after a restart?
• Is this a replay, or is an earlier broker outcome still unknown?
• Is this caller allowed to send the request?

Those checks are deliberately boring. A configured limit should not require a model to reconstruct what the operator meant. A pause should remain a pause when the agent restarts. An uncertain outcome should be reconciled before another request turns a question mark into a second side effect.

This does not make a strategy good. It does not prevent losses. It gives you a smaller and more useful advantage: when something behaves strangely, you can separate strategy behavior from the mess created by the agent, its tools, the network, a restart, or the broker interaction. That leaves you more time to test the strategy itself and less time turning logs into a crime scene.

Recover an uncertain request

Here is the recovery sequence for the example: record one client order ID, send the request once, and treat a timeout as an unknown outcome. Look up that same ID at the broker. If the original order is there, return its result instead of sending another order. If the lookup is inconclusive, the outcome remains unresolved; these steps do not imply that retrying is safe.

1. Record the request — Give it a client order ID
2. Send once — Timeout leaves the outcome unknown
3. Look up the same ID — Ask the broker what happened
4. Resolve the original result — Found order; no second send

Put the roles in the right places

The agent still has plenty to do. It can research, propose a strategy change, explain a decision, and surface failure cases worth testing. The broker still owns market access and execution.

The layer between them should record the request, the rule that allowed or rejected it, the broker response, and the uncertainty after a restart. That is the design problem TradeAssembly addresses.

TradeAssembly Core is being prepared as the local, open-source foundation for auditable workflow capabilities around an agent-driven trading system. TradeAssembly Relay is planned to add off-device monitoring, retained evidence, and supported remote inspection or controls. Core does not host your agent. Relay does not keep a local workflow running while it is off. The point is to make the surrounding system easier to inspect and recover. It does not choose strategies for you.

Related reading

• Alpaca: retrieve an order by client order ID — This API lets you check whether the broker has an order by the client-provided ID. When you retry a failed request, you can call this endpoint with the same ID to see if the broker already has the order.

• AWS Builders' Library: Making retries safe with idempotent APIs — This article explains that network timeouts are unreliable indicators of whether a request succeeded. The broker may have processed the request before the timeout occurred. Use unique request identifiers and idempotent APIs to make retries safe.

Scope note

This is a design note about workflow controls. It does not offer trading advice, recommend strategies, report performance, or claim that any control eliminates trading risk.