Post
2448
Aiden: a physical AI agent that controls phones over USB HID
Most GUI agent work assumes the agent lives inside the device or
drives it through a debugging interface. We went the other way.
Aiden is a small board that sits outside the host. It captures the
screen over HDMI-to-CSI, runs the agent loop on-device, and sends
actions back as a standard USB HID device — the host sees a keyboard
and a mouse, nothing else. No app install, no root, no ADB, no cloud.
Runtime is Go. Frame capture, full-duplex audio with VAD, the agent
loop, and HID output all run as independent goroutines. There's no
backend — nothing leaves the device, which is the only defensible
design when the input is a live feed of someone's phone screen.
Open questions we haven't solved:
· Action verification — inferring success from a re-read of the
screen breaks when loading states lie
· Prompt injection — an agent that reads screens reads whatever an
attacker puts on them
· iOS pointer control requires AssistiveTouch
Repo, including the HID gadget config and capture pipeline:
github.com/AidenAI-IO/aiden-hardware-demo
Wrote up how this differs from cloud-based computer use agents here:
https://aidenai.io/blog/mobile-ai-agent-vs-computer-use-agent-whats-the-difference/
Note: current hardware is a dev board, not a finished product.
Most GUI agent work assumes the agent lives inside the device or
drives it through a debugging interface. We went the other way.
Aiden is a small board that sits outside the host. It captures the
screen over HDMI-to-CSI, runs the agent loop on-device, and sends
actions back as a standard USB HID device — the host sees a keyboard
and a mouse, nothing else. No app install, no root, no ADB, no cloud.
Runtime is Go. Frame capture, full-duplex audio with VAD, the agent
loop, and HID output all run as independent goroutines. There's no
backend — nothing leaves the device, which is the only defensible
design when the input is a live feed of someone's phone screen.
Open questions we haven't solved:
· Action verification — inferring success from a re-read of the
screen breaks when loading states lie
· Prompt injection — an agent that reads screens reads whatever an
attacker puts on them
· iOS pointer control requires AssistiveTouch
Repo, including the HID gadget config and capture pipeline:
github.com/AidenAI-IO/aiden-hardware-demo
Wrote up how this differs from cloud-based computer use agents here:
https://aidenai.io/blog/mobile-ai-agent-vs-computer-use-agent-whats-the-difference/
Note: current hardware is a dev board, not a finished product.
- 3 replies