[chenchenSimular]

Why

Ang asked for the in-chat live view to take input ("if it's a small change we should make it controllable", #sai-api, 10-31). The task view, shown inside Claude (MCP Apps) and as the local watch_url page for Claude Code / Codex CLI, streamed the screen view-only. The Playground and the hosted live_url page already take input over the same Guacamole session.

What

• When input is on. Only while Sai isn't working on the computer, the same rule as the live_url page:
  • The task waits for the person's approval, which is often a sign-in only they can do.
  • After the task ends, through a new Use this computer button. It reconnects the screen; Done turns input off at once, then disconnects and keeps the last frame. If the computer is asleep or taken back, the view says so and stops instead of waiting.
  • While the person uses the computer, the view keeps polling (one sai_view_poll per 4 s). Input goes off, with a note, once Sai starts another task on that computer: the task is archived (a newer task from this sai-mcp on the same session or the same machine, including new_session), or the session starts a new turn. Use this computer runs this check before it connects, and is hidden after another viewer took the screen over.
• While Sai works. Input is paused, and a note under the screen says so.
• Keyboard binding.
  • The keyboard binds to the screen element only, never the document, so the host's chat box keeps its keys.
  • Control+Alt+Escape hands the keyboard back.
  • Turning input off, the page losing focus, or the stream closing releases any held key or mouse button.
• Input module. ui/input.ts mirrors the Playground's bindComputerInput (sai-platform src/playground/guacamole/input.ts).

Not covered:

• A task started on the same machine from somewhere this sai-mcp can't see (the Playground, another sai-mcp process, another session) doesn't stop Use this computer. It ends only if another viewer opens the screen (the conflict closes this one), or the person presses Done.
• An approval made outside the card (link-only sign-in, Playground, live_url) leaves input on for up to one poll (about 4 s) while Sai resumes. 4 s is the /events rate-limit floor, so the poll can't go faster.

Not included: a free machine's control lease. The Playground takes one on input to keep the network on; the live_url page doesn't, and this matches live_url.

Tests

• ui/__tests__/input.test.ts:
  • nothing is sent while off;
  • mouse, touch and keys are forwarded once on;
  • held keys and buttons are released on off and on window blur;
  • Control+Alt+Escape hands the keyboard back.
• ui/__tests__/gating.test.ts: input on during needs_approval, off once running (after decide), after the task only while using (Done turns it off); using stops on archived / expired / a non-done status, and when the computer has no tunnel or is unreachable after the task.
• app-view.test.ts: a newer task on the same machine, in another session, archives the older one.
• npm run typecheck, lint, test (122) and build pass.
• Not yet tested by hand in Claude (web and desktop) or on the local watch_url page; that test runs before merge.

Docs: sai-platform coding-agents/live-view.md still says the task view is view-only; a docs PR follows once this is tested.

🤖 Generated with Claude Code

[lihautan]

I read the diff at 1742f1b against the full task-view.ts (poll loop, decide, followScreen) and input.ts. Checked: when input is turned on and off around approval and task end; releasing held keys and buttons on off, blur, close and dispose; keyboard scope (bound to the screen element, never the document); and whether the view can now see more of the tunnel or its token. input.ts is clean: release() runs before on flips, and dispose() runs before the client disconnects, so key-ups still go out. The in-card Approve path also turns input off before the approve call, because decide sets status: 'running' and then calls render().

Blocking

• task-view.ts:76 + :228: after the task ends, "Use this computer" can stay on while Sai works on that computer again. inputAllowed() is isDone(view), and keepLive() is using. Both are about this task only. poll() stops once the task is done (:181), so the card never learns that a newer task started on the same machine. The person keeps mouse and keyboard on the screen while that task runs. The archived signal at :197 would catch it, but only while polling. The bad case: use the computer from an old card, the model sends the next task, and both drive the screen. A related path: after a takeover, use stays visible (:118 ignores takenOver). followScreen resets takenOver = false (:259), so this old card takes the screen back from the new task's card, with input on.
  Fix: while using, keep polling (one sai_view_poll every 4 s already fits the rate limit). On archived, or on a status that isn't done, call stopUsing() with a note. Also gate use on the machine not running another task. If the server can't tell us that, at least say in the PR that this case isn't covered.

Non-blocking

• task-view.ts:76: an approval made outside this card leaves input on for up to one poll (about 4 s) while Sai resumes. That covers the link-only approvalUrl sign-in case and approvals in the Playground or live_url. Today the card can't detect it sooner. Worth a line in the PR, or a quicker poll while needs_approval.
• task-view.ts stop-use.onclick: Done sets using = false, but render() still calls setInput(liveConnected && isDone(view)), which is still true. Input stays on until untilEnd's next 1 s tick closes the stream. Call live?.setInput(false) in the handler, or make inputAllowed() require using once the task is done.
• No tests cover the gating in task-view.ts: on during needs_approval, off after decide, use → stopUsing when sai_view_live has no URL or the stream is unreachable, and Done. input.test.ts covers the binding only.
• The PR says the hand test in Claude web and desktop, and on the watch_url page, hasn't run yet. Check Control+Alt+Escape inside the Claude iframe: hosts may take that chord before the iframe sees it.
• Tunnel and token: no change. websocket_url was already in the view. Input gating is client-side only, as it is on live_url. Anyone who has the tunnel URL could always send input, so this PR adds no server-side exposure.

[chenchenSimular]

@lihautan thanks, addressed in 9f8ebb1.

Blocking: input stays on while a newer task drives the computer

• While the person uses the computer, the view keeps polling (one sai_view_poll per 4 s). It calls stopUsing() with a note on archived, on expired, or when the session's status is no longer done (a new turn), and doesn't fold that newer turn into the old card.
• Use this computer runs that check before it connects.
• TaskLog now also archives a task when a newer one starts on the same machine, so a new_session task from this sai-mcp counts too.
• use is hidden after a takeover, so an old card can't take the screen back from a newer task's card.
• Not covered, and now said in the PR: a task on the same machine that this sai-mcp can't see (Playground, another process, another session). The server can't tell us that today.

Non-blocking

• Done: inputAllowed now requires using after the task, so Done turns input off at once.
• Approval outside the card: 4 s is the /events rate-limit floor, so I didn't add a faster poll. The PR now says input can stay on for up to one poll.
• Tests: the gating rules moved to view-model.ts (inputAllowed, useEnd, useStopsOnLoss) and ui/__tests__/gating.test.ts covers them: on during needs_approval, off after decide, Done, and stopping on no tunnel / unreachable / archived / new turn. app-view.test.ts covers the same-machine archive. typecheck, lint, test (122) and build pass.
• Hand test in Claude web/desktop and on watch_url (including Control+Alt+Escape inside the iframe) still runs before merge.

[lihautan]

Re-review of 1742f1b..9f8ebb1. The blocking issue is fixed, so I'm approving.

Checked:

• Newer task on the same computer. After the task, the poll loop keeps running while using ((!isDone(view) || using)). It stops on archived, on expired, or on a status that isn't done, and it doesn't fold the new turn's events into the old card. sai_view_poll always returns status from deps.wait, so useEnd can't stop use by mistake when status is missing.
• Use this computer now polls once before followScreen(). render() runs first, but liveConnected is still false, so input stays off until the check passes.
• TaskLog.state() now returns archived when a newer task starts on the same machineId from another session (new_session). app-view.test.ts covers this.
• After a takeover, use is hidden, so an old card can't take the screen back.
• Done turns input off at once: inputAllowed(view, using) needs using after the task.
• stopUsing() turns input off in the same tick (render()). It shows the note now, or when the screen closes.
• gating.test.ts covers the rules that moved to view-model.ts. CI passes on ubuntu, macOS and Windows.
• The PR now states the cases it doesn't cover: a task this sai-mcp can't see, and an approval made outside the card (up to one 4 s poll).

Non-blocking:

• useStopsOnLoss(view, loss: 'no_tunnel' | string): the union becomes plain string. 'no_tunnel' | 'unreachable' (or the stream's kind type) would catch a typo at the call site.
• The hand test in Claude web/desktop and on watch_url, including Control+Alt+Escape inside the iframe, is still to do before merge, as you said.