No description
  • Rust 85.9%
  • C# 6.8%
  • Shell 6.6%
  • Python 0.6%
  • Dockerfile 0.1%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-01 09:37:31 +01:00
.forgejo/workflows feat: phase 1 core (proto types+schemas, policy, leases, audit, ratelimit, model, config, auth) and backend trait 2026-10-03 19:43:18 +02:00
compositor/cosmic-comp feat: honest input results (keys delivery unverified + warnings, landed candidates, stale window-bound frames checked against a fresh snapshot), coseatctl run --seat, compositor patches 0012/0013 so X11 programs open on their agent's seat 2026-10-30 17:52:18 +01:00
crates feat: honest input results (keys delivery unverified + warnings, landed candidates, stale window-bound frames checked against a fresh snapshot), coseatctl run --seat, compositor patches 0012/0013 so X11 programs open on their agent's seat 2026-10-30 17:52:18 +01:00
dev feat: e2e scenario on real headless sway (per-seat focus, concurrent pilots, drift pause, Unicode), coseatctl call 2026-10-03 20:23:39 +02:00
docs feat: honest input results (keys delivery unverified + warnings, landed candidates, stale window-bound frames checked against a fresh snapshot), coseatctl run --seat, compositor patches 0012/0013 so X11 programs open on their agent's seat 2026-10-30 17:52:18 +01:00
gui fix(cosmic): review findings (client thread lifecycle and errors, seat pairing, required globals, windows only listed with geometry, injector retry vs emergency stop, bounded release_all), compositor patch 0004, tracked build artifacts removed, stricter e2e checks, injector supervision tests 2026-10-04 11:53:54 +02:00
install docs: README overview, status table, authorship note and feedback section; https clone URL 2026-11-01 09:37:31 +01:00
protocols feat: agents join at runtime (session.join): a pool of agent seats, the human approves in a dialog, one seat and cursor per MCP server 2026-10-04 21:46:57 +02:00
schemas feat: one physical device is one tile on cosmic (udev merge shared with sway), moving a tile moves all its nodes and its remapper, a person's last keyboard or pointer cannot be moved away without force, a second human seat shows its cursor 2026-10-05 18:39:13 +02:00
tests feat: agents join at runtime (session.join): a pool of agent seats, the human approves in a dialog, one seat and cursor per MCP server 2026-10-04 21:46:57 +02:00
.gitignore fix(cosmic): review findings (client thread lifecycle and errors, seat pairing, required globals, windows only listed with geometry, injector retry vs emergency stop, bounded release_all), compositor patch 0004, tracked build artifacts removed, stricter e2e checks, injector supervision tests 2026-10-04 11:53:54 +02:00
Cargo.lock feat: one physical device is one tile on cosmic (udev merge shared with sway), moving a tile moves all its nodes and its remapper, a person's last keyboard or pointer cannot be moved away without force, a second human seat shows its cursor 2026-10-05 18:39:13 +02:00
Cargo.toml feat: phase 1 core (proto types+schemas, policy, leases, audit, ratelimit, model, config, auth) and backend trait 2026-10-03 19:43:18 +02:00
deny.toml feat: phase 1 core (proto types+schemas, policy, leases, audit, ratelimit, model, config, auth) and backend trait 2026-10-03 19:43:18 +02:00
justfile feat: phase 1 core (proto types+schemas, policy, leases, audit, ratelimit, model, config, auth) and backend trait 2026-10-03 19:43:18 +02:00
README.md docs: README overview, status table, authorship note and feedback section; https clone URL 2026-11-01 09:37:31 +01:00

Coseat

A second cursor and keyboard for an AI agent on your own desktop. Coseat gives every keyboard/mouse group — and every AI agent — its own cursor, its own keyboard focus and its own permissions, on the same screens. A person and a computer-use agent (for example Claude through MCP) can work side by side on one desktop without fighting over a single pointer, and the person stays in control: a kill switch, a per-agent policy (apps, outputs, shortcuts, rate limits) and an audit log.

Typical uses: letting an agent operate a browser, a game or a desktop app while you keep working; testing computer-use agents safely; two people with two mice on one machine.

Authorship: the idea, the requirements and the testing are Kiki's. The design documents and the code were written by Claude (Anthropic's AI model) working under Kiki's direction, so expect an AI-written code base and review it accordingly.

Status

Part State
Core, control API, CLI (coseatd, coseatctl) done
MCP server for agents (coseat-mcp, 24 tools incl. sequence and manual) done
sway backend, nested/headless lab in a container done, verified end to end (tests/e2e/m2.sh–m4.sh)
X11 (MPX) backend done; no client isolation (see Security notes)
COSMIC backend (patched cosmic-comp, per-agent seat sockets) working; installed on the author's COSMIC desktop with ./install/install.sh (no root, see install/README.md); the "COSMIC (Coseat)" login entry is untested on hardware
Avalonia GUI (device tiles, drag & drop, kill/pause, permission prompts) done, see docs/GUI.md
QEMU/QMP backend (Windows VM) basic

Try the COSMIC variant nested first with compositor/cosmic-comp/try.sh. Docs: docs/DESIGN.md (concept), docs/PLAN.md (detailed plan, amendments in §14), docs/AGENT-MANUAL.md (rules for agents), docs/compat/day0-results.md (what real apps do with a second seat).

Heads up on X11: an agent that can open a terminal on the shared X server can bypass the policy (XTEST, XGetImage, XRecord). Deny terminals in deny_apps or run agents on a nested X server.

Feedback welcome

Issues and suggestions are welcome, in particular on making agent control faster and more reliable (screenshot latency, "wait for the screen to change" primitives, pointer/keyboard injection quirks) and on the seat model for Wayland compositors (see docs/upstream.md).

Try it (fake backend)

cargo build
cat > /tmp/coseat.toml <<'TOML'
[[seat]]
id = "main"
kind = "human"
[[seat]]
id = "agent-1"
kind = "agent"
[policy.agent-default]
capabilities = ["pointer", "keyboard", "windows.list"]
TOML
target/debug/coseatd --config /tmp/coseat.toml --runtime-dir /tmp/coseat-run --state-dir /tmp/coseat-state &
target/debug/coseatctl --socket /tmp/coseat-run/admin.sock status
target/debug/coseatctl --socket /tmp/coseat-run/admin.sock agent add claude-1 --seat agent-1

Run it on sway

Needs sway ≥ 1.9 (1.12 tested), grim (screenshots) and wl-clipboard (per-seat clipboard).

cargo build --release
# 1. seats, kill hotkey, per-seat focus bindings and shortcut shadows for your sway
coseatctl sway-config --config ~/.config/coseat/coseat.toml --sway-config ~/.config/sway/config \
    > ~/.config/coseat/sway.generated.conf
echo 'include ~/.config/coseat/sway.generated.conf' >> ~/.config/sway/config   # then restart sway
# 2. daemon (finds coseat-injector next to itself) and an agent
coseatd --backend sway --generated-config ~/.config/coseat/sway.generated.conf &
coseatctl agent add claude-1 --seat agent-1 --policy agent-default

Seats must exist before apps start and the pool must be static: any swaymsg seat ... configuration command at runtime makes sway re-attach virtual devices to the human seat (day-0 finding F1). coseatd detects that, kills the affected injector and pauses the seat. coseat-panic (bind it to a hotkey, the generated config does) stops all agent input in well under a millisecond without needing the daemon.

Tests against a real sway (no window on your desktop; needs Docker and the coseat-lab image from dev/container):

cargo build && COSEAT_SWAY_CONF=$PWD/dev/lab/sway-generated.conf dev/headless.sh up && dev/headless.sh exec /work/tests/e2e/m2.sh

No multi-seat compositor? Use the lab

coseatctl lab up starts sway + coseatd in a container (headless with a VNC preview, or as a window on your desktop), so Coseat works on COSMIC, GNOME and KDE today. coseatctl launch --x11 -- COMMAND starts apps in the lab (XWayland for Chromium/Electron). Details, take-over and the real-VT checklist: docs/LAB.md.

Give an agent a seat (MCP)

coseatctl agent add claude-1 --seat agent-1 --policy agent-default --token-out ~/.config/coseat/claude-1.token
# writes the token to a private file (never printed, never on a command line) and prints ready-to-paste host config:
#   claude mcp add coseat -- coseat-mcp --agent claude-1 --token-file ~/.config/coseat/claude-1.token

Without --token-out the token is printed once and the snippet uses COSEAT_TOKEN=... (it will land in shell history and the host's config file). coseat-mcp only accepts a token file that is yours, mode 0600 and small.

Agents without a token (coseat-mcp --join, what the installer sets up): each MCP server asks coseatd for a free agent seat the first time a tool is used (session.join). You get a dialog naming the agent and its process (or coseatctl requests / approve <id> / deny <id>); [join] mode = "auto" skips the question, "off" allows only registered agents. The seat, its colour and its cursor belong to that server until it exits (plus 15 s for a reconnect), then go back to the pool; so two Claude Code sessions, or Claude and a local model, each get their own cursor from the same MCP config. Idle agent seats draw no cursor. No agent can see or click the dialog (zenity and Coseat's own windows are denied to every policy).

coseat-mcp speaks MCP over stdio (one process per agent, spawned by the host). The seat comes from the token, never from a tool argument. The model gets 24 tools: whoami, list_outputs, list_windows, screenshot (image content; pointer tools use the pixels of the last full one; region screenshots are zooms), pointer_move, click, drag, scroll, type_text, key_combo, focus_window, move_window, acquire_lease, release_lease, give_window, clipboard_read/write, share_clipboard, send_file (not available yet), list_seats, request_permission, wait, sequence (up to 50 input steps in one call on one screenshot's pixels, stops at the first failure, optional screenshot after) and manual. manual serves docs/AGENT-MANUAL.md by topic: short rules for using the tools well, written so that small local models can follow them too; the MCP server instructions tell every model to read it. Calls are paced under the policy's rate limit on the client side, so long sequences slow down instead of getting the seat paused. Failures come back as tool results with isError: true and JSON {"code","kind","message","retry_after_ms"} so the model can react. If the connection to the daemon drops after a click or typed text was sent, the tool answers outcome_unknown (never repeats it); read-only calls are repeated. Window titles and clipboard text are stripped of control/bidi characters and shortened before they reach the model. Debug without a host: coseat-mcp call <tool> '<json>', coseat-mcp list-tools.

Everything a window title or the clipboard contains reaches the model as tool output, so treat it as untrusted: the policy (apps, outputs, shortcuts, rate limits, human-yield) is what keeps a prompt-injected agent inside its lane.

Layout

Crate Role
coseat-proto Wire types (JSON-RPC envelope, model, methods, events, errors). JSON Schemas in schemas/ are generated from it (just schemas).
coseat-core Pure logic: model + reducer, policy, leases, human-yield, audit hash chain, rate limiter, config, agent registry.
coseat-backend Backend trait and BackendEvent.
coseat-backend-fake In-memory backend for tests and demos.
coseatd Daemon: admin/agent Unix sockets, auth, enforcement, event fan-out.
coseatctl CLI: status, seats, devices, windows, agent add/list/revoke, audit tail/verify, panic/pause/resume, config-check, lab up/down/status, launch.
coseat-wl Wayland client helpers: seat-targeted virtual pointer/keyboard, Unicode typing via generated keymaps.
coseat-injector One process per seat holding its virtual devices; NDJSON commands on stdin. Killing it is a complete kill switch.
coseat-backend-sway sway backend: IPC, state diffing, config generator, per-seat focus, drift watchdog, grim capture, wl-clipboard.
coseat-panic Emergency stop that works even when coseatd is hung.
coseat-client Async JSON-RPC client used by coseatctl and coseat-mcp.
coseat-mcp MCP stdio server for one agent (rmcp), 24 tools including sequence and manual.
coseat-probe Dev helper for the day-0 experiments (seats, hold virtual input, watch per-seat events).

just check runs fmt, clippy (-D warnings) and all tests.

Running an agent with a shell: coseatctl agent run

coseatctl agent add claude-1 --seat agent-1 --token-out ~/.config/coseat/claude-1.token
coseatctl agent run --token-file ~/.config/coseat/claude-1.token --workdir ~/project --ro ~/coseat/target/debug -- claude

The program runs in a bubblewrap sandbox with no network (--net shares the host's network namespace; see below), a fresh home, a private /run and /tmp, /usr read-only, and no admin socket, compositor IPC (swaymsg), display sockets or /dev/uinput. Coseat is reachable only through agent.sock and the token file. It writes only in --workdir (plus --rw paths and the throw-away tmpfs). Mounts that would expose Coseat or secrets are refused even when spelled with .., trailing slashes or symlinks (the runtime directory, $HOME, ~/.ssh, /run, /dev, ...), and the work directory may not contain the socket or the token file.

With --net the program shares the host's network: abstract unix sockets of the host (X11 without a cookie, dbus) and every local TCP service become reachable, which defeats the sandbox on the X11 backend. Use --net only for agents that must reach the internet, on a Wayland backend. The sandbox shares the kernel and is not a boundary against a determined local attacker. Details: docs/PLAN.md §16.

MCP over HTTP, and a Windows VM

coseat-mcp --http 127.0.0.1:8123 --agent claude-1 --token-file ... serves the same tools over streamable HTTP (/mcp, Authorization: Bearer <token>, loopback only unless --allow-non-loopback). coseatd --backend qemu --qmp-socket PATH drives a QEMU VM (its screen as one window; keys, pointer and screenshots through QMP); see docs/PLAN.md §16.

Security notes (read before giving an agent a shell)

  • Coseat is a guardrail, not a sandbox. admin.sock is protected by the owning uid only. An agent that runs under the same uid with a shell can connect to it (or run coseatctl) and act as admin. Run agents under a separate uid or in a sandbox that cannot reach $XDG_RUNTIME_DIR/coseat/admin.sock, the compositor socket or /dev/uinput (PLAN §8.6). A per-agent uid allow-list for agent.sock is not implemented yet.
  • Policy fails closed: unknown focus, unidentifiable windows and points outside every output are refused for seats that are clamped to outputs or that have app lists.
  • The audit log is hash-chained but unkeyed; someone with write access to the file can recompute it, and truncating the tail is not detectable. There is no retention yet. Keep it on a path the agent cannot write.
  • A stuck backend call holds the daemon's state lock for up to 5 s before it is abandoned; the kill switch through the daemon queues behind it. The independent emergency stop coseat-panic (PLAN §8.1) does not depend on the daemon.
  • Screenshots are refused whenever a window the policy forbids (or cannot identify) overlaps the captured area, because a crop of the output shows whatever is on top. Layer-shell surfaces (notifications, lock screens) are not windows to the daemon and cannot be told apart; toplevel capture is planned. The session guard watches loginctl's LockedHint, which swaylock does not set, so a screen locked that way is not detected.
  • The emergency stop latches in memory the first time the KILLED marker is seen; deleting the file does not lift it. coseat-panic only believes pid files that name a real coseat-injector of the same user.
  • Typed text and clipboard contents never reach the audit log (salted hash / size only).
  • A second connection with the same token is allowed but audited; there is no per-agent uid binding or takeover handling yet (PLAN §5.1), so keep tokens private. When the last connection of an agent closes, its held buttons and keys are released and its leases dropped.