- Rust 85.9%
- C# 6.8%
- Shell 6.6%
- Python 0.6%
- Dockerfile 0.1%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| .forgejo/workflows | ||
| compositor/cosmic-comp | ||
| crates | ||
| dev | ||
| docs | ||
| gui | ||
| install | ||
| protocols | ||
| schemas | ||
| tests | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| deny.toml | ||
| justfile | ||
| README.md | ||
Coseat
A second cursor and keyboard for an AI agent on your own desktop. Coseat gives every keyboard/mouse group — and every AI agent — its own cursor, its own keyboard focus and its own permissions, on the same screens. A person and a computer-use agent (for example Claude through MCP) can work side by side on one desktop without fighting over a single pointer, and the person stays in control: a kill switch, a per-agent policy (apps, outputs, shortcuts, rate limits) and an audit log.
Typical uses: letting an agent operate a browser, a game or a desktop app while you keep working; testing computer-use agents safely; two people with two mice on one machine.
Authorship: the idea, the requirements and the testing are Kiki's. The design documents and the code were written by Claude (Anthropic's AI model) working under Kiki's direction, so expect an AI-written code base and review it accordingly.
Status
| Part | State |
|---|---|
Core, control API, CLI (coseatd, coseatctl) |
done |
MCP server for agents (coseat-mcp, 24 tools incl. sequence and manual) |
done |
| sway backend, nested/headless lab in a container | done, verified end to end (tests/e2e/m2.sh–m4.sh) |
| X11 (MPX) backend | done; no client isolation (see Security notes) |
COSMIC backend (patched cosmic-comp, per-agent seat sockets) |
working; installed on the author's COSMIC desktop with ./install/install.sh (no root, see install/README.md); the "COSMIC (Coseat)" login entry is untested on hardware |
| Avalonia GUI (device tiles, drag & drop, kill/pause, permission prompts) | done, see docs/GUI.md |
| QEMU/QMP backend (Windows VM) | basic |
Try the COSMIC variant nested first with compositor/cosmic-comp/try.sh.
Docs: docs/DESIGN.md (concept), docs/PLAN.md (detailed plan, amendments in §14),
docs/AGENT-MANUAL.md (rules for agents), docs/compat/day0-results.md
(what real apps do with a second seat).
Heads up on X11: an agent that can open a terminal on the shared X server can bypass the policy (XTEST, XGetImage, XRecord). Deny terminals in
deny_appsor run agents on a nested X server.
Feedback welcome
Issues and suggestions are welcome, in particular on making agent control faster and more reliable (screenshot latency, "wait for the screen to change" primitives, pointer/keyboard injection quirks) and on the seat model for Wayland compositors (see docs/upstream.md).
Try it (fake backend)
cargo build
cat > /tmp/coseat.toml <<'TOML'
[[seat]]
id = "main"
kind = "human"
[[seat]]
id = "agent-1"
kind = "agent"
[policy.agent-default]
capabilities = ["pointer", "keyboard", "windows.list"]
TOML
target/debug/coseatd --config /tmp/coseat.toml --runtime-dir /tmp/coseat-run --state-dir /tmp/coseat-state &
target/debug/coseatctl --socket /tmp/coseat-run/admin.sock status
target/debug/coseatctl --socket /tmp/coseat-run/admin.sock agent add claude-1 --seat agent-1
Run it on sway
Needs sway ≥ 1.9 (1.12 tested), grim (screenshots) and wl-clipboard (per-seat clipboard).
cargo build --release
# 1. seats, kill hotkey, per-seat focus bindings and shortcut shadows for your sway
coseatctl sway-config --config ~/.config/coseat/coseat.toml --sway-config ~/.config/sway/config \
> ~/.config/coseat/sway.generated.conf
echo 'include ~/.config/coseat/sway.generated.conf' >> ~/.config/sway/config # then restart sway
# 2. daemon (finds coseat-injector next to itself) and an agent
coseatd --backend sway --generated-config ~/.config/coseat/sway.generated.conf &
coseatctl agent add claude-1 --seat agent-1 --policy agent-default
Seats must exist before apps start and the pool must be static: any swaymsg seat ... configuration command at
runtime makes sway re-attach virtual devices to the human seat (day-0 finding F1). coseatd detects that, kills the
affected injector and pauses the seat. coseat-panic (bind it to a hotkey, the generated config does) stops all agent
input in well under a millisecond without needing the daemon.
Tests against a real sway (no window on your desktop; needs Docker and the coseat-lab image from dev/container):
cargo build && COSEAT_SWAY_CONF=$PWD/dev/lab/sway-generated.conf dev/headless.sh up && dev/headless.sh exec /work/tests/e2e/m2.sh
No multi-seat compositor? Use the lab
coseatctl lab up starts sway + coseatd in a container (headless with a VNC preview, or as a window on your desktop),
so Coseat works on COSMIC, GNOME and KDE today. coseatctl launch --x11 -- COMMAND starts apps in the lab (XWayland for
Chromium/Electron). Details, take-over and the real-VT checklist: docs/LAB.md.
Give an agent a seat (MCP)
coseatctl agent add claude-1 --seat agent-1 --policy agent-default --token-out ~/.config/coseat/claude-1.token
# writes the token to a private file (never printed, never on a command line) and prints ready-to-paste host config:
# claude mcp add coseat -- coseat-mcp --agent claude-1 --token-file ~/.config/coseat/claude-1.token
Without --token-out the token is printed once and the snippet uses COSEAT_TOKEN=... (it will land in shell
history and the host's config file). coseat-mcp only accepts a token file that is yours, mode 0600 and small.
Agents without a token (coseat-mcp --join, what the installer sets up): each MCP server asks coseatd for a free
agent seat the first time a tool is used (session.join). You get a dialog naming the agent and its process (or
coseatctl requests / approve <id> / deny <id>); [join] mode = "auto" skips the question, "off" allows only
registered agents. The seat, its colour and its cursor belong to that server until it exits (plus 15 s for a
reconnect), then go back to the pool; so two Claude Code sessions, or Claude and a local model, each get their own
cursor from the same MCP config. Idle agent seats draw no cursor. No agent can see or click the dialog (zenity and
Coseat's own windows are denied to every policy).
coseat-mcp speaks MCP over stdio (one process per agent, spawned by the host). The seat comes from the token, never
from a tool argument. The model gets 24 tools: whoami, list_outputs, list_windows, screenshot (image content;
pointer tools use the pixels of the last full one; region screenshots are zooms), pointer_move, click, drag, scroll, type_text, key_combo,
focus_window, move_window, acquire_lease, release_lease, give_window, clipboard_read/write,
share_clipboard, send_file (not available yet), list_seats, request_permission, wait, sequence (up to 50
input steps in one call on one screenshot's pixels, stops at the first failure, optional screenshot after) and manual.
manual serves docs/AGENT-MANUAL.md by topic: short rules for using the tools well, written so
that small local models can follow them too; the MCP server instructions tell every model to read it. Calls are paced
under the policy's rate limit on the client side, so long sequences slow down instead of getting the seat paused. Failures come back as
tool results with isError: true and JSON {"code","kind","message","retry_after_ms"} so the model can react. If the
connection to the daemon drops after a click or typed text was sent, the tool answers outcome_unknown (never repeats
it); read-only calls are repeated. Window titles and clipboard text are stripped of control/bidi characters and
shortened before they reach the model. Debug
without a host: coseat-mcp call <tool> '<json>', coseat-mcp list-tools.
Everything a window title or the clipboard contains reaches the model as tool output, so treat it as untrusted: the policy (apps, outputs, shortcuts, rate limits, human-yield) is what keeps a prompt-injected agent inside its lane.
Layout
| Crate | Role |
|---|---|
coseat-proto |
Wire types (JSON-RPC envelope, model, methods, events, errors). JSON Schemas in schemas/ are generated from it (just schemas). |
coseat-core |
Pure logic: model + reducer, policy, leases, human-yield, audit hash chain, rate limiter, config, agent registry. |
coseat-backend |
Backend trait and BackendEvent. |
coseat-backend-fake |
In-memory backend for tests and demos. |
coseatd |
Daemon: admin/agent Unix sockets, auth, enforcement, event fan-out. |
coseatctl |
CLI: status, seats, devices, windows, agent add/list/revoke, audit tail/verify, panic/pause/resume, config-check, lab up/down/status, launch. |
coseat-wl |
Wayland client helpers: seat-targeted virtual pointer/keyboard, Unicode typing via generated keymaps. |
coseat-injector |
One process per seat holding its virtual devices; NDJSON commands on stdin. Killing it is a complete kill switch. |
coseat-backend-sway |
sway backend: IPC, state diffing, config generator, per-seat focus, drift watchdog, grim capture, wl-clipboard. |
coseat-panic |
Emergency stop that works even when coseatd is hung. |
coseat-client |
Async JSON-RPC client used by coseatctl and coseat-mcp. |
coseat-mcp |
MCP stdio server for one agent (rmcp), 24 tools including sequence and manual. |
coseat-probe |
Dev helper for the day-0 experiments (seats, hold virtual input, watch per-seat events). |
just check runs fmt, clippy (-D warnings) and all tests.
Running an agent with a shell: coseatctl agent run
coseatctl agent add claude-1 --seat agent-1 --token-out ~/.config/coseat/claude-1.token
coseatctl agent run --token-file ~/.config/coseat/claude-1.token --workdir ~/project --ro ~/coseat/target/debug -- claude
The program runs in a bubblewrap sandbox with no network (--net shares the host's network namespace; see below), a
fresh home, a private /run and /tmp, /usr read-only, and no admin socket, compositor IPC (swaymsg), display
sockets or /dev/uinput. Coseat is reachable only through agent.sock and the token file. It writes only in --workdir
(plus --rw paths and the throw-away tmpfs). Mounts that would expose Coseat or secrets are refused even when spelled with
.., trailing slashes or symlinks (the runtime directory, $HOME, ~/.ssh, /run, /dev, ...), and the work directory
may not contain the socket or the token file.
With --net the program shares the host's network: abstract unix sockets of the host (X11 without a cookie, dbus) and
every local TCP service become reachable, which defeats the sandbox on the X11 backend. Use --net only for agents that
must reach the internet, on a Wayland backend. The sandbox shares the kernel and is not a boundary against a determined
local attacker. Details: docs/PLAN.md §16.
MCP over HTTP, and a Windows VM
coseat-mcp --http 127.0.0.1:8123 --agent claude-1 --token-file ... serves the same tools over streamable HTTP
(/mcp, Authorization: Bearer <token>, loopback only unless --allow-non-loopback).
coseatd --backend qemu --qmp-socket PATH drives a QEMU VM (its screen as one window; keys, pointer and screenshots
through QMP); see docs/PLAN.md §16.
Security notes (read before giving an agent a shell)
- Coseat is a guardrail, not a sandbox.
admin.sockis protected by the owning uid only. An agent that runs under the same uid with a shell can connect to it (or runcoseatctl) and act as admin. Run agents under a separate uid or in a sandbox that cannot reach$XDG_RUNTIME_DIR/coseat/admin.sock, the compositor socket or/dev/uinput(PLAN §8.6). A per-agent uid allow-list foragent.sockis not implemented yet. - Policy fails closed: unknown focus, unidentifiable windows and points outside every output are refused for seats that are clamped to outputs or that have app lists.
- The audit log is hash-chained but unkeyed; someone with write access to the file can recompute it, and truncating the tail is not detectable. There is no retention yet. Keep it on a path the agent cannot write.
- A stuck backend call holds the daemon's state lock for up to 5 s before it is abandoned; the kill switch through
the daemon queues behind it. The independent emergency stop
coseat-panic(PLAN §8.1) does not depend on the daemon. - Screenshots are refused whenever a window the policy forbids (or cannot identify) overlaps the captured area, because a
crop of the output shows whatever is on top. Layer-shell surfaces (notifications, lock screens) are not windows to
the daemon and cannot be told apart; toplevel capture is planned. The session guard watches
loginctl'sLockedHint, which swaylock does not set, so a screen locked that way is not detected. - The emergency stop latches in memory the first time the
KILLEDmarker is seen; deleting the file does not lift it.coseat-paniconly believes pid files that name a realcoseat-injectorof the same user. - Typed text and clipboard contents never reach the audit log (salted hash / size only).
- A second connection with the same token is allowed but audited; there is no per-agent uid binding or takeover handling yet (PLAN §5.1), so keep tokens private. When the last connection of an agent closes, its held buttons and keys are released and its leases dropped.