TUI-Use: Operating TUI Programs via tui-agent#

This skill teaches you how to use the tui-agent CLI tool to start, interact with, and control TUI (Terminal User Interface) programs. Think of tui-agent as your eyes and hands for terminal programs — it lets you see what's on screen, type text, press keys, click with the mouse, and wait for things to happen.

Prerequisites#

tui-agent must be installed and available in PATH, you can clone from tui-use github. Install from the sibling tui-agent/ directory:

bash

pip install -e tui-agent/

Architecture at a Glance#

text

You (via Bash tool)  -->  tui-agent CLI  -->  Daemon (auto-started)  -->  PTY  -->  TUI Program
                     <--  stdout/JSON    <--  Unix Socket             <--  pyte virtual screen

The daemon starts automatically on first use — no manual setup needed.

Core Workflow#

Every TUI interaction follows this pattern:

1. Start a session (launches the TUI program)
2. Wait for the program to be ready
3. Capture the screen to see what's displayed
4. Interact (type text, send keys, click, scroll)
5. Capture again to verify the result
6. Stop the session when done

This capture-act-capture loop is the fundamental rhythm. Always capture the screen before and after important actions so you can verify what happened.

Command Reference#

Session Management#

Start a TUI program:

bash

tui-agent start --name <session-name> [--cols 120] [--rows 40] -- <command> [args...]

• --name: A unique identifier for this session (you'll use it in all subsequent commands)
• --cols/ --rows: Terminal dimensions (default: 80x24). Use larger values for complex UIs
• Everything after -- is the command to run

List active sessions:

bash

tui-agent list

Check session status:

bash

tui-agent status --name <session-name>

Stop a session:

bash

tui-agent stop --name <session-name>
# Or stop everything:
tui-agent stop --all

Seeing the Screen#

Capture full screen:

bash

tui-agent capture --name <session-name>

Capture with cursor position:

bash

tui-agent capture --name <session-name> --cursor

This appends cursor coordinates to the output — helpful for knowing where you are in an editor.

Capture a specific region:

bash

tui-agent capture --name <session-name> --top 0 --left 0 --height 5 --width 40

Save a snapshot for later comparison:

bash

tui-agent capture --name <session-name> --save my_snapshot

Compare current screen to a saved snapshot:

bash

tui-agent diff --name <session-name> --snapshot my_snapshot

Read scrollback history (content that scrolled off the top):

bash

tui-agent scrollback --name <session-name> --lines 200

Typing and Keys#

Type text:

bash

tui-agent type --name <session-name> "hello world"

Type text and press Enter:

bash

tui-agent type --name <session-name> --enter "ls -la"

Send special keys:

bash

tui-agent key --name <session-name> <key1> [key2] [key3...]

Important: key vs type — The key command only accepts special key names listed below. For regular characters (letters, numbers, symbols) like vim's i, o, dd, :wq, always use type. Use key only for keys that aren't printable characters.

Supported key names (case-insensitive):

• Navigation: up, down, left, right, home, end, pageup, pagedown
• Editing: enter/ return, tab, backspace, delete, insert, space, escape/ esc
• Function keys: f1 through f12
• Ctrl combos: ctrl-a through ctrl-z

Paste text (uses bracketed paste mode, safe for editors):

bash

tui-agent paste --name <session-name> "multi-line\ntext content"

Use paste instead of type when inserting multi-line content into editors — bracketed paste mode prevents the editor from interpreting line breaks as commands.

Mouse Operations#

Click at a position (0-indexed row and column):

bash

tui-agent click --name <session-name> --row 5 --col 10

Scroll up/down:

bash

tui-agent scroll-up --name <session-name> --row 12 --col 40 --lines 3
tui-agent scroll-down --name <session-name> --row 12 --col 40 --lines 3

Waiting for Conditions#

Waiting is critical for reliable automation. TUI programs take time to render, and acting too fast leads to errors.

Wait for specific text to appear on screen:

bash

tui-agent wait --name <session-name> --text "Ready" --timeout 10

Wait for text to disappear:

bash

tui-agent wait --name <session-name> --text "Loading..." --absent --timeout 15

Wait for screen to stop changing (stable for N seconds):

bash

tui-agent wait --name <session-name> --stable 2.0 --timeout 10

Use --stable when you don't know exactly what text to expect, but need the program to finish rendering.

Resizing the Terminal#

bash

tui-agent resize --name <session-name> --cols 160 --rows 50

Best Practices#

Always wait before capturing#

TUI programs need time to render. After starting a session or sending input, always wait before capturing:

bash

tui-agent start --name ed -- vim file.py
tui-agent wait --name ed --stable 1.0         # Wait for vim to fully load
tui-agent capture --name ed --cursor           # Now safe to read the screen

Use meaningful session names#

Name sessions after their purpose, not the program:

• Good: editor, monitor, file-browser
• Bad: s1, test, abc

Capture before and after actions#

This is how you verify that your actions had the intended effect:

bash

tui-agent capture --name ed --save before_edit
tui-agent type --name ed "i"                   # Enter insert mode (single char, use type)
tui-agent type --name ed "new line of code"
tui-agent key --name ed escape                 # Escape is a special key, use key
tui-agent capture --name ed                    # Check the result

Choose appropriate terminal size#

• For simple programs (bash, basic TUIs): 80x24 (default) is fine
• For editors or complex UIs: use 120x40 or larger
• For programs with wide tables: increase cols to 160+

Clean up sessions when done#

Always stop sessions you no longer need:

bash

tui-agent stop --name ed

Or stop all sessions if you're done:

bash

tui-agent stop --all

Handle errors gracefully#

If a wait command times out (exit code 1), capture the screen to understand what happened instead of blindly retrying:

bash

tui-agent wait --name ed --text "Success" --timeout 5
# If this fails, capture to see the actual state:
tui-agent capture --name ed

Common Workflows#

Editing a File with nano#

bash

# Start nano
tui-agent start --name ed --cols 120 --rows 40 -- nano myfile.py
tui-agent wait --name ed --stable 1.0
tui-agent capture --name ed

# Type some code
tui-agent type --name ed "print('hello world')"
tui-agent key --name ed enter
tui-agent type --name ed "print('goodbye')"
tui-agent capture --name ed

# Save (Ctrl-O, then Enter to confirm filename)
tui-agent key --name ed ctrl-o
tui-agent wait --name ed --stable 0.5
tui-agent key --name ed enter
tui-agent wait --name ed --text "Wrote"

# Exit (Ctrl-X)
tui-agent key --name ed ctrl-x
tui-agent stop --name ed

Editing a File with vim#

Note: vim may not respond to input on some platforms (see Platform Notes). If vim works in your environment:

bash

# Start vim (use -u NONE to avoid config issues)
tui-agent start --name ed --cols 120 --rows 40 -- vim -u NONE -N myfile.py
tui-agent wait --name ed --stable 1.0
tui-agent capture --name ed

# Navigate to a specific line (e.g., line 10)
tui-agent type --name ed ":10"
tui-agent key --name ed enter
tui-agent capture --name ed

# Enter insert mode and add text
tui-agent type --name ed "o"                    # Open new line below (normal char, use type)
tui-agent type --name ed "    print('hello')"
tui-agent key --name ed escape                  # Back to normal mode

# Save and quit
tui-agent type --name ed ":wq"
tui-agent key --name ed enter
tui-agent wait --name ed --stable 1.0

Monitoring System with htop#

bash

# Start htop
tui-agent start --name mon --cols 160 --rows 50 -- htop
tui-agent wait --name mon --stable 2.0
tui-agent capture --name mon

# Search for a process
tui-agent key --name mon f3                     # F3 is a special key, use key
tui-agent type --name mon "python"
tui-agent key --name mon enter
tui-agent capture --name mon

# Scroll through the list
tui-agent scroll-down --name mon --row 25 --col 80 --lines 10
tui-agent capture --name mon

# Quit
tui-agent type --name mon "q"                   # Regular char, use type
tui-agent stop --name mon

Running a Shell Command and Reading Output#

bash

# Start a bash session
tui-agent start --name sh -- bash --norc --noprofile
tui-agent wait --name sh --stable 1.0

# Run a command
tui-agent type --name sh --enter "ls -la /tmp"
tui-agent wait --name sh --stable 1.0
tui-agent capture --name sh

# If output is long, check scrollback
tui-agent scrollback --name sh --lines 100

# Exit
tui-agent type --name sh --enter "exit"
tui-agent stop --name sh

Interacting with a ncurses Application#

bash

# Start any ncurses app (example: midnight commander)
tui-agent start --name fm --cols 120 --rows 40 -- mc
tui-agent wait --name fm --stable 2.0
tui-agent capture --name fm

# Navigate using arrow keys
tui-agent key --name fm down down down
tui-agent capture --name fm

# Press enter to open a directory
tui-agent key --name fm enter
tui-agent wait --name fm --stable 1.0
tui-agent capture --name fm

# Use function keys
tui-agent key --name fm f5                      # Copy dialog
tui-agent capture --name fm

# Quit
tui-agent key --name fm f10
tui-agent stop --name fm

Platform Notes#

vim on macOS: The system vim ( /usr/bin/vim) may not respond to type input when launched directly via tui-agent start. This appears to be related to how macOS vim detects terminal capabilities. Workarounds:

• Use nano or pico instead — they work reliably with tui-agent
• Use vim -u NONE -N to disable vimrc and run in nocompatible mode
• Install a different vim build (e.g., via Homebrew: brew install vim)
• For simple file edits, consider using shell commands ( sed, awk) instead of a TUI editor

Other programs: Most terminal programs (bash, htop, mc, ncurses apps) work well out of the box with tui-agent. If a program doesn't respond to input, try increasing the wait time or using --stable with a longer duration.

Troubleshooting#

"session not found" error: The session may have been stopped or the program exited. Check with tui-agent list.

Screen capture is empty or garbled: The program may not have finished rendering. Add a longer --stable wait or increase --timeout.

Keys don't seem to work: Some programs expect specific key sequences. Capture the screen to verify the program's current state (e.g., is vim in insert mode or normal mode?).

Program exited unexpectedly: Use tui-agent status --name <session> to check the exit code. The program may have crashed or received a signal.

Daemon not running: The daemon starts automatically. If issues persist, check if the socket file exists at /tmp/tui-agent-*.sock. You can manually start the daemon or simply retry the command.