Skip to content

Media nodes: transcribe, speak, imagine

These three activity nodes call multimodal models: transcribe turns speech into text, speak turns text into speech, and imagine generates an image from a text prompt. They are the building blocks of voice agents and image-generating agents.

They pass audio and image data by reference, never as inline blobs. A transcribe node reads an audio reference; speak and imagine produce an audio or image reference. The bytes themselves are stored as local artifact files — they are never written into run history or session turns, and a resumed durable run replays the reference rather than the media.

Artifacts, in brief

An artifact is a blob of media (audio or image) stored on your machine and addressed by an id like art_.... Generated bytes live on the local filesystem under THEYGENT_ARTIFACT_DIR (default: a theygent-artifacts folder in the system temp directory). To turn a reference back into playable/viewable media, the chat client downloads it from GET /artifacts/{ref}, which serves only stored art_... ids — it cannot read arbitrary files.

See Voice input/output and Image input and generation for how these nodes show up in chat.

Models these nodes need

Each media node references a model by its logical id — the same way an llm node does — and each needs a model registered with a matching modality:

Node Required modality Local engine that serves it
transcribe audio.transcription whisper.cpp (whisper-server)
speak audio.speech the MLX audio server (mlx_audio.server) on Apple Silicon
imagine images.generation stable-diffusion.cpp (sd-cli) via the llama.cpp binding, or FLUX (mflux) on Apple Silicon via the MLX binding

If a node's model binding names an engine instead of a logical id, the run is rejected up front (engine_name_not_allowed). See Installing models and the engines for how to register each kind.

transcribe — speech to text

Reads audio and returns its transcript.

Ports

Port Direction Required Description
audio in yes An audio reference: a stored artifact id, an http(s) URL, or a local file path.
text out — The transcript text.
err out — An error-typed handle carrying a fetch or transport failure.

Configuration

Field Type Default Description
model string — (required) A logical model id with the audio.transcription modality.
params JSON object {} Passed through to transcription. Common keys: language, prompt, temperature, response_format (json | text | verbose_json).

Behavior

The node fetches the audio bytes from the reference and streams them to the inference plane's transcription endpoint — the transcription model call goes straight to your inference plane, not through a control-plane inference route. (The audio itself is stored as a control-plane artifact; it is the model call, not the storage, that bypasses the control plane.) On a fetch or transport failure the error binds to err and the run continues; wire err to handle it. The inspector gives this node a dedicated parameter panel with per-field help.

speak — text to speech

Turns text into an audio artifact.

Ports

Port Direction Required Description
text in yes The text to synthesize.
audio out — A reference to the produced audio artifact.
err out — An error-typed handle carrying a synthesis failure.

Configuration

Field Type Default Description
model string — (required) A logical model id with the audio.speech modality.
params JSON object {} Voice controls: voice, speed, format (mp3 | wav | opus | flac | aac | pcm).

Behavior

The node calls the inference speech endpoint, stores the returned audio as a new artifact, and emits the reference on audio. On the wire, the node's format key becomes the request's response_format, and it sets the artifact's MIME type. When params omits them, the node defaults voice to alloy and format to mp3.

Set a voice your engine actually has

alloy is a common hosted-API voice name and may not exist on a local TTS engine, which uses its own voice names. Set voice in params to one your engine supports rather than relying on the default.

imagine — text to image

Generates an image from a text prompt.

Ports

Port Direction Required Description
in in yes The text prompt describing the image.
out out — A reference to the produced image artifact.

Configuration

Field Type Default Description
model string — (required) A logical model id with the images.generation modality.
params JSON object {} Generation controls: size (WxH, snapped to multiples of 64, default 512x512), n (clamped 1–4), steps.

Behavior

The node sends the prompt to the inference image endpoint, stores the produced bytes as an image artifact, and emits the reference on out. Image generation loads weights per request and is serialized on the inference plane (one render at a time), so it can be slow — the run stream sends keepalives during the wait so the connection is not mistaken for a dead one.

Unlike transcribe and speak, imagine has no err port. If generation fails, the node emits nothing, and the run surfaces an honest empty-output reason naming the cause (the run stays completed with a note rather than a hard failure). In the inspector this node uses the generic form — a plain model field plus a raw JSON params box.

Example: a voice agent

An audio-in / audio-out agent transcribes your speech, answers with a language model, and speaks the answer back. This is exactly the shape chat's voice agents run.

graph LR
  input["input (audio)"] --> transcribe
  transcribe -->|text| llm
  llm --> speak
  speak -->|audio| output["output (audio)"]
  • The input boundary node declares modality: "audio", so chat gives it a microphone composer; the recorded clip is uploaded as an artifact and the run input becomes that reference.
  • transcribe reads the reference and produces text on text.
  • The llm node answers (see the llm node).
  • speak synthesizes the answer and emits an audio reference.
  • The output boundary node declares modality: "audio", so the run's output is the audio reference, which chat downloads into a playable bubble.

An image-generating agent follows the same idea with imagine feeding an output node whose modality is image.

Works well with