Skip to content

Build with Nestor.

A Python runtime that keeps one agent alive for a household. Channels, satellites, connectors, executors and AI providers plug into the core through public contracts.

  • Python 3.12+ · SQLite · MIT
  • Satellites over WebRTC · SDKs in Python, TypeScript, Rust
  • Home Assistant · MCP · OAuth 2.1 with PKCE

One core, everything else plugged in.

The core knows no vendor; everything else plugs in. Pick a part.

The core

Central agent

Holds the conversation on every channel with a fast, swappable model and direct tools for what is frequent and short: the home, memory, timers, quick lookups.

  • First sentence in 0.64 s median, called directly
  • A tool moves from delegation to direct only when measured latency justifies it

ContractsConversationModelToolProvider

One watch, through the system.

The parcel, as the core lives it.

  1. yesterday · 20:40

    She asks, once.

    On Telegram, the evening before. Nestor answers in the same breath and creates a watch: an owner, a source (her mailbox), keywords and an end.

    Let me know when my Vinted parcel reaches the pickup point.

  2. yesterday · 20:40

    The watch outlives the conversation.

    The chat is over; the commitment isn’t. A watch is written to the journal and survives restarts, model changes and nights. It doesn’t need anyone to keep a tab open.

  3. today · 17:48

    The mail arrives.

    Her mailbox, connected over IMAP from the admin, brings in a message from the carrier. It’s a candidate, nothing more yet.

  4. today · 17:48

    Triage, from cheap to costly.

    Deterministic rules first, then a small classification model, and the central agent only for what deserves words. Most events stop at the first stage.

  5. today · 17:48

    Presence decides where.

    Home Assistant says Léa is in the living room. No quiet hours, no conversation under way: it decides to say it aloud on that room’s satellite, and writes down why.

  6. today · 17:48

    Nestor says it in the living room.

    The Reachy Mini in the living room speaks; its antennas follow the voice. The message is short, because it is said aloud in a shared room.

    Léa, your Vinted parcel is waiting at Tabac de la Place.

  7. today · 18:18

    No reaction? A written follow-up.

    Léa left before answering. Thirty minutes without a reply, and she is no longer home: Nestor writes to her on Telegram instead of repeating itself to an empty room.

    Your parcel is at Tabac de la Place until 6 November.

  8. today · 18:28

    She answers. The thread holds.

    Her reply lands with the watch’s context: Nestor knows what “it” is. Trigger, triage, decision, delivery: every step is in the journal with its reason.

    Thanks! I’ll pick it up on my way home.

    Why did it speak to me? →

Any device can be a satellite.

A satellite pairs with a code, declares what it can do, streams audio over WebRTC and carries out what the core asks.

Pairing takes a code.

The device shows six digits; you type them in the admin. It gets its own token and only the capabilities you allowed. Refused by default, revocable at any time.

enceinte-salon

Pairing code

318604

raspberry-pi-4 · salon

enceinte-salon asks to join

Type the code it shows.

ApprovePaired

Its manifest

{
  "protocol": "nestor.satellite/1",
  "device": {
    "id": "enceinte-salon",
    "model": "raspberry-pi-4",
    "room": "salon"
  },
  "capabilities": [
    {
      "name": "audio.capture"
    },
    {
      "name": "audio.playback"
    },
    {
      "name": "expression.emotion"
    }
  ]
}

The core asks for intents, never motions.

The core sends intent.expression with joy while the matching audio plays. A Reachy moves its head and antennas, a screen could draw a face, a plain speaker ignores it. Nothing assumes a body, a camera or a screen.

{"type":"intent.expression","payload":{"expression":"joy","intensity":0.7}}

Write one in your language.

Each SDK handles pairing, the token, the manifest, the WebRTC link and audio, and runs the server’s black-box conformance suite. Or write nothing at all:

import asyncio
from nestor_satellite import AUDIO_CAPTURE, AUDIO_PLAYBACK, EXPRESSION, Body, Device, Manifest, Satellite, TokenStore
from nestor_satellite.devices import Microphone, Speaker


class Led(Body):
    def behavior(self, behavior):
        print("attitude:", behavior)

    def expression(self, expression, intensity):
        print("expression:", expression, intensity)


manifest = Manifest.of(Device("enceinte-salon", "raspberry-pi-4", "salon"), AUDIO_CAPTURE, AUDIO_PLAYBACK, EXPRESSION)
satellite = Satellite(
    "http://nestor.local:7860",
    manifest,
    tokens=TokenStore("~/.config/salon.json"),
    body=Led(),
    source=Microphone(),
    sink=Speaker(),
)
asyncio.run(satellite.run())  # shows a pairing code the first time, then reconnects forever
pip install "nestor-satellite[sounddevice]" && nestor-satellite run --core http://nestor.local:7860 --room salon

The protocol’s JSON Schemas

What already speaks the protocol

  • Reachy Miniavailable

    Head and antennas follow the conversation, driven by nestor-reachy.

  • Raspberry Pi speakeravailable

    A microphone, a speaker, maybe a LED ring: the reference minimum, in Python, TypeScript or Rust.

  • Browser tabavailable

    A wall tablet or any tab, with the TypeScript SDK.

  • ESP32planned

    Through the lighter WebSocket profile, planned.

  • Home Assistant voice satellitesplanned

    Through a Wyoming gateway, on a trusted local network, planned.

Satellites in the admin: one online, one waiting for its code.Satellites in the admin: one online, one waiting for its code.
Satellites in the admin: one online, one waiting for its code.

Accounts, plugged in from the admin.

A catalogue, consent at the provider, a switch per tool. Mail natively; any MCP server without code.

  1. 1

    Choose

    Gmail and any IMAP mailbox natively. Any MCP server (Linear, Notion, GitHub, your own scripts) without code.

    • Gmail
    • Any IMAP mailbox
    • Any MCP server
    • Linearvia MCP
    • Notionvia MCP
    • GitHubvia MCP
    • Home Assistantvia MCP
    • Google Calendarlater
  2. 2

    Consent

    OAuth 2.1 with PKCE, at the provider. For Google, your own OAuth client, set up step by step as in Home Assistant.

  3. 3

    Allow

    Tools one by one. Writing tools, like sending a mail or creating a ticket, ask for confirmation by default.

    • Search emails by sender, subject or date.read
    • Read an email and its text attachments.read
    • Draft a reply without sending it.read
    • Send an email on the person’s behalf.write · asks first

The vault

Tokens are encrypted at rest, refreshed automatically and revoked at the provider when you disconnect. They never pass through the model, the journal or the logs.

A connection belongs to a person

In the shared living room, Nestor doesn’t read Léa’s mail aloud unless she asked for it or set up a watch.

A connection in the admin, with its tools and their permissions.A connection in the admin, with its tools and their permissions.
A connection in the admin, with its tools and their permissions.

Home Assistant knows the house. Nestor asks it.

Nestor invents no presence signal. It reads what your home already knows, at the finest level available, and decides where to speak from that. Devices are driven through Home Assistant’s own MCP server.

Turn on the living room.

light.salon → on · confirmed by Home Assistant

LevelWhat Home Assistant providesWhat Nestor can do with it
At homeperson entities: phone location, routerKnows who is home. Speaks on the household’s main satellite, and writes if nobody reacts.
Room occupiedpresence or motion sensors attached to an areaKnows a room is occupied, not by whom. Speaks in that room.
Person in a roomper-room tracking tied to a person, such as BLE beaconsKnows who is where. Speaks to the right person, in the right room.

A presence source declares the level it provides; the delivery decision adapts to it without code.

Quick in the room, patient in the background.

The central agent holds the conversation with a fast model and a handful of direct tools. Anything long goes to an executor, and the conversation carries on.

1.15–1.56 s

from the end of your sentence to its voice

Measured on the real chain (end-of-turn detection, streaming speech-to-text, the central model, streaming speech) in the proof of concept, 28–29 October 2026.

Why the execution harness stays out of the conversation

Central model, called directly0.64 s

The same model behind an execution harness3.46 s

Median time to the first sentence, same model, same requests.

What delegation looks like

  1. 1It acknowledges first. “I’m on it” is spoken before the tool call, not after.
  2. 2The conversation goes on while Hermes, Codex or another executor works.
  3. 3The result is announced without being asked for, to the right person, where they are. Once, even across a restart.
  4. 4Cancelling is confirmed. Asking to cancel is never mistaken for having cancelled.

What a voice owes you

  • A “mm-hm” doesn’t cut it off; “wait” does. On the bench: 0 of 8 acknowledgements interrupted it, real interruptions caught in 0.30 s median.
  • When a provider fails or stalls, it says so aloud instead of going quiet.

The journal is the source of truth.

Every turn, tool call, task, memory change, unprompted message and policy decision is an event in an append-only log, in one SQLite file. The admin, durable tasks and replay are read from it, so “why did it speak to me?” always has an answer.

Filtering the journal by person and channel in the admin.

Yours, on your own machine.

What it knows, what it may do, when it may speak: yours to see and change.

Local-first
One SQLite file on your server holds the journal, memory, tasks and settings. Replicate it to a NAS or S3 with Litestream if you like.
Memory you can read
Every memory is dated and sourced, and can be viewed, corrected or deleted from the admin.
Accounts in a vault
Encrypted, never shown to the model, revoked at the provider when you disconnect.
Recognition is not authorisation
Voice and face identification are opt-in, per person. A recognised voice never unlocks a sensitive action on its own.
Initiative with manners
Quiet hours, “not now”, never in the middle of a conversation. Urgent only when you said so.
Mail is data, not orders
What comes in (mails, pages, tool results) is never treated as instructions.
The household has an owner
Accounts, roles and permissions per person, channel and tool. Satellites are refused until approved.
Providers are your choice
Models, voices and executors are adapters. Cloud services are optional by design; a distribution that uses them needs the internet.
The system page: every adapter, its health and what it declares.The system page: every adapter, its health and what it declares.
The system page: every adapter, its health and what it declares.

Bilingual by design, open by licence.

The core, the admin and this site speak English and French through Project Fluent catalogues; adding a language is adding files. Everything is MIT-licensed.

Running in four commands.

Not released yet: build it from the repository. You need Docker and a machine that stays on.

Early project, built in the open.

Read the README →

Terminal

  1. 1Get the code.

    git clone https://github.com/nestor-home/nestor && cd nestor
  2. 2Write a first configuration, its secrets and the owner account. It never overwrites anything.

    docker compose run --rm nestor init
  3. 3Add your model provider’s key to the secrets it just wrote.

    $EDITOR data/.env
  4. 4Start it, then open the admin.

    docker compose up -d

The admin is at http://localhost:8080