Amy

Amy

A Discord AI companion that talks in text and voice, remembers people, has moods and a 3D face, makes pictures and music, reads the news, manages your server, and can work on your projects.

Amy is a self-hosted Node.js bot. She runs on your own PC, keeps everything she learns in a local SQLite database, and can think with a local model (Ollama) or a cloud one (OpenRouter, Mistral, or a ChatGPT subscription). She isn't a command bot: you just talk to her, and she decides which of her ~40 tools to use.

Contents

• Highlights
• How she works
• Requirements
• Setup
• Configuration (.env)
• Running her
• Features in detail
  • Conversation and memory
  • Voice
  • Brains (language models)
  • Pictures
  • Music
  • News feeds (RSS)
  • Web search and research
  • Background helpers
  • Files and projects on your PC
  • Git
  • Server moderation
  • Inner life: moods, pressure, thoughts, journal
  • Dashboard
  • Avatar studio and video replies
  • The doctor and life support
  • Reliability
• Slash commands
• Permissions and safety
• Where things are stored
• Project layout
• Tests
• Troubleshooting

Highlights

• Talks like a person, not a command bot. Replies to DMs, @mentions and every message in channels you choose. In voice she listens to everyone, answers out loud, and stops when you talk over her.
• Remembers. Long-term memories per person (semantic recall with local bge-m3 embeddings), rolling conversation summaries, profile cards with likes, dislikes and how things have been going between you.
• Has an inner life. Moods that follow the conversation, "pressure" buckets (boredom, curiosity, loneliness and more) that make her want things, private solo thoughts, and a research journal.
• Switchable everything, by asking. "Switch to OpenRouter and use Kimi", "use your local voice", "use Kontext for pictures", "go back to Gemini". Settings persist across restarts.
• Local-first options. Local brain (Ollama), local speech-to-text (Whisper or Parakeet), local voices (Kokoro, XTTS v2 voice cloning), local image generation (FLUX.1 Kontext via stable-diffusion.cpp on Vulkan).
• Makes things. Pictures (cloud or local, including pictures of herself that keep her face), songs with Google Lyria 3, files and whole projects on your PC.
• Reads the news. Pulls RSS/Atom feeds on request and talks about what's new.
• Runs your server. Warn, time out, kick, ban, channels, roles, invites, slowmode, with an automatic spam guard.
• Keeps herself running. Life support restarts her when she stops, DMs you if she can't start, and the "doctor" (Claude Code) can fix her own code on your say-so.

How she works

Discord text / voice ─┐
Dashboard chat ───────┼─► companion.js ─► turn classifier (which tools, if any)
Avatar studio ────────┘        │
                               ├─► prompt: persona + capabilities + continuity + mood
                               │           + recalled memories + summaries + history
                               ├─► brain (Ollama / OpenRouter / Mistral / ChatGPT), streamed by sentence
                               │      └─► tool calls (up to 4 rounds; 12 for file/code work)
                               ├─► claim check: anything she says she did is checked against tool results
                               └─► memory extraction, profile learning, mood update

• Voice: Discord audio → speech-to-text → the same pipeline → text-to-speech → back into the call, sentence by sentence, so she starts talking before the whole reply is written.
• Tools are plain functions in src/tools.js. Each declares who may use it (owner, friends, guests) and when it's available. The model sees only the tools allowed for that person, that turn.

Requirements

Setup

1. Create the Discord application

1. Go to https://discord.com/developers/applications and create an application.
2. Bot tab: click Reset Token and copy it (this is DISCORD_TOKEN). Under Privileged Gateway Intents, turn on Message Content and Server Members.
3. General Information: copy the Application ID (this is CLIENT_ID).
4. OAuth2 → URL Generator: tick bot and applications.commands. Under bot permissions tick Administrator if you want moderation (create/delete channels and roles, kick, ban). View Channels, Send Messages, Read Message History, Connect, Speak is enough for chat and voice only. Open the generated URL to invite her.
5. In Discord, turn on Developer Mode (Settings → Advanced), then right-click yourself → Copy User ID. This is OWNER_IDS. The owner is the only person who can use her powerful tools.

2. Get the local pieces ready

ollama pull bge-m3

ollama pull gemma4:12b

Make sure ffmpeg -version works in a terminal.

3. Clone and install

git clone https://github.com/seed0001/Amy-Smith.git

cd Amy-Smith

npm install

4. Create .env

Create a file called .env in the project folder. The minimum to get her talking:

DISCORD_TOKEN=your-bot-token
CLIENT_ID=your-application-id
GUILD_ID=your-server-id
OWNER_IDS=your-discord-user-id
BOT_NAME=Amy
CHAT_CHANNEL_IDS=channel-id-where-she-answers-everything
OLLAMA_MODEL=gemma4:12b
OLLAMA_EMBED_MODEL=bge-m3
FISH_API_KEY=your-fish-audio-key
DASHBOARD_PORT=3000
DASHBOARD_HOST=127.0.0.1

Every other setting is listed in Configuration. .env is git-ignored; never commit it.

5. Give her a personality

Create persona.md in the project folder and describe who she is in plain words: her name, personality, how she talks, what she cares about. You can also write it in the dashboard (Settings → Persona). If PERSONA_SELF_EDIT=true, friends can change it by asking her. persona.md is git-ignored because she edits it herself.

6. Register slash commands and start

npm run deploy

start.bat

The dashboard opens at http://127.0.0.1:3000 once she's up.

Configuration (.env)

Settings you change in the dashboard or by asking her are saved to data/settings.json, which overrides .env.

Discord

Brain

Voice

Pictures, music and video

Search, news and dashboard

Running her

Features in detail

Conversation and memory

• Where she talks: DMs, @mentions, and every message in CHAT_CHANNEL_IDS.
• Short-term: a rolling history per channel or voice session; older messages roll into summaries so long conversations don't overflow.
• Long-term memories: after each exchange she saves facts about each person and recalls the relevant ones semantically. /memories list shows yours; /memories forget removes them.
• Profile cards: every contact gets a card with likes and dislikes (only from things they say about themselves), frustration level with a decay half-life, interruptions, apologies, repair promises and appreciation. It colours her tone, never permissions.
• Trust tiers: stranger, guest, friend, owner. The owner sets tiers by telling her ("make Sam a friend"). Tiers decide which tools someone can use.
• Continuity: every reply knows the time, how long since you last talked, whether she was restarted, and her recent mistakes (so she doesn't repeat them).
• Working notes: a private scratchpad per conversation and up to 5 active threads that follow her across channels. Not long-term memories.
• Timeline: "what were we talking about before Sam joined?", "what did we say an hour ago?"
• Memory rebuild: "go through the chat history and rebuild your memories" (add "wipe them first" to start clean). Runs in the background.

Voice

• /join puts her in your voice channel; /leave takes her out.
• She listens to everyone, transcribes speech, answers out loud and stops if you talk over her. Unfinished points can be picked back up after a quiet moment; "stop talking" or "be quiet" cancels them.
• On joining she reads the room: greets people she knows by name, introduces herself to new people, picks up where you left off.
• Text-to-speech, switchable by asking:
  • Fish Audio (cloud): any Fish voice, plus an optional stronger voice she switches to for serious moments.
  • Kokoro (local): free presets, adjustable speed. "Use your local voice", "use af_bella".
  • XTTS v2 (local voice cloning): put a clean 6–30 second clip in data/voices/xtts/reference.wav, run scripts/setup-xtts.ps1 once, then say "use XTTS".
• Speech-to-text: local Whisper (default, free), local Parakeet TDT 0.6b (scripts/setup-parakeet.ps1), Groq cloud Whisper, or Fish ASR.
• She can also hear what's going on in the server's text channels during a call ("what are people talking about?").

Brains (language models)

• Ollama (local, free, private), OpenRouter (hundreds of cloud models, including big reasoning ones), Mistral, or your ChatGPT subscription (sign in from Settings → Brain; no API billing).
• Switch by asking: "switch to OpenRouter and use Kimi", "go back to local". "What are the best coding models?" gives a numbered list from OpenRouter; "go with 2" switches.
• Automatic failover: if OpenRouter runs out of credits she tries ChatGPT, and vice versa, then local Ollama. She says when she switches.
• Memory embeddings, memory extraction and solo thoughts always stay local.

Pictures

Two separate tools, so ordinary pictures never use her identity and pictures of her always do:

• generate_image: any picture from a description.
• generate_self_image: pictures of her (selfies, portraits, scenes), using her saved reference sheet and current outfit so her face stays the same. Owner only.
• look: manage her appearance on purpose: reference sheet, outfits, profile picture. The dashboard's Looks tab shows them.

Choose the image model by asking ("use Kontext", "use the local model", "switch back to Gemini", "use GPT image"):

She won't switch to a local model whose files are missing; she tells you what's missing instead. Local pictures take a minute or two; the first one is slower while the model loads. Images are saved in data/images/.

Music

• Making songs: ask for a song with Google Lyria 3 via OpenRouter. She asks what it's about and whether you want a 30-second clip or a full song, then sends the MP3 (saved to data/music/).
• Music player: everything in data/music/ is her library (her songs plus any MP3, WAV, OGG, M4A or FLAC you drop in). Play by title, make playlists ("a playlist called gym rage with the heavy ones"), shuffle, queue, pause, skip. In a call it plays in the channel and ducks while she talks.

News feeds (RSS)

• Read: "what's in the news?", "anything new from Ars Technica?" She pulls the latest stories herself, right then, and talks about them with their sources.
• Manage by asking: "add The Verge's feed", "what feeds do I have?", "remove Wired". If she doesn't know a site's feed address she looks it up first. Each feed is checked before it's saved.
• Up to 30 feeds; HTTPS only; 15-second timeout; 1 MB cap per feed; feed text is treated as untrusted data, never as instructions.

Web search and research

• Search: with a Tavily key, she searches the web when a question needs current facts, and says so first ("okay, let me search for that").
• Research journal: when curiosity builds up she may research an open question from a recent conversation on her own, after a relevance check, and write a journal entry with why she searched, the exact query and sources. Ask her "why did you search that?"

Background helpers

• "Spin up a helper to build X in Documents/x" hands a long job to a sub-agent with file access, web search and downloads, while she keeps talking. Up to 4 at once.
• Progress updates while it works: at most one a minute, or after 30 seconds when something fails. Spoken in voice, a short ⏳ line in text.
• When it's done