AI assistant¶

A SDK-free REST pipeline on top of Gemini: function-calling chat, voice messages, speech-to-text (STT), text-to-speech (TTS), live translation, and a pgvector knowledge base.

UI: /me/ai, /me/translate, and the anonymous widget on the landing page.

The shape of it¶

Browser (/me/ai, /me/translate)
   │  same-origin fetch (CSRF header)
   ▼
Next.js BFF  /api/ai/{chat,stt,tts,translate}   ← validates shape, attaches JWT
   │  server→server
   ▼
Go API  /api/v1/ai/*   (JWT + rate limit ~20/min)
   │
   ▼
usecases/ai ──────────► pkg/gemini ──────► Gemini REST API
   │   ▲                 (3× retry + backoff on 429/5xx/network)
   │   └─ functionResponse
   ▼
ToolDef.Execute()  ← runs ON THE SERVER with the request context
   ├─ search_knowledge → ai_knowledge (pgvector)
   └─ get_server_time  → sample tool

The core principle

The model decides which tool to call; the backend executes it. The model never runs code, and because tools run with the request context, database access is subject to RLS and timeouts.

The chat loop¶

Build contents from the history (≤20 turns) plus the new prompt. Voice messages arrive as an inline base64 audio part — the chat model is multimodal, so no separate STT step is needed.
Call Gemini with the layered system instruction and the tool declarations.
If the response contains function calls: execute each tool, append the model turn and a functionResponse turn, and repeat (up to MaxSteps, default 4). Each executed call is returned to the client as Step{Tool, Args, Result} so the UI can show what the AI did.
If the response is text, return it.

Error semantics:

Condition	Response
Transient Gemini failure (even after 3 retries)	A fallback message in the user's language plus `degraded: true`
`GEMINI_API_KEY` missing	A genuine `500` (cause logged)
Unknown / failing tool	`{"error": …}` back to the model, which explains gracefully

Degraded must not become a 5xx

A transient Gemini failure should be a soft degradation for the user. Turning it into a 500 makes the whole chat look broken.

The three prompt layers¶

Layer	Source	Editable?	Purpose
1. Base rules	Compiled-in constant	never	Answer language, scope discipline, prompt-injection resistance
2. Scope	`ai_prompts` → `AI_SCOPE_PROMPT` → built-in	admin, at runtime	What the assistant helps with
3. Extra instructions	`ai_prompts` (optional)	admin, at runtime	Tone, additional rules

The guardrail layer must never become configurable

Layer 1 carries prompt-injection resistance and scope discipline. Making it database-driven would let any admin — or an attacker with an admin account — switch the protection off.

Answer language: the request's lang field (the frontend sends its UI language: mn/en/zh/ru; unknown ⇒ mn) is the decider. Even if the user writes in another language or the knowledge base is in another language, the answer comes back in the UI language, translating sources as needed. The instruction appears twice — at the start and the end of the prompt (primacy and recency).

Caching: prompts are cached for 60 seconds; SetPrompt invalidates the cache. SetPrompt only updates the seeded keys (scope, instructions) — the prompt surface cannot be widened through the API.

Admin UI: Admin → Settings. API: GET/PUT /v1/admin/ai/prompts/{key} (settings.manage).

Tools¶

ai.ToolDef{
    Declaration: gemini.FunctionDeclaration{
        Name:        "my_tool",
        Description: "This is how the model knows when to call it…",
        Parameters:  map[string]any{ /* JSON Schema */ },
    },
    Execute: func(ctx context.Context, args map[string]any) (map[string]any, error) {
        // runs on the backend, with the request identity in ctx (RLS applies)
        return map[string]any{"result": "…"}, nil
    },
}

Registered in server.go:

aiTools := append(ai.DefaultTools(), ai.KnowledgeSearchTool(aiRepo), myTool)

Tool	What
`search_knowledge`	Semantic search over `ai_knowledge`
`get_server_time`	The simplest example (Ulaanbaatar time)

Knowledge base (RAG)¶

The corpus lives in ai_knowledge — about 58 chunks written from the code and the docs (migration 48). Each row has a stable slug, source, lang, and a vector(768) embedding (migration 47, pgvector with an HNSW index).

Search logic:

Embed the question (RETRIEVAL_QUERY).
Take the top 8 candidates by cosine distance (embedding <=> $1).
Filter relative to the best match — drop anything further than relativeScoreMargin (0.03) and keep 2–4 rows.
If no embedder is configured, embedding fails, or nothing survives, fall back to an ILIKE keyword search.

The tool result reports which mode ran ("mode": "vector" | "keyword"). Logs record the mode, hit count, best score and slugs — never the user's question text.

Why a relative threshold?

Measured on this corpus, two entirely unrelated chunks could still score 0.64+ cosine similarity, so a fixed threshold (previously 0.55) filtered nothing. minVectorScore (0.35) is now only a garbage floor.

Backfill: after boot the API embeds rows with a NULL embedding or a stale content_hash in batches of 20, in the background — boot does not wait for it.

Editing the corpus: add or change rows inside a migration (keep the slug), then reboot or call POST /v1/admin/ai/knowledge/reindex (settings.manage).

Model: with GEMINI_EMBED_MODEL empty the client picks one itself — gemini-embedding-001 → text-embedding-004 → embedding-001. Every request asks for outputDimensionality: 768, so vectors always match the column.

Anonymous chat¶

The landing page carries a floating widget that works without an account: POST /v1/public/ai/chat. Same pipeline, but a separate usecase instance — wired with only the knowledge-search tool.

That separation is a security boundary

A tool that reads user data can be added to the authenticated assistant without ever becoming reachable by an anonymous visitor.

Three extra limits:

Limit	Value
Rate limit	~6/min per IP, burst 3
Payload	Message ≤1000 characters, history ≤6 turns
Prompt	An extra compiled-in layer stating that this is a guest

Push-to-talk and streaming: holding the large round button records, and on release a ~250 KB base64 clip (≈15 s) goes to POST /v1/public/ai/chat/stream, answered over Server-Sent Events. Because the chat model is multimodal, a single call returns both the transcript and the answer: the first line is the transcript (the server turns it into a transcript event) and the rest streams as delta events.

Spoken replies: the client splits the streaming text at sentence boundaries and sends each finished sentence to POST /v1/public/ai/tts, playing them in turn — so speech starts at the first sentence, not the whole answer.

Voice¶

Capability	Endpoint	How
Voice chat message	`POST /v1/ai/chat` + `audio`	Audio rides inline in the user turn
Speech → text	`POST /v1/ai/stt`	A single call with a strict "return exactly what you heard" instruction
Text → speech	`POST /v1/ai/tts`	A separate TTS model; raw PCM (L16/24 kHz) wrapped in a WAV header
Live translation	`POST /v1/ai/translate`	Text → translation; audio → two steps (STT then translate)

The TTS 503

The model occasionally returns 200 with no audio inside (the same text comes back complete on the next call). Speak therefore retries up to three times before returning 503 — not 500, because this is a transient hiccup.

Live-translation UX: the microphone records in ~7-second segments, with a fresh MediaRecorder per segment (timeslice chunks only carry a container header on the first one). Silent segments return an empty field, which is not an error.

Audio limits: a mime whitelist (webm/ogg/wav/mpeg/mp3/mp4/m4a/aac/flac) plus ~700 KB base64 (~30 s of opus), enforced in both the BFF (lib/aiBff.ts) and the backend DTO.

Answer variety¶

The same question never produces a byte-identical answer:

The system prompt has a style section — an anti-repetition rule plus one randomly chosen style hint per request.
Sampling: temperature 1.0, topP 0.95.

Only the wording varies — facts, numbers, steps and sources do not.

Configuration¶

GEMINI_API_KEY=                                # required; without it /ai/* returns 500
GEMINI_MODEL=gemini-2.5-flash                  # chat / STT / translation
GEMINI_TTS_MODEL=gemini-2.5-flash-preview-tts  # TTS
GEMINI_EMBED_MODEL=                            # empty = auto-select
GEMINI_VOICE=Kore                              # prebuilt voice
GEMINI_API_BASE=                               # proxy / tests
AI_SCOPE_PROMPT=                               # fallback scope when the DB layer is empty

Rate limit: /v1/ai/* ~20 requests/min (burst 5) — enough headroom for live translation's ~8 chunks/min. The timeout on this path is 50 seconds (30 elsewhere).

Tests¶

Everything runs without Gemini:

gemini.Generator is an interface — usecase tests use a fakeGenerator.
repointerface.AIRepository is faked in prompt / tool tests.
The HTTP client itself is tested against an httptest server (retry/backoff, no-retry on 4xx, function-call parsing).