fix(aiui): the content surface renders what the assistant found

Four defects, one visible symptom: a correct prose answer beside an
empty grid.

1. The assistant's curated RPC bridge had an arm only for
   `content.list-mine`. `tools.rs` mapped the `peers`, `purchased` and
   `films` scopes onto three real, dispatcher-registered handlers that
   `assistant_dispatch_tool` had never heard of, so every non-"own"
   scope died on its catch-all. Downstream that read as "the peers have
   no content" — it was a missing match arm, and the tool never ran.
   Regression test added: every scope the schema advertises must reach a
   real handler.

2. `content.browse-all-peers` wrapped its whole fan-out in one
   `timeout(..).unwrap_or_default()`, which DISCARDED every completed
   batch the moment the budget expired. One slow peer turned a
   partly-successful browse into "0 reached, 16 unreachable". Observed
   live on archi-dev-box: back-to-back calls returned real peer items,
   then nothing. Now accumulates per batch and checks a deadline between
   them, so partial results always survive. Budget 20s -> 45s: two
   batches of eight at a 10s per-peer timeout had no headroom at all.

3. `assistant.chat` returned only `{ text }`. The structured results of
   any content tool the turn ran were dropped inside the loop, so the
   surface had nothing to render. The turn now carries them through
   (captured raw, before the untrusted wrap, since they go to a renderer
   that treats every field as inert data, never back into the prompt).

4. The adapter classified images as 'excluded' and dropped them. A node
   sharing mostly photos rendered as an empty grid while AIUI's image
   grid sat unused. Images now have a bucket, with the paid-lock and
   extension-fallback handling audio and video already had.

Also: the panel says "Loading…" while a turn is in flight and "Nothing
found" when it comes back empty, instead of leaving the previous
query's heading standing as though it answered this one; the system
prompt tells the model to call the content tool and summarise rather
than re-list what the cards already show; and a refused tool now names
its permission category so the trusted chrome can offer the settings
screen instead of leaving "I don't have a tool for that" as the only
clue.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-07 05:00:54 -04:00
co-authored by Claude Opus 5
parent f7c541e867
commit 9abc162394
18 changed files with 650 additions and 56 deletions
@@ -30,11 +30,18 @@ describe('classifyByMime', () => {
expect(classifyByMime(item({ mime_type: 'audio/mpeg', filename: 'song.mp3' }))).toBe('audio')
})
it('excludes image and document mimes rather than mis-typing them', () => {
expect(classifyByMime(item({ mime_type: 'image/jpeg', filename: 'photo.jpg' }))).toBe('excluded')
it('classifies images as image, and still excludes documents', () => {
expect(classifyByMime(item({ mime_type: 'image/jpeg', filename: 'photo.jpg' }))).toBe('image')
expect(classifyByMime(item({ mime_type: 'application/pdf', filename: 'doc.pdf' }))).toBe('excluded')
})
it('classifies images by extension when the mime is generic', () => {
// A node whose catalog is mostly photos shared with an unidentified
// mime must not present as empty.
expect(classifyByMime(item({ mime_type: 'application/octet-stream', filename: 'p.webp' }))).toBe('image')
expect(classifyByMime(item({ mime_type: 'application/octet-stream', filename: 'p.heic' }))).toBe('image')
})
it('classifies m4a, aac, opus and wma as audio via extension fallback (ShareModal blind spot)', () => {
expect(classifyByMime(item({ mime_type: 'application/octet-stream', filename: 'track.m4a' }))).toBe('audio')
expect(classifyByMime(item({ mime_type: 'application/octet-stream', filename: 'track.aac' }))).toBe('audio')
@@ -74,7 +81,7 @@ describe('adaptContentItems', () => {
expect(bundle.films).toHaveLength(0)
})
it('excludes image and document mimes from all three buckets', () => {
it('routes images to the images bucket and still excludes documents', () => {
const bundle = adaptContentItems(
[
item({ id: 'img-1', filename: 'photo.jpg', mime_type: 'image/jpeg' }),
@@ -85,6 +92,20 @@ describe('adaptContentItems', () => {
expect(bundle.films).toHaveLength(0)
expect(bundle.songs).toHaveLength(0)
expect(bundle.podcasts).toHaveLength(0)
// The photo renders; the PDF has no grid, so it stays out of every bucket.
expect(bundle.images).toHaveLength(1)
expect(bundle.images[0]!.id).toBe('img-1')
expect(bundle.images[0]!.url).toBe('/content/img-1')
})
it('locks a paid image: price carried, no URL to fetch bytes the user has not bought', () => {
const bundle = adaptContentItems(
[item({ id: 'p-1', filename: 'photo.jpg', mime_type: 'image/jpeg', access: { paid: { price_sats: 100 } } })],
{ source: 'own' },
)
expect(bundle.images[0]!.locked).toBe(true)
expect(bundle.images[0]!.priceSats).toBe(100)
expect(bundle.images[0]!.url).toBe('')
})
it('maps an access:Paid item with a price and a locked flag, and no playable source URL', () => {
@@ -139,12 +160,12 @@ describe('adaptContentItems', () => {
it('an empty input array produces empty films/songs/podcasts arrays, not undefined or an error', () => {
const bundle = adaptContentItems([], { source: 'own' })
expect(bundle).toEqual({ films: [], songs: [], podcasts: [] })
expect(bundle).toEqual({ films: [], songs: [], podcasts: [], images: [] })
})
it('handles null/undefined input the same as an empty array', () => {
expect(adaptContentItems(null, { source: 'own' })).toEqual({ films: [], songs: [], podcasts: [] })
expect(adaptContentItems(undefined, { source: 'own' })).toEqual({ films: [], songs: [], podcasts: [] })
expect(adaptContentItems(null, { source: 'own' })).toEqual({ films: [], songs: [], podcasts: [], images: [] })
expect(adaptContentItems(undefined, { source: 'own' })).toEqual({ films: [], songs: [], podcasts: [], images: [] })
})
it('maps a null/absent description to an empty string, never the literal "null"', () => {
@@ -106,6 +106,23 @@ export interface Podcast {
priceSats?: number
}
/** Structurally matches `aiui/packages/core/src/types/content.ts`'s
* `ImageItem` — declared locally for the same D-19 reason as the shapes
* above (neode-ui does not depend on `@aiui/core`). */
export interface ImageItem {
id: string
url: string
title?: string
description?: string
alt?: string
width?: number
height?: number
source?: string
attribution?: string
locked?: boolean
priceSats?: number
}
// ─── Source shape: content_server.rs's ContentItem, as seen over RPC ───
/** `AccessControl` (`core/archipelago/src/content_server.rs`) serializes via
@@ -138,6 +155,12 @@ export interface ArchyContentBundle {
films: Film[]
songs: Song[]
podcasts: Podcast[]
/** Shared photos. Images were previously classified 'excluded' and
* dropped on the floor, so a node sharing mostly photos rendered as an
* empty grid — the single biggest gap between what the assistant could
* DESCRIBE and what the surface could SHOW. AIUI has had an image grid
* (`panelImages`/`ImageGrid`) the whole time; nothing fed it. */
images: ImageItem[]
}
export interface AdaptContentOptions {
@@ -153,7 +176,7 @@ export interface AdaptContentOptions {
// ─── Classification ───────────────────────────────────────────────────────
type ContentBucket = 'video' | 'audio' | 'excluded'
type ContentBucket = 'video' | 'audio' | 'image' | 'excluded'
// `m4a`, `aac`, `opus` and `wma` classify as audio via this extension
// fallback — `ShareModal.vue`'s mime map omits exactly these four today, so
@@ -175,6 +198,11 @@ const AUDIO_EXT_FALLBACK = new Set([
const VIDEO_EXT_FALLBACK = new Set(['mp4', 'mkv', 'avi', 'mov', 'webm', 'm4v'])
// Same reasoning as the audio fallback above: a photo shared with a mime
// this node could not identify still has an unambiguous extension, and a
// node whose catalog is mostly photos should not present as empty.
const IMAGE_EXT_FALLBACK = new Set(['jpg', 'jpeg', 'png', 'gif', 'webp', 'avif', 'heic', 'bmp'])
function extensionOf(filename: string): string {
const base = filename.includes('/') ? filename.slice(filename.lastIndexOf('/') + 1) : filename
const idx = base.lastIndexOf('.')
@@ -204,10 +232,12 @@ export function classifyByMime(item: Pick<ArchyContentItem, 'mime_type' | 'filen
const mime = (item.mime_type || '').toLowerCase().trim()
if (mime.startsWith('video/')) return 'video'
if (mime.startsWith('audio/')) return 'audio'
if (mime.startsWith('image/')) return 'image'
const ext = extensionOf(item.filename || '')
if (AUDIO_EXT_FALLBACK.has(ext)) return 'audio'
if (VIDEO_EXT_FALLBACK.has(ext)) return 'video'
if (IMAGE_EXT_FALLBACK.has(ext)) return 'image'
return 'excluded'
}
@@ -310,6 +340,30 @@ export function adaptToFilm(item: ArchyContentItem, opts: AdaptContentOptions):
}
}
/**
* Map a shared photo onto AIUI's `ImageItem`. A locked (paid, not yet
* bought) image gets an EMPTY `url` for the same reason a locked film
* does: the card should render its price and lock, not silently fetch
* bytes the operator has not paid for.
*/
export function adaptToImage(item: ArchyContentItem, opts: AdaptContentOptions): ImageItem {
const priceSats = paidPriceSats(item.access)
const locked = priceSats !== null
const title = stripExtension(item.filename || '')
return {
id: item.id,
url: locked ? '' : buildMediaUrl(item, opts),
title,
description: item.description ?? '',
// `alt` falls back to the title rather than being left empty — a photo
// grid with no alt text is unreadable to a screen reader.
alt: title,
source: SOURCE_LABEL[opts.source],
locked,
...(priceSats !== null ? { priceSats } : {}),
}
}
export function adaptToSong(item: ArchyContentItem, opts: AdaptContentOptions): Song {
const priceSats = paidPriceSats(item.access)
const locked = priceSats !== null
@@ -396,15 +450,18 @@ export function adaptContentItems(
const films: Film[] = []
const songs: Song[] = []
const podcasts: Podcast[] = []
const images: ImageItem[] = []
for (const item of sorted) {
const bucket = classifyByMime(item)
if (bucket === 'video') films.push(adaptToFilm(item, opts))
else if (bucket === 'audio') songs.push(adaptToSong(item, opts))
// 'excluded' (image/document/other) — not mistyped into any bucket.
else if (bucket === 'image') images.push(adaptToImage(item, opts))
// 'excluded' (documents/archives/other) — no grid renders these, so
// they stay out of every bucket rather than being mistyped into one.
}
return { films, songs, podcasts }
return { films, songs, podcasts, images }
}
// ─── Library mapping (13-11) ────────────────────────────────────────────
+3 -1
View File
@@ -368,7 +368,9 @@
"loadingAssistant": "Loading AI assistant...",
"aiAssistant": "AI Assistant",
"notConfigured": "AI Assistant needs to be enabled before use.",
"deployCta": "Go to Settings to configure your AI provider API key, then return here to start chatting."
"deployCta": "Go to Settings to configure your AI provider API key, then return here to start chatting.",
"permissionNeeded": "The assistant needed access to {categories}, which is switched off.",
"openAISettings": "Open AI settings"
},
"web5": {
"title": "Web5",
+2
View File
@@ -361,6 +361,8 @@
},
"chat": {
"close": "Cerrar",
"permissionNeeded": "El asistente necesitaba acceso a {categories}, que está desactivado.",
"openAISettings": "Abrir ajustes de IA",
"aiuiConnected": "AIUI conectado",
"closeAssistant": "Cerrar asistente de IA",
"loadingAssistant": "Cargando asistente de IA...",
@@ -183,6 +183,7 @@ describe('ContextBroker', () => {
films: [],
songs: [],
podcasts: [],
images: [],
},
expect.any(String),
)
@@ -354,6 +355,7 @@ describe('ContextBroker', () => {
films: [],
songs: [],
podcasts: [],
images: [],
}),
expect.any(String),
)
@@ -232,7 +232,14 @@ describe('tool confirmation — ContextBroker half', () => {
)
await vi.advanceTimersByTimeAsync(0)
expect(confirmRequests).toHaveLength(0)
expect(rpcClient.call).not.toHaveBeenCalled()
// `start()` hydrates AI permissions over RPC, so "no RPC at all" is too
// broad an assertion to state the property under test. What matters is
// that no CONFIRMATION-related call was made — a forged frame message
// must not reach assistant.pending or assistant.confirm-tool.
const confirmCalls = (rpcClient.call as unknown as { mock: { calls: [{ method: string }][] } }).mock.calls
.map(([arg]) => arg.method)
.filter((m) => m.startsWith('assistant.'))
expect(confirmCalls).toEqual([])
// 2) With a REAL confirmation open, a frame message shaped like the
// response must not resolve it — the response listener is for the
+65 -2
View File
@@ -6,6 +6,7 @@ import type {
ArchyContextResponse,
ArchyActionResponse,
ArchyChatResponse,
ArchyChatSurface,
ArchyContentPush,
} from '@/types/aiui-protocol'
import { useAIPermissionsStore } from '@/stores/aiPermissions'
@@ -21,6 +22,16 @@ import {
type ArchyLibraryTrack,
} from '@/composables/archyContentAdapter'
/** Wire shape of one entry in `assistant.chat`'s `surfaces` — the raw
* result of a content-producing tool the turn ran, exactly as its RPC
* handler returned it (`crate::assistant::Surface`). Every such handler
* answers `{ items: [...] }`, which is what `adaptChatSurfaces` reads. */
interface NodeChatSurface {
tool: string
scope?: string
data?: { items?: ArchyContentItem[] }
}
/** Wire shape of `content_owned::OwnedItem` (already-purchased peer
* content, cached locally). Its fields don't match `ArchyContentItem` —
* this is intentional; `normalizeOwnedItem` below bridges the gap rather
@@ -65,7 +76,7 @@ interface PendingToolConfirm {
}
function emptyBundle(): ArchyContentBundle {
return { films: [], songs: [], podcasts: [] }
return { films: [], songs: [], podcasts: [], images: [] }
}
function mergeBundles(bundles: ArchyContentBundle[]): ArchyContentBundle {
@@ -74,6 +85,7 @@ function mergeBundles(bundles: ArchyContentBundle[]): ArchyContentBundle {
films: [...acc.films, ...cur.films],
songs: [...acc.songs, ...cur.songs],
podcasts: [...acc.podcasts, ...cur.podcasts],
images: [...acc.images, ...cur.images],
}),
emptyBundle(),
)
@@ -233,16 +245,33 @@ export class ContextBroker {
// on the next turn, over and over. 420s must stay below AIUI's
// bridge timeout (430s) so this error, not the bridge's, is the one
// the user sees.
const result = await rpcClient.call<{ text: string }>({
const result = await rpcClient.call<{
text: string
surfaces?: NodeChatSurface[]
refused_categories?: string[]
}>({
method: 'assistant.chat',
params: { text },
timeout: 420_000,
})
// A tool was blocked by an ungranted category. Offer the toggle in
// the TRUSTED chrome — the iframe must not be able to draw anything
// that looks like a permission prompt (same reasoning as the confirm
// dialog), and the model's own "I don't have a tool for that" gives
// the operator no idea the capability is one switch away.
if (result.refused_categories?.length) {
window.dispatchEvent(
new CustomEvent('aiui:permission-needed', {
detail: { categories: result.refused_categories },
}),
)
}
this.postToIframe({
type: 'chat:response',
id,
success: true,
text: result.text,
surfaces: this.adaptChatSurfaces(result.surfaces),
} satisfies ArchyChatResponse)
} catch (err) {
this.postToIframe({
@@ -256,6 +285,40 @@ export class ContextBroker {
}
}
/**
* Turn the node's raw content-tool results into the SAME adapted grid
* records `content:push` already delivers, so AIUI renders them with
* the film/song/image components it has rather than needing a second,
* chat-only shape.
*
* Gated on media/files exactly like `handleContentRequest`: this
* carries node data (own files, peer catalogues, purchases) into the
* iframe, so it is a consent surface and is checked HERE rather than
* trusting the node's own grant check to be the only one. Dropping the
* surfaces never drops the answer — the prose still goes through.
*/
private adaptChatSurfaces(surfaces?: NodeChatSurface[]): ArchyChatSurface[] | undefined {
if (!surfaces?.length) return undefined
const perms = useAIPermissionsStore()
if (!perms.isEnabled('media') && !perms.isEnabled('files')) return undefined
const adapted = surfaces.flatMap((s) => {
const items = Array.isArray(s.data?.items) ? s.data.items : []
if (!items.length) return []
// The adapter uses `source` to decide the badge and how a playable
// URL is built, so a wrong value renders a peer's paid item as
// freely local — map each scope to what it actually is.
const source =
s.scope === 'peers' || s.scope === 'purchased'
? 'peer'
: s.scope === 'films'
? 'indeehub'
: 'own'
return [{ tool: s.tool, scope: s.scope, bundle: adaptContentItems(items, { source }) }]
})
return adapted.length ? adapted : undefined
}
private beginConfirmPolling() {
this.activeChatTurns += 1
if (this.confirmPollTimer) return
+15
View File
@@ -124,6 +124,21 @@ export interface ArchyChatResponse {
success: boolean
text?: string
error?: string
/** Content-producing tool results from this turn, already adapted into
* the same grid records `content:push` delivers, so AIUI can RENDER
* what the answer describes instead of leaving its surface empty
* beside a correct paragraph. Absent when the turn ran no such tool. */
surfaces?: ArchyChatSurface[]
}
/** One content tool result from a chat turn. `scope` is the tool's own
* argument (`own` | `peers` | `purchased` | `films`) — it is what lets the
* surface title itself with what was actually asked for rather than
* inferring it from the payload's shape. */
export interface ArchyChatSurface {
tool: string
scope?: string
bundle: ArchyContentBundle
}
/**
+110
View File
@@ -85,6 +85,36 @@
@dismiss="dismissToolConfirm"
/>
<!-- A tool the operator asked for was blocked by an ungranted
category. Trusted chrome, and Teleported to body for the same
reason ToolConfirmModal is: a transformed ancestor would trap
position:fixed. This only OFFERS the settings screen — it never
changes a grant itself, so nothing the iframe or the model says
can widen permissions. -->
<Teleport to="body">
<Transition name="fade">
<div v-if="permissionNeeded.length" class="chat-permission-offer" role="status">
<p class="text-sm text-white/85">
{{ t('chat.permissionNeeded', { categories: permissionNeededLabels }) }}
</p>
<div class="flex items-center gap-2 shrink-0">
<button class="chat-permission-btn" @click="openAISettings">
{{ t('chat.openAISettings') }}
</button>
<button
class="chat-permission-dismiss"
:aria-label="t('common.dismiss')"
@click="permissionNeeded = []"
>
<svg class="w-4 h-4" aria-hidden="true" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
</svg>
</button>
</div>
</div>
</Transition>
</Teleport>
</div>
</template>
@@ -94,6 +124,7 @@ import { useRoute, useRouter } from 'vue-router'
import { useI18n } from 'vue-i18n'
import { ContextBroker } from '@/services/contextBroker'
import ToolConfirmModal from '@/components/ToolConfirmModal.vue'
import { AI_PERMISSION_CATEGORIES } from '@/stores/aiPermissions'
import { IS_DEMO } from '@/composables/useDemoIntro'
const { t } = useI18n()
@@ -227,6 +258,32 @@ function onToolConfirmExpired(e: Event) {
}
}
// A tool the operator's question needed was refused because its category
// is off. The node reports WHICH categories; we name them and offer the
// screen that owns the toggles. Never flips a toggle here — the operator
// decides, on the settings screen, in the trusted chrome.
const permissionNeeded = ref<string[]>([])
const permissionNeededLabels = computed(() =>
permissionNeeded.value
.map((id) => AI_PERMISSION_CATEGORIES.find((c) => c.id === id)?.label ?? id)
.join(', '),
)
function onPermissionNeeded(e: Event) {
const detail = (e as CustomEvent).detail as { categories?: unknown }
const categories = Array.isArray(detail?.categories) ? detail.categories : []
const known = categories.filter(
(c): c is string => typeof c === 'string' && AI_PERMISSION_CATEGORIES.some((k) => k.id === c),
)
if (known.length) permissionNeeded.value = known
}
function openAISettings() {
permissionNeeded.value = []
router.push({ path: '/dashboard/settings', hash: '#ai-data-access' })
}
function onAiuiMessage(event: MessageEvent) {
if (!aiuiUrl.value) return
// Validate origin — only accept messages from AIUI
@@ -258,6 +315,8 @@ function armChatLive() {
window.addEventListener('aiui:tool-confirm-request', onToolConfirmRequest)
window.removeEventListener('aiui:tool-confirm-expired', onToolConfirmExpired)
window.addEventListener('aiui:tool-confirm-expired', onToolConfirmExpired)
window.removeEventListener('aiui:permission-needed', onPermissionNeeded)
window.addEventListener('aiui:permission-needed', onPermissionNeeded)
broker?.stop()
broker = null
if (aiuiUrl.value) {
@@ -279,6 +338,7 @@ onDeactivated(() => {
window.removeEventListener('message', onAiuiMessage)
window.removeEventListener('aiui:tool-confirm-request', onToolConfirmRequest)
window.removeEventListener('aiui:tool-confirm-expired', onToolConfirmExpired)
window.removeEventListener('aiui:permission-needed', onPermissionNeeded)
broker?.stop()
broker = null
if (loadTimeout) { clearTimeout(loadTimeout); loadTimeout = null }
@@ -294,6 +354,7 @@ onBeforeUnmount(() => {
window.removeEventListener('message', onAiuiMessage)
window.removeEventListener('aiui:tool-confirm-request', onToolConfirmRequest)
window.removeEventListener('aiui:tool-confirm-expired', onToolConfirmExpired)
window.removeEventListener('aiui:permission-needed', onPermissionNeeded)
broker?.stop()
broker = null
if (loadTimeout) { clearTimeout(loadTimeout); loadTimeout = null }
@@ -301,6 +362,55 @@ onBeforeUnmount(() => {
</script>
<style scoped>
/* Teleported to body, so this is positioned against the viewport, not the
chat panel. Sits above the iframe but below the confirm modal — a
blocking decision must always win over a passive offer. */
.chat-permission-offer {
position: fixed;
left: 50%;
transform: translateX(-50%);
bottom: calc(1.25rem + var(--safe-bottom, 0px));
z-index: 60;
max-width: min(40rem, calc(100vw - 2rem));
display: flex;
align-items: center;
gap: 0.75rem;
padding: 0.75rem 0.875rem;
border-radius: 0.875rem;
background: rgba(24, 24, 27, 0.92);
border: 1px solid rgba(255, 255, 255, 0.12);
backdrop-filter: blur(12px);
box-shadow: 0 8px 32px rgba(0, 0, 0, 0.45);
}
.chat-permission-btn {
padding: 0.375rem 0.75rem;
border-radius: 0.5rem;
font-size: 0.8125rem;
font-weight: 500;
white-space: nowrap;
color: #fdba74;
background: rgba(251, 146, 60, 0.14);
border: 1px solid rgba(251, 146, 60, 0.3);
transition: background 0.15s ease;
}
.chat-permission-btn:hover {
background: rgba(251, 146, 60, 0.24);
}
.chat-permission-dismiss {
padding: 0.375rem;
border-radius: 0.5rem;
color: rgba(255, 255, 255, 0.5);
transition: color 0.15s ease, background 0.15s ease;
}
.chat-permission-dismiss:hover {
color: rgba(255, 255, 255, 0.9);
background: rgba(255, 255, 255, 0.08);
}
.chat-loading {
position: absolute;
inset: 0;