Architecture - Digcrate

Overview

Digcrate is a Next.js application with a Convex real-time backend. The browser communicates with serverless API routes on Vercel, which in turn talk to Convex (database), Anthropic (AI), and MCP tool servers (music data).

Browser (Next.js App Router)
  │
  ├── Clerk (auth middleware)
  │
  ├── POST /api/chat ─────────────────────── Vercel serverless
  │     │                                         │
  │     │                               agentic-loop.ts
  │     │                                    │         │
  │     │                             Anthropic SDK   OpenRouter
  │     │                                    │
  │     │                             tool_use calls
  │     │                                    │
  │     │                             crate-cli MCP servers
  │     │                          (Discogs, MusicBrainz, Last.fm, …)
  │     │
  │     └── SSE stream ──────────────────── Browser
  │           (thinking, tool_start, tool_end, answer_token, done)
  │
  └── Convex subscriptions (real-time)
        (sessions, messages, artifacts, playlists, usage)

Tech stack

Layer	Technology	Purpose
Framework	Next.js 16 (App Router)	SSR, API routes, Turbopack dev
Deployment	Vercel	Serverless functions
Auth	Clerk	User sign-in, OAuth
Connected services	Auth0 Token Vault	OAuth connections for Spotify, Slack, Google
Database	Convex	Real-time sessions, messages, playlists, collections, influence graph, subscriptions, skills, shares
Dynamic UI	OpenUI (`@openuidev/react-lang`)	Agent-generated interactive components
Agent	Anthropic SDK + agentic loop	Tool-use loop with crate-cli MCP servers
Billing	Stripe	Checkout, billing portal, webhooks
Styling	Tailwind CSS v4	Dark theme, responsive mobile
Audio	YouTube IFrame API	Persistent player bar
Analytics	PostHog	Product analytics, LLM observability

Project structure

crate-web/
├── convex/                          # Convex backend (schema + query/mutation functions)
│   ├── schema.ts                    # Database schema (26 tables)
│   ├── sessions.ts                  # Chat session CRUD
│   ├── messages.ts                  # Message persistence
│   ├── playlists.ts                 # Playlist management
│   ├── subscriptions.ts             # Stripe subscription state
│   ├── usage.ts                     # Usage tracking and quotas
│   ├── userSkills.ts                # Custom skill CRUD
│   ├── shares.ts                    # Published Deep Cut shares
│   └── users.ts                     # User sync (Clerk → Convex)
├── src/
│   ├── app/
│   │   └── api/
│   │       ├── chat/route.ts        # SSE streaming — agentic loop entry point
│   │       ├── auth0/               # OAuth connect, callback, status
│   │       ├── stripe/              # Checkout, billing portal, webhooks
│   │       ├── skills/              # Custom skill management
│   │       └── keys/route.ts        # User API key management
│   ├── components/
│   │   ├── workspace/               # Chat panel, Deep Cuts panel, workspace shell
│   │   ├── sidebar/                 # Desktop and mobile sidebar
│   │   ├── settings/                # Connected services, plan, skills, keys
│   │   └── player/                  # YouTube audio player
│   └── lib/
│       ├── agentic-loop.ts          # Core agent loop (Anthropic + OpenRouter)
│       ├── openui/                  # 27+ OpenUI component definitions and registry
│       ├── web-tools/               # Connected service tools (Spotify, Slack, Google)
│       ├── auth0-token-vault.ts     # OAuth token exchange via Auth0 Management API
│       └── plans.ts                 # Subscription tiers and rate limiting

The agentic loop

Every user message goes through a multi-turn tool-use loop implemented in src/lib/agentic-loop.ts.

The browser sends a POST to /api/chat with the message, session history, model choice, and API key.
The route handler builds the tool list from crate-cli MCP servers and calls agenticLoop().
The loop calls the Anthropic (or OpenRouter) API with the message and available tools.
If the model returns tool_use blocks, the loop executes each tool call against the appropriate MCP server.
Tool results are fed back to the model as tool_result blocks. Steps 3–5 repeat up to 25 turns.
Each turn emits Server-Sent Events to the browser: thinking, tool_start, tool_end, answer_token, and finally done.
The browser’s chat panel renders tokens as they arrive and parses OpenUI components from the final answer.

The loop supports both Anthropic (direct SDK) and OpenRouter (OpenAI-compatible API) with the same event format. The useOpenRouter flag on AgenticLoopOptions selects the path.

Key data flows

Authentication

Request arrives
  → Clerk middleware checks session token
  → Authenticated: request proceeds with userId
  → Unauthenticated: redirect to /sign-in

User records are synced from Clerk to Convex via a webhook at /api/webhooks/clerk. The Convex users table stores the Clerk user ID, email, encrypted API keys, and Stripe customer ID.

Agent query

POST /api/chat
  → Validate Clerk session, load subscription + usage
  → Check quota (free: 10/month, pro: 50/month, team: 200/month pooled)
  → Build tool groups from crate-cli MCP servers
  → agenticLoop() → Anthropic SDK → tool calls → MCP servers
  → Stream CrateEvents as SSE
  → On done: write message + artifacts to Convex

Real-time state (Convex)

The browser subscribes to Convex queries that push updates in real time:

Subscription	What updates
`sessions.listByUser`	Sidebar session list
`messages.listBySession`	Chat message history
`artifacts.listBySession`	Deep Cuts panel
`playlists.listByUser`	Playlist library
`usage.getForCurrentPeriod`	Usage meter in settings

Connected services (Auth0 Token Vault)

User clicks "Connect Spotify" in Settings
  → GET /api/auth0/connect?service=spotify
  → Generates CSRF nonce, redirects to Auth0 /authorize
  → User grants permissions on Auth0 / Spotify consent screen
  → Auth0 redirects to /api/auth0/callback
  → Callback extracts Auth0 user ID, sets secure cookie

Agent calls read_spotify_library tool
  → getTokenVaultToken("spotify", auth0UserId)
  → Fetches Management API token (cached 23 hours)
  → GET /api/v2/users/{userId}?fields=identities
  → Returns access_token from the Spotify identity
  → Tool calls Spotify Web API with the token

Database schema

The Convex database has 24 tables. Key tables:

Table	Purpose
`users`	User accounts with Clerk ID, encrypted keys, Stripe customer ID
`crates`	Named session folders for organizing research
`sessions`	Chat sessions (title, crate membership, starred, archived)
`messages`	Chat messages (user and assistant roles)
`artifacts`	Saved research components (Deep Cuts)
`toolCalls`	Tool execution log per session
`playerQueue`	Audio player queue tracks per session
`playlists` / `playlistTracks`	User playlists with track metadata
`collection`	Vinyl collection records
`subscriptions`	Stripe subscription state
`usageEvents`	Per-user, per-period query count
`userSkills`	Custom slash commands
`shares`	Published Deep Cut share records
`influenceArtists` / `influenceEdges`	Cached influence graph nodes and edges
`telegraphAuth` / `telegraphEntries`	Telegraph publishing credentials and posts
`tumblrAuth` / `tumblrPosts`	Tumblr publishing credentials and posts
`orgKeys`	Team-level shared API keys by domain

​Overview

​Tech stack

​Project structure

​The agentic loop

​Key data flows

​Authentication

​Agent query

​Real-time state (Convex)

​Connected services (Auth0 Token Vault)

​Database schema