mirror of https://github.com/usememos/memos
chore(mcp): improve tool discoverability, add orientation tools and evals
Make the OpenAPI-driven MCP surface more usable by agents, following the mcp-builder guidance. - Enrich proto descriptions (single source of truth, flows to OpenAPI + MCP tool descriptions): document the memo `filter` CEL grammar with fields and examples (replacing the dangling "Refer to Shortcut.filter"), clarify the created_ts/updated_ts vs create_time/update_time naming, the visibility enum, the declarative replace semantics of Set* ops, and steer tag filters to `"x" in tags` (not the unsupported `tag == "x"`). - Mark SetMemoAttachments / SetMemoRelations idempotent via a per-operation override the HTTP-method heuristic can't express. - Curate two read-only orientation tools: shortcut_list_shortcuts (surfaces reusable CEL filters) and auth_get_current_user (the single allowed auth/identity op, for resolving the current user); guard test updated to keep the rest of the auth/user surface excluded. - Add a task-level evaluation suite (server/router/mcp/evals) with 10 verified questions, pinned to the deterministic demo seed.pull/6058/head
boojack
2 weeks ago
parent
daa71d0456
commit
047175dbed
@ -0,0 +1,67 @@
|
||||
# MCP Evaluations
|
||||
|
||||
Task-level evaluations for the memos MCP server. Where the `*_test.go` files
|
||||
verify the server *plumbing* (schema resolution, tool naming, annotations),
|
||||
these check the thing that actually matters for an MCP server: **can an LLM
|
||||
accomplish realistic tasks by composing the tools?** They are the regression
|
||||
net for tool descriptions and discoverability — e.g. a bad `filter` description
|
||||
leaves the unit tests green but makes question 3/5/9 unanswerable.
|
||||
|
||||
`memos_eval.xml` holds 10 question/answer pairs in the format used by the
|
||||
mcp-builder skill. Each question is independent, read-only, requires multiple
|
||||
tool calls, and has a single string-comparable answer.
|
||||
|
||||
## Why a fresh seeded instance (not the public demo)
|
||||
|
||||
Answers are pinned to the deterministic seed in
|
||||
[`store/seed/sqlite/01__dump.sql`](../../../../store/seed/sqlite/01__dump.sql)
|
||||
(10 memos — 7 top-level + 3 comments — 2 users, 12 reactions, no attachments,
|
||||
no shortcuts).
|
||||
|
||||
The public demo (`demo.usememos.com`) signs everyone into the **same shared
|
||||
`demo` account**, so visitors continually add/edit/delete memos and reactions.
|
||||
Its data has already diverged from the seed — do **not** evaluate against it.
|
||||
|
||||
The seed uses **relative** timestamps (`strftime('now','-N days')`), so the
|
||||
questions avoid absolute dates and rely only on relative ordering, counts, and
|
||||
content, all of which are stable across re-seeds.
|
||||
|
||||
## Running an evaluation
|
||||
|
||||
1. Launch a throwaway demo-mode instance (SQLite, auto-seeded) on a free port:
|
||||
|
||||
```bash
|
||||
go run ./cmd/memos --demo --driver sqlite \
|
||||
--port 8099 --data "$(mktemp -d)" \
|
||||
--instance-url http://localhost:8099
|
||||
```
|
||||
|
||||
2. The MCP endpoint is `http://localhost:8099/mcp`. Authenticate with the seed's
|
||||
demo personal access token:
|
||||
|
||||
```
|
||||
Authorization: Bearer memos_pat_demo
|
||||
```
|
||||
|
||||
3. Point an MCP client / eval harness at that endpoint and have the model answer
|
||||
each `<question>`, then string-compare against each `<answer>`.
|
||||
|
||||
Quick manual check of a single tool call:
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8099/mcp \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'Accept: application/json, text/event-stream' \
|
||||
-H 'Authorization: Bearer memos_pat_demo' \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"memo_list_memos","arguments":{"filter":"pinned == true"}}}'
|
||||
```
|
||||
|
||||
## Notes for whoever extends this
|
||||
|
||||
- The CEL `tag` field does **not** support `==`; filter tags with
|
||||
`"work" in tags` (or `tags.exists(t, t == "work")`), not `tag == "work"`.
|
||||
- `memo_list_memos` returns only top-level memos; comments are reached via
|
||||
`memo_list_memo_comments`.
|
||||
- The seed defines no shortcuts and no attachments, so `shortcut_list_shortcuts`
|
||||
and the attachment tools return empty sets against a fresh seed. Add seed rows
|
||||
before writing questions that depend on them.
|
||||
@ -0,0 +1,62 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!--
|
||||
Task-level evaluation for the memos MCP server.
|
||||
|
||||
These questions test whether an LLM can accomplish realistic tasks by
|
||||
composing the curated MCP tools (memo_list_memos with CEL filters,
|
||||
memo_list_memo_comments, memo_list_memo_reactions, shortcut_list_shortcuts,
|
||||
auth_get_current_user, ...). They are NOT unit tests of the server plumbing —
|
||||
see *_test.go for that.
|
||||
|
||||
Answers are pinned to the deterministic seed in
|
||||
store/seed/sqlite/01__dump.sql and were verified by querying a fresh
|
||||
demo-mode instance. Run against a freshly seeded instance ONLY (the public
|
||||
demo at demo.usememos.com is a shared, mutable account and will not match).
|
||||
See README.md in this directory for how to launch one.
|
||||
|
||||
Questions deliberately avoid absolute dates: the seed uses relative
|
||||
timestamps (strftime('now','-N days')), so only relative ordering, counts,
|
||||
and content are stable.
|
||||
-->
|
||||
<evaluation>
|
||||
<qa_pair>
|
||||
<question>Look at every memo in the instance and add up all the emoji reactions across them. What is the total number of reactions?</question>
|
||||
<answer>12</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>How many memos are pinned?</question>
|
||||
<answer>2</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Considering only the memos that are NOT pinned, one of them has more reactions than any other. That memo is about a book the author started reading. What is the title of that book?</question>
|
||||
<answer>Deep Work</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Two memos are pinned. One contains external hyperlinks and the other does not. Give the single tag of the pinned memo that contains hyperlinks.</question>
|
||||
<answer>sponsors</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>How many memos contain at least one fenced code block?</question>
|
||||
<answer>2</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>How many distinct users have authored the memos in this instance?</question>
|
||||
<answer>2</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Find the memo that is a cheat sheet of git commands. How many comments does it have?</question>
|
||||
<answer>2</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Find the memo containing a movie watchlist. What is the username of the user who created it?</question>
|
||||
<answer>alice</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Exactly one memo is a travel bucket list. It names a Nordic country the author plans to visit in winter to see the Northern Lights. Which country?</question>
|
||||
<answer>Iceland</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Using the authenticated session, what is the username (login name) of the account these memos belong to?</question>
|
||||
<answer>demo</answer>
|
||||
</qa_pair>
|
||||
</evaluation>
|
||||
Loading…
Reference in New Issue