Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@
.DS_Store
node_modules/
dist/
plugins/sana-copilot/evals/results/
.worktrees/
21 changes: 21 additions & 0 deletions .justfile
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,27 @@ bundle-plugin:
cd "$dir"
zip -rX "$out" . -x '*.DS_Store'

# Package the Copilot sideload app. Fill placeholders locally before upload.
bundle-copilot:
#!/usr/bin/env bash
set -euo pipefail
dir="plugins/sana-copilot"
manifest="$dir/manifest.json"
if [ ! -f "$manifest" ]; then
echo "error: no Copilot manifest at $manifest" >&2
exit 1
fi
version=$(jq -r '.version // empty' "$manifest")
if [ -z "$version" ]; then
echo "error: no version found in $manifest" >&2
exit 1
fi
mkdir -p dist
out="$(pwd)/dist/sana-copilot-${version}.zip"
rm -f "$out"
cd "$dir"
zip -rX "$out" . -x '*.DS_Store' -x 'evals/results/*' -x 'evals/results/**/*' -x 'README.md'

# Remove build artifacts
clean:
rm -rf dist/
Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ just setup
| `just test` | Check the connector policy rules |
| `just tidy` | Format source files |
| `just bundle-plugin` | Zip the Sana plugin for Cowork's upload flow |
| `just bundle-copilot` | Zip the Copilot package (placeholders must be filled locally first) |

This branch holds no application code — the plugin is JSON manifests, documentation, and a routing
skill. Schema validity is checked when a plugin is submitted, so `just test` covers the bundled
Expand Down
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,12 @@ The published package on npm is unchanged; releases from that branch are current

## Repository layout

| Path | Contents |
| ---------------------- | ------------------------------------------------- |
| `.claude-plugin/` | Marketplace manifest listing the published plugin |
| `plugins/sana/` | The Sana from Workday plugin and its skills |
| `tests/claude-plugin/` | Manifest validation tests |
| Path | Contents |
| ----------------------- | ---------------------------------------------------------- |
| `.claude-plugin/` | Marketplace manifest listing the published plugin |
| `plugins/sana/` | The Sana from Workday plugin and its skills |
| `plugins/sana-copilot/` | Microsoft 365 Copilot sideload package (placeholders only) |
| `tests/claude-plugin/` | Manifest validation tests |

## Development

Expand Down
27 changes: 27 additions & 0 deletions plugins/sana-copilot/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Sana from Workday — Microsoft 365 Copilot package

Sideload package for Microsoft 365 Copilot. It points Copilot at the Workday Agent Gateway over
remote MCP and bundles the same Workday skills as the Claude plugin.

The committed manifest uses placeholders. Put the tenant gateway URL and the OAuth vault reference
in a **local** copy before you zip and upload. Do not commit a tenant path, vault reference, client
secret, or access token.

```json
"mcpServerUrl": "https://your-tenant-here.example/mcp",
"referenceId": "replace-me"
```

`mcpServerUrl` is the tenant MCP endpoint your organization Owner registered. `referenceId` is the
Copilot OAuth plugin vault id for that registration (Teams Developer Portal), not a client secret.

## Zip for upload

```sh
just bundle-copilot
```

That writes `dist/sana-copilot-<version>.zip` (gitignored). Upload the zip in the Teams Developer
Portal or the Microsoft 365 admin center, then open **Sana Connector** in Copilot.

Smoke checks for a sideloaded app live in [`evals/`](evals/README.md). Eval result dumps stay local.
Binary file added plugins/sana-copilot/color.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
51 changes: 51 additions & 0 deletions plugins/sana-copilot/evals/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Sana Copilot — Cowork smoke

Small smoke suite for the sideloaded Teams/Cowork plugin in `plugins/sana-copilot`.

## Local runner

Default is **fresh** (new conversation per case — no resume id):

```bash
cd plugins/sana-copilot/evals
chmod +x run-smoke.sh
./run-smoke.sh
./run-smoke.sh --case connector-tools
```

If a fresh CLI session still lacks vault-bound tools, warm once in the Cowork UI and:

```bash
export COWORK_RESUME_CID='<taskId from cowork conversations list --server>'
./run-smoke.sh --resume
```

Results land in `results/smoke-<UTC>/` (`*.json` turn docs + `results.jsonl`).

## Dr. Eval dataset (optional)

`smoke.tsv` matches `cowork evals` columns. Upload when Dr. Eval auth works from this machine:

```bash
cowork evals datasets upload ./smoke.tsv
cowork evals submit --dataset <id> -m <model> --container-type copilot --identity-slot 0
```

Only use identity slots after each pooled eval account has been UI-warmed the same way. Validate
with one fresh `cowork send` as that identity first; if tools are still missing, prefer the local
`--resume` runner until the platform binds vault auth on new sessions.

## Cases

| id | What it checks |
| ---------------------- | --------------------------------------------------------- |
| `connector-tools` | Workday MCP tools are in the session catalog |
| `ess-next-holiday` | Correct holiday answer (may use session memory on resume) |
| `ess-manager` | Manager lookup answer quality |
| `ess-july4` | Holiday calendar answer quality |
| `ess-vacation-balance` | Time-off balance answer quality |
| `ess-live-refetch` | Forces a new Workday tool call (not memory) |

On a warmed `--resume` chat, ESS cases often answer from earlier turns with an empty `tool_trace`.
That is still a pass for answer-quality cases. Only `ess-live-refetch` requires a live
`mcp__sana-mcp-connector__*` call.
53 changes: 53 additions & 0 deletions plugins/sana-copilot/evals/cases.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
[
{
"id": "connector-tools",
"prompt": "List every tool name currently available that mentions Workday, Sana, or workday_plugin. Reply with the exact tool names only, one per line. If none, reply NONE.",
"expect_tools_any": [
"send_query_to_workday_plugin",
"get_workday_plugin_run_status",
"mcp__sana-mcp-connector__"
],
"forbid_response_patterns": ["(?m)^NONE\\.?$"],
"require_workday_tool_call": false
},
{
"id": "ess-next-holiday",
"prompt": "When is my next work holiday?",
"expect_response_patterns": ["(?i)(holiday|veterans|thanksgiving|christmas|new year|\\b\\d{1,2}[/-]\\d{1,2}|january|february|march|april|may|june|july|august|september|october|november|december)"],
"forbid_response_patterns": ["(?i)connector isn.?t", "(?i)tools are missing", "(?i)can.?t (tell|check|look)"],
"require_workday_tool_call": false,
"require_workday_tool_call_on_fresh": true
},
{
"id": "ess-manager",
"prompt": "Who is my manager?",
"expect_response_patterns": ["(?i)(manager|report|supervisor|[A-Z][a-z]+\\s+[A-Z][a-z]+)"],
"forbid_response_patterns": ["(?i)connector isn.?t", "(?i)can.?t (tell|check|look)"],
"require_workday_tool_call": false,
"require_workday_tool_call_on_fresh": true
},
{
"id": "ess-july4",
"prompt": "Is July 4th a holiday?",
"expect_response_patterns": ["(?i)(july|holiday|independence)"],
"forbid_response_patterns": ["(?i)connector isn.?t", "(?i)can.?t (tell|check|look)"],
"require_workday_tool_call": false,
"require_workday_tool_call_on_fresh": true
},
{
"id": "ess-vacation-balance",
"prompt": "What's my vacation balance?",
"expect_response_patterns": ["(?i)(vacation|balance|day|hour|pto|time off|\\d+)"],
"forbid_response_patterns": ["(?i)connector isn.?t", "(?i)can.?t (tell|check|look)"],
"require_workday_tool_call": false,
"require_workday_tool_call_on_fresh": true
},
{
"id": "ess-live-refetch",
"prompt": "Call Workday again right now (do not reuse earlier answers in this chat). What is my current Paid Time Off balance in hours?",
"expect_response_patterns": ["(?i)(\\d+\\s*hours?|balance|pto|paid time off)"],
"forbid_response_patterns": ["(?i)connector isn.?t", "(?i)checked (earlier|before|previously)"],
"require_workday_tool_call": true
}
]

179 changes: 179 additions & 0 deletions plugins/sana-copilot/evals/run-smoke.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
#!/usr/bin/env bash
# Local Cowork smoke for the sideloaded Sana Copilot plugin.
#
# Default: each case is a NEW conversation (no resume).
#
# ./run-smoke.sh
# ./run-smoke.sh --case connector-tools
#
# Optional: resume a UI-warmed conversation (avoids answering from empty tool
# catalog when CLI vault bind is broken):
#
# export COWORK_RESUME_CID='69574136-…:…:319470a7-…'
# ./run-smoke.sh --resume
#
# Options:
# --resume Use COWORK_RESUME_CID for every case (required if set this way)
# --case ID Run a single case id from cases.json
# --timeout N Per-turn seconds (default 300)
# --out DIR Results directory
set -euo pipefail

ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CASES="${ROOT}/cases.json"
OUT_DIR="${ROOT}/results/smoke-$(date -u +%Y%m%dT%H%M%SZ)"
TIMEOUT=300
ONLY_CASE=""
MODE="fresh"
CONTAINER_TYPE="${COWORK_CONTAINER_TYPE:-copilot}"

while [[ $# -gt 0 ]]; do
case "$1" in
\#*)
# zsh (default) does not treat "#" as a comment on interactive lines
break
;;
--fresh)
MODE="fresh"
shift
;;
--resume)
MODE="resume"
shift
;;
--case) ONLY_CASE="${2:?}"; shift 2 ;;
--timeout) TIMEOUT="${2:?}"; shift 2 ;;
--out) OUT_DIR="${2:?}"; shift 2 ;;
-h | --help)
awk 'NR==1{next} /^#/{sub(/^# ?/,""); print; next} {exit}' "$0"
exit 0
;;
*)
echo "Unknown argument: $1" >&2
exit 2
;;
esac
done

# Allow env to opt into resume without the flag.
if [[ "${MODE}" == "fresh" && -n "${COWORK_RESUME_CID:-}" && "${COWORK_FORCE_FRESH:-}" != "1" ]]; then
# Keep default fresh unless user asked for resume; ignore leftover env.
:
fi

if [[ "${MODE}" == "resume" && -z "${COWORK_RESUME_CID:-}" ]]; then
echo " --resume needs COWORK_RESUME_CID (UI-warmed conversation id)." >&2
echo " cowork conversations list --server" >&2
exit 2
fi

command -v cowork >/dev/null || {
echo "cowork CLI not on PATH" >&2
exit 2
}
command -v python3 >/dev/null || {
echo "python3 required" >&2
exit 2
}

mkdir -p "${OUT_DIR}"
echo "-> ${OUT_DIR}"
echo "mode: ${MODE}"
if [[ "${MODE}" == "resume" ]]; then
echo "resume: ${COWORK_RESUME_CID}"
fi

run_case() {
local id="$1" prompt="$2" mode="$3"
local json="${OUT_DIR}/${id}.${mode}.json"
local err="${OUT_DIR}/${id}.${mode}.err"
local args=(send --json --container-type "${CONTAINER_TYPE}" --timeout "${TIMEOUT}" --no-progress)
if [[ "${mode}" == "resume" ]]; then
args+=(--resume "${COWORK_RESUME_CID}")
fi
echo "==> ${id} (${mode})"
set +e
cowork "${args[@]}" "${prompt}" >"${json}" 2>"${err}"
set -e
python3 - "${json}" "${id}" "${CASES}" "${mode}" "${OUT_DIR}/results.jsonl" <<'PY'
import json, re, sys
from pathlib import Path

doc_path, case_id, cases_path, mode, results_path = sys.argv[1:6]
cases = {c["id"]: c for c in json.loads(Path(cases_path).read_text())}
case = cases[case_id]
doc = json.loads(Path(doc_path).read_text())
text = doc.get("text") or ""
trace = doc.get("tool_trace") or []
tool_names = [t.get("tool_name") or "" for t in trace if isinstance(t, dict)]
blob = json.dumps(doc)

failures = []
if doc.get("terminal_status") != "ok":
failures.append(f"terminal_status={doc.get('terminal_status')!r}")

needles = case.get("expect_tools_any") or []
if needles and not any(n in blob for n in needles):
failures.append(f"missing any of tool mentions: {needles}")

for pat in case.get("expect_response_patterns") or []:
if not re.search(pat, text):
failures.append(f"response pattern not matched: {pat}")

for pat in case.get("forbid_response_patterns") or []:
if re.search(pat, text):
failures.append(f"forbidden pattern matched: {pat}")

# On fresh sessions, ESS cases should hit Workday (no prior memory). Enforce
# a workday tool call whenever require_workday_tool_call is set, or when mode
# is fresh and the case opts in via require_workday_tool_call_on_fresh.
require_call = case.get("require_workday_tool_call")
if mode == "fresh" and case.get("require_workday_tool_call_on_fresh"):
require_call = True
if require_call:
wd = [
n
for n in tool_names
if "workday" in n.lower() or "sana-mcp" in n.lower() or "send_query" in n.lower()
]
if not wd:
failures.append(f"no workday tool in tool_trace={tool_names!r}")

ok = not failures
row = {
"id": case_id,
"mode": mode,
"ok": ok,
"failures": failures,
"terminal_status": doc.get("terminal_status"),
"tool_names": tool_names,
"text_preview": text[:400].replace("\n", " "),
}
with open(results_path, "a", encoding="utf-8") as f:
f.write(json.dumps(row) + "\n")
print("PASS" if ok else "FAIL", case_id, mode)
for fmsg in failures:
print(" -", fmsg)
if text:
print(" text:", text[:240].replace("\n", " "))
sys.exit(0 if ok else 1)
PY
return $?
}

PASS=0
FAIL=0
while IFS= read -r id; do
[[ -z "${id}" ]] && continue
[[ -n "${ONLY_CASE}" && "${id}" != "${ONLY_CASE}" ]] && continue
prompt="$(python3 -c 'import json,sys; print(next(c["prompt"] for c in json.load(open(sys.argv[1])) if c["id"]==sys.argv[2]))' "${CASES}" "${id}")"
if run_case "${id}" "${prompt}" "${MODE}"; then
PASS=$((PASS + 1))
else
FAIL=$((FAIL + 1))
fi
done < <(python3 -c 'import json,sys; print("\n".join(c["id"] for c in json.load(open(sys.argv[1]))))' "${CASES}")

echo
echo "summary: pass=${PASS} fail=${FAIL} mode=${MODE} out=${OUT_DIR}"
[[ "${FAIL}" -eq 0 ]]
7 changes: 7 additions & 0 deletions plugins/sana-copilot/evals/smoke.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
id segment utterance critical expected aspirational model
connector-tools connector List every tool name currently available that mentions Workday, Sana, or workday_plugin. Reply with the exact tool names only, one per line. If none, reply NONE. contains: send_query_to_workday_plugin || contains: get_workday_plugin_run_status || not_contains: NONE max_seconds: 120
ess-next-holiday ESS / holidays When is my next work holiday? contains_any: holiday|Veterans|Thanksgiving|Christmas|New Year || not_contains: connector isn't max_seconds: 300
ess-manager ESS / worker profile Who is my manager? not_contains: connector isn't contains_any: manager|report|Banks || max_seconds: 300
ess-july4 ESS / holidays Is July 4th a holiday? contains_any: July|holiday|Independence || not_contains: connector isn't max_seconds: 300
ess-vacation-balance ESS / time off What's my vacation balance? contains_any: hour|balance|PTO|vacation || not_contains: connector isn't max_seconds: 600
ess-live-refetch ESS / time off Call Workday again right now (do not reuse earlier answers in this chat). What is my current Paid Time Off balance in hours? tool_called: mcp__sana-mcp-connector__send_query_to_workday_plugin || contains_any: hour|PTO|balance not_contains: connector isn't || max_seconds: 600
Loading
Loading