Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions evals/RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,3 +128,73 @@ native, the not-in-build native and the undeclared permission every time, with l

Not changed: `open77_data` has no enum for weather presets (they live in the guide prose), and
the reason lists of `setTime`/`setWeather` differ between card and guide section.

## Round 3, 2026-09-16 20:00–21:40: two agents on 0.1.1, then a probe on the server

Two subagents, `@open2077/mcp@0.1.1` only (`harness/mcp-call.mjs`), one task each; the
resources are in `round3/`. Both validated clean on the first pass and ran on the private stock
release-75 server through the MCP's live tools (`refresh`, `ensure`, `announce`, `motd`, `lock`,
`whosin`; console refusals correct, no runtime error). What they reported wrong or ambiguous was
then **measured** with `round3/probe-op77.75.lua` (a resource that prints what each call answers
at chunk top level, in `onResourceStart` and from a command) before anything was rewritten.

| Resource | Task | MCP calls | `open77_validate` | Live |
|---|---|---|---|---|
| `eval_announce` | admin `/announce` toast + chat, `/motd`, suggestions on `chat:ready` | 44 | OK first try | started; `announce` reached toast + chat, `motd` printed |
| `eval_carlock` | `/lock` `/whosin` on the caller's vehicle, `onPlayerLeftVehicle` | 40 | OK first try | started; console refusals correct |

**Measured on op77.75** (each line is the probe's output, not a reading of the source):

- `Open77.chat.broadcast / addSuggestions / send` at chunk top level → `false, resource_preparing`;
`Open77.notifications.broadcast` and `TriggerClientEvent` at top level → **work**. The
0.1.1 cards said the opposite for notifications.
- `Open77.chat.send("1", …)` → `false, invalid_chat_target`: a string id is refused, and every
host event (`onPlayerReady`, `onPlayerDisconnected`, …) delivers string ids. The identity
guide's own `onPlayerReady` example did exactly that.
- `os`, `io`, `debug`, `package`, `require`, `load`, `loadfile`, `dofile`, `collectgarbage` →
nil; `math`, `string`, `table`, `utf8`, `coroutine`, `json` present (`math.type` too).
- `Open77.vehicles.flags.paintApplied` = 512; `occupantInSeat(unknown, "driver")` →
`nil, "vehicle_not_found"`; `getPlayerSeat(unknown)` → a single `nil`.
- `RegisterCommand` handler: `source` is a number (0 = console), `args` is `table.pack` of
**string** tokens (`args.n`), `raw` is `name arg arg` without the `/`; the console splits on
spaces only (quotes stay literal), chat splits shell-style.

**What changed because of it** (base #25, app #4, devkit 0.1.2):

- The constant tables (`Open77.vehicles.seats/flags/…`) read "NOT AVAILABLE on op77.75
(unreleased)" while every guide used them: the release surface never listed a table of
scalars. They carry `since op77.45` now, and `open77_validate` checks a dotted reference that
is not a call (`Open77.vehicles.flags.locked`, `local send = Open77.chat.send`) the same way
it checks a call — both agents had guarded the tables with `pcall`.
- `TriggerEvent`, `Open77.chat.*`, `Open77.events.emit*` listed none of the bus refusals
(`resource_preparing`, `event_queue_limit`, `invalid_event_name`, …); the readers follow the
`EventFailure(...)` helpers, tuple returns, the array-registered bus doors, the
`local eventTrigger = __open77_trigger_event` aliases, and a member that returns another
member. `Open77.chat.broadcast` (which "lists no reasons while claiming to be exactly
send(-1)") now carries `send`'s.
- `open77_events` showed payloads for 4 of 92 lifecycle events, one of them wrong
(`onResourceStart (resourceName, revision)` was the neighbouring bus row's). 76 of 92 carry a
payload; a table row feeds only the events in its first cell, prose signatures and handler
parameter lists fill the rest. `chat:ready` is documented (client net event, no arguments,
`source` = the player).
- `open77_validate` accepted `os.time()` and `require`; it names each absent library and
function with the replacement (`Open77.time.unix()`, more `server_script` lines), and on the
client side `setmetatable` / `coroutine.create`. It warned that the notifications guide's
`dependency "open77_notifications >=1.0.0"` "is not a resource slug": the runtime grammar
(`name`, then `>=`/`<=`/`>`/`<`/`=`/`==` constraints) is parsed, and a malformed constraint is
an error.
- `open77_search "register command admin only restricted"` ranked the client `RegisterCommand`
above the server one; a side named in the question is now a filter, and the tool says so.
- `open77_changes to=main` lists what exists on main and in no published build; a card prints
"reasons: none found in the handler" instead of omitting the line, and a description equal to
the summary is not printed twice (`GetCurrentResourceName`, now `shared`).
- `server-api#resource-lifecycle-events` returned 40 lines of cyberware/dash/reflex prose: the
guide filed five feature summaries and the namespaced-equivalents table under that heading.
They have their own headings, and the guide has a "What the sandbox provides" section.
- The manifest schema lacked `exports` / `server_exports` (A20), which made
`extract-schemas.py` refuse to run.

Not changed: whether a server toast sent at `onPlayerReady` is queued or lost while the client
WebUI is still coming up (the agent waited 3 s by analogy with the identity guide); whether
`tostring(integer id)` equals the event's string id is asserted in the seat card from the host's
`ToString()` and not measured with a real vehicle.
14 changes: 14 additions & 0 deletions evals/round3/eval_announce/open77.lua
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
resource "eval_announce"
version "1.0.0"
open77_version "*"
auto_start true

-- Open77.notifications.send / broadcast and RegisterNetEvent all require network.events
-- (per their native cards). Open77.chat.* and RegisterCommand check no permission.
permissions { "network.events" }

-- The server notification natives route to the official open77_notifications client
-- package; the notifications guide declares it as a dependency.
dependency "open77_notifications"

server_script "server/main.lua"
226 changes: 226 additions & 0 deletions evals/round3/eval_announce/server/main.lua
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
-- eval_announce / server/main.lua
--
-- Server-only resource.
-- /announce <text...> admin-only (RegisterCommand restricted = true, so the caller must hold
-- the ACL right `command.announce`; the dedicated console is always allowed).
-- Shows the text to every player as a toast and as a chat line.
-- /motd any player: the current message of the day as a private toast.
--
-- Anything that reaches clients (notifications, chat suggestions) is refused with
-- `resource_preparing` while the chunk is loading, so nothing below runs at top level:
-- everything is inside onResourceStart, a command, or an event handler.

local RESOURCE_NAME = "eval_announce"

local MAX_MESSAGE_BYTES = 384 -- notifications definition reference: message max 384 UTF-8 bytes
local MAX_TITLE_BYTES = 96 -- title max 96 UTF-8 bytes
local MOTD_DURATION_MS = 8000 -- timed values are 750-120000
local ANNOUNCE_DURATION_MS = 10000
local MOTD_DELAY_MS = 3000 -- onPlayerReady is a barrier lifting, not "world is up"

local motd = nil

-- Every reason the natives document, mapped to a readable sentence. Unknown reasons still
-- produce a message: nothing here may crash on an unexpected string or on nil.
local REASON_TEXT = {
-- Open77.notifications.send / Open77.notifications.broadcast
["network_unavailable"] = "the network transport is unavailable",
["permission_denied:network.events"] = "the manifest does not declare network.events",
["duplicate_notification_id"] = "a notification with that id is already live for this resource",
["resource_preparing"] = "the resource chunk is still loading, clients cannot be reached yet",
-- Open77.chat.send / broadcast / addSuggestion / addSuggestions
["invalid_chat_message"] = "the chat message is malformed",
["invalid_chat_target"] = "the chat target is not a connected player id or -1",
["invalid_chat_command"] = "the suggested command text is invalid",
["invalid_chat_suggestion"] = "the suggestion descriptor is malformed",
}

local function describe(reason)
if reason == nil then
return "no reason given"
end
return REASON_TEXT[reason] or ("unexpected reason '" .. tostring(reason) .. "'")
end

local function log(fmt, ...)
print(("[%s] " .. fmt):format(RESOURCE_NAME, ...))
end

local function truncateBytes(text, maxBytes)
text = tostring(text or "")
if #text <= maxBytes then
return text
end
return text:sub(1, maxBytes - 3) .. "..."
end

-- source is the player id, or 0 for the dedicated console.
local function isConsole(source)
return source == nil or tonumber(source) == 0
end

-- Answer the caller: chat for a player, the log for the console. Never assumes the chat
-- facade succeeded.
local function reply(source, text)
if isConsole(source) then
log("%s", text)
return
end
local ok, reason = Open77.chat.send(source, { type = "system", author = RESOURCE_NAME, text = text })
if not ok then
log("reply to player %s failed: %s", tostring(source), describe(reason))
end
end

-- Chat autocomplete entries for both commands.
local SUGGESTIONS = {
{
command = "/announce",
help = "Admin: show a message to every player (toast + chat)",
parameters = { { name = "text", help = "The announcement text" } },
},
{
command = "/motd",
help = "Show the message of the day",
},
}

local function publishSuggestions(target)
local ok, reason = Open77.chat.addSuggestions(target, SUGGESTIONS)
if not ok then
log("could not publish chat suggestions to %s: %s", tostring(target), describe(reason))
return false
end
return true
end

-- Private toast with the current message of the day. Returns true on success, or
-- false plus a sentence explaining why.
local function sendMotd(playerId)
if motd == nil then
return false, "no message of the day is set"
end
local id, reason = Open77.notifications.send(playerId, {
type = "info",
title = "Message of the day",
message = motd,
icon = "MOTD",
position = "top_center",
durationMs = MOTD_DURATION_MS,
})
if id == nil then
return false, "motd toast failed: " .. describe(reason)
end
return true
end

-- Resource start: set the message of the day, and advertise the commands to everyone
-- already connected (their chat:ready fired before this resource existed).
AddEventHandler("onResourceStart", function(name)
if name ~= GetCurrentResourceName() then
return
end
motd = truncateBytes("Welcome to the server. Be nice, have fun, and type /motd to read this again.", MAX_MESSAGE_BYTES)
log("message of the day set: %s", motd)
publishSuggestions(-1)
end)

-- A player's chat UI came up: publish this resource's suggestions to that player.
-- `source` is the authenticated connection (RegisterNetEvent). A RegisterNetEvent handler also
-- receives host-wide bus events, where source is not set, so validate it before use.
RegisterNetEvent("chat:ready", function(...)
local target = source
if isConsole(target) then
local first = ...
if first ~= nil then
target = first
else
log("chat:ready without a usable source; suggestions not published")
return
end
end
publishSuggestions(target)
end)

-- The join-time readiness gate lifted for a player: show them the message of the day.
-- playerId arrives as a string, which Open77.notifications.send accepts.
AddEventHandler("onPlayerReady", function(playerId, detail)
if playerId == nil then
log("onPlayerReady without a player id (detail=%s); motd skipped", tostring(detail))
return
end
CreateThread(function()
Wait(MOTD_DELAY_MS)
local ok, why = sendMotd(playerId)
if not ok then
log("motd for player %s not shown: %s", tostring(playerId), tostring(why))
end
end)
end)

-- /announce <text...> -- restricted: ACL right command.announce (console always allowed).
RegisterCommand("announce", function(source, args, raw)
local words = {}
for _, value in ipairs(args or {}) do
words[#words + 1] = tostring(value)
end
local text = table.concat(words, " ")
text = text:match("^%s*(.-)%s*$") or ""
if text == "" then
reply(source, "Usage: /announce <text>")
return
end

local body = truncateBytes(text, MAX_MESSAGE_BYTES)
local who = "SERVER"
if not isConsole(source) then
who = Open77.players.name(source) or ("player " .. tostring(source))
end

local problems = {}

-- 1. toast to everyone
local id, reason = Open77.notifications.broadcast({
type = "warning",
title = truncateBytes("Announcement from " .. who, MAX_TITLE_BYTES),
message = body,
icon = "!",
position = "top_center",
durationMs = ANNOUNCE_DURATION_MS,
})
if id == nil then
problems[#problems + 1] = "toast: " .. describe(reason)
end

-- 2. chat line to everyone (system message, positional RGB colour)
local ok, chatReason = Open77.chat.broadcast({
type = "system",
author = "ANNOUNCE",
text = ("[%s] %s"):format(who, body),
color = { 255, 196, 0 },
})
if not ok then
problems[#problems + 1] = "chat: " .. describe(chatReason)
end

if #problems == 0 then
log("announcement by %s: %s", who, body)
reply(source, "Announcement sent.")
else
local summary = table.concat(problems, "; ")
log("announcement by %s partially failed (%s): %s", who, summary, body)
reply(source, "Announcement problems: " .. summary)
end
end, true)

-- /motd -- any player; the console gets the text in its log.
RegisterCommand("motd", function(source, args, raw)
if isConsole(source) then
log("message of the day: %s", motd or "(none set)")
return
end
local ok, why = sendMotd(source)
if not ok then
reply(source, "Could not show the message of the day: " .. tostring(why))
end
end, false)
11 changes: 11 additions & 0 deletions evals/round3/eval_carlock/open77.lua
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
resource "eval_carlock"
version "0.1.0"
auto_start true

-- Every native this resource calls that is gated by a permission
-- (getPlayerSeat, get, setLocked, occupantInSeat) requires world.vehicles.
-- seatName, isLocked, players.name, chat.send, RegisterCommand and
-- AddEventHandler check no permission.
permissions { "world.vehicles" }

server_script "server/main.lua"
Loading
Loading