[OMEGA-468] Carry the OmegaClaw memory and auth secret over on upgrade

[TossSky]

Description

An upgrade from v0.1.19 started with empty memory. The launcher mounts omega-memory, the v0.1.19 memory stays in omegaclaw-memory, and the old omegaclaw container keeps running next to the new one. A setup that starts the container itself and still passes OMEGACLAW_AUTH_SECRET ran with authorization off, because the proxy reads only OMEGA_AUTH_SECRET.

start now copies the old memory before it creates the container. This happens only when omegaclaw-memory exists and has not been copied before, and --memory-import is not set. An existing omega-memory is left alone, because the new version may already keep memory there. The launcher stops omegaclaw if it is running, fills omega-memory from the image and copies everything except prompt*.txt and tg_prompt.txt, which the new version ships itself.

Before it creates omega-memory, the launcher writes .migration-started into the old volume, and after the copy it writes .migrated-to-omega. Finding only the first marker means that the previous run was killed in the middle of the copy, so the launcher removes omega-memory and copies again. The second marker stays in the old volume, so clean followed by start gives an empty memory again. The old volume is mounted read-only for the copy and is never removed. If any step fails, the launcher removes omega-memory, starts omegaclaw again if it was running, and exits before docker run. It removes the start marker only once omega-memory is gone, so a volume it could not remove is copied again on the next start. If the markers cannot be read, start stops with an error. After a successful copy it warns that the copy is experimental. The installer goes through start, so it migrates as well.

proxy/nginx.sh falls back to OMEGACLAW_AUTH_SECRET when OMEGA_AUTH_SECRET is empty and logs that the old name is deprecated. When both are set, OMEGA_AUTH_SECRET wins. The paths to the template and to the generated config can be overridden through NGINX_TEMPLATE and NGINX_CONFIG, so the script runs in a test.

How Has This Been Tested?

• tests/test_omega_launcher_migration.py runs the launcher against a fake docker that keeps volumes as directories and runs the copy for real. It covers the copy itself, the prompts, the old container, clean followed by start, an existing omega-memory, --memory-import, a failed copy, a failed file copy, an image without the Omega memory layout, a copy interrupted before and after the image files reached omega-memory, the start marker staying out of omega-memory, a failed copy that could not remove omega-memory, and markers that cannot be read.
• tests/test_proxy_auth_secret.py runs proxy/nginx.sh with the real template and checks the generated config. The old name alone sets the secret and logs a warning, the new name wins over the old one, and without a secret the value stays empty. tests/pytest.sh gives 107 passed and 1 skipped.
• On Docker, starting from a running v0.1.19 installation with two facts in memory, both scripts/omega start and the installer copied history.metta and chroma_db unchanged, used the prompts of the image and stopped omegaclaw, and the agent recalled both facts through query. clean followed by start gave an empty memory.
• On Docker, a start killed in the middle of the copy was redone by the next start, and every copied file matched its original. With omega-memory limited to a 64 KB tmpfs, the launcher exited with code 1, removed omega-memory and the start marker, and started omegaclaw again.
• With only OMEGACLAW_AUTH_SECRET=4242 passed to the image, the new nginx.sh gives {"enabled":true} on /auth/status and rejects a wrong token. The current one gives {"enabled":false} and accepts any token.

Checklist

• PR contains autogenerated code
• Self-review completed
• Test scenarios above are passed with the version of the code from PR

[paul-v-snet]

Looks good to me, but I'm concerned about what will happen if migration process crashes for some reason?

The script only check whether (excluding memory import file):

1. omega-memory doesn't exist OR
2. omegalcaw-memory doesn't exist OR
3. omegaclaw-memory doesn't contain the marker

And check 3 runs after check 1, so if the migration process was already initiated and crashed for some reason, then omega-memory was created and partially filled with data, but the migration was not completed. In this case the check 1 will not pass and migration process will not be initiated, so this could lead to knowledge loss.

So I'd like to ask: should we address this, or should we just hope this never happens?

cc: @alyona-snet

[alyona-snet]

@kabirkbr Could you please review Paul's comment and provide your opinion?

[kabirkbr]

Thanks @TossSky, this is what I had in mind: the manual upgrade steps from the release notes, done by the launcher. Keeping the old volume and starting omegaclaw again on failure are good choices.

re @paul-v-snet concern -- as i understand it, if the run is killed half way (Ctrl-C, a reboot, the Docker daemon restarting), omega-memory already exists and the marker in omegaclaw-memory is not written yet. The next start sees omega-memory and skips the migration. The agent then starts with part of the old memory, or with none if the stop came before the copy began, and nobody is told. The old data is still safe in omegaclaw-memory, but getting it back is manual.

We cannot just drop the "omega-memory exists" check: it protects people who already run v0.1.20 and have stored new memories there. A marker inside omega-memory does not work either: Docker fills a volume from the image only while it is empty, so a marker written first stops the fill.

Proposal: mark the start of the migration in the old volume, next to the marker you already write at the end. On the next start, the old volume is checked first:

• .migrated-to-omega → done, skip (as now).
• only .migration-started → the last run did not finish: remove omega-memory and copy again.
• neither → an existing omega-memory means "skip" (as now).

On a handled failure the start marker is removed again, so the old volume stays exactly as it was and your existing failure tests still hold.

The code below (as well as the idea of the proposal :). was generated by my AI assistant. It is a proposal, not a finished change: please review it and take it, adapt it or do it your own way.

scripts/omega:

--- a/scripts/omega
+++ b/scripts/omega
@@ -752,17 +752,35 @@
     local new_volume="omega-memory"
     local memory_path="/PeTTa/repos/Omega/memory"
     local marker=".migrated-to-omega"
+    local started_marker=".migration-started"
     local stopped_old_container=0
+    local old_state
 
     if [ -n "${memory_import_file}" ] \
-        || docker volume inspect "${new_volume}" >/dev/null 2>&1 \
         || ! docker volume inspect "${old_volume}" >/dev/null 2>&1; then
         return 0
     fi
-    if docker run --rm --entrypoint sh --volume "${old_volume}:/from:ro" "${image}" \
-        -c "test -e /from/${marker}"; then
-        return 0
-    fi
+    old_state=$(docker run --rm --entrypoint sh --volume "${old_volume}:/from:ro" "${image}" \
+        -c "if [ -e /from/${marker} ]; then echo migrated; elif [ -e /from/${started_marker} ]; then echo interrupted; fi") \
+        || old_state=""
+    case "${old_state}" in
+        migrated)
+            return 0
+            ;;
+        interrupted)
+            echo "The last copy from ${old_volume} did not finish, copying again"
+            if docker volume inspect "${new_volume}" >/dev/null 2>&1 \
+                && ! docker volume rm "${new_volume}" >/dev/null; then
+                echo "Could not remove the partly copied volume ${new_volume}" >&2
+                return 1
+            fi
+            ;;
+        *)
+            if docker volume inspect "${new_volume}" >/dev/null 2>&1; then
+                return 0
+            fi
+            ;;
+    esac
 
     echo "Copying memory from volume ${old_volume} to ${new_volume}"
     if [ "$(docker inspect -f '{{.State.Running}}' "${old_container}" 2>/dev/null)" = "true" ]; then
@@ -774,7 +792,9 @@
         echo "Stopped container ${old_container}"
     fi
 
-    if docker volume create "${new_volume}" >/dev/null \
+    if docker run --rm --entrypoint sh --volume "${old_volume}:/from" "${image}" \
+            -c "date -u > /from/${started_marker}" \
+        && docker volume create "${new_volume}" >/dev/null \
         && docker run --rm --entrypoint sh --volume "${new_volume}:${memory_path}" "${image}" \
             -c "test -f ${memory_path}/prompt.txt" \
         && docker run --rm --entrypoint sh \
@@ -788,6 +808,8 @@
 
     echo "Could not copy memory from ${old_volume}, the old installation is left as it was" >&2
     docker volume rm "${new_volume}" >/dev/null 2>&1 || true
+    docker run --rm --entrypoint sh --volume "${old_volume}:/from" "${image}" \
+        -c "rm -f /from/${started_marker}" >/dev/null 2>&1 || true
     if [ "${stopped_old_container}" = "1" ]; then
         docker start "${old_container}" >/dev/null 2>&1 || true
     fi

Two tests for it, using your fake docker (tests/test_omega_launcher_migration.py):

--- a/tests/test_omega_launcher_migration.py
+++ b/tests/test_omega_launcher_migration.py
@@ -338,3 +338,37 @@
     assert _container_state(docker_root, "omegaclaw") == "running"
     assert _snapshot(old) == before
     assert not _started_agent(docker_root)
+
+
+def _interrupted_migration(root, image_files):
+    old = _install_omegaclaw(root, running=False)
+    (old / ".migration-started").write_text("")
+    new = root / "volumes" / "omega-memory"
+    new.mkdir()
+    if image_files:
+        shutil.copytree(root / "image-memory", new, dirs_exist_ok=True)
+    return old, new
+
+
+def test_interrupted_copy_is_redone(docker_root):
+    old, new = _interrupted_migration(docker_root, image_files=True)
+
+    result = _launcher(docker_root, "start", "-d", IMAGE)
+
+    assert result.returncode == 0, result.stderr
+    assert _read(new / "history.metta") == "(old history)\n"
+    assert _read(new / "chroma_db" / "chroma.sqlite3") == "old long-term memory"
+    assert _read(new / "prompt.txt") == "omega prompt\n"
+    assert (old / ".migrated-to-omega").exists()
+    assert _started_agent(docker_root)
+
+
+def test_run_interrupted_before_the_image_files_is_redone(docker_root):
+    old, new = _interrupted_migration(docker_root, image_files=False)
+
+    result = _launcher(docker_root, "start", "-d", IMAGE)
+
+    assert result.returncode == 0, result.stderr
+    assert _read(new / "history.metta") == "(old history)\n"
+    assert _read(new / "prompt.txt") == "omega prompt\n"
+    assert _started_agent(docker_root)

On the current PR head, both new tests fail (the migration is skipped). With the proposed change, all 14 tests in the file pass, run against the fake docker only.

With a fix for this in, in whatever form, I am fine for it to go to testing.

One more point for the release: it may be a good idea if the launcher say clearly that carrying memory over on upgrade is experimental, and that the old omegaclaw-memory volume should be kept until the user has checked that the agent remembers what it knew.

[TossSky]

Fixed it, see e30dc0a. I took the proposed patch and both tests as they were. I also left the start marker out of the copy and made the message after a successful copy say that the copy is experimental. Now a start killed in the middle of the copy is redone by the next start.
@kabirkbr @paul-v-snet

[paul-v-snet]

If docker volume rm on line 810 fails, line 811 will still remove the migration marker. This can recreate the previously reported issue: the incomplete omega-memory volume remains, but on the next start it may be treated as a fully migrated volume.

I suggest explicitly checking that the volume has been removed successfully before removing the migration marker.

[TossSky]

Fixed it, see 60e922f. The start marker is now removed only when omega-memory is gone, so if the volume cannot be removed, the next start copies the memory again. The test covers it.

[paul-v-snet]

If this docker run fails for any reason, old_state is silently set to an empty string, which is then treated the same as "no migration markers found".
If omega-memory already exists after an interrupted migration, this can cause the script to skip migration and start the agent with an incomplete volume.
I suggest failing here instead of treating this case as an empty state.

[TossSky]

Fixed it, see 60e922f. If the markers cannot be read, start now stops with an error. The test covers it.

[paul-v-snet]

@TossSky, you need to resolve the conflicts with the main branch. I'll reapprove it after that's done.

[TossSky]

@paul-v-snet conflicts were resolved. Can you re-approve please.