perf: move remember login tokens out of oc_preferences

[cristianscheid]

• Resolves: Move remember me token out of oc_preferences #61728

Summary

Currently, when logging in with "Remember me" selected, a token is stored in oc_preferences table:

MariaDB [nextcloud]> SELECT * FROM oc_preferences WHERE appId="login_token";
+--------+-------------+----------------------------------+-------------+------+------+-------+---------+
| userid | appid       | configkey                        | configvalue | lazy | type | flags | indexed |
+--------+-------------+----------------------------------+-------------+------+------+-------+---------+
| john   | login_token | eqqbZqE6U1t7GsYyd/hCNtMQdGXDq1k0 | 1789069718  |    0 |    0 |     0 |         |
+--------+-------------+----------------------------------+-------------+------+------+-------+---------+
1 row in set (0.001 sec)

Changes introduced by this PR:

• new dedicated table for this purporse: oc_remember_login_tokens

MariaDB [nextcloud]> SELECT * FROM oc_remember_login_tokens;
+----+------+----------------------+------------+
| id | uid  | token                | created    |
+----+------+----------------------+------------+
|  1 | john | <hashed-token-here>  | 1789070212 |
+----+------+----------------------+------------+
1 row in set (0.000 sec)

• mechanism to migrate old remember login tokens from oc_preferences to oc_remember_login_tokens
  • first look for token on oc_remember_login_tokens, if not there, look at oc_preferences
  • if found in oc_preferences, insert token on oc_remember_login_tokens and remove from oc_preferences
  • old tokens from oc_preferences should all be eventually removed either by mechanism above or by OC\User\BackgroundJobs\CleanupLoginTokens
  • OC\User\BackgroundJobs\CleanupLoginTokens cleans stale tokens, by default the ones created > 15 days

Note 1

While brainstorming how to implement this, one suggested approach was to use OCP\Security\ICredentialsManager to store the tokens.

• credentials are stored in oc_storages_credentials (columns: id, user, identifier, credentials)
• credentials column value is encrypted like $this->crypto->encrypt(json_encode($credentials))
• to store the timestamp, as is currently done in oc_preferences, the token would need to be set as the identifier (to be able to search by token), with credentials holding the timestamp:

$this->credentialsManager->store($userId, $token, $timestamp);

• this would store the token in plain text. If we wanted to hash it instead (e.g. hash('sha512', $token)), the 128-character string would be too large for the column, since identifier is defined as:

$table->addColumn('identifier', Types::STRING, [
    'notnull' => true,
    'length' => 64,
]);

• even without hash, there's still OC\User\BackgroundJobs\CleanupLoginTokens, which removes all login tokens older than a certain threshold. Since credentials column value is encrypted before being stored, cleanup would require decrypting the timestamp for each one just to determine whether an entry is stale

For these reasons, I went with a dedicated table instead, since we can store the hashed value of the token, and easily delete stale records with a direct query, since the timestamp itself is not hashed.

Note 2

After this get merged in master, existing dev instances will need to apply new migration to avoid table not found error:

# to run pending migrations for core
occ migrations:migrate core

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)
• Labels added where applicable (ex: bug/enhancement, 3. to review, feature component)
• Milestone added for target branch/version (ex: 32.x for stable32)

AI (if applicable)

• The content of this PR was partly or fully generated using AI

[CarlSchwan]

After this get merged in master, existing dev instances will need to apply new migration to avoid table not found error:

Bump the version in version.php

[come-nc]

I think it would make sense to update the entity when rotating the token, rather than deleting+inserting.
I’m also wondering whether we could use snowflake ids instead of numeric id + created at. The snowflake contains the creation timestamp.

[come-nc]

Also it would be better to switch to the new entity system.
See #63288 for an example.

[cristianscheid]

I think it would make sense to update the entity when rotating the token, rather than deleting+inserting. I’m also wondering whether we could use snowflake ids instead of numeric id + created at. The snowflake contains the creation timestamp.

I think we cannot have both things together:

• if we rotate the token by updating the entry, same snowflake ID would be kept, meaning timestamp would still hold creation time
• CleanupLoginTokens cron job would eventually cleanup this entry from DB, even if it was updated, since timestamp is still from creation time

So I think we have three options:

option 1

• use snowflake ID and use it's own timestamp to check when token was generated instead of created column
• delete + insert entry when rotating token, to generate correct timestamp

option 2

• use regular numeric ID + dedicated timestamp column
• update entry instead of delete + insert when rotating token

option 3

• use snowflake ID + dedicated timestamp column
• update entry instead of delete + insert when rotating token

cc @come-nc

[come-nc]

@cristianscheid Is it not possible to update the id as well?

[cristianscheid]

Bump the version in version.php

@CarlSchwan Thanks for the tip! Should the commit below be enough?
commit: d7aa083

[cristianscheid]

@cristianscheid Is it not possible to update the id as well?

@come-nc I did not consider this approach at first since EntityManager::update() does not allow updating ID, but looking at codebase again I think we can follow a similar approach to AccessTokenMapper::rotateToken(), which does a more direct update using query builder instead of EntityManager::update().

We could do something similar to update both the token and the snowflake ID, refreshing it's timestamp. I implemented this on this commit: 7eade0a

There's one caveat with this approach: when migrating from oc_preferences, the old token's creation timestamp gets reset, since the row is recreated in the new table with snowflake id, encoding current timestamp instead of the original one:
SnowflakeGenerator::nextId()

edit: caveat above (strikethrough text) isn't actually a problem. Before this change, token was already regenerated at that same point in loginWithCookie(), which produces a fresh timestamp.

[github-actions]

Possible performance regression detected

Show Output

7 queries added

= /remote.php/dav/files/test
= /remote.php/dav/files/test/test.txt
= /remote.php/dav/files/test/many_files
= /remote.php/dav/files/test/new_file.txt
≠ /remote.php/dav/files/test/new_file.txt with 7 queries added
  - UPDATE "oc_filecache" SET "size" = :dcValue4 WHERE ("fileid" = :dcValue1) AND ("storage" = :dcValue2) AND (("size" <> :dcValue3) OR ("size" IS NULL))
  - UPDATE "oc_filecache" SET "size" = :dcValue4 WHERE ("fileid" = :dcValue1) AND ("storage" = :dcValue2) AND (("size" <> :dcValue3) OR ("size" IS NULL))
  - SELECT "path" FROM "oc_filecache" WHERE ("storage" = :dcValue1) AND ("fileid" = :dcValue2)
  - SELECT "path" FROM "oc_filecache" WHERE ("storage" = :dcValue1) AND ("fileid" = :dcValue2)
  + SELECT "storage", "path", "mimetype" FROM "oc_filecache" WHERE "fileid" = :dcValue1
  + SELECT "storage_id", "root_id", "user_id", "mount_point", "mount_id", "f"."path", "mount_provider_class" FROM "oc_mounts" "m" INNER JOIN "oc_filecache" "f" ON "m"."root_id" = "f"."fileid" WHERE ("m"."storage_id" = :dcValue1) AND ("f"."storage" = :dcValue2) AND (("f"."fileid" = :dcValue3) OR ("f"."path" = '') OR (("f"."path" || :dcValue4) = SUBSTR(:dcValue5, :dcValue6, LENGTH("f"."path") + :dcValue7))) AND ("user_id" = :dcValue8)
  + SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."parent", "filecache"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "filecache"."etag", "filecache"."permissions", "checksum", "unencrypted_size", "metadata_etag", "creation_time", "upload_time", "meta"."json" AS "meta_json", "meta"."sync_token" AS "meta_sync_token" FROM "oc_filecache" "filecache" LEFT JOIN "oc_filecache_extended" "fe" ON "filecache"."fileid" = "fe"."fileid" LEFT JOIN "oc_files_metadata" "meta" ON "filecache"."fileid" = "meta"."file_id" WHERE ("filecache"."fileid" = :dcValue1) AND ("storage" = :dcValue2)
  + DELETE FROM "oc_filecache" WHERE ("storage" = :dcValue1) AND ("fileid" = :dcValue2)
  + DELETE FROM "oc_filecache_extended" WHERE "fileid" = :dcValue1
  + SELECT "file_id", "json", "sync_token" FROM "oc_files_metadata" WHERE "file_id" IN (:dcValue1)
  + DELETE FROM "oc_files_metadata" WHERE "file_id" IN (:dcValue1)
  + DELETE FROM "oc_files_metadata_index" WHERE "file_id" IN (:dcValue1)
  + SELECT "fileid" FROM "oc_filecache" WHERE ("storage" = :dcValue1) AND ("path_hash" = :dcValue2)
  + UPDATE "oc_file_locks" SET "lock" = :dcValue1, "ttl" = :dcValue2 WHERE ("key" = :dcValue3) AND ("lock" = :dcValue4)
  + SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."parent", "filecache"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "filecache"."etag", "filecache"."permissions", "checksum", "unencrypted_size", "metadata_etag", "creation_time", "upload_time", "meta"."json" AS "meta_json", "meta"."sync_token" AS "meta_sync_token" FROM "oc_filecache" "filecache" LEFT JOIN "oc_filecache_extended" "fe" ON "filecache"."fileid" = "fe"."fileid" LEFT JOIN "oc_files_metadata" "meta" ON "filecache"."fileid" = "meta"."file_id" WHERE ("path_hash" = :dcValue1) AND ("storage" = :dcValue2)
= ./occ profiler:list

[cristianscheid]

Possible performance regression detected

Show Output

This is unrelated to this PR changes. Comment was triggered by failed "Performance testing" CI action, after re-run, job completed sucessfully.

[icewind1991]

I believe this is only needed if you're editing an existing table

[cristianscheid]

isn't this supposed to be used as metadata? saw the same pattern being used in other migrations where new tables are created, like:

https://github.com/nextcloud/server/blob/master/core/Migrations/Version34000Date20260521110333.php
https://github.com/nextcloud/server/blob/master/core/Migrations/Version30000Date20240429122720.php

https://docs.nextcloud.com/server/stable/developer_manual/basics/storage/migrations.html#migrations-and-metadata

[icewind1991]

The example in the docs show it mentioning the modified column on another table than the one it's creating, and nothing about the columns/indexes of the new table.