diff --git a/changelog.d/2-features/WPB-28665-user-search-backend-choice b/changelog.d/2-features/WPB-28665-user-search-backend-choice new file mode 100644 index 00000000000..b7f2b8cb9a1 --- /dev/null +++ b/changelog.d/2-features/WPB-28665-user-search-backend-choice @@ -0,0 +1 @@ +brig user search can now be served from PostgreSQL (`searchBackend: postgres`); ElasticSearch remains the default backend and is deprecated. diff --git a/docs/src/developer/reference/config-options.md b/docs/src/developer/reference/config-options.md index 062f891f03c..f774847bce6 100644 --- a/docs/src/developer/reference/config-options.md +++ b/docs/src/developer/reference/config-options.md @@ -1875,6 +1875,28 @@ accessible to services (and not the private key.) The corresponding Cassandra options are described in Cassandra’s documentation: [client_encryption_options](https://cassandra.apache.org/doc/stable/cassandra/configuration/cass_yaml_file.html#client_encryption_options) +## Configure brig's user search backend + +brig's user search can be served from two backends, selected with the +`searchBackend` option in brig's config: + +- `elasticsearch` (the default): users are indexed into ElasticSearch and + search is served from there. This backend is **deprecated** and will be + removed in a future release. +- `postgres`: search is served directly from brig's PostgreSQL user store and + brig stops writing to the ElasticSearch index (so the index becomes stale). + +```yaml +brig: + config: + searchBackend: postgres +``` + +The `elasticsearch` config block remains required, but it is ignored when +`searchBackend: postgres`. See +[the migration notes](../../how-to/install/infrastructure-configuration.md) +for how to migrate an existing installation. + ## Configure Elasticsearch basic authentication When the Wire backend is configured to work against a custom Elasticsearch diff --git a/docs/src/how-to/install/infrastructure-configuration.md b/docs/src/how-to/install/infrastructure-configuration.md index 34e1eb2c19d..7e717798305 100644 --- a/docs/src/how-to/install/infrastructure-configuration.md +++ b/docs/src/how-to/install/infrastructure-configuration.md @@ -407,6 +407,43 @@ adding this information for your cloud provider, feel free to read the creating such automation, feel free to read the [contributing guidelines](https://github.com/wireapp/wire-server-deploy/blob/master/CONTRIBUTING.md) and open a PR. +## ElasticSearch user-search backend deprecation + +The ElasticSearch-based user search backend of brig is **deprecated** and will +be removed in a future release. brig can now serve user search directly from +its PostgreSQL user store by setting `searchBackend: postgres` in brig's +config; ElasticSearch remains the default (`searchBackend: elasticsearch`). + +To migrate an existing installation: + +1. Upgrade wire-server: the database migration + `20260911000000-user-search-postgres.sql` adds the `wire_user.name_normalized` + column (with indexes) and the `team_search_visibility` table. +2. Run the one-off backfill (this only writes to PostgreSQL, the + ElasticSearch index keeps serving search in the meantime): + + ```bash + brig-index backfill-normalized-names --pg-settings "host=... dbname=... user=... password=..." + ``` + + Note: brig itself maintains `name_normalized` on every user creation and + name change, so the one-off backfill only needs to cover users that + existed before this upgrade. +3. Set `searchBackend: postgres` in brig's config. brig will stop writing + to the ElasticSearch index. +4. After verifying that search works as expected, ElasticSearch can be + decommissioned. + + Note: the `team_search_visibility` table starts out empty. Teams that + currently have the inbound search-visibility feature enabled should + toggle it off and on again after cutover; otherwise inbound search + silently falls back to own-team-only for those teams. + +To roll back, set `searchBackend: elasticsearch` again: the index is only kept +fresh while brig runs with the ElasticSearch backend, so user changes made +while running with `searchBackend: postgres` will only appear in the index +after a re-index (e.g. `brig-index reindex`). + ## Persistence and high-availability Currently, due to the way kubernetes and cassandra diff --git a/libs/wire-subsystems/postgres-migrations/20260911000000-user-search-postgres.sql b/libs/wire-subsystems/postgres-migrations/20260911000000-user-search-postgres.sql new file mode 100644 index 00000000000..ed1099c20ce --- /dev/null +++ b/libs/wire-subsystems/postgres-migrations/20260911000000-user-search-postgres.sql @@ -0,0 +1,24 @@ +-- Stores the ICU-transliterated, lowercased display name +-- (see 'Wire.UserSearch.Normalize.normalized'). Required for +-- case/diacritic-insensitive prefix search of user names, which plain +-- lower() cannot provide ("Björn" -> "bjorn"). +ALTER TABLE wire_user ADD COLUMN name_normalized text; + +-- The CREATE INDEX CONCURRENTLY statements for this migration live in +-- 2026091100000{1,2,3}-user-search-*.sql: they cannot run inside a +-- transaction and are registered in 'Wire.PostgresMigrations.nonTransactionMigrations' +-- (one statement per file: statements within a single script run in one +-- implicit transaction, which CREATE INDEX CONCURRENTLY forbids). + +-- Replaces the per-user search_visibility_inbound field formerly +-- denormalized into the ElasticSearch user documents. One row per team; +-- a missing row means 'SearchableByOwnTeam'. +CREATE TABLE team_search_visibility ( + team uuid PRIMARY KEY, + search_visibility_inbound integer NOT NULL +); + +-- Deployment note: rows created before this migration have a NULL +-- name_normalized. Run `brig-index backfill-normalized-names` once before +-- switching user search over to Postgres, otherwise name search misses +-- every pre-migration user (handle/email search are unaffected). diff --git a/libs/wire-subsystems/postgres-migrations/20260911000001-user-search-name-normalized-index.sql b/libs/wire-subsystems/postgres-migrations/20260911000001-user-search-name-normalized-index.sql new file mode 100644 index 00000000000..1320a756790 --- /dev/null +++ b/libs/wire-subsystems/postgres-migrations/20260911000001-user-search-name-normalized-index.sql @@ -0,0 +1,24 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2026 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +-- Prefix-search index for name_normalized (LIKE 'abc%', no pg_trgm needed); +-- companion to 20260911000000-user-search-postgres.sql. Runs outside a +-- transaction (CREATE INDEX CONCURRENTLY); see +-- 'Wire.PostgresMigrations.nonTransactionMigrations'. One statement per file: +-- statements within a single script run in one implicit transaction, which +-- CREATE INDEX CONCURRENTLY forbids. +CREATE INDEX CONCURRENTLY IF NOT EXISTS wire_user_name_normalized_pattern_idx ON wire_user (name_normalized text_pattern_ops); diff --git a/libs/wire-subsystems/postgres-migrations/20260911000002-user-search-handle-index.sql b/libs/wire-subsystems/postgres-migrations/20260911000002-user-search-handle-index.sql new file mode 100644 index 00000000000..3a12a535a15 --- /dev/null +++ b/libs/wire-subsystems/postgres-migrations/20260911000002-user-search-handle-index.sql @@ -0,0 +1,24 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2026 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +-- Prefix-search index on lower(handle); companion to +-- 20260911000000-user-search-postgres.sql. Runs outside a transaction +-- (CREATE INDEX CONCURRENTLY); see +-- 'Wire.PostgresMigrations.nonTransactionMigrations'. One statement per file: +-- statements within a single script run in one implicit transaction, which +-- CREATE INDEX CONCURRENTLY forbids. +CREATE INDEX CONCURRENTLY IF NOT EXISTS wire_user_lower_handle_pattern_idx ON wire_user (lower(handle) text_pattern_ops); diff --git a/libs/wire-subsystems/postgres-migrations/20260911000003-user-search-team-index.sql b/libs/wire-subsystems/postgres-migrations/20260911000003-user-search-team-index.sql new file mode 100644 index 00000000000..39eaea3e6c0 --- /dev/null +++ b/libs/wire-subsystems/postgres-migrations/20260911000003-user-search-team-index.sql @@ -0,0 +1,24 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2026 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +-- Index on wire_user.team; companion to +-- 20260911000000-user-search-postgres.sql. Runs outside a transaction +-- (CREATE INDEX CONCURRENTLY); see +-- 'Wire.PostgresMigrations.nonTransactionMigrations'. One statement per file: +-- statements within a single script run in one implicit transaction, which +-- CREATE INDEX CONCURRENTLY forbids. +CREATE INDEX CONCURRENTLY IF NOT EXISTS wire_user_team_idx ON wire_user (team); diff --git a/libs/wire-subsystems/src/Wire/PostgresMigrations.hs b/libs/wire-subsystems/src/Wire/PostgresMigrations.hs index 75b696f8e74..bab269067f9 100644 --- a/libs/wire-subsystems/src/Wire/PostgresMigrations.hs +++ b/libs/wire-subsystems/src/Wire/PostgresMigrations.hs @@ -46,7 +46,10 @@ nonTransactionMigrations = Set.fromList [ "20260428072649-create-conv-parent-index.sql", "20260708090000-meetings-recurrence-eff-end-index.sql", - "20260708100000-meetings-end-time-nonrecurring-index.sql" + "20260708100000-meetings-end-time-nonrecurring-index.sql", + "20260911000001-user-search-name-normalized-index.sql", + "20260911000002-user-search-handle-index.sql", + "20260911000003-user-search-team-index.sql" ] data PostgresMigrationError diff --git a/libs/wire-subsystems/src/Wire/UserSearch/Normalize.hs b/libs/wire-subsystems/src/Wire/UserSearch/Normalize.hs new file mode 100644 index 00000000000..5354fbd1b46 --- /dev/null +++ b/libs/wire-subsystems/src/Wire/UserSearch/Normalize.hs @@ -0,0 +1,35 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2026 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +module Wire.UserSearch.Normalize + ( normalized, + ) +where + +import Data.Text (Text) +import Data.Text.ICU.Translit (trans, transliterate) + +-- | Normalizes a name (or search term) for matching: transliterate to +-- Latin, strip diacritics, lowercase. ("Björn" -> "bjorn") +-- +-- This is the same function that used to be applied when writing the +-- ElasticSearch user documents (formerly 'Wire.UserStore.IndexUser.normalized'); +-- it is applied wherever @wire_user.name_normalized@ is written (brig's +-- user store inserts/updates and the one-off brig-index backfill) and +-- when building search queries. +normalized :: Text -> Text +normalized = transliterate (trans "Any-Latin; Latin-ASCII; Lower") diff --git a/libs/wire-subsystems/src/Wire/UserSearchStore.hs b/libs/wire-subsystems/src/Wire/UserSearchStore.hs new file mode 100644 index 00000000000..49d56155d0a --- /dev/null +++ b/libs/wire-subsystems/src/Wire/UserSearchStore.hs @@ -0,0 +1,71 @@ +{-# LANGUAGE TemplateHaskell #-} + +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2026 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +module Wire.UserSearchStore where + +import Data.Id +import Data.Qualified (Local) +import Imports +import Polysemy +import Wire.API.Team.Size +import Wire.API.User.Search +import Wire.UserSearch.Types (BrowseTeamFilters, SearchVisibilityInbound, TeamSearchInfo) + +-- | User search queries backed directly by brig's user store (Postgres), +-- replacing the former ElasticSearch-backed 'Wire.IndexedUserStore'. +data UserSearchStore m a where + -- | Full-text search over user names and handles (prefix matching, see the + -- swagger docs of @/users/search@ for rank ordering). Excludes the searcher + -- and exact-handle matches (the latter are fetched from the user store by + -- the caller). + SearchUsers :: + -- | The searcher; its domain qualifies the returned contacts. + Local UserId -> + Maybe TeamId -> + TeamSearchInfo -> + Text -> + Int -> + Maybe [UserTypeFilter] -> + UserSearchStore m (SearchResult Contact) + -- | Team member browse (formerly @/teams/:tid/browse@ via ES). Fills + -- 'TeamContact.teamContactRole'; 'teamContactUserGroups' is left empty and is + -- filled by the caller. + PaginateTeamMembers :: + BrowseTeamFilters -> + Int -> + Maybe PagingState -> + UserSearchStore m (SearchResult TeamContact) + -- | Number of activated, non-deleted team members, split by regular users + -- and apps. Used for max-team-size enforcement. + GetTeamSize :: TeamId -> UserSearchStore m TeamSize + -- | Inbound federated search (a remote backend searching this backend). + -- @Nothing@ allows non-team users and members of teams opted in to + -- searchable-by-all-teams; @Just []@ matches nothing; @Just teams@ allows + -- only opted-in members of the given teams. + SearchUsersFederated :: + Maybe [TeamId] -> + Text -> + Int -> + Maybe [UserTypeFilter] -> + UserSearchStore m (SearchResult Contact) + -- | Upsert the inbound search visibility setting of a team (pushed from + -- galley when the corresponding team feature changes). + SetTeamSearchVisibilityInbound :: TeamId -> SearchVisibilityInbound -> UserSearchStore m () + +makeSem ''UserSearchStore diff --git a/libs/wire-subsystems/src/Wire/UserSearchStore/ElasticSearch.hs b/libs/wire-subsystems/src/Wire/UserSearchStore/ElasticSearch.hs new file mode 100644 index 00000000000..fe9d669ad4c --- /dev/null +++ b/libs/wire-subsystems/src/Wire/UserSearchStore/ElasticSearch.hs @@ -0,0 +1,60 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2026 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +-- | ElasticSearch adapter for 'UserSearchStore'. +-- +-- Delegates to the 'IndexedUserStore' effect (the ElasticSearch-backed +-- implementation), converting 'UserDoc' results to contacts exactly like the +-- pre-rework local search did. This keeps ElasticSearch the default search +-- backend while 'Wire.UserSearchStore.Postgres' is being rolled out. +module Wire.UserSearchStore.ElasticSearch + ( interpretUserSearchStoreElasticSearch, + ) +where + +import Data.Id +import Data.Qualified +import Imports +import Polysemy +import Wire.API.User (fromName) +import Wire.API.User.Search +import Wire.IndexedUserStore (IndexedUserStore) +import Wire.IndexedUserStore qualified as IndexedUserStore +import Wire.UserSearch.Types +import Wire.UserSearchStore + +interpretUserSearchStoreElasticSearch :: + (Member IndexedUserStore r) => + InterpreterFor UserSearchStore r +interpretUserSearchStoreElasticSearch = interpret \case + SearchUsers luid mtid info q n mtypes -> + fmap (fmap (docToContact luid)) (IndexedUserStore.searchUsers (tUnqualified luid) mtid info q n mtypes) + PaginateTeamMembers filters maxResults paging -> + fmap (userDocToTeamContact []) <$> IndexedUserStore.paginateTeamMembers filters maxResults paging + GetTeamSize tid -> IndexedUserStore.getTeamSize tid + SearchUsersFederated {} -> + error "Wire.UserSearchStore.ElasticSearch: federated search goes through Brig.User.Search.SearchIndex when searchBackend=elasticsearch" + SetTeamSearchVisibilityInbound tid vis -> + IndexedUserStore.updateTeamSearchVisibilityInbound tid vis + +docToContact :: Local UserId -> UserDoc -> Contact +docToContact luid userDoc = + runIdentity $ + userDocToContact + (tUntagged $ qualifyAs luid userDoc.udId) + (Identity . maybe "" fromName) + userDoc diff --git a/libs/wire-subsystems/src/Wire/UserSearchStore/Postgres.hs b/libs/wire-subsystems/src/Wire/UserSearchStore/Postgres.hs new file mode 100644 index 00000000000..604dd2f032f --- /dev/null +++ b/libs/wire-subsystems/src/Wire/UserSearchStore/Postgres.hs @@ -0,0 +1,865 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2026 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +-- | Postgres interpreter for 'UserSearchStore'. +-- +-- Queries brig's @wire_user@ table directly, replacing the former +-- ElasticSearch projection ('Wire.IndexedUserStore.ElasticSearch' is the +-- semantic spec for the predicates below): +-- +-- * candidate filter: activated, status active/suspended, no service +-- accounts (mirrors the former @shouldIndex@ write filter); +-- +-- * 'searchUsers': prefix matching on @name_normalized@ (ICU-folded) and +-- handle, rank-ordered like the documented swagger ordering (exact +-- handle, exact name, handle prefix, name prefix), visibility via the +-- @team_search_visibility@ table (left join; a missing row defaults to +-- own-team-only); +-- +-- * 'paginateTeamMembers': keyset pagination on @(sort_value, id)@ for +-- SQL-side sorts; role filter and role/SAML-idp sorts are resolved via +-- the galley role map and paginated by offset in Haskell (team +-- membership is bounded by @hardTruncationLimit@); +-- +-- * 'getTeamSize': count of activated, active/suspended team members. +module Wire.UserSearchStore.Postgres + ( interpretUserSearchStorePostgres, + ) +where + +import Control.Lens ((^.)) +import Data.Aeson qualified as Aeson +import Data.Bifunctor (first) +import Data.ByteString.Conversion qualified as BSC +import Data.ByteString.Lazy qualified as LBS +import Data.Domain (Domain) +import Data.Functor.Contravariant ((>$<)) +import Data.Id +import Data.Json.Util (toUTCTimeMillis) +import Data.Map.Strict qualified as Map +import Data.Qualified (Local, Qualified (..), tDomain, tUnqualified) +import Data.Text qualified as Text +import Data.Text.Ascii (decodeBase64Url, encodeBase64Url) +import Data.Text.Encoding qualified as TE +import Data.Time (UTCTime) +import Hasql.Decoders qualified as Dec +import Hasql.Encoders qualified as Enc +import Hasql.Statement (Statement, refineResult) +import Hasql.TH +import Imports +import Polysemy +import Polysemy.Input +import Polysemy.TinyLog (TinyLog) +import Polysemy.TinyLog qualified as TinyLog +import System.Logger.Class qualified as Log +import Wire.API.PostgresMarshall +import Wire.API.Team.HardTruncationLimit (hardTruncationLimit, hardTruncationLimitRange) +import Wire.API.Team.Member qualified as Team +import Wire.API.Team.Member.Info (TeamMemberInfo (..), TeamMemberInfoList (members)) +import Wire.API.Team.Role (Role, permissionsToRole, roleName) +import Wire.API.Team.Size (TeamSize (..)) +import Wire.API.User +import Wire.API.User.Search +import Wire.GalleyAPIAccess (GalleyAPIAccess) +import Wire.GalleyAPIAccess qualified as GalleyAPIAccess +import Wire.Postgres +import Wire.StoredUser (inferUserType) +import Wire.UserSearch.Normalize (normalized) +import Wire.UserSearch.Types (BrowseTeamFilters (..), SearchVisibilityInbound (..), TeamSearchInfo (..)) +import Wire.UserSearchStore + +-- | Only users with these account statuses were indexed into ElasticSearch +-- (see the former @shouldIndex@); encoded as in +-- @instance PostgresMarshall Int32 AccountStatus@. +accountStatusIndexed :: [AccountStatus] +accountStatusIndexed = [Active, Suspended] + +interpretUserSearchStorePostgres :: + ( PGConstraints r, + Member GalleyAPIAccess r, + Member TinyLog r, + Member (Input (Local ())) r + ) => + InterpreterFor UserSearchStore r +interpretUserSearchStorePostgres = + interpret $ \case + SearchUsers lSearcher mSearcherTeam teamSearchInfo term maxResults mTypes -> + searchUsersImpl lSearcher mSearcherTeam teamSearchInfo term maxResults mTypes + PaginateTeamMembers filters maxResults mPagingState -> + paginateTeamMembersImpl filters maxResults mPagingState + SearchUsersFederated mOnlyInTeams term maxResults mTypes -> + searchUsersFederatedImpl mOnlyInTeams term maxResults mTypes + GetTeamSize tid -> getTeamSizeImpl tid + SetTeamSearchVisibilityInbound tid vis -> + runStatement (tid, searchVisibilityInboundToInt vis) upsertTeamSearchVisibilityStatement + +-------------------------------------------------------------------------------- +-- SearchUsers + +searchUsersImpl :: + (PGConstraints r) => + Local UserId -> + Maybe TeamId -> + TeamSearchInfo -> + Text -> + Int -> + Maybe [UserTypeFilter] -> + Sem r (SearchResult Contact) +searchUsersImpl lSearcher mSearcherTeam teamSearchInfo searchTerm maxResults mTypes = + case (Text.words term, visibilityCondition mSearcherTeam teamSearchInfo) of + -- The former ES query could not match the empty term either. + ([], _) -> pure emptyResult + (_, VisImpossible) -> pure emptyResult + (tokens, VisCondition vis) -> + runContactSearch + localDomain + (contactSearchQuery (Just searcher) mSearcherTeam tokens term vis maxResults mTypes) + rankedOrder + where + searcher = tUnqualified lSearcher + localDomain = tDomain lSearcher + term = Text.unwords (analyzeTerm searchTerm) + + rankedOrder = literal "order by" <> rankFragment term <> literal "asc, wu.id asc" + + emptyResult = SearchResult 0 0 0 [] FullSearch Nothing Nothing + +-- | Inbound federated search: the searcher is unknown, so no self +-- exclusion applies and results are not relevance-ordered (the former ES +-- implementation scored all matches equally here). +searchUsersFederatedImpl :: + (PGConstraints r, Member (Input (Local ())) r) => + Maybe [TeamId] -> + Text -> + Int -> + Maybe [UserTypeFilter] -> + Sem r (SearchResult Contact) +searchUsersFederatedImpl mOnlyInTeams searchTerm maxResults mTypes = + case Text.words term of + [] -> pure emptyResult + tokens -> case federatedVisibility mOnlyInTeams of + VisImpossible -> pure emptyResult + VisCondition vis -> do + loc <- input + runContactSearch (tDomain loc) (contactSearchQuery Nothing Nothing tokens term vis maxResults mTypes) (literal "order by wu.id asc") + where + term = normalized searchTerm + + emptyResult = SearchResult 0 0 0 [] FullSearch Nothing Nothing + + federatedVisibility = \case + Nothing -> + VisCondition $ + literal "(wu.team is null or tsv.search_visibility_inbound =" + <> intParam searchVisibilityInboundAllTeams + <> literal ")" + Just [] -> + -- Impossible to fulfill (safety net, handled earlier by the caller). + VisImpossible + Just teams -> + VisCondition $ + andList + [ paramLiteral + (const (map toUUID teams) >$< Enc.param (Enc.nonNullable (Enc.foldableArray (Enc.nonNullable Enc.uuid)))) + (\i -> "wu.team = any(" <> argPattern "uuid[]" i <> ")"), + literal "tsv.search_visibility_inbound =" <> intParam searchVisibilityInboundAllTeams + ] + +runContactSearch :: + (PGConstraints r) => + Domain -> + QueryFragment -> + QueryFragment -> + Sem r (SearchResult Contact) +runContactSearch localDomain query orderFragment = do + rows <- + runStatement + () + ( refineResult + (traverse (\(total, raw) -> (,) (fromIntegral (total :: Int64)) <$> rawToContact localDomain raw)) + (buildStatement (query <> orderFragment) (Dec.rowList contactRow)) + ) + pure + SearchResult + { searchFound = maybe 0 fst (listToMaybe rows), + searchReturned = length rows, + searchTook = 0, + searchResults = map snd rows, + searchPolicy = FullSearch, + searchPagingState = Nothing, + searchHasMore = Nothing + } + +-- | Shared filter and matching logic of 'SearchUsers' and +-- 'SearchUsersFederated' (the former @defaultUserQuery@ / @mkUserQuery@). +-- The ordering clause is supplied by the caller. +contactSearchQuery :: + Maybe UserId -> + Maybe TeamId -> + [Text] -> + Text -> + QueryFragment -> + Int -> + Maybe [UserTypeFilter] -> + QueryFragment +contactSearchQuery mSearcher mSearcherTeam tokens term vis maxResults mTypes = + literal "select (count(*) over ()) :: bigint, wu.id :: uuid, wu.name :: text, wu.accent_id :: int, wu.handle :: text, wu.team :: uuid, wu.user_type :: int" + <> literal "from wire_user wu left join team_search_visibility tsv on tsv.team = wu.team" + <> literal "where" + <> andList conditions + <> limitParam maxResults + where + conditions = + candidateCondition + ++ [ -- Exact handle matches are fetched by the caller (user store + -- lookup for local search, exact-handle search for federation). + -- Handle-less users pass (they are still findable by name). + literal "(wu.handle is null or lower(wu.handle) <>" + <> textParam term + <> literal ")", + literal "(wu.searchable is null or wu.searchable)", + appExclusionCondition mSearcherTeam, + vis + ] + ++ catMaybes [selfExclusion mSearcher, userTypeCondition mTypes] + ++ map (tokenMatchCondition False) tokens + + selfExclusion = \case + Nothing -> Nothing + Just searcher -> Just (clause1 "wu.id" "<>" searcher) + +-- | Rank ordering documented in the swagger docs of @/users/search@: +-- exact name, handle prefix, name prefix. (Exact handle matches are +-- excluded from this query entirely - the swagger's first tier is +-- resolved by the caller's exact-handle lookup.) +rankFragment :: Text -> QueryFragment +rankFragment term = + literal "case when wu.name_normalized =" + <> textParam term + <> literal "then 0 when lower(wu.handle) like" + <> textParam (escapeLike term <> "%") + <> literal "then 1 when wu.name_normalized like" + <> textParam (escapeLike term <> "%") + <> literal "then 2 else 3 end" + +contactRow :: Dec.Row (Int64, (UserId, Text, Int32, Maybe Text, Maybe TeamId, Int32)) +contactRow = + (,) + <$> Dec.column (Dec.nonNullable Dec.int8) + <*> ( (,,,,,) + <$> (Id <$> Dec.column (Dec.nonNullable Dec.uuid)) + <*> Dec.column (Dec.nonNullable Dec.text) + <*> Dec.column (Dec.nonNullable Dec.int4) + <*> Dec.column (Dec.nullable Dec.text) + <*> ((Id <$>) <$> Dec.column (Dec.nullable Dec.uuid)) + <*> Dec.column (Dec.nonNullable Dec.int4) + ) + +rawToContact :: Domain -> (UserId, Text, Int32, Maybe Text, Maybe TeamId, Int32) -> Either Text Contact +rawToContact dom (uid, name, accent, mHandle, mTeam, utype) = do + ty <- postgresUnmarshall utype + pure + Contact + { contactQualifiedId = Qualified uid dom, + contactName = name, + contactColorId = Just (fromIntegral accent), + contactHandle = mHandle, + contactTeam = mTeam, + contactType = inferUserType Nothing (Just ty) + } + +-------------------------------------------------------------------------------- +-- PaginateTeamMembers + +data BrowseCursor + = -- | Offset into the Haskell-sorted result set (role / SAML idp sorts). + OffsetCursor Int + | -- | Keyset: null-flag of the sort value, the value itself, and the id + -- of the last returned row. + KeysetCursor Bool Aeson.Value UserId + +-- | Raw @wire_user@ row served to the browse endpoint. +data TeamRow = TeamRow + { trFound :: Int64, + trId :: UserId, + trType :: Int32, + trName :: Text, + trAccent :: Int32, + trHandle :: Maybe Text, + trTeam :: Maybe TeamId, + trEmail :: Maybe Text, + trEmailUnvalidated :: Maybe Text, + trCreatedAt :: UTCTime, + trManagedBy :: Maybe Int32, + trSsoId :: Maybe UserSSOId, + trSearchable :: Maybe Bool + } + +teamRow :: Dec.Row TeamRow +teamRow = + TeamRow + <$> Dec.column (Dec.nonNullable Dec.int8) + <*> (Id <$> Dec.column (Dec.nonNullable Dec.uuid)) + <*> Dec.column (Dec.nonNullable Dec.int4) + <*> Dec.column (Dec.nonNullable Dec.text) + <*> Dec.column (Dec.nonNullable Dec.int4) + <*> Dec.column (Dec.nullable Dec.text) + <*> ((Id <$>) <$> Dec.column (Dec.nullable Dec.uuid)) + <*> Dec.column (Dec.nullable Dec.text) + <*> Dec.column (Dec.nullable Dec.text) + <*> Dec.column (Dec.nonNullable Dec.timestamptz) + <*> Dec.column (Dec.nullable Dec.int4) + <*> Dec.column (Dec.nullable (Dec.jsonbBytes decodeJsonStrict)) + <*> Dec.column (Dec.nullable Dec.bool) + +decodeJsonStrict :: (Aeson.FromJSON a) => ByteString -> Either Text a +decodeJsonStrict = first Text.pack . Aeson.eitherDecodeStrict' + +teamRowToTeamContact :: TeamRow -> Either Text TeamContact +teamRowToTeamContact row = do + ty <- postgresUnmarshall row.trType + managedBy <- traverse postgresUnmarshall row.trManagedBy + email <- traverse decodeEmail row.trEmail + emailUnvalidated <- traverse decodeEmail row.trEmailUnvalidated + pure + TeamContact + { teamContactUserId = row.trId, + teamContactUserType = ty, + teamContactName = row.trName, + teamContactColorId = Just (fromIntegral row.trAccent), + teamContactHandle = row.trHandle, + teamContactTeam = row.trTeam, + teamContactEmail = email, + teamContactCreatedAt = Just (toUTCTimeMillis row.trCreatedAt), + teamContactManagedBy = managedBy, + teamContactSAMLIdp = fst <$> (ssoIssuerAndNameId =<< row.trSsoId), + teamContactRole = Nothing, + teamContactScimExternalId = join (scimExternalId <$> managedBy <*> row.trSsoId), + teamContactSso = fmap (uncurry Sso) (row.trSsoId >>= ssoIssuerAndNameId), + teamContactEmailUnvalidated = emailUnvalidated, + teamContactUserGroups = [], + teamContactSearchable = fromMaybe True row.trSearchable + } + where + decodeEmail = first Text.pack . BSC.runParser BSC.parser . TE.encodeUtf8 + +-- | @refineResult@ step keeping the raw row alongside the contact, so that +-- keyset cursors are built from the exact DB values. +withRaw :: TeamRow -> Either Text (TeamContact, TeamRow) +withRaw row = (,row) <$> teamRowToTeamContact row + +paginateTeamMembersImpl :: + forall r. + (PGConstraints r, Member GalleyAPIAccess r, Member TinyLog r) => + BrowseTeamFilters -> + Int -> + Maybe PagingState -> + Sem r (SearchResult TeamContact) +paginateTeamMembersImpl filters maxResults mPagingState = do + -- The role filter is resolved via the galley role map; sorts on role and + -- SAML idp are not SQL-expressible on wire_user. + let needRoleMap = isJust filters.mRoleFilter || filters.mSortBy `elem` [Just SortByRole, Just SortBySAMLIdp] + mRoleMap <- + if needRoleMap + then Just <$> fetchRoleMap filters.teamId + else pure Nothing + let mCursor = mPagingState >>= decodeCursor + case filters.mSortBy of + Just SortByRole -> haskellSortPage (Just SortByRole) mRoleMap mCursor + Just SortBySAMLIdp -> haskellSortPage (Just SortBySAMLIdp) mRoleMap mCursor + _ -> sqlKeysetPage mRoleMap mCursor + where + -- Full result set, filtered/sorted/paged in Haskell. Only used for + -- sorts that depend on the galley role map; team membership is bounded + -- by `hardTruncationLimit` (see `fetchRoleMap`). + haskellSortPage :: + Maybe TeamUserSearchSortBy -> + Maybe (Map UserId Role) -> + Maybe BrowseCursor -> + Sem r (SearchResult TeamContact) + haskellSortPage mSortBy' mRoleMap mCursor = do + let startOffset = case mCursor of + Just (OffsetCursor n) -> n + _ -> 0 + rows <- + runStatement + () + (refineResult (traverse withRaw) (buildStatement (browseQuery filters mRoleMap Nothing) (Dec.rowList teamRow))) + let total = maybe 0 (trFound . snd) (listToMaybe rows) + dir = fromMaybe SortOrderAsc filters.mSortOrder + sorted = sortTeamContacts mSortBy' dir (map fst rows) + page = take maxResults (drop startOffset sorted) + hasMore = length sorted > startOffset + length page + applyRoleFill mRoleMap (mkSearchResult (fromIntegral total) page (Just (OffsetCursor (startOffset + length page))) hasMore) + + -- Keyset-paginated SQL path for all sorts directly expressible on + -- wire_user. + sqlKeysetPage :: + Maybe (Map UserId Role) -> + Maybe BrowseCursor -> + Sem r (SearchResult TeamContact) + sqlKeysetPage mRoleMap mCursor = do + let sortSpec@(_, dir) = effectiveSort filters + keyset = mCursor >>= keysetPredicate (Just (fst sortSpec)) dir + rows <- + runStatement + () + ( refineResult + (traverse withRaw) + ( buildStatement + ( browseQuery filters mRoleMap keyset + <> browseOrderBy sortSpec + <> limitParam (maxResults + 1) + ) + (Dec.rowList teamRow) + ) + ) + let total = maybe 0 (trFound . snd) (listToMaybe rows) + pageRows = take maxResults rows + hasMore = length rows > maxResults + nextCursor = keysetCursor sortSpec . snd <$> listToMaybe (reverse pageRows) + applyRoleFill mRoleMap (mkSearchResult (fromIntegral total) (map fst pageRows) nextCursor hasMore) + + -- Fill roles from the role map when one was fetched (role filter/sort), + -- otherwise resolve the page roles via the galley batch RPC. + applyRoleFill :: + Maybe (Map UserId Role) -> + SearchResult TeamContact -> + Sem r (SearchResult TeamContact) + applyRoleFill (Just roleMap) result = + pure result {searchResults = map setRole (searchResults result)} + where + setRole tc = tc {teamContactRole = Map.lookup tc.teamContactUserId roleMap} + applyRoleFill Nothing result = do + results <- fillRolesFromGalley filters.teamId (searchResults result) + pure result {searchResults = results} + +-- | Browse query on @wire_user wu@; @keyset@ (if given) is the pagination +-- continuation predicate, @mRoleMap@ provides the optional role filter. +browseQuery :: + BrowseTeamFilters -> + Maybe (Map UserId Role) -> + Maybe QueryFragment -> + QueryFragment +browseQuery filters mRoleMap keyset = + literal "select (count(*) over ()) :: bigint, wu.id :: uuid, wu.user_type :: int, wu.name :: text, wu.accent_id :: int, wu.handle :: text, wu.team :: uuid, wu.email :: text, wu.email_unvalidated :: text, wu.created_at :: timestamptz, wu.managed_by :: int, wu.sso_id :: jsonb, wu.searchable :: bool" + <> literal "from wire_user wu" + <> literal "where" + <> andList + ( candidateCondition + ++ [ clause1 "wu.team" "=" filters.teamId, + searchableCondition filters.mSearchable, + emailVerificationCondition filters.mEmailVerificationFilter + ] + ++ catMaybes [roleFilterCondition mRoleMap filters.mRoleFilter, keyset] + ++ map (tokenMatchCondition True) (maybe [] analyzeTerm filters.mQuery) + ) + +-- | Keyset continuation cursor for the last returned row. +keysetCursor :: (TeamUserSearchSortBy, TeamUserSearchSortOrder) -> TeamRow -> BrowseCursor +keysetCursor (mSortBy', _) row = + KeysetCursor (isNothing val) (maybe (Aeson.toJSON ()) sortValToAeson val) row.trId + where + val = rowSortVal (Just mSortBy') row + +rowSortVal :: Maybe TeamUserSearchSortBy -> TeamRow -> Maybe SortVal +rowSortVal mSortBy' row = case mSortBy' of + Just SortByName -> Just (SortText row.trName) + Just SortByHandle -> SortText <$> row.trHandle + Just SortByEmail -> SortText <$> row.trEmail + Just SortByManagedBy -> SortInt32 <$> row.trManagedBy + Just SortByCreatedAt -> Just (SortTime row.trCreatedAt) + _ -> Nothing + +-- | The effective SQL sort: explicit @sort-by@ wins (default direction +-- ascending); without it, browse is ordered by creation date, newest first +-- (same as the former ES query). +effectiveSort :: BrowseTeamFilters -> (TeamUserSearchSortBy, TeamUserSearchSortOrder) +effectiveSort filters = case filters.mSortBy of + Just sb -> (sb, fromMaybe SortOrderAsc filters.mSortOrder) + Nothing -> (SortByCreatedAt, SortOrderDesc) + +-- | Ordering clause for the SQL keyset path. Nulls sort last when +-- ascending and first when descending (the former ES implementation sorted +-- missing values last in BOTH directions - a small, documented divergence); +-- the user id is the deterministic tie breaker. +browseOrderBy :: (TeamUserSearchSortBy, TeamUserSearchSortOrder) -> QueryFragment +browseOrderBy (mSortBy', dir) = + literal "order by" + <> literal ("(" <> col <> " is null)") + <> literal dirSql + <> literal "," + <> literal col + <> literal dirSql + <> literal "," + <> literal ("wu.id " <> dirSql) + where + col = fromMaybe "wu.created_at" (sortColumnExpr (Just mSortBy')) + dirSql = case dir of SortOrderAsc -> "asc"; SortOrderDesc -> "desc" + +-- | SQL expressions of the SQL-side sort columns. +sortColumnExpr :: Maybe TeamUserSearchSortBy -> Maybe Text +sortColumnExpr = \case + Just SortByName -> Just "wu.name" + Just SortByHandle -> Just "wu.handle" + Just SortByEmail -> Just "wu.email" + Just SortByManagedBy -> Just "wu.managed_by" + Just SortByCreatedAt -> Just "wu.created_at" + _ -> Nothing + +-- | Keyset continuation predicate for @(null-flag, sort value, id)@ tuples. +-- @Nothing@ when the cursor does not match the requested sort (the page +-- restarts). +keysetPredicate :: + Maybe TeamUserSearchSortBy -> + TeamUserSearchSortOrder -> + BrowseCursor -> + Maybe QueryFragment +keysetPredicate mSortBy dir = \case + OffsetCursor _ -> Nothing + KeysetCursor flag val uid -> do + col <- sortColumnExpr mSortBy + typed <- sortValFromAeson col val + let (op, idOp, dirSql) = case dir of + SortOrderAsc -> (">", ">", "asc") + SortOrderDesc -> ("<", "<", "desc") + pure $ + literal "(((" + <> literal ("(" <> col <> " is null)") + <> literal dirSql + <> literal "," + <> literal col + <> literal dirSql + <> literal "," + <> literal ("wu.id " <> dirSql) + <> literal (") " <> op <> " (") + <> boolParam flag + <> literal "," + <> sortValParam typed + <> literal "," + <> uidParam uid + <> literal "))" + <> literal "or (" + <> literal col + <> literal "is null and" + <> boolParam flag + <> literal ("and wu.id " <> idOp) + <> uidParam uid + <> literal ")" + +-- | Sort value of a row, used for keyset pagination cursors. +data SortVal = SortText Text | SortInt32 Int32 | SortTime UTCTime + +sortValToAeson :: SortVal -> Aeson.Value +sortValToAeson = \case + SortText t -> Aeson.toJSON t + SortInt32 n -> Aeson.toJSON n + SortTime t -> Aeson.toJSON t + +-- | Decode a cursor sort value according to the SQL type of the column. +sortValFromAeson :: Text -> Aeson.Value -> Maybe SortVal +sortValFromAeson col v = case col of + "wu.managed_by" -> tryVal SortInt32 + "wu.created_at" -> tryVal SortTime + _ -> tryVal SortText + where + tryVal :: forall a. (Aeson.FromJSON a) => (a -> SortVal) -> Maybe SortVal + tryVal mk = case Aeson.fromJSON v of + Aeson.Success x -> Just (mk x) + Aeson.Error _ -> Nothing + +sortValParam :: SortVal -> QueryFragment +sortValParam = \case + SortText t -> textParam t + SortInt32 n -> intParam n + SortTime t -> timeParam t + +sortTeamContacts :: Maybe TeamUserSearchSortBy -> TeamUserSearchSortOrder -> [TeamContact] -> [TeamContact] +sortTeamContacts mSortBy' dir = arrange dir . sortOn key + where + key tc = case mSortBy' of + Just SortByRole -> roleName @Text <$> tc.teamContactRole + Just SortBySAMLIdp -> tc.teamContactSAMLIdp + _ -> Nothing + arrange = \case + SortOrderAsc -> id + SortOrderDesc -> reverse + +-- | Role filter as an @id = any(...)@ condition over the members with a +-- matching role in the galley role map. +roleFilterCondition :: Maybe (Map UserId Role) -> Maybe RoleFilter -> Maybe QueryFragment +roleFilterCondition mRoleMap mRoleFilter = do + roleMap <- mRoleMap + RoleFilter roles <- mRoleFilter + -- The former ES query treated an empty role list as "no filter". + if null roles + then pure (literal "true") + else + let roleNames = map (roleName @Text) roles + matching = + [toUUID uid | (uid, role) <- Map.toList roleMap, roleName @Text role `elem` roleNames] + in pure $ + paramLiteral + (const matching >$< Enc.param (Enc.nonNullable (Enc.foldableArray (Enc.nonNullable Enc.uuid)))) + (\i -> "wu.id = any(" <> argPattern "uuid[]" i <> ")") + +searchableCondition :: Maybe Bool -> QueryFragment +searchableCondition = \case + Nothing -> literal "true" + -- Former `searchableFilter`: false = exactly false; true = not false + -- (i.e. true or unset). + Just False -> literal "wu.searchable is false" + Just True -> literal "not (wu.searchable is false)" + +-- | Former @emailFilter@: verified = has a verified email and no unvalidated +-- one; unverified = has an unvalidated email. +emailVerificationCondition :: Maybe EmailVerificationFilter -> QueryFragment +emailVerificationCondition = \case + Nothing -> literal "true" + Just EmailVerified -> literal "(wu.email is not null and wu.email_unvalidated is null)" + Just EmailUnverified -> literal "wu.email_unvalidated is not null" + +-- | Fetches the team member role map from galley. Like the other +-- team-member endpoints, the list is truncated at @hardTruncationLimit@; +-- the truncation is logged and the (possibly incomplete) map is used. +fetchRoleMap :: (Member GalleyAPIAccess r, Member TinyLog r) => TeamId -> Sem r (Map UserId Role) +fetchRoleMap tid = do + teamMemberList <- GalleyAPIAccess.getTeamMembersWithLimit tid (Just (hardTruncationLimitRange @Int32)) + let teamMembers = teamMemberList ^. Team.teamMembers + when (length teamMembers >= hardTruncationLimit) $ + TinyLog.warn $ + Log.msg (Log.val "UserSearchStore: team member list truncated, browse role filter/sort may be incomplete") + . Log.field "team" (idToText tid) + pure $ + Map.fromList + [ (m ^. Team.userId, role) + | m <- teamMembers, + Just role <- [permissionsToRole (m ^. Team.permissions)] + ] + +-- | Fills 'TeamContact.teamContactRole' for the returned page via the +-- galley batch RPC (only needed for pages produced without a role map). +fillRolesFromGalley :: (Member GalleyAPIAccess r) => TeamId -> [TeamContact] -> Sem r [TeamContact] +fillRolesFromGalley tid contacts = do + infos <- members <$> GalleyAPIAccess.selectTeamMemberInfos tid (map (.teamContactUserId) contacts) + let roleOf = Map.fromList [(i.userId, permissionsToRole i.permissions) | i <- infos] + pure [tc {teamContactRole = join (Map.lookup tc.teamContactUserId roleOf)} | tc <- contacts] + +-------------------------------------------------------------------------------- +-- GetTeamSize + +-- | Counts activated team members with an active/suspended status, split by +-- regulars and apps (the former ES implementation aggregated over index +-- documents, which never contained deactivated/deleted/service users). +-- +-- The user type literals mirror @instance PostgresMarshall Int32 UserType@ +-- (regular = 0, app = 2); the status literals mirror +-- @instance PostgresMarshall Int32 AccountStatus@ (active = 0, suspended = 1). +getTeamSizeImpl :: (PGConstraints r) => TeamId -> Sem r TeamSize +getTeamSizeImpl tid = do + (regulars, apps) <- + runStatement tid select + pure TeamSize {regulars = fromIntegral regulars, apps = fromIntegral apps} + where + select :: Statement TeamId (Int64, Int64) + select = + dimapPG + [singletonStatement| + select + count(*) filter (where wu.user_type = 0) :: bigint, + count(*) filter (where wu.user_type = 2) :: bigint + from wire_user wu + where wu.team = ($1 :: uuid) + and wu.activated + and (wu.account_status is null or wu.account_status in (0, 1)) + and wu.service is null + |] + +-------------------------------------------------------------------------------- +-- SetTeamSearchVisibilityInbound + +-- | Integral encoding shared with @instance C.Cql SearchVisibilityInbound@. +searchVisibilityInboundToInt :: SearchVisibilityInbound -> Int32 +searchVisibilityInboundToInt = \case + SearchableByOwnTeam -> 0 + SearchableByAllTeams -> 1 + +upsertTeamSearchVisibilityStatement :: Statement (TeamId, Int32) () +upsertTeamSearchVisibilityStatement = + dimapPG + [resultlessStatement| + insert into team_search_visibility (team, search_visibility_inbound) + values ($1 :: uuid, $2 :: int) + on conflict (team) do update set search_visibility_inbound = excluded.search_visibility_inbound + |] + +-------------------------------------------------------------------------------- +-- Shared query fragments + +-- | Normalizes a search term and splits it into tokens, dropping the +-- leading '@' of each token (the former ES analyzer did this implicitly; +-- the swagger documents that '@' does nothing special). +analyzeTerm :: Text -> [Text] +analyzeTerm = map (Text.dropWhile (== '@')) . Text.words . normalized + +-- | Prefix match on a single (whitespace-split) token of the normalized +-- search term. A name token must start at the beginning of the name or be +-- preceded by whitespace; handle starts with the token. Email is matched +-- only for team browse: matching email in contact search would enable +-- email-based user enumeration on those broader surfaces, which the +-- former ES contact-search queries did not do either. +tokenMatchCondition :: Bool -> Text -> QueryFragment +tokenMatchCondition matchEmail tok = + literal "(" + <> likeParam "wu.name_normalized" prefix + <> literal "or" + <> likeParam "wu.name_normalized" midWord + <> literal "or" + <> likeParam "lower(wu.handle)" prefix + <> (if matchEmail then literal "or" <> likeParam "lower(wu.email)" prefix else literal "true") + <> literal ")" + where + prefix = escapeLike tok <> "%" + midWord = "% " <> escapeLike tok <> "%" + +-- | Mirrors the former @shouldIndex@ write filter: only activated users with +-- an active/suspended status and no service account are searchable. +candidateCondition :: [QueryFragment] +candidateCondition = + [ literal "wu.activated", + literal "(wu.account_status is null or wu.account_status in" + <> commaList (intParam . postgresMarshall @Int32 <$> accountStatusIndexed) + <> literal "))", + literal "wu.service is null" + ] + +-- | The user_type filter semantics of the former ES implementation: no +-- filter for @Nothing@ and @Just []@. +userTypeCondition :: Maybe [UserTypeFilter] -> Maybe QueryFragment +userTypeCondition = \case + Nothing -> Nothing + Just [] -> Nothing + Just uts -> + Just $ + literal "(" + <> foldr1 + (\a b -> a <> literal "or" <> b) + [clause1 "wu.user_type" "=" (postgresMarshall @Int32 (userTypeFilterToUserType ut)) | ut <- uts] + <> literal ")" + +-- | Apps are only searchable within their own team (former +-- @matchAppsFromOtherTeams@, expressed as a negated condition). +appExclusionCondition :: Maybe TeamId -> QueryFragment +appExclusionCondition = \case + Nothing -> + literal "not (wu.user_type =" + <> intParam (postgresMarshall @Int32 UserTypeApp) + <> literal "and wu.team is not null)" + Just st -> + literal "not (wu.user_type =" + <> intParam (postgresMarshall @Int32 UserTypeApp) + <> literal "and (wu.team is null or wu.team <>" + <> uidParam st + <> literal "))" + +-- | Inbound search visibility (former @restrictSearchSpaceByTeam@). +data Visibility = VisImpossible | VisCondition QueryFragment + +visibilityCondition :: Maybe TeamId -> TeamSearchInfo -> Visibility +visibilityCondition mSearcherTeam teamSearchInfo = case (mSearcherTeam, teamSearchInfo) of + (Nothing, _) -> VisCondition (literal "wu.team is null") + (Just _, NoTeam) -> VisCondition (literal "wu.team is null") + (Just searcherTeam, TeamOnly t) + | searcherTeam == t -> VisCondition (clause1 "wu.team" "=" t) + | otherwise -> VisImpossible + (Just searcherTeam, AllUsers) -> + -- Team members of other teams are only visible if their team set + -- search_visibility_inbound = searchable-by-all-teams. The left join + -- yields null (own-team-only default) when the team has no row. + VisCondition $ + literal "(wu.team is null or" + <> clause1 "wu.team" "=" searcherTeam + <> literal "or tsv.search_visibility_inbound =" + <> intParam searchVisibilityInboundAllTeams + <> literal ")" + +searchVisibilityInboundAllTeams :: Int32 +searchVisibilityInboundAllTeams = 1 + +andList :: [QueryFragment] -> QueryFragment +andList = foldr1 (\a b -> a <> literal "and" <> b) + +commaList :: [QueryFragment] -> QueryFragment +commaList = foldr1 (\a b -> a <> literal "," <> b) + +-- | Single (non-null) parameter fragments. +intParam :: Int32 -> QueryFragment +intParam n = paramLiteral (const n >$< Enc.param (Enc.nonNullable Enc.int4)) (argPattern "int") + +textParam :: Text -> QueryFragment +textParam t = paramLiteral (const t >$< Enc.param (Enc.nonNullable Enc.text)) (argPattern "text") + +timeParam :: UTCTime -> QueryFragment +timeParam t = paramLiteral (const t >$< Enc.param (Enc.nonNullable Enc.timestamptz)) (argPattern "timestamptz") + +boolParam :: Bool -> QueryFragment +boolParam b = paramLiteral (const b >$< Enc.param (Enc.nonNullable Enc.bool)) (argPattern "bool") + +uidParam :: Id a -> QueryFragment +uidParam u = paramLiteral (const (toUUID u) >$< Enc.param (Enc.nonNullable Enc.uuid)) (argPattern "uuid") + +limitParam :: Int -> QueryFragment +limitParam n = paramLiteral (const (fromIntegral n) >$< Enc.param (Enc.nonNullable Enc.int4)) (\i -> "limit " <> argPattern "int" i) + +-- | @ like $n@ with the given (already escaped) pattern. +likeParam :: Text -> Text -> QueryFragment +likeParam field pat = paramLiteral (const pat >$< Enc.param (Enc.nonNullable Enc.text)) (\i -> field <> " like " <> argPattern "text" i) + +escapeLike :: Text -> Text +escapeLike = Text.replace "_" "\\_" . Text.replace "%" "\\%" . Text.replace "\\" "\\\\" + +mkSearchResult :: Int -> [a] -> Maybe BrowseCursor -> Bool -> SearchResult a +mkSearchResult found results cursor hasMore = + SearchResult + { searchFound = found, + searchReturned = length results, + searchTook = 0, + searchResults = results, + searchPolicy = FullSearch, + searchPagingState = encodeCursor <$> cursor, + searchHasMore = Just hasMore + } + +encodeCursor :: BrowseCursor -> PagingState +encodeCursor c = + PagingState + . encodeBase64Url + . LBS.toStrict + . Aeson.encode + $ case c of + OffsetCursor n -> Aeson.toJSON n + KeysetCursor flag val uid -> Aeson.toJSON (flag, val, uid) + +decodeCursor :: PagingState -> Maybe BrowseCursor +decodeCursor (PagingState ps) = do + bs <- decodeBase64Url ps + v <- either (const Nothing) Just (Aeson.eitherDecode (LBS.fromStrict bs)) + case Aeson.fromJSON @Int v of + Aeson.Success n -> Just (OffsetCursor n) + Aeson.Error _ -> case Aeson.fromJSON @(Bool, Aeson.Value, UserId) v of + Aeson.Success (flag, val, uid) -> Just (KeysetCursor flag val uid) + Aeson.Error _ -> Nothing diff --git a/libs/wire-subsystems/src/Wire/UserStore/IndexUser.hs b/libs/wire-subsystems/src/Wire/UserStore/IndexUser.hs index 09ac630d191..e59e10c002f 100644 --- a/libs/wire-subsystems/src/Wire/UserStore/IndexUser.hs +++ b/libs/wire-subsystems/src/Wire/UserStore/IndexUser.hs @@ -17,7 +17,7 @@ -- You should have received a copy of the GNU Affero General Public License along -- with this program. If not, see . -module Wire.UserStore.IndexUser where +module Wire.UserStore.IndexUser (module Wire.UserStore.IndexUser, normalized) where import Cassandra.Util import Data.ByteString.Builder @@ -28,7 +28,6 @@ import Data.Id import Data.Json.Util import Data.Text.Encoding qualified as Text import Data.Text.Encoding.Error qualified as Text -import Data.Text.ICU.Translit import Data.Time import Database.CQL.Protocol import Imports @@ -37,6 +36,7 @@ import URI.ByteString import Wire.API.Team.Role (Role) import Wire.API.User hiding (userId) import Wire.API.User.Search +import Wire.UserSearch.Normalize (normalized) import Wire.UserSearch.Types type Activated = Bool @@ -184,11 +184,6 @@ indexUserToDoc searchVisInbound mRole IndexUser {..} = (issuer, nameid) <- ssoIssuerAndNameId userSsoId pure $ Sso {ssoIssuer = issuer, ssoNameId = nameid} --- Transliteration could also be done by ElasticSearch (ICU plugin), but this would --- require a data migration. -normalized :: Text -> Text -normalized = transliterate (trans "Any-Latin; Latin-ASCII; Lower") - emptyUserDoc :: UserId -> UserDoc emptyUserDoc uid = UserDoc diff --git a/libs/wire-subsystems/src/Wire/UserStore/Postgres.hs b/libs/wire-subsystems/src/Wire/UserStore/Postgres.hs index ca9e002bfc5..521461a1c6c 100644 --- a/libs/wire-subsystems/src/Wire/UserStore/Postgres.hs +++ b/libs/wire-subsystems/src/Wire/UserStore/Postgres.hs @@ -104,7 +104,8 @@ type InsertUserRow = ( UserId, Name, Maybe TextStatus, Pict, Maybe EmailAddress, Maybe UserSSOId, ColourId, Maybe Password, Bool, AccountStatus, Maybe UTCTimeMillis, Language, Maybe Country, Maybe ProviderId, Maybe ServiceId, - Maybe TeamId, ManagedBy, Set BaseProtocolTag, Bool, UserType + Maybe TeamId, ManagedBy, Set BaseProtocolTag, Bool, UserType, + Text ) type SelectUserRow = @@ -191,7 +192,8 @@ createUserImpl new mbConv = new.managedBy, new.supportedProtocols, new.searchable, - new.userType + new.userType, + normalized (fromName new.name) ) insertUser :: Hasql.Statement InsertUserRow () @@ -202,14 +204,17 @@ createUserImpl new mbConv = (id, name, text_status, picture, email, sso_id, accent_id, password, activated, account_status, expires, language, country, provider, service, - team, managed_by, supported_protocols, searchable, user_type) + team, managed_by, supported_protocols, searchable, user_type, + name_normalized) VALUES ($1 :: uuid, $2 :: text, $3 :: text?, $4 :: jsonb, $5 :: text?, $6 :: jsonb?, $7 :: integer, $8 :: text?, $9 :: boolean, $10 :: integer, $11 :: timestamptz?, $12 :: text, $13 :: text?, $14 :: uuid?, $15 :: uuid?, - $16 :: uuid?, $17 :: integer, $18 :: integer, $19 :: boolean, $20 :: integer) + $16 :: uuid?, $17 :: integer, $18 :: integer, $19 :: boolean, $20 :: integer, + $21 :: text) ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, + name_normalized = EXCLUDED.name_normalized, text_status = EXCLUDED.text_status, picture = EXCLUDED.picture, email = EXCLUDED.email, @@ -503,7 +508,7 @@ updateUserImpl uid MkStoredUserUpdate {..} = do warn $ Log.msg (Log.val "Updating user") . Log.field "locale" (show locale) runTransaction Serializable Write $ do Transaction.statement - (uid, name, textStatus, pict, accentId, supportedProtocols) + (uid, name, textStatus, pict, accentId, supportedProtocols, fmap (normalized . fromName) name) updateUserFields for_ locale $ \newLocale -> Transaction.statement (uid, newLocale.lLanguage, newLocale.lCountry) updateLocale @@ -511,12 +516,13 @@ updateUserImpl uid MkStoredUserUpdate {..} = do Transaction.statement uid deleteAssetsStatement Transaction.statement (mkAssetRows uid newAssets) insertAssetsStatement where - updateUserFields :: Hasql.Statement (UserId, Maybe Name, Maybe TextStatus, Maybe Pict, Maybe ColourId, Maybe (Set BaseProtocolTag)) () + updateUserFields :: Hasql.Statement (UserId, Maybe Name, Maybe TextStatus, Maybe Pict, Maybe ColourId, Maybe (Set BaseProtocolTag), Maybe Text) () updateUserFields = lmapPG [resultlessStatement| UPDATE wire_user SET name = COALESCE($2 :: text?, name), + name_normalized = COALESCE($7 :: text?, name_normalized), text_status = COALESCE($3 :: text?, text_status), picture = COALESCE($4 :: jsonb?, picture), accent_id = COALESCE($5 :: integer?, accent_id), diff --git a/libs/wire-subsystems/src/Wire/UserSubsystem/Interpreter.hs b/libs/wire-subsystems/src/Wire/UserSubsystem/Interpreter.hs index d5cb2dfee62..09825de112d 100644 --- a/libs/wire-subsystems/src/Wire/UserSubsystem/Interpreter.hs +++ b/libs/wire-subsystems/src/Wire/UserSubsystem/Interpreter.hs @@ -107,6 +107,8 @@ import Wire.UserGroupStore (UserGroupStore, getUserGroupIdsForUsers) import Wire.UserKeyStore import Wire.UserSearch.Metrics import Wire.UserSearch.Types +import Wire.UserSearchStore (UserSearchStore) +import Wire.UserSearchStore qualified as UserSearchStore import Wire.UserStore as UserStore import Wire.UserStore.IndexUser import Wire.UserSubsystem as UserSubsystem @@ -135,6 +137,7 @@ runUserSubsystem :: FederationMonad fedM, Typeable fedM, Member IndexedUserStore r, + Member UserSearchStore r, Member FederationConfigStore r, Member Metrics r, Member InvitationStore r, @@ -279,7 +282,7 @@ internalFindTeamInvitationImpl :: Member (Error UserSubsystemError) r, Member (Input UserSubsystemConfig) r, Member (GalleyAPIAccess) r, - Member IndexedUserStore r, + Member UserSearchStore r, Member TinyLog r, Member DRS.DomainRegistrationStore r ) => @@ -309,7 +312,7 @@ internalFindTeamInvitationImpl (Just e) c = NotAllowed -> throwGuardFailed TeamInviteSetToNotAllowed maxSize <- maxTeamSize <$> input - teamSize <- teamSizeTotal <$> IndexedUserStore.getTeamSize tid + teamSize <- teamSizeTotal <$> UserSearchStore.getTeamSize tid when (teamSize >= fromIntegral maxSize) $ throw UserSubsystemTooManyTeamMembers -- FUTUREWORK: The above can easily be done/tested in the intra call. @@ -880,9 +883,9 @@ syncUserIndex uid = ) <$> permissionsToRole info.permissions -updateTeamSearchVisibilityInboundImpl :: (Member IndexedUserStore r) => TeamStatus SearchVisibilityInboundConfig -> Sem r () +updateTeamSearchVisibilityInboundImpl :: (Member UserSearchStore r) => TeamStatus SearchVisibilityInboundConfig -> Sem r () updateTeamSearchVisibilityInboundImpl teamStatus = - IndexedUserStore.updateTeamSearchVisibilityInbound teamStatus.team $ + UserSearchStore.setTeamSearchVisibilityInbound teamStatus.team $ searchVisibilityInboundFromFeatureStatus teamStatus.status searchUsersImpl :: @@ -890,7 +893,7 @@ searchUsersImpl :: ( Member UserStore r, Member GalleyAPIAccess r, Member (Error UserSubsystemError) r, - Member IndexedUserStore r, + Member UserSearchStore r, Member FederationConfigStore r, RunClient (fedM 'Brig), Member (FederationAPIAccess fedM) r, @@ -926,7 +929,7 @@ searchLocally :: forall r. ( Member GalleyAPIAccess r, Member UserStore r, - Member IndexedUserStore r, + Member UserSearchStore r, Member (Input UserSubsystemConfig) r ) => Local (UserId, Maybe TeamId) -> @@ -946,8 +949,8 @@ searchLocally searcher searchTerm maybeMaxResults mTypes = do esResult <- if esMaxResults > 0 then - IndexedUserStore.searchUsers - (tUnqualified searcherId) + UserSearchStore.searchUsers + searcherId (tUnqualified searcherTeamId) teamSearchInfo searchTerm @@ -955,7 +958,7 @@ searchLocally searcher searchTerm maybeMaxResults mTypes = do mTypes else pure $ SearchResult 0 0 0 [] FullSearch Nothing Nothing - let esContacts = map userDocToContact' (searchResults esResult) + let esContacts = searchResults esResult -- Prepend results matching exact handle and results from ES. allContacts = case maybeExactHandleMatch of Nothing -> esContacts @@ -971,14 +974,6 @@ searchLocally searcher searchTerm maybeMaxResults mTypes = do handleTeamVisibility _ SearchVisibilityStandard = AllUsers handleTeamVisibility t SearchVisibilityNoNameOutsideTeam = TeamOnly t - userDocToContact' :: UserDoc -> Contact - userDocToContact' userDoc = - runIdentity $ - userDocToContact - (tUntagged $ qualifyAs searcher userDoc.udId) - (Identity . maybe "" fromName) - userDoc - mkTeamSearchInfo :: Maybe TeamId -> Sem r TeamSearchInfo mkTeamSearchInfo searcherTeamId = do config <- input @@ -1059,7 +1054,7 @@ searchRemotely rDom mTid searchTerm mTypes = do browseTeamImpl :: ( Member (Error UserSubsystemError) r, - Member IndexedUserStore r, + Member UserSearchStore r, Member TeamSubsystem r, Member UserGroupStore r ) => @@ -1075,13 +1070,10 @@ browseTeamImpl uid filters mMaxResults mPagingState = do ensurePermissions uid filters.teamId [Permission.AddTeamMember] let maxResults = maybe 15 fromRange mMaxResults - result <- IndexedUserStore.paginateTeamMembers filters maxResults mPagingState - let docs = result.searchResults - uids = fmap (.udId) docs + result <- UserSearchStore.paginateTeamMembers filters maxResults mPagingState + let uids = fmap (.teamContactUserId) result.searchResults ugMap <- getUserGroupIdsForUsers (toList uids) - for result $ \userDoc -> do - let ugids = fromMaybe [] (Map.lookup userDoc.udId ugMap) - pure $ userDocToTeamContact ugids userDoc + pure $ fmap (\tc -> tc {teamContactUserGroups = fromMaybe [] (Map.lookup tc.teamContactUserId ugMap)}) result getAccountsByEmailNoFilterImpl :: forall r. @@ -1180,6 +1172,7 @@ acceptTeamInvitationImpl :: Member (Error UserSubsystemError) r, Member InvitationStore r, Member IndexedUserStore r, + Member UserSearchStore r, Member Metrics r, Member Events r, Member AuthenticationSubsystem r, diff --git a/libs/wire-subsystems/test/unit/Wire/MiniBackend.hs b/libs/wire-subsystems/test/unit/Wire/MiniBackend.hs index 1324d919db3..20920a6272e 100644 --- a/libs/wire-subsystems/test/unit/Wire/MiniBackend.hs +++ b/libs/wire-subsystems/test/unit/Wire/MiniBackend.hs @@ -143,6 +143,8 @@ import Wire.TeamSubsystem.GalleyAPI import Wire.UserClientIndexStore (UserClientIndexStore) import Wire.UserGroupStore (UserGroupStore) import Wire.UserKeyStore +import Wire.UserSearchStore (UserSearchStore) +import Wire.UserSearchStore.ElasticSearch (interpretUserSearchStoreElasticSearch) import Wire.UserStore import Wire.UserSubsystem import Wire.UserSubsystem.Error @@ -285,6 +287,7 @@ type MiniBackendLowerEffects = AppStore, TeamCollaboratorsStore, UserKeyStore, + UserSearchStore, IndexedUserStore, FederationConfigStore, DRS.DomainRegistrationStore, @@ -342,6 +345,7 @@ miniBackendLowerEffectsInterpreters mb@(MiniBackendParams {..}) = . inMemoryDomainRegistrationStoreInterpreter . runFederationConfigStoreInMemory . inMemoryIndexedUserStoreInterpreter + . interpretUserSearchStoreElasticSearch . inMemoryUserKeyStoreInterpreter . inMemoryTeamCollaboratorsStoreInterpreter . inMemoryAppStoreInterpreter diff --git a/libs/wire-subsystems/wire-subsystems.cabal b/libs/wire-subsystems/wire-subsystems.cabal index a8e33868abd..4401eb657d0 100644 --- a/libs/wire-subsystems/wire-subsystems.cabal +++ b/libs/wire-subsystems/wire-subsystems.cabal @@ -496,7 +496,11 @@ library Wire.UserPendingActivationStore.Cassandra Wire.UserSearch.Metrics Wire.UserSearch.Migration + Wire.UserSearch.Normalize Wire.UserSearch.Types + Wire.UserSearchStore + Wire.UserSearchStore.ElasticSearch + Wire.UserSearchStore.Postgres Wire.UserStore Wire.UserStore.Cassandra Wire.UserStore.IndexUser diff --git a/services/brig/brig.cabal b/services/brig/brig.cabal index 613e931c9ef..29368b3b08f 100644 --- a/services/brig/brig.cabal +++ b/services/brig/brig.cabal @@ -240,6 +240,7 @@ library , fsnotify >=0.4 , galley-types >=0.75.3 , hashable >=1.2 + , hasql , hasql-resource-pool , hs-opentelemetry-instrumentation-wai , hs-opentelemetry-sdk @@ -328,9 +329,13 @@ executable brig-index , base , brig , extended + , hasql , imports , optparse-applicative + , text , tinylog + , uuid + , wire-subsystems executable brig-integration import: common-all diff --git a/services/brig/default.nix b/services/brig/default.nix index e431f9c93db..33395362a10 100644 --- a/services/brig/default.nix +++ b/services/brig/default.nix @@ -47,6 +47,7 @@ , fsnotify , galley-types , hashable +, hasql , hasql-resource-pool , hs-opentelemetry-instrumentation-wai , hs-opentelemetry-sdk @@ -192,6 +193,7 @@ mkDerivation { fsnotify galley-types hashable + hasql hasql-resource-pool hs-opentelemetry-instrumentation-wai hs-opentelemetry-sdk @@ -287,6 +289,7 @@ mkDerivation { federator filepath galley-types + hasql hscim HsOpenSSL http-api-data diff --git a/services/brig/src/Brig/API/Federation.hs b/services/brig/src/Brig/API/Federation.hs index d109db22531..1ef5517cddf 100644 --- a/services/brig/src/Brig/API/Federation.hs +++ b/services/brig/src/Brig/API/Federation.hs @@ -75,6 +75,8 @@ import Wire.GalleyAPIAccess (GalleyAPIAccess) import Wire.MlsKeyPackageSubsystem (MlsKeyPackageSubsystem) import Wire.NotificationSubsystem import Wire.Sem.Concurrency +import Wire.UserSearchStore (UserSearchStore) +import Wire.UserSearchStore qualified as UserSearchStore import Wire.UserStore import Wire.UserStore qualified as UserStore import Wire.UserSubsystem (UserSubsystem) @@ -91,7 +93,8 @@ federationSitemap :: Member UserStore r, Member ClientStore r, Member MlsKeyPackageSubsystem r, - Member ClientSubsystem r + Member ClientSubsystem r, + Member UserSearchStore r ) => ServerT FederationAPI (Handler r) federationSitemap = @@ -226,7 +229,8 @@ searchUsers :: forall r. ( Member FederationConfigStore r, Member UserSubsystem r, - Member UserStore r + Member UserStore r, + Member UserSearchStore r ) => Domain -> SearchRequest -> @@ -252,10 +256,13 @@ searchUsers domain (SearchRequest searchTerm mTeam mOnlyInTeams mbUserTypeFilter go contacts maxResult (search : searches) = do contactsNew <- search maxResult go (contacts <> contactsNew) (maxResult - length contactsNew) searches - fullSearch :: Int -> ExceptT HttpError (AppT r) [Contact] fullSearch n - | n > 0 = lift $ searchResults <$> Q.searchIndex (Q.FederatedSearch mOnlyInTeams mbUserTypeFilter) searchTerm n + | n > 0 = do + backend <- lift $ asks (.searchBackend) + lift $ case backend of + SearchBackendElasticSearch -> searchResults <$> Q.searchIndex (Q.FederatedSearch mOnlyInTeams mbUserTypeFilter) searchTerm n + SearchBackendPostgres -> searchResults <$> liftSem (UserSearchStore.searchUsersFederated mOnlyInTeams searchTerm n mbUserTypeFilter) | otherwise = pure [] exactHandleSearch :: Int -> ExceptT HttpError (AppT r) [Contact] diff --git a/services/brig/src/Brig/API/Internal.hs b/services/brig/src/Brig/API/Internal.hs index 86f9f2e9b2d..07262ee5a19 100644 --- a/services/brig/src/Brig/API/Internal.hs +++ b/services/brig/src/Brig/API/Internal.hs @@ -115,7 +115,6 @@ import Wire.FederationConfigStore import Wire.FederationConfigStore qualified as E import Wire.GalleyAPIAccess (GalleyAPIAccess) import Wire.HashPassword (HashPassword) -import Wire.IndexedUserStore (IndexedUserStore, getTeamSize) import Wire.InvitationStore import Wire.MlsKeyPackageSubsystem (MlsKeyPackageSubsystem) import Wire.MlsKeyPackageSubsystem qualified as Mls @@ -135,6 +134,8 @@ import Wire.TeamSubsystem (TeamSubsystem) import Wire.UserGroupSubsystem import Wire.UserKeyStore import Wire.UserPendingActivationStore (UserPendingActivationStore) +import Wire.UserSearchStore (UserSearchStore) +import Wire.UserSearchStore qualified as UserSearchStore import Wire.UserStore as UserStore import Wire.UserSubsystem import Wire.UserSubsystem qualified as User @@ -170,7 +171,7 @@ servantSitemap :: Member PasswordResetCodeStore r, Member PropertySubsystem r, Member (Input (Local ())) r, - Member IndexedUserStore r, + Member UserSearchStore r, Member (Polysemy.Error UserSubsystemError) r, Member HashPassword r, Member (Embed IO) r, @@ -320,7 +321,7 @@ teamsAPI :: Member (Polysemy.Error UserSubsystemError) r, Member Events r, Member (Input (Local ())) r, - Member IndexedUserStore r, + Member UserSearchStore r, Member AuthenticationSubsystem r ) => ServerT BrigIRoutes.TeamsAPI (Handler r) @@ -330,7 +331,7 @@ teamsAPI = :<|> Named @"get-invitation-code" (\tid iid -> lift . liftSem $ Team.getInvitationCode tid iid) :<|> Named @"suspend-team" Team.suspendTeam :<|> Named @"unsuspend-team" Team.unsuspendTeam - :<|> Named @"team-size" (lift . liftSem . getTeamSize) + :<|> Named @"team-size" (lift . liftSem . UserSearchStore.getTeamSize) :<|> Named @"create-invitations-via-scim" Team.createInvitationViaScim userAPI :: (Member UserSubsystem r) => ServerT BrigIRoutes.UserAPI (Handler r) diff --git a/services/brig/src/Brig/API/Public.hs b/services/brig/src/Brig/API/Public.hs index 60cc7a53bf6..5801540d6bc 100644 --- a/services/brig/src/Brig/API/Public.hs +++ b/services/brig/src/Brig/API/Public.hs @@ -173,7 +173,6 @@ import Wire.FederationConfigStore (FederationConfigStore) import Wire.GalleyAPIAccess import Wire.GalleyAPIAccess qualified as GalleyAPIAccess import Wire.HashPassword (HashPassword) -import Wire.IndexedUserStore (IndexedUserStore) import Wire.InvitationStore import Wire.JwtTools (JwtTools) import Wire.MlsKeyPackageSubsystem (MlsKeyPackageSubsystem) @@ -198,6 +197,7 @@ import Wire.UserGroupSubsystem qualified as UserGroup import Wire.UserKeyStore import Wire.UserPendingActivationStore (UserPendingActivationStore) import Wire.UserSearch.Types +import Wire.UserSearchStore (UserSearchStore) import Wire.UserStore (UserStore) import Wire.UserStore qualified as UserStore import Wire.UserSubsystem hiding (checkHandle, checkHandles, requestEmailChange) @@ -389,7 +389,7 @@ servantSitemap :: Member VerificationCodeSubsystem r, Member (Concurrency 'Unsafe) r, Member BlockListStore r, - Member IndexedUserStore r, + Member UserSearchStore r, Member (ConnectionStore InternalPaging) r, Member HashPassword r, Member (Input UserSubsystemConfig) r, diff --git a/services/brig/src/Brig/App.hs b/services/brig/src/Brig/App.hs index da0c62a188a..0bb2a5fc52f 100644 --- a/services/brig/src/Brig/App.hs +++ b/services/brig/src/Brig/App.hs @@ -75,6 +75,7 @@ module Brig.App rateLimitEnvLens, amqpJobsPublisherChannelLens, postgresMigrationLens, + searchBackendLens, initZAuth, initLogger, initPostgresPool, @@ -220,6 +221,7 @@ data Env = Env digestSHA256 :: Digest, digestMD5 :: Digest, indexEnv :: IndexEnv, + searchBackend :: Opt.SearchBackend, randomPrekeyLocalLock :: Maybe (MVar ()), keyPackageLocalLock :: MVar (), rabbitmqChannel :: MVar Q.Channel, @@ -338,6 +340,7 @@ newEnv opts = do digestMD5 = md5, digestSHA256 = sha256, indexEnv = idxEnv, + searchBackend = fromMaybe Opt.SearchBackendElasticSearch opts.searchBackend, randomPrekeyLocalLock = prekeyLocalLock, keyPackageLocalLock = kpLock, rabbitmqChannel = rabbitChan, diff --git a/services/brig/src/Brig/CanonicalInterpreter.hs b/services/brig/src/Brig/CanonicalInterpreter.hs index 05365ca28a8..c55b0af464d 100644 --- a/services/brig/src/Brig/CanonicalInterpreter.hs +++ b/services/brig/src/Brig/CanonicalInterpreter.hs @@ -23,7 +23,7 @@ import Brig.DeleteQueue.Interpreter as DQ import Brig.Effects.ConnectionStore (ConnectionStore) import Brig.Effects.ConnectionStore.Cassandra (connectionStoreToCassandra) import Brig.IO.Intra (runEvents) -import Brig.Options (Settings (consumableNotifications), federationDomainConfigs, federationStrategy) +import Brig.Options (SearchBackend (..), Settings (consumableNotifications), federationDomainConfigs, federationStrategy) import Brig.Options qualified as Opt import Brig.Template (InvitationUrlTemplates) import Brig.User.Search.Index (IndexEnv (..)) @@ -106,6 +106,7 @@ import Wire.GundeckAPIAccess import Wire.HashPassword import Wire.HashPassword.Interpreter import Wire.IndexedUserStore +import Wire.IndexedUserStore qualified as IndexedUserStore import Wire.IndexedUserStore.ElasticSearch import Wire.InvitationStore (InvitationStore) import Wire.InvitationStore.Cassandra (interpretInvitationStoreToCassandra) @@ -166,6 +167,9 @@ import Wire.UserKeyStore import Wire.UserKeyStore.Cassandra import Wire.UserPendingActivationStore (UserPendingActivationStore) import Wire.UserPendingActivationStore.Cassandra (userPendingActivationStoreToCassandra) +import Wire.UserSearchStore +import Wire.UserSearchStore.ElasticSearch (interpretUserSearchStoreElasticSearch) +import Wire.UserSearchStore.Postgres (interpretUserSearchStorePostgres) import Wire.UserStore import Wire.UserStore.Cassandra import Wire.UserStore.Postgres (interpretUserStorePostgres) @@ -217,6 +221,7 @@ type BrigLowerLevelEffects = MlsKeyPackageStore, UserStore, UserGroupStore, + UserSearchStore, DomainRegistrationStore, DomainVerificationChallengeStore, Error AppSubsystemError, @@ -471,7 +476,10 @@ runBrigToIO e (AppT ma) = do . interpretVerificationCodeStoreCassandra e.casClient . interpretPasswordStore e.casClient . interpretSessionStoreCassandra e.casClient - . interpretIndexedUserStoreES indexedUserStoreConfig + . ( case e.searchBackend of + SearchBackendElasticSearch -> interpretIndexedUserStoreES indexedUserStoreConfig + SearchBackendPostgres -> interpretIndexedUserStoreNoop + ) . interpretClientStoreCassandra clientStoreCassandraEnv . runHashPassword e.settings.passwordHashingOptions . runCryptoSign @@ -494,6 +502,13 @@ runBrigToIO e (AppT ma) = do . mapError appSubsystemErrorToHttpError . domainVerificationChallengeStore . domainRegistrationStore + . ( case e.searchBackend of + SearchBackendElasticSearch -> interpretUserSearchStoreElasticSearch + SearchBackendPostgres + | CassandraStorage <- e.postgresMigration.user -> + error "UserSearchStore requires the brig user store in Postgres: set postgresMigration.user to PostgresqlStorage or MigrationToPostgresql" + | otherwise -> interpretUserSearchStorePostgres + ) . interpretUserGroupStoreToPostgres . userStoreInterpreter . interpretMlsKeyPackageStoreToCassandra e.casClient @@ -560,3 +575,16 @@ emailSendingInterpreter e = do case e.smtpEnv of Just smtp -> emailViaSMTPInterpreter e.appLogger smtp Nothing -> emailViaSESInterpreter (e.awsEnv ^. amazonkaEnv) + +-- | Disables the ElasticSearch index when brig serves user search from +-- Postgres ('SearchBackendPostgres'): no writes go to the index any more, so +-- ElasticSearch can be decommissioned. +interpretIndexedUserStoreNoop :: InterpreterFor IndexedUserStore r +interpretIndexedUserStoreNoop = interpret \case + IndexedUserStore.Upsert {} -> pure () + IndexedUserStore.BulkUpsert {} -> pure () + IndexedUserStore.UpdateTeamSearchVisibilityInbound {} -> pure () + IndexedUserStore.DoesIndexExist -> pure False + IndexedUserStore.SearchUsers {} -> error "IndexedUserStore: disabled when searchBackend=postgres" + IndexedUserStore.PaginateTeamMembers {} -> error "IndexedUserStore: disabled when searchBackend=postgres" + IndexedUserStore.GetTeamSize {} -> error "IndexedUserStore: disabled when searchBackend=postgres" diff --git a/services/brig/src/Brig/Index/Eval.hs b/services/brig/src/Brig/Index/Eval.hs index f931a032769..727bece8214 100644 --- a/services/brig/src/Brig/Index/Eval.hs +++ b/services/brig/src/Brig/Index/Eval.hs @@ -36,12 +36,24 @@ import Data.Aeson (FromJSON) import Data.Aeson qualified as Aeson import Data.ByteString.Lazy.UTF8 qualified as UTF8 import Data.Credentials (Credentials (..)) +import Data.Functor.Contravariant ((>$<)) import Data.Id +import Data.Text qualified as Text +import Data.Text.IO qualified as TextIO +import Data.UUID (UUID) import Database.Bloodhound qualified as ES import Database.Bloodhound.Internal.Client (BHEnv (..)) +import Hasql.Connection qualified as HasqlConn +import Hasql.Connection.Settings qualified as HasqlConnSettings +import Hasql.Decoders qualified as Decoders +import Hasql.Encoders qualified as Encoders +import Hasql.Errors (IsError (..), toDetailedText) import Hasql.Pool (UsageError) import Hasql.Pool.Extended import Hasql.Pool.Extended qualified as Hasql +import Hasql.Session qualified as Session +import Hasql.Statement +import Hasql.Statement qualified as Statement import Imports import Network.HTTP.Client (Manager) import Polysemy @@ -51,6 +63,7 @@ import Polysemy.Error import Polysemy.Input import Polysemy.Resource (Resource, runResource) import Polysemy.TinyLog (TinyLog) +import System.Exit (exitFailure) import System.Logger qualified as Log import System.Logger.Class (Logger) import Util.Options @@ -72,6 +85,7 @@ import Wire.Sem.Metrics.IO import Wire.UserKeyStore (UserKeyStore) import Wire.UserKeyStore.Cassandra import Wire.UserSearch.Migration (MigrationException) +import Wire.UserSearch.Normalize (normalized) import Wire.UserStore (UserStore) import Wire.UserStore.Cassandra import Wire.UserStore.Postgres (interpretUserStorePostgres) @@ -191,6 +205,8 @@ runCommand l = \case Migrate es cas pg userStorageLocation galley pageSize -> do semDeps <- mkSemDeps (es ^. esConnection) cas pg l IndexedUserStoreBulk.migrateData (runSem semDeps userStorageLocation galley l) pageSize + BackfillNormalizedNames backfillOpts -> + runBackfillNormalizedNames backfillOpts.backfillPgSettings backfillOpts.backfillBatchSize ReindexFromAnotherIndex reindexSettings -> do mgr <- initHttpManagerWithTLSConfig @@ -269,3 +285,51 @@ newtype ReindexFromAnotherIndexError = ReindexFromAnotherIndexError String deriving (Show) instance Exception ReindexFromAnotherIndexError + +-- | One-off backfill of @wire_user.name_normalized@ (required by +-- 'Wire.UserSearchStore' when brig serves user search from Postgres). +runBackfillNormalizedNames :: Text -> Int32 -> IO () +runBackfillNormalizedNames pgSettings batchSize = do + conn <- do + r <- HasqlConn.acquire (HasqlConnSettings.connectionString pgSettings) + either failWith' pure r + let loop total = do + rows <- runSession conn (Session.statement batchSize selectBatch) + case rows of + [] -> + TextIO.putStrLn + ("backfill-normalized-names: done, updated " <> Text.pack (show (total :: Int)) <> " users") + batch -> do + forM_ batch $ \(uid, name) -> + runSession conn (Session.statement (uid, normalized name) updateOne) + loop (total + length batch) + loop 0 + where + failWith' = failWith "connecting to postgres" + +-- | Fetches up to N users with a missing @name_normalized@. +selectBatch :: Statement Int32 [(UUID, Text)] +selectBatch = + Statement.preparable + "SELECT id :: uuid, name :: text FROM wire_user WHERE name_normalized IS NULL AND name IS NOT NULL ORDER BY id LIMIT ($1 :: int4)" + (const (0 :: Int32) >$< Encoders.param (Encoders.nonNullable Encoders.int4)) + (Decoders.rowList ((,) <$> Decoders.column (Decoders.nonNullable Decoders.uuid) <*> Decoders.column (Decoders.nonNullable Decoders.text))) + +updateOne :: Statement (UUID, Text) () +updateOne = + Statement.preparable + "UPDATE wire_user SET name_normalized = ($2 :: text) WHERE id = ($1 :: uuid)" + ( (fst >$< Encoders.param (Encoders.nonNullable Encoders.uuid)) + <> (snd >$< Encoders.param (Encoders.nonNullable Encoders.text)) + ) + Decoders.noResult + +failWith :: (IsError e) => Text -> e -> IO a +failWith context err = do + TextIO.putStrLn ("backfill-normalized-names: " <> context <> ": " <> toDetailedText err) + exitFailure + +runSession :: HasqlConn.Connection -> Session.Session a -> IO a +runSession conn sess = do + r <- HasqlConn.use conn sess + either (failWith "postgres query") pure r diff --git a/services/brig/src/Brig/Index/Options.hs b/services/brig/src/Brig/Index/Options.hs index b19c1f3124f..fe33e9ede2a 100644 --- a/services/brig/src/Brig/Index/Options.hs +++ b/services/brig/src/Brig/Index/Options.hs @@ -43,6 +43,7 @@ module Brig.Index.Options mkCreateIndexSettings, toESServer, ReindexFromAnotherIndexSettings (..), + BackfillNormalizedNamesOpts (..), reindexDestIndex, reindexTimeoutSeconds, reindexEsConnection, @@ -82,9 +83,17 @@ data Command | -- | 'ElasticSettings' has shards and other settings that are not needed here. UpdateMapping ESConnectionSettings Endpoint | Migrate ElasticSettings CassandraSettings PostgresSettings UserStorageLocation Endpoint Int32 + | BackfillNormalizedNames BackfillNormalizedNamesOpts | ReindexFromAnotherIndex ReindexFromAnotherIndexSettings deriving (Show) +-- | Options for @brig-index backfill-normalized-names@. +data BackfillNormalizedNamesOpts = BackfillNormalizedNamesOpts + { backfillPgSettings :: Text, + backfillBatchSize :: Int32 + } + deriving (Show) + data ESConnectionSettings = ESConnectionSettings { esServer :: URIRef Absolute, esIndex :: ES.IndexName, @@ -483,6 +492,23 @@ pageSizeParser = <> showDefault ) +backfillNormalizedNamesOptsParser :: Parser BackfillNormalizedNamesOpts +backfillNormalizedNamesOptsParser = + BackfillNormalizedNamesOpts + <$> strOption + ( long "pg-settings" + <> metavar "SETTINGS" + <> help "libpq connection settings, e.g. \"host=... dbname=... user=... password=...\"" + ) + <*> option + auto + ( long "batch-size" + <> metavar "N" + <> value 500 + <> showDefault + <> help "number of users per batch" + ) + commandParser :: Parser Command commandParser = hsubparser @@ -522,6 +548,12 @@ commandParser = (Migrate <$> elasticSettingsParser <*> cassandraSettingsParser <*> postgresSettingsParser <*> userStorageLocationParser <*> galleyEndpointParser <*> pageSizeParser) (progDesc "Migrate data in elastic search") ) + <> command + "backfill-normalized-names" + ( info + (BackfillNormalizedNames <$> backfillNormalizedNamesOptsParser) + (progDesc "Backfill wire_user.name_normalized with the ICU-folded lowercase display name") + ) <> command "reindex-from-another-index" ( info diff --git a/services/brig/src/Brig/Options.hs b/services/brig/src/Brig/Options.hs index 396843a0163..5b2a0f4b718 100644 --- a/services/brig/src/Brig/Options.hs +++ b/services/brig/src/Brig/Options.hs @@ -325,6 +325,15 @@ instance ToSchema ListAllSFTServers where element "disabled" HideAllSFTServers ] +data SearchBackend = SearchBackendElasticSearch | SearchBackendPostgres + deriving (Show, Eq, Generic) + +instance FromJSON SearchBackend where + parseJSON = withText "SearchBackend" $ \case + "elasticsearch" -> pure SearchBackendElasticSearch + "postgres" -> pure SearchBackendPostgres + _ -> fail "expected \"elasticsearch\" or \"postgres\"" + -- | Options that are consumed on startup data Opts = Opts -- services @@ -348,6 +357,11 @@ data Opts = Opts cassandra :: !CassandraOpts, -- | ElasticSearch settings elasticsearch :: !ElasticSearchOpts, + -- | Backend for user search. "elasticsearch" (default) indexes users + -- into ElasticSearch and serves search from there (deprecated); + -- "postgres" serves search from the PostgreSQL user store and stops + -- writing the ElasticSearch index. + searchBackend :: !(Maybe SearchBackend), -- | Postgresql settings, the key values must be in libpq format. -- https://www.postgresql.org/docs/17/libpq-connect.html#LIBPQ-PARAMKEYWORDS postgresql :: !(Map Text Text), diff --git a/services/brig/src/Brig/Team/API.hs b/services/brig/src/Brig/Team/API.hs index ce88b50a63f..12199a7f20d 100644 --- a/services/brig/src/Brig/Team/API.hs +++ b/services/brig/src/Brig/Team/API.hs @@ -75,7 +75,6 @@ import Wire.Error import Wire.Events (Events) import Wire.GalleyAPIAccess (GalleyAPIAccess, ShowOrHideInvitationUrl (..)) import Wire.GalleyAPIAccess qualified as GalleyAPIAccess -import Wire.IndexedUserStore (IndexedUserStore, getTeamSize) import Wire.InvitationStore (InvitationStore (..), PaginatedResult (..), StoredInvitation (..)) import Wire.InvitationStore qualified as Store import Wire.NotificationSubsystem (NotificationSubsystem) @@ -91,6 +90,8 @@ import Wire.UserGroupSubsystem (UserGroupSubsystem) import Wire.UserKeyStore import Wire.UserPendingActivationStore (UserPendingActivationStore) import Wire.UserPendingActivationStore qualified as UserPendingActivationStore +import Wire.UserSearchStore (UserSearchStore) +import Wire.UserSearchStore qualified as UserSearchStore import Wire.UserStore import Wire.UserSubsystem import Wire.UserSubsystem.Error @@ -105,7 +106,7 @@ servantAPI :: Member (Input InvitationUrlTemplates) r, Member (Input (Local ())) r, Member (Error UserSubsystemError) r, - Member IndexedUserStore r, + Member UserSearchStore r, Member TeamSubsystem r, Member SparAPIAccess r, Member (Embed App.HttpClientIO) r, @@ -133,7 +134,7 @@ servantAPI = teamSizePublic :: ( Member (Error UserSubsystemError) r, - Member IndexedUserStore r, + Member UserSearchStore r, Member TeamSubsystem r ) => UserId -> @@ -142,7 +143,7 @@ teamSizePublic :: teamSizePublic uid tid = do -- limit this to team admins to reduce risk of involuntary DOS attacks ensurePermissions uid tid [AddTeamMember] - getTeamSize tid + UserSearchStore.getTeamSize tid getInvitationCode :: ( Member Store.InvitationStore r, diff --git a/services/brig/test/unit/Test/Brig/Options.hs b/services/brig/test/unit/Test/Brig/Options.hs index 6bc1701609d..2d7a12ab333 100644 --- a/services/brig/test/unit/Test/Brig/Options.hs +++ b/services/brig/test/unit/Test/Brig/Options.hs @@ -55,5 +55,16 @@ tests = case eitherDecode "\"noreply@diana.123\"" :: Either String StricterDomain of Left _ -> pure () Right _ -> assertFailure "expected invalid sender domain" + ], + testGroup + "searchBackend" + [ testCase "elasticsearch parses" $ + (eitherDecode "\"elasticsearch\"" :: Either String SearchBackend) @?= Right SearchBackendElasticSearch, + testCase "postgres parses" $ + (eitherDecode "\"postgres\"" :: Either String SearchBackend) @?= Right SearchBackendPostgres, + testCase "invalid value fails to parse" $ + case eitherDecode "\"sqlite\"" :: Either String SearchBackend of + Left _ -> pure () + Right _ -> assertFailure "expected invalid searchBackend to fail parsing" ] ]