feat: add authz schema pipeline and load_authz_schema command

CHANGELOG.rst

@@ -14,6 +14,51 @@ Change Log
 Unreleased
 **********
 
+Added
+=====
+
+* Added the static authorization schema, a versioned YAML format for declaring permissions,
+  permission categories, roles, and role extensions (ADR 0017). The permissions and roles that
+  ``authz.policy`` defines are now also expressed as schema files under
+  ``openedx_authz/authz/schema/``.
+* Added the schema loading pipeline in ``openedx_authz/engine/schema/``, covering the discover,
+  load, validate and compile phases of the lifecycle (ADR 0018), plus render and apply in
+  ``openedx_authz/engine/renderer.py``.
+* Added schema discovery through the ``authz.schema`` entry-point group and the
+  ``OPENEDX_AUTHZ_SCHEMA_DIRECTORIES`` setting, so applications can ship authorization definitions
+  with their code and operators can contribute them through deployment configuration (ADR 0019).
+* Added the ``load_authz_schema`` management command, the single non-interactive deployment entry
+  point, with ``--dry-run`` to print the change report without writing, ``--force`` to allow
+  removing roles that still have assignments, and repeatable ``--dir`` for CI and local runs.
+* Added ``role_extensions`` support: an application or deployment can add or remove permissions and
+  replace the display metadata or ``hidden`` flag of an existing static role without copying its
+  definition. ``priority`` resolves conflicts; an unresolvable equal-priority conflict stops the run
+  before any database change (ADR 0023).
+* Added first-class tables for compiled definitions and their provenance, in migration
+  ``0011_authz_schema_definitions``: permission categories, permission definitions, role
+  definitions, role-permission grants, schema sources, and one source-link table per definition kind
+  recording whether a contribution was a base definition or an extension (ADR 0025).
+* Added source attribution at the role-permission grain, so contributions from different
+  applications to the same role remain distinguishable and queryable through
+  ``origins_for_role``, ``origins_for_permission``, ``origins_for_category`` and
+  ``origin_for_role_permission``.
+* Added a change report before any write: the command lists the policy rows and the definitions that
+  would be added, updated or removed, including metadata-only edits that change no policy row
+  (ADR 0018 §6).
+
+Notes
+=====
+
+* No authorization behavior changes in this release. Loading ``authz.policy`` works as before, and
+  the new pipeline runs only when ``load_authz_schema`` is invoked.
+* Applying a schema is idempotent and preserves data the loader does not own: user assignments,
+  dynamic roles, legacy ``g2`` action-inheritance rows, and pre-existing policy rows that no schema
+  declares. Rows that already exist are adopted, gaining definition and source records rather than
+  being rewritten (ADR 0025 §6).
+* Removing a static role that still has user assignments stops the deployment and reports the
+  assignments. ``--force`` removes the role together with its assignments and writes a
+  ``RoleAssignmentAudit`` record for each one.
+
 1.24.0 - 2026-09-14
 *******************
 

docs/decisions/0018-authorization-schema-lifecycle.rst

@@ -87,6 +87,26 @@ An explicit force option allows an operator to remove the role and its remaining
 
 Application and plugin code cannot edit generated static rows directly. During an incident, however, an administrator may use the Django admin, which records who changed what and why, and explains whether the next schema deployment will replace the change.
 
+8. Serialize concurrent applies
+===============================
+
+Section 5 keeps the writes of a single apply atomic. This section covers what happens when two applies run at the same time — for example two deployments racing, or a deployment overlapping an administrative re-run.
+
+Apply is a read-modify-write cycle: it reads the current policy and definitions, computes the difference against the compiled schema, then writes the generated policy and the definition records. Two applies that run concurrently can each read the same starting state, each compute a difference against it, and then both write. The result is a lost update in which the generated policy and the definition records disagree — for instance the definitions retain a permission that the policy no longer grants — even though each apply individually did an atomic write.
+
+An apply therefore runs under a lock that lets only one apply proceed at a time. A second apply waits until the first finishes and then reads the state the first committed, so its difference is computed against current data rather than a stale snapshot. The lock is released automatically when the apply's transaction ends — on success, failure, or a lost connection — so a failed or interrupted apply cannot leave the lock held.
+
+Taking the lock is not sufficient on its own; the apply must also read current data once it holds the lock. Two things make that true. First, the apply reads committed database state under the lock rather than a snapshot captured before the lock was granted, because some databases fix a transaction's read view at its first read. Second, the apply refreshes any in-memory copy of the policy it keeps, so the difference is computed from what the database actually holds and not from a cached view a previous apply left behind.
+
+The signal that tells running services their cached policy is stale is updated inside the same transaction as the policy writes. If that signal were updated after the transaction, the new policy would be committed and visible while the signal still indicated the old version, and a service could serve the superseded policy for that window. Updating the signal together with the writes closes that window: a service either sees both the new policy and the new signal, or neither.
+
+This ADR records the decision to serialize applies and to read current state under the lock. It does not prescribe the specific locking primitive, isolation level, or signalling mechanism; those are implementation choices that depend on the deployment's database.
+
+Two limitations are accepted:
+
+* **Single-database assumption.** Serialization holds only when the generated policy and the lock live in the same database. The engine already assumes one database for its policy storage, so this is consistent with the rest of the system.
+* **Blocking wait without observability.** A second apply waits for the first with no timeout and no distinct signal, so a blocked apply is operationally indistinguishable from a slow one. This is acceptable because apply runs at deployment time, not on the request path.
+
 Consequences
 ************
 
@@ -97,6 +117,7 @@ Consequences
 * Applying a schema requires a database transaction or another mechanism that updates all generated rows together.
 * Operators receive the change report before the database is updated.
 * An assignment to a role being removed blocks deployment unless an operator explicitly removes the role and its assignments together.
+* Concurrent applies are serialized so they cannot interleave into a lost update, and each apply reads current committed state under the lock and updates the cache-staleness signal together with its writes.
 
 References
 **********

src/openedx_authz/engine/policy.py

@@ -199,3 +199,15 @@ def remove_grouping_row(self, row: GroupingRow) -> bool:
         than a reconstructed one.
         """
         return self._resolved_enforcer.remove_grouping_policy(*row.as_policy())
+
+    def reload(self) -> None:
+        """Reload the enforcer's in-memory model from the database (ADR 0018 §8).
+
+        The schema applier calls this right after taking the serialization lock
+        so the subsequent ``get_policy_rows`` read reflects the committed ``p``
+        rows the lock just made visible, rather than a prior run's version-gated
+        in-memory snapshot. Delegating through the store (instead of reaching for
+        the global enforcer) keeps the applier's "never hold the enforcer handle"
+        contract and lets injected fakes model the reload.
+        """
+        self._resolved_enforcer.load_policy()

src/openedx_authz/engine/renderer.py

@@ -1,4 +1,4 @@
-"""Render compiled definitions to Casbin rows and apply them (ADR 0018 §1, §5).
+"""Render compiled definitions to Casbin rows and apply them (ADR 0018 §1, §5, §8).
 
 This is the only Casbin/Django-aware part of the schema pipeline. It implements
 the ``render`` and ``apply`` lifecycle steps:
@@ -43,6 +43,7 @@
 from __future__ import annotations
 
 import logging
+from contextlib import contextmanager
 from dataclasses import dataclass, field
 
 from crum import get_current_user
@@ -56,6 +57,7 @@
 from openedx_authz.engine.schema.types import CompiledSchema, RoleDefinition
 from openedx_authz.models import schema as schema_models
 from openedx_authz.models.core import RoleAssignmentAudit
+from openedx_authz.models.engine import PolicyCacheControl
 
 logger = logging.getLogger(__name__)
 
@@ -233,32 +235,44 @@ def apply(
         invalidated so the enforcer reloads the last committed state rather than
         keeping the uncommitted in-memory rows (ADR 0018 §5).
 
+        Concurrency (ADR 0018 §8): the whole read-modify-write cycle runs inside
+        :meth:`_serialized_section`, which takes an exclusive row lock as the
+        transaction's first statement, reloads the enforcer, and — on a change —
+        bumps the policy cache version *inside* the same transaction. Two
+        concurrent ``load_authz_schema`` runs would otherwise each diff against
+        the same starting snapshot and both write, leaving the Casbin ``p`` rows
+        and the ``Authz*`` definition tables disagreeing (a lost update); the lock
+        serializes them so the second reads the already-applied state. Bumping the
+        version in-transaction also closes the reader stale window: without it the
+        policy writes commit before the version changes, so a concurrent enforcer
+        could serve stale in-memory policy against already-updated rows.
+
         Raises:
             SchemaApplyError: If the plan has blocking assignments and ``force``
                 is False.
         """
-        plan = self.plan(rendered, schema)
-
-        if plan.blocking_assignments and not force:
-            details = ", ".join(f"{role} (assigned to {subject})" for role, subject in plan.blocking_assignments)
-            raise SchemaApplyError(
-                "Refusing to proceed: static roles with existing assignments would be removed: "
-                f"{details}. Re-run with force to remove them together with their assignments."
-            )
-
         policy_store = self._policy_store
 
-        # Reconcile p rows and sync definition/source records atomically.
-        # Definitions are synced even when p rows are unchanged so metadata-only
-        # edits land and pre-existing p rows get adopted on first run.
-        #
-        # add_policy_row/remove_policy_row mutate the enforcer's in-memory model
-        # as well as the database, so a rollback would otherwise leave this
-        # process enforcing rows the database no longer has. Bumping the policy
-        # cache version on the failure path forces a reload from the committed
-        # state, keeping Casbin on the last working version (ADR 0018 §5).
         try:
-            with transaction.atomic():
+            with self._serialized_section():
+                plan = self.plan(rendered, schema)
+
+                # Invariant: this gate must stay *before* any add_policy_row/
+                # remove_policy_row below. SchemaApplyError is caught separately
+                # and re-raised without invalidating the policy cache, which is
+                # only safe while nothing has mutated the in-memory enforcer. If a
+                # future change needs to raise SchemaApplyError after a write, it
+                # must invalidate the cache (like the generic except below) so the
+                # process does not keep enforcing rolled-back rows.
+                if plan.blocking_assignments and not force:
+                    details = ", ".join(
+                        f"{role} (assigned to {subject})" for role, subject in plan.blocking_assignments
+                    )
+                    raise SchemaApplyError(
+                        "Refusing to proceed: static roles with existing assignments would be removed: "
+                        f"{details}. Re-run with force to remove them together with their assignments."
+                    )
+
                 for row in plan.added_rows:
                     policy_store.add_policy_row(row)
                 for row in plan.removed_rows:
@@ -267,20 +281,39 @@ def apply(
                 if force and plan.blocking_assignments:
                     removed_assignments = self._remove_assignments(policy_store, plan.blocking_assignments)
                 self._store_sources(schema)
+
+                changed = bool(plan.added_rows or plan.removed_rows)
+                if changed:
+                    # Bump the cache version *inside* the transaction so it commits
+                    # atomically with the p-row writes. If it ran after the block
+                    # (as it once did), the rows would be committed-and-visible
+                    # while the version still read old, and a concurrent enforcer
+                    # would skip the reload and serve stale policy for that window.
+                    AuthzEnforcer.invalidate_policy_cache()
+
                 # Emit the audit events only if the transaction commits, mirroring
                 # unassign_role_from_subject_in_scope, so no audit row is written for
                 # an assignment removal that gets rolled back.
                 if removed_assignments:
                     transaction.on_commit(lambda: self._emit_assignment_deleted(removed_assignments))
+        except SchemaApplyError:
+            # The force gate fired before any write (see the invariant above):
+            # nothing was mutated and the cache does not need bumping. Re-raise
+            # without the failure handling below, which exists for partial-write
+            # rollbacks.
+            raise
         except Exception:
-            # Runs outside the rolled-back block, so the new version commits.
+            # The transaction rolled back the database, but add_policy_row/
+            # remove_policy_row already mutated the enforcer's in-memory model, so
+            # this process would keep enforcing rows the database no longer has.
+            # The in-transaction bump above was rolled back too, so bump again
+            # here — outside the rolled-back block, where it commits — to force a
+            # reload from the committed (pre-apply) state (ADR 0018 §5).
             AuthzEnforcer.invalidate_policy_cache()
             logger.exception("Authz schema apply failed; policy cache invalidated to force a reload.")
             raise
 
-        changed = bool(plan.added_rows or plan.removed_rows)
-        if changed:
-            AuthzEnforcer.invalidate_policy_cache()
+        if plan.added_rows or plan.removed_rows:
             logger.info(
                 "Authz schema apply: added %d p row(s), removed %d p row(s).",
                 len(plan.added_rows),
@@ -296,6 +329,52 @@ def apply(
             plan=plan,
         )
 
+    @contextmanager
+    def _serialized_section(self):
+        """Open the serialized critical section for an apply (ADR 0018 §8).
+
+        Owns the concurrency contract so :meth:`apply` reads as plain
+        read-diff-write. On entry it, in order:
+
+        1. ``ensure_exists`` the lock row **before** the transaction opens — a
+           creating ``SELECT``/INSERT inside the transaction would be its first
+           read and, under InnoDB Repeatable Read, pin the consistent-read
+           snapshot before the lock is taken, leaving every post-lock read stale.
+        2. opens ``transaction.atomic()``;
+        3. takes the exclusive row lock as the transaction's first read, so a
+           second runner blocks until the first commits and this runner's reads
+           see the committed state;
+        4. reloads the enforcer, because the lock only makes the *database*
+           current — the version-gated in-memory model would otherwise still hold
+           a prior run's rows, so the ``plan`` read of ``get_policy`` could be
+           stale even with the lock held.
+
+        The caller does its read-diff-write inside the ``with`` block. The lock
+        (a row lock) releases automatically when the transaction commits, rolls
+        back, or the connection dies, so no stale lock can outlive a crashed
+        apply. Keeping the lock, reload, and the cache-version bump the caller
+        performs all within this one transaction is what makes the critical
+        section cohesive and leaves no committed-but-unversioned reader window.
+
+        Two accepted limitations, deferred (ADR 0018 §8):
+
+        * ``PolicyCacheControl`` (the lock) and the ``transaction.atomic()``
+          default to the ``default`` database, while the Casbin ``p`` rows live
+          in ``CASBIN_DB_ALIAS``. Serialization is only meaningful while those
+          coincide, which matches the engine's existing single-database policy
+          storage assumption. If ``CASBIN_DB_ALIAS`` is ever pointed at a
+          separate database, the lock must move there too.
+        * The lock wait is unbounded and silent: a second apply blocks with no
+          ``nowait``/timeout and no log line, so a blocked apply looks like a slow
+          one. Acceptable because apply runs at deploy time, not on the request
+          path.
+        """
+        PolicyCacheControl.ensure_exists()
+        with transaction.atomic():
+            PolicyCacheControl.lock()
+            self._policy_store.reload()
+            yield
+
     @staticmethod
     def _remove_assignments(
         policy_store: PolicyStore, blocking_assignments: list[tuple[str, str]]