diff --git a/.gitattributes b/.gitattributes index 17c3856c4..601478bbd 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,17 +1,9 @@ * text=auto eol=lf -# The rsync-web/ subdirectory holds the project website source content -# (mirrors what gets pushed to https://rsync.samba.org). Exclude it from -# `git archive` output so the release source tarball produced by -# packaging/release.py step_7_tarball does not bloat with HTML the -# tarball doesn't need. +# Website content is published separately from the release source archive. /rsync-web/ export-ignore -# old_versions/ holds static binaries of historical rsync releases, used by the -# version-mixing test suite (.github/workflows/ubuntu-version-mix.yml) to run -# the current code against a real old peer over the daemon / remote-shell. -# Mark the binaries as binary so the `text=auto eol=lf` rule above can't try to -# normalise line endings and corrupt them; export-ignore keeps them out of the -# release source tarball. +# Archived peers must retain their bytes and stay out of release archives. /old_versions/rsync_* binary /old_versions/rsync_* export-ignore +/old_versions/binaries.json export-ignore diff --git a/.github/INTEROPERABILITY.md b/.github/INTEROPERABILITY.md index c4d23ff9f..d0f9d86a7 100644 --- a/.github/INTEROPERABILITY.md +++ b/.github/INTEROPERABILITY.md @@ -1,26 +1,32 @@ -# Network checks +# Cross-platform transfers -CI / Interoperability contains Linux, FreeBSD, NetBSD, OpenBSD, macOS, Solaris and Cygwin endpoints. Linux, macOS and Cygwin execute on hosted runners; the BSD and Solaris binaries execute inside their guests. Guest forwarding exposes fixture ports only. Administrative SSH stays local to the VM host. +We run Linux, FreeBSD, NetBSD, OpenBSD, macOS, Solaris and Cygwin endpoints. Linux, macOS and Cygwin run directly on hosted runners; BSD and Solaris run inside guests. We expose only the fixture ports through Tailscale and keep administrative SSH local to each guest host. -The 168 cases cover every directed platform pair, push and pull initiation and SSH and daemon transport. Separate trees prevent concurrent cases from sharing destinations. Initial copies finish before verification and source updates; checksum updates finish before destination verification. Results include data, modes, integer timestamps, symlink targets and hard-link groups. +We run push and pull transfers over SSH and daemon connections for every ordered platform pair, giving 168 data cases. We use separate trees so concurrent cases never share a destination. The first phase verifies the initial transfer. The second verifies checksum updates and deletion. We compare data, modes, integer timestamps, symlink targets and hard-link groups. -The parent CI run and attempt identify each cohort. The tested revision comes from GitHub's resolved reusable-workflow commit and is checked against the native checkout. Endpoint records also contain the operating system, effective UID/GID and executable SHA-256. Required depends on all seven endpoint jobs, complete case results and confirmed network cleanup. +We test compatible POSIX ACLs and xattrs separately. We return fake-super backups unchanged with `-XX` and decode them only on their originating platform. We require explicit errors for unsupported metadata and empty NTFS xattrs. + +We bind every record to the parent CI run, attempt and tested revision. Endpoint records include the operating system, effective UID, effective GID and executable SHA-256. We accept the final result only after all seven endpoints complete every assigned case and we confirm network cleanup. Artefact names end in `attempt-N`. Context and lease artefacts use the controller attempt; other interoperability artefacts use the parent CI attempt. The suffix prevents a rerun from consuming records from an earlier attempt. +We publish each phase result before waiting at the barrier, which keeps a failed endpoint available until its peers finish the current phase. We block later phases but still run local cleanup. A missing, failed or incomplete endpoint result fails the final check. + ## Authentication -The Network controller uses trusted default-branch code through a workflow_run listener. It is deployed independently of the PR endpoint workflow and ignores CI runs which have not selected that workflow. Its existing OIDC identity has write access to auth_keys, policy_file, devices:core and devices:posture_attributes in the dedicated test tailnet. The posture scope is a dependency of policy-file writes. Enrolment is restricted to tag:quarantine. +The Network controller runs trusted default-branch code through a `workflow_run` listener and ignores CI runs that did not select the endpoint workflow. We configure the dedicated test tailnet outside this repository. The controller identity needs write access to `auth_keys`, `policy_file`, `devices:core` and `devices:posture_attributes`. We restrict enrolment to `tag:quarantine`. + +The controller checks CI and endpoint liveness before obtaining authority, requesting keys and publishing enrolment. -Endpoint jobs have read-only GitHub permissions and no OIDC permission. Each generates a temporary RSA recipient and receives one encrypted, single-use enrolment key. RSA-OAEP uses 3072-bit keys and SHA-256. Key files and recipient private keys are removed after enrolment. SSH private keys are removed when fixture services stop. Plaintext credentials are excluded from artifacts and job outputs. +We give endpoint jobs read-only GitHub permissions and no OIDC permission. Each endpoint generates a temporary RSA recipient and receives one encrypted, single-use enrolment key. RSA-OAEP uses 3072-bit keys and SHA-256. We remove the enrolment files immediately and remove SSH private keys when fixture services stop. We never include plaintext credentials in artefacts or job output. -Endpoints authenticate through the supported Tailscale CLI. Client package checksums and the macOS source revision are pinned in interop-client.py. Route acceptance and DNS changes are disabled and checked through the client's active preferences. SSH host keys are pinned to endpoint identities. +We pin the Tailscale CLI in `interop-client.py` and pin SSH host keys to endpoint identities. We disable route acceptance and DNS changes and verify the active preferences. -Admission permits only the cohort's registered IPv4 addresses on TCP ports 18731 and 18733. Conditional policy updates preserve other cohorts. Local service checks precede and follow denial probes. Key expiry limits unused enrolment credentials; active nodes require explicit removal. +We admit only the cohort's registered IPv4 addresses on TCP ports 18731 and 18733. Policy updates preserve other cohorts. We check local services before and after the denial probes. Enrolment keys expire; we remove active nodes explicitly. ## Configuration -The production federated identity uses these claims: +We configure the controller identity with these claims: Field | Value --- | --- @@ -32,14 +38,14 @@ event_name | workflow_run workflow_ref | RsyncProject/rsync/.github/workflows/interoperability*.yml@refs/heads/master runner_environment | github-hosted -Endpoint jobs have no federated identity. Controller discovery uses the parent CI run and attempt and verifies the controller's default-branch revision. No per-PR identity, branch or repository variable is required. The controller checks CI and endpoint liveness before obtaining authority, requesting keys and publishing enrolment. +We give endpoint jobs no federated identity. We find the controller from the parent CI run and attempt. We then verify its default-branch revision. We do not need a per-PR identity, branch or repository variable. ## Recovery -Normal cleanup revokes the recorded grant, removes owned devices and revokes issued keys. An individual failure does not prevent attempts to remove the remaining resources. Device deletion checks the recorded node identity; reassigned devices remain untouched. +We always try to revoke the cohort grant, remove its devices and revoke its enrolment keys. We remove a device only while its current node ID matches the lease. If a run fails before any key request, we record that no resources need cleanup without marking transfers as passed. -Keys created before enrolment delivery can be revoked from the local journal without an admission lease. An interrupted request with an unknown key ID or enrolment delivered without a lease requires reconciliation. A failed transfer remains failed even when resource cleanup succeeds. +Keys created before enrolment delivery can be revoked from the local journal without an admission lease. An interrupted request with an unknown key ID or enrolment delivered without a lease requires reconciliation. -Admission leases are attested before access is granted. Recovery verifies the original lease bytes, repository, controller workflow, signing commit, source ref and run attempt before modifying the tailnet. Failed or cancelled controller runs trigger recovery from trusted code. +We attest the admission lease before granting access. A failed or cancelled controller triggers the cleanup workflow. We first check whether cleanup already succeeded. Otherwise we verify the lease bytes, repository, controller workflow, signing commit, source reference and run attempt before changing the tailnet. -Missing leases and provider failures require reconciliation. Partial enrolment before a lease exists remains quarantined; hostnames alone are insufficient grounds for device deletion. Cleanup failure keeps the CI result unsuccessful. +If automated cleanup fails, we use the attested `interoperability-lease-attempt-N` artefact from the controller run to identify the cohort. We remove only the recorded grant, keys and devices while each device's node ID still matches. Without an attested lease, we compare the `interoperability-ready-PLATFORM-attempt-N` artefacts with quarantined node IDs before making any change. A hostname alone is not enough to delete a device. diff --git a/.github/actions/interoperability/action.yml b/.github/actions/interoperability/action.yml new file mode 100644 index 000000000..1dc8ac198 --- /dev/null +++ b/.github/actions/interoperability/action.yml @@ -0,0 +1,216 @@ +name: Directed transfers +description: Native SSH and daemon transfers between seven endpoints +inputs: + platform: + required: true + description: Native endpoint + guest: + default: 'false' + description: Use the existing local VM SSH connection +outputs: + blocked: + description: A shared phase could not run + value: ${{ steps.outcome.outputs.blocked }} +runs: + using: composite + steps: + - name: Install Tailscale + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py install "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Generate enrolment recipient + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py recipient "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Upload recipient + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-public-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/public-key.json + if-no-files-found: error + retention-days: 7 + - name: Join quarantine + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py connect "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Start native fixture services + if: env.INTEROP_BLOCKED != 'true' + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py prepare "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Upload endpoint identity + if: env.INTEROP_BLOCKED != 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-ready-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/ready.json + if-no-files-found: error + retention-days: 7 + - name: Check quarantine + if: env.INTEROP_BLOCKED != 'true' + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py quarantine "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Upload quarantine result + if: env.INTEROP_BLOCKED != 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-quarantine-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/quarantine.json + if-no-files-found: error + retention-days: 7 + - name: Check admission + if: env.INTEROP_BLOCKED != 'true' + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py admit "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Upload endpoint admission + if: env.INTEROP_BLOCKED != 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-admitted-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/admitted.json + if-no-files-found: error + retention-days: 7 + - name: Initial transfers + id: transfer1 + if: env.INTEROP_BLOCKED != 'true' + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py transfer1 "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Upload initial transfer completion + id: transfer1_upload + if: ${{ always() && !cancelled() && env.INTEROP_BLOCKED != 'true' && (steps.transfer1.outcome == 'success' || steps.transfer1.outcome == 'failure') }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-transfer1-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/transfer1.json + if-no-files-found: error + retention-days: 7 + - name: Complete initial transfer phase + if: ${{ always() && !cancelled() && steps.transfer1_upload.outcome == 'success' }} + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + run: python3 -B -S .github/scripts/interop-host.py wait1 "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Verify initial data and update sources + if: env.INTEROP_BLOCKED != 'true' + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py next "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Upload update readiness + if: env.INTEROP_BLOCKED != 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-next-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/next.json + if-no-files-found: error + retention-days: 7 + - name: Checksum updates and deletion + id: transfer2 + if: env.INTEROP_BLOCKED != 'true' + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py transfer2 "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Upload update completion + id: transfer2_upload + if: ${{ always() && !cancelled() && env.INTEROP_BLOCKED != 'true' && (steps.transfer2.outcome == 'success' || steps.transfer2.outcome == 'failure') }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-transfer2-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/transfer2.json + if-no-files-found: error + retention-days: 7 + - name: Complete updated transfer phase + if: ${{ always() && !cancelled() && steps.transfer2_upload.outcome == 'success' }} + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + run: python3 -B -S .github/scripts/interop-host.py wait2 "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Verify updated native data + if: env.INTEROP_BLOCKED != 'true' + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py verify "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Upload transfer result + if: env.INTEROP_BLOCKED != 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-result-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/result.json + if-no-files-found: error + retention-days: 7 + - name: Check revocation + if: env.INTEROP_BLOCKED != 'true' + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py revoked "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Upload revocation result + if: env.INTEROP_BLOCKED != 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-revoked-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/revoked.json + if-no-files-found: error + retention-days: 7 + - name: Upload blocked phase + if: ${{ always() && env.INTEROP_BLOCKED == 'true' }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: interoperability-blocked-${{ inputs.platform }}-attempt-${{ github.run_attempt }} + path: ${{ runner.temp }}/interoperability/blocked.json + if-no-files-found: error + retention-days: 7 + - name: Stop fixture services and leave the network + if: always() + shell: bash + env: + GH_TOKEN: ${{ github.token }} + INTEROP_PLATFORM: ${{ inputs.platform }} + INTEROP_GUEST: ${{ inputs.guest }} + run: python3 -B -S .github/scripts/interop-host.py stop "$INTEROP_PLATFORM" --root "$RUNNER_TEMP/interoperability" + - name: Record endpoint outcome + id: outcome + if: always() + shell: bash + run: | + python3 -B -S - <<'PY' + import os + with open(os.environ['GITHUB_OUTPUT'], 'a') as output: + output.write('blocked=' + os.environ.get('INTEROP_BLOCKED', 'false') + '\n') + PY diff --git a/.github/scripts/android-emulator.py b/.github/scripts/android-emulator.py new file mode 100644 index 000000000..c7abc9995 --- /dev/null +++ b/.github/scripts/android-emulator.py @@ -0,0 +1,183 @@ +import argparse +from contextlib import redirect_stdout +import hashlib +import json +import os +from pathlib import Path +import re +import shutil +import signal +import subprocess +import sys +import tempfile +import time + +from android import check_transfer as transfer, quote_command, run +from termux import check_transfer as check_termux + + +def wait_for_boot(adb, proc, timeout=300): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if proc.poll() is not None: + raise RuntimeError('Emulator exited before boot completed') + try: + if adb('shell', 'getprop sys.boot_completed', capture_output=True, timeout=10).stdout.strip() == '1': + return + except (subprocess.CalledProcessError, subprocess.TimeoutExpired): + pass + time.sleep(2) + raise TimeoutError('Android boot did not complete') + + +def stop_emulator(adb, proc): + if proc.poll() is not None: + return + try: + adb('emu', 'kill', timeout=10, capture_output=True) + proc.wait(timeout=20) + return + except subprocess.SubprocessError: + proc.terminate() + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait(timeout=10) + + +def check_device(adb): + values = {} + commands = {'api': ('getprop', 'ro.build.version.sdk'), + 'abi': ('getprop', 'ro.product.cpu.abi'), + 'emulator': ('getprop', 'ro.kernel.qemu'), + 'selinux': ('getenforce',), 'uid': ('id', '-u'), + 'kernel': ('uname', '-r'), 'fingerprint': ('getprop', 'ro.build.fingerprint')} + for name, command in commands.items(): + values[name] = adb('shell', quote_command(command), capture_output=True).stdout.strip() + for name, expected in dict(api='36', abi='x86_64', emulator='1', selinux='Enforcing', uid='2000').items(): + if values[name] != expected: + raise ValueError(f'Android {name}: expected {expected}, got {values[name]!r}') + print(json.dumps(values, indent=2), flush=True) + return values + + +def check_transfer(adb, binary, output): + device = check_device(adb) + work = adb('shell', 'mktemp -d /data/local/tmp/rsync-ci.XXXXXXXX', capture_output=True).stdout.strip() + if not re.fullmatch(r'/data/local/tmp/rsync-ci\.[A-Za-z0-9]{8}', work): + raise ValueError('Invalid Android scratch directory') + try: + adb('push', str(binary), f'{work}/{binary.name}', timeout=60) + digest = hashlib.sha256(binary.read_bytes()).hexdigest() + remote = adb('shell', quote_command(['sha256sum', f'{work}/{binary.name}']), + capture_output=True).stdout.split()[0] + if digest != remote: + raise ValueError('Android binary hash does not match the candidate') + adb('shell', quote_command(['chmod', '700', f'{work}/{binary.name}'])) + def command(*args, **kwargs): + return adb('shell', '-T', quote_command(args), **kwargs) + with tempfile.TemporaryDirectory(prefix='rsync-android-data-') as directory: + local = Path(directory) / 'data' + def write_file(path, data): + local.write_bytes(data) + adb('push', str(local), path, capture_output=True) + def read_file(path): + adb('pull', path, str(local), capture_output=True) + return local.read_bytes() + with (output / 'transfer.log').open('w') as log, redirect_stdout(log): + transfer(command, f'{work}/{binary.name}', work, hardlinks=False, + write_file=write_file, read_file=read_file) + check_permissions(adb, binary, work, output) + receipt = {'device': device, 'binary_sha256': digest, + 'source': run('git', 'rev-parse', 'HEAD', capture_output=True).stdout.strip()} + (output / 'receipt.json').write_text(json.dumps(receipt, indent=2) + '\n') + finally: + failed = sys.exc_info()[0] is not None + try: + adb('shell', quote_command(['rm', '-rf', work])) + except (OSError, subprocess.SubprocessError) as error: + if not failed: + raise + print(f'Android scratch cleanup failed: {error}', file=sys.stderr) + + +def check_permissions(adb, binary, work, output): + with tempfile.TemporaryDirectory(prefix='rsync-unreadable-') as directory: + private = Path(directory) / 'private' + private.write_text('must not be copied\n') + adb('shell', quote_command(['mkdir', f'{work}/unreadable', f'{work}/refused'])) + adb('push', str(private), f'{work}/unreadable/private') + adb('shell', quote_command(['chmod', '000', f'{work}/unreadable/private'])) + result = adb('shell', quote_command([f'{work}/{binary.name}', '-r', f'{work}/unreadable/', + f'{work}/refused/']), capture_output=True, check=False) + (output / 'permissions.log').write_text(result.stdout + result.stderr) + if result.returncode != 23 or 'Permission denied' not in result.stderr: + raise ValueError('Unreadable Android source did not fail with permission denied and exit 23') + adb('shell', quote_command(['test', '!', '-e', f'{work}/refused/private'])) + adb('shell', 'test -d /data/system') + adb('shell', 'test ! -r /data/system') + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('--binary', type=Path, default=Path('rsync')) + parser.add_argument('--output', type=Path, default=Path('test-results/android-emulator')) + parser.add_argument('--termux', action='store_true', help='Run in the Termux application sandbox') + args = parser.parse_args() + binary = args.binary.resolve(strict=True) + args.output.mkdir(parents=True) + sdk = Path(os.environ['ANDROID_HOME']) + os.environ['PATH'] = os.pathsep.join(str(sdk / path) for path in ( + 'platform-tools', 'emulator', 'cmdline-tools/latest/bin')) + os.pathsep + os.environ.get('PATH', '') + image = sdk / 'system-images/android-36/google_apis/x86_64/source.properties' + properties = image.read_text() + if 'Pkg.Revision=7' not in properties.splitlines(): + raise ValueError('Expected Android API 36 Google APIs X86-64 image revision 7') + (args.output / 'image.properties').write_text(properties) + emulator = shutil.which('emulator') + if not emulator: + raise ValueError('Android emulator is not on PATH') + version = run(emulator, '-version', capture_output=True, check=False) + (args.output / 'emulator-version.log').write_text(version.stdout + version.stderr) + if version.returncode: + raise ValueError(f'Emulator version probe failed ({version.returncode}): {version.stdout}{version.stderr}') + run(emulator, '-accel-check') + serial = 'emulator-5560' + def adb(*command, **kwargs): + return run('adb', '-s', serial, *command, **kwargs) + if serial in run('adb', 'devices', capture_output=True).stdout.split(): + raise ValueError(f'{serial} is already in use') + def cancel(signum, frame): + raise SystemExit(128 + signum) + previous = {sig: signal.signal(sig, cancel) for sig in (signal.SIGINT, signal.SIGTERM)} + try: + with tempfile.TemporaryDirectory(prefix='rsync-avd-') as directory: + env = {**os.environ, 'ANDROID_AVD_HOME': directory} + run('avdmanager', 'create', 'avd', '--name', 'rsync', '--package', + 'system-images;android-36;google_apis;x86_64', input='no\n', env=env, timeout=60) + with (args.output / 'emulator.log').open('w') as log: + proc = subprocess.Popen([emulator, '-avd', 'rsync', '-port', '5560', '-accel', 'on', + '-no-window', '-no-audio', '-no-boot-anim', '-no-snapshot', + '-gpu', 'swiftshader', '-memory', '2048', '-cores', '2'], + env=env, stdout=log, stderr=subprocess.STDOUT) + try: + wait_for_boot(adb, proc) + if args.termux: + check_termux(adb, binary, args.output, check_device(adb)) + else: + check_transfer(adb, binary, args.output) + finally: + stop_emulator(adb, proc) + finally: + for sig, handler in previous.items(): + signal.signal(sig, handler) + print((args.output / 'transfer.log').read_text(), end='') + + +if __name__ == '__main__': + try: + main() + except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as error: + print(f'Android emulator check failed: {error}', file=sys.stderr) + raise SystemExit(1) diff --git a/.github/scripts/android.py b/.github/scripts/android.py new file mode 100644 index 000000000..c7717544d --- /dev/null +++ b/.github/scripts/android.py @@ -0,0 +1,149 @@ +import argparse +import hashlib +import os +from pathlib import Path +import re +import shlex +import shutil +import subprocess +import sys +import tempfile + +def quote_command(arguments): + return ' '.join(map(shlex.quote, arguments)) + + +def run(*command, timeout=30, **kwargs): + kwargs.setdefault('check', True) + kwargs.setdefault('text', True) + return subprocess.run(command, timeout=timeout, **kwargs) + + +def check_transfer(command, binary, work, executor=(), hardlinks=True, write_file=None, read_file=None): + source, destination = f'{work}/source', f'{work}/destination' + data_path, delta_path = f'{source}/dir/data', f'{source}/dir/delta' + data = b'initial data\n' + delta = b''.join(f'{i:08d}:0123456789abcdef0123456789abcdef\n'.encode() for i in range(8192)) + + if write_file is None: + def write_file(path, data): + command('tee', path, input=data, text=False, stdout=subprocess.DEVNULL) + if read_file is None: + def read_file(path): + return command('cat', path, capture_output=True, text=False).stdout + + def stat(path, field): + return command('stat', '-c', field, path, capture_output=True).stdout.strip() + + command('mkdir', '-p', f'{source}/dir', destination) + write_file(data_path, data) + write_file(delta_path, delta) + command('chmod', '640', data_path) + linked = command('ln', data_path, f'{source}/dir/hard-link', capture_output=True, check=False) + if hardlinks: + linked.check_returncode() + elif linked.returncode == 0 or 'Permission denied' not in linked.stderr: + raise ValueError('Expected Android hard-link creation to be denied') + command('ln', '-s', 'data', f'{source}/dir/symbolic-link') + write_file(f'{source}/obsolete', b'remove me\n') + print(command(*executor, binary, '--version', capture_output=True).stdout.splitlines()[0], flush=True) + for updated in (False, True): + options = ['-aH'] + if updated: + data = b'updated Android data\n' + change = b'changed delta block' + delta = delta[:65536] + change + delta[65536 + len(change):] + write_file(data_path, data) + write_file(delta_path, delta) + command('chmod', '600', data_path) + command('rm', f'{source}/obsolete') + options += ['--delete', '--no-whole-file', '--stats'] + command('touch', '-t', '200201020304.05' if updated else '200101020304.05', data_path, delta_path) + result = command('env', 'LC_ALL=C', *executor, binary, *options, source + '/', destination + '/', + capture_output=True, check=False, timeout=120) + if result.returncode: + raise ValueError(f'Android transfer failed ({result.returncode}): {result.stdout}{result.stderr}') + for name, expected in (('data', data), ('delta', delta)): + path = f'{destination}/dir/{name}' + received = read_file(path) + if received != expected: + raise ValueError(f'Android transfer changed {name}: expected {expected[:32]!r}, got {received[:32]!r}') + if stat(f'{source}/dir/{name}', '%Y') != stat(path, '%Y'): + raise ValueError(f'Android transfer lost the {name} timestamp') + if stat(f'{destination}/dir/data', '%a') != ('600' if updated else '640'): + raise ValueError('Android transfer lost file permissions') + if hardlinks and stat(f'{destination}/dir/data', '%i') != stat(f'{destination}/dir/hard-link', '%i'): + raise ValueError('Android transfer lost a hard link') + if command('readlink', f'{destination}/dir/symbolic-link', capture_output=True).stdout.strip() != 'data': + raise ValueError('Android transfer changed a symlink') + if updated: + command('test', '!', '-e', f'{destination}/obsolete') + counts = {} + for name in ('Matched', 'Literal'): + match = re.search(rf'^{name} data: ([0-9,]+) bytes$', result.stdout, re.MULTILINE) + if not match: + raise ValueError(f'Missing Android {name.lower()} data statistics: {result.stdout}') + counts[name] = int(match[1].replace(',', '')) + if counts['Matched'] <= 0 or not 0 < counts['Literal'] < len(delta): + raise ValueError(f'Android update did not use the delta algorithm: {counts}') + print(f'Delta: {counts["Matched"]} matched bytes, {counts["Literal"]} literal bytes', flush=True) + print('Android transfer passed', flush=True) + + +def restore_binary(directory, abi, destination): + binaries = list(directory.glob(f'rsync-*-android-{abi}')) + if len(binaries) != 1: + raise ValueError('Expected one Android binary') + binary = binaries[0] + data = binary.read_bytes() + digest = hashlib.sha256(data).hexdigest() + if binary.with_name(binary.name + '.sha256').read_text() != f'{digest} {binary.name}\n': + raise ValueError('Android binary checksum does not match') + with destination.open('xb') as stream: + stream.write(data) + destination.chmod(0o755) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('task', choices=['test', 'package', 'restore']) + parser.add_argument('--executor') + parser.add_argument('--abi', choices=['arm64-v8a', 'armeabi-v7a', 'x86_64']) + args = parser.parse_args() + if args.task == 'test': + description = subprocess.check_output(['file', 'rsync'], text=True, timeout=30) + print(description, end='') + if 'statically linked' not in description or 'dynamically linked' in description: + raise ValueError('Android binary is not statically linked') + if not args.executor: + parser.error('--executor is required for Bionic testing') + executor = [] if args.executor == 'direct' else [args.executor] + with tempfile.TemporaryDirectory(prefix='rsync-android-') as work: + check_transfer(run, str(Path('rsync').resolve()), work, executor) + return 0 + if not args.abi: + parser.error('--abi is required for packaging or restoring a binary') + if args.task == 'restore': + restore_binary(Path('dist'), args.abi, Path('rsync')) + return 0 + match = re.search(r'RSYNC_VERSION\s+"([A-Za-z0-9.+-]+)"', Path('version.h').read_text()) + if not match: + raise ValueError('Cannot determine the rsync version') + name = f'rsync-{match[1]}-android-{args.abi}' + output = Path('dist') + output.mkdir(exist_ok=True) + binary = output / name + shutil.copy2('rsync', binary) + digest = hashlib.sha256(binary.read_bytes()).hexdigest() + (output / f'{name}.sha256').write_text(f'{digest} {name}\n') + with open(os.environ['GITHUB_ENV'], 'a') as stream: + stream.write(f'ARTIFACT_NAME=rsync-android-{args.abi}\n') + return 0 + + +if __name__ == '__main__': + try: + raise SystemExit(main()) + except (OSError, ValueError, subprocess.SubprocessError) as error: + print(f'Android check failed: {error}', file=sys.stderr) + raise SystemExit(1) diff --git a/.github/scripts/archive.py b/.github/scripts/archive.py new file mode 100644 index 000000000..d2e800de3 --- /dev/null +++ b/.github/scripts/archive.py @@ -0,0 +1,173 @@ +import argparse +import hashlib +import json +import os +from pathlib import Path, PurePosixPath +import re +import shutil +import subprocess +import sys +import tarfile +import tempfile + +sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'packaging')) +from source_archive import GEN_FILES, make_archive + + +def run(*args, **kwargs): + return subprocess.run(args, check=True, timeout=900, **kwargs) + + +def create(archive): + version = re.search(r'#define RSYNC_VERSION "([A-Za-z0-9.+-]+)"', Path('version.h').read_text()) + if not version: + raise ValueError('Cannot read release version') + with tempfile.TemporaryDirectory(prefix='rsync-release-') as directory: + stage = Path(directory) / ('rsync-' + version[1]) + stage.mkdir() + for name in GEN_FILES: + shutil.copy2(name, stage / name) + make_archive('HEAD', stage, archive) + + +def qualify(archive, output): + digest = hashlib.sha256(archive.read_bytes()).hexdigest() + with tempfile.TemporaryDirectory(prefix='rsync-archive-test-') as directory: + root = Path(directory) + root.chmod(0o755) + with tarfile.open(archive) as source: + members = source.getmembers() + tops = {member.name.split('/')[0] for member in members} + if len(tops) != 1 or not next(iter(tops)).startswith('rsync-'): + raise ValueError('Invalid release archive root') + if any(member.uid or member.gid for member in members): + raise ValueError('Release archive contains non-root ownership') + release_root = root / next(iter(tops)) + for member in members: + parts = PurePosixPath(member.name).parts + if PurePosixPath(member.name).is_absolute() or '..' in parts or '.git' in parts or '.github' in parts: + raise ValueError('Invalid release archive entry') + path = (root / member.name).resolve() + if path != release_root and release_root not in path.parents: + raise ValueError('Archive entry escapes extraction directory') + if member.isdir(): + path.mkdir(parents=True, exist_ok=True) + elif member.isfile(): + path.parent.mkdir(parents=True, exist_ok=True) + with source.extractfile(member) as incoming, path.open('xb') as target: + shutil.copyfileobj(incoming, target) + elif member.issym() or member.islnk(): + target = ((path.parent if member.issym() else root) / member.linkname).resolve() + if target != release_root and release_root not in target.parents: + raise ValueError('Archive link escapes extraction directory') + if member.issym(): + path.symlink_to(member.linkname) + else: + os.link(target, path) + else: + raise ValueError('Unsupported release archive entry') + if not member.issym(): + path.chmod(member.mode) + os.utime(path, (member.mtime, member.mtime), follow_symlinks=False) + source = root / tops.pop() + for name in GEN_FILES: + if not (source / name).is_file() or not (source / name).stat().st_size: + raise ValueError(f'Missing generated release file: {name}') + for name in ('configure', 'configure.sh', 'rsync-ssl', 'support/rrsync'): + if not os.access(source / name, os.X_OK): + raise ValueError(f'Release file is not executable: {name}') + blocked = root / 'blocked' + blocked.mkdir() + for name in ('git', 'autoconf', 'autoheader', 'automake', 'aclocal'): + tool = blocked / name + tool.write_text(f'#!{sys.executable}\nraise SystemExit("Unexpected build dependency: {name}")\n') + tool.chmod(0o755) + env = {**os.environ, 'PATH': str(blocked) + os.pathsep + os.environ['PATH'], + 'RSYNC_TEST_TMP': '/dev/shm'} + build, prefix = root / 'build', root / 'installed' + build.mkdir() + with (output / 'build.log').open('w') as log: + run(str(source / 'configure'), '--with-rrsync', '--disable-md2man', '--prefix=' + str(prefix), + cwd=build, env=env, stdout=log, stderr=subprocess.STDOUT) + run('make', '-j2', 'check-progs', cwd=build, env=env, stdout=log, stderr=subprocess.STDOUT) + privilege = 'root' if os.geteuid() == 0 else 'nonroot' + for transport in ('pipe', 'tcp'): + profiles = f'non-asan,{privilege},self-peer,linux,no-userns,archive' + args = [] if transport == 'pipe' else ['--use-tcp', '--daemon-tests-only'] + if transport == 'pipe': + profiles = 'pipe,' + profiles + results = root / 'results' / transport + try: + with (output / f'{transport}.log').open('w') as log: + run(sys.executable, '-B', '-S', str(source / 'testsuite/tools/check.py'), '.', './rsync', + profiles, str(results), *args, '-j4', cwd=build, env=env, + stdout=log, stderr=subprocess.STDOUT) + finally: + retained = output / transport + retained.mkdir() + for name in ('receipt.json', 'run.log'): + if (results / name).is_file(): + shutil.copy2(results / name, retained / name) + run('make', 'install', cwd=build, env=env) + required = ('bin/rsync', 'bin/rsync-ssl', 'bin/rrsync', 'share/man/man1/rsync.1', + 'share/man/man1/rsync-ssl.1', 'share/man/man1/rrsync.1', 'share/man/man5/rsyncd.conf.5') + for name in required: + if not (prefix / name).is_file() or not (prefix / name).stat().st_size: + raise ValueError(f'Missing installed release file: {name}') + data = root / 'data' + data.mkdir() + (data / 'file').write_bytes(b'installed rsync\n') + destination = root / 'copy' + run(str(prefix / 'bin/rsync'), '-a', str(data) + '/', str(destination)) + if (destination / 'file').read_bytes() != (data / 'file').read_bytes(): + raise ValueError('Installed rsync changed file contents') + rrsync = prefix / 'bin/rrsync' + configured, count = re.subn(r'(?m)^RSYNC = .+$', + lambda match: f'RSYNC = {str(prefix / "bin/rsync")!r}', + rrsync.read_text(), count=1) + if count != 1: + raise ValueError('Cannot configure installed rrsync executable') + rrsync.write_text(configured) + remote = root / 'remote' + remote.write_text(f'#!{sys.executable}\nimport os, shlex, sys\n' + 'os.environ["SSH_ORIGINAL_COMMAND"] = " ".join(map(shlex.quote, sys.argv[2:]))\n' + f'os.execv({str(prefix / "bin/rrsync")!r}, ["rrsync", {str(data)!r}])\n') + remote.chmod(0o755) + installed_env = {**env, 'PATH': str(prefix / 'bin') + os.pathsep + env['PATH']} + restricted = root / 'restricted' + run(str(prefix / 'bin/rsync'), '-a', '-e', str(remote), 'dummy:file', str(restricted), env=installed_env) + if restricted.read_bytes() != (data / 'file').read_bytes(): + raise ValueError('Installed rrsync changed file contents') + run(sys.executable, '-B', '-S', str(source / 'testsuite/tools/check-rsync-ssl.py'), + '--rsync', str(prefix / 'bin/rsync'), '--rsync-ssl', str(prefix / 'bin/rsync-ssl'), + '--openssl', shutil.which('openssl'), env=env) + sentinel = prefix / 'unrelated' + sentinel.write_text('retained\n') + run('make', 'uninstall', cwd=build, env=env) + remaining = {path.relative_to(prefix).as_posix() for path in prefix.rglob('*') + if path.is_file() or path.is_symlink()} + if remaining != {'unrelated'} or sentinel.read_text() != 'retained\n': + raise ValueError(f'Incorrect uninstall result: {remaining}') + (output / 'qualification.json').write_text(json.dumps({'archive_sha256': digest, 'result': 'pass'}, indent=2) + '\n') + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('task', choices=('create', 'test')) + parser.add_argument('--archive', type=Path, required=True) + parser.add_argument('--output', type=Path) + args = parser.parse_args() + archive = args.archive.resolve() + if args.task == 'create': + archive.parent.mkdir(parents=True, exist_ok=True) + create(archive) + else: + if args.output is None: + parser.error('--output is required for testing') + output = args.output.resolve() + output.mkdir(parents=True) + qualify(archive, output) + + +if __name__ == '__main__': + main() diff --git a/.github/scripts/changes.py b/.github/scripts/changes.py new file mode 100644 index 000000000..33b8b8786 --- /dev/null +++ b/.github/scripts/changes.py @@ -0,0 +1,196 @@ +import json +import os +from pathlib import Path, PurePosixPath +import re +import subprocess +import sys + + +FAMILIES = ('linux', 'bsd', 'macos', 'other', 'versions', 'android', + 'archive', 'diagnostics', 'valgrind', 'interoperability') +OUTPUTS = (*FAMILIES, 'asan', 'coverage', 'clang') +ARCHIVE_DOCUMENTS = { + 'rsync-ssl.1.md', 'rsyncd.conf.5.md', 'support/rrsync.1.md', + 'rsync.1', 'rsync.1.html', 'rsync-ssl.1', 'rsync-ssl.1.html', + 'rsyncd.conf.5', 'rsyncd.conf.5.html', 'rrsync.1', 'rrsync.1.html', +} + + +def empty(): + return {name: False for name in OUTPUTS} + + +def enable(selected, *names): + for name in names: + selected[name] = True + + +def full(): + return {name: True for name in OUTPUTS} + + +def documentation(path): + name = PurePosixPath(path).name + return (PurePosixPath(path).suffix in ('.md', '.markdown', '.rst', '.adoc') + or name in ('README', 'NEWS', 'COPYING', 'COPYRIGHT', 'AUTHORS', + 'CONTRIBUTING', 'SUPPORT', 'GOVERNANCE', 'INSTALL')) + + +def workflow(selected, name): + mapping = { + 'ci.yml': OUTPUTS, + 'linux.yml': ('linux', 'diagnostics'), + 'bsd.yml': ('bsd', 'diagnostics'), + 'macos.yml': ('macos', 'diagnostics'), + 'other.yml': ('other', 'diagnostics'), + 'versions.yml': ('versions', 'diagnostics'), + 'android.yml': ('android', 'diagnostics'), + 'archive.yml': ('archive', 'diagnostics'), + 'diagnostics.yml': ('diagnostics', 'asan', 'coverage', 'clang'), + 'valgrind.yml': ('valgrind', 'diagnostics'), + 'interoperability.yml': ('interoperability', 'diagnostics'), + 'interoperability-cleanup.yml': ('interoperability', 'diagnostics'), + 'interoperability-tests.yml': ('interoperability', 'diagnostics'), + } + if name not in mapping: + return False + enable(selected, *mapping[name]) + return True + + +def script(selected, name): + stem = name.rsplit('.', 1)[0] + test = stem.startswith('test-') + base = stem[5:] if test else stem + if base.startswith('interop') or base in ('transfers',): + enable(selected, 'interoperability', 'diagnostics') + elif base in ('android', 'android-emulator', 'termux'): + enable(selected, 'android', 'diagnostics') + elif base == 'archive': + enable(selected, 'archive', 'diagnostics') + elif base == 'versions': + enable(selected, 'versions', 'diagnostics') + elif base == 'guest': + enable(selected, 'linux', 'bsd', 'other', 'diagnostics') + elif base == 'storage': + enable(selected, 'linux', 'diagnostics') + elif base == 'diagnostics': + enable(selected, 'diagnostics') + if not test: + enable(selected, 'valgrind', 'asan', 'coverage', 'clang') + elif base == 'changes' and not test: + enable(selected, *OUTPUTS) + elif base == 'check-results' and not test: + enable(selected, *OUTPUTS) + elif test and base in ('check-results', 'results', 'changes'): + enable(selected, 'diagnostics') + elif base == 'check-platform': + enable(selected, 'linux', 'macos', 'other', 'diagnostics') + elif base == 'installcheck': + enable(selected, 'linux', 'diagnostics') + elif name == 'install-cygwin.ps1': + enable(selected, 'other', 'diagnostics') + else: + return False + return True + + +def testsuite(selected, path): + native = ('linux', 'bsd', 'macos', 'other', 'versions', 'diagnostics') + if path in ('testsuite/fixtures/valgrind.supp', 'testsuite/profiles/valgrind.json'): + enable(selected, 'valgrind', 'diagnostics') + return + if path == 'testsuite/profiles/archive.json': + enable(selected, 'archive', 'diagnostics') + return + if path == 'testsuite/profiles/asan.json': + enable(selected, 'diagnostics', 'asan') + return + if path == 'testsuite/harness/interop_metadata.py': + enable(selected, 'interoperability', 'diagnostics') + return + if path in ('testsuite/tools/check-rsync-ssl.py', 'testsuite/tools/check-ssh.py', + 'testsuite/tools/check-storage.py'): + enable(selected, 'linux', 'diagnostics') + return + enable(selected, *native) + enable(selected, 'asan') + if (path == 'testsuite/runtests.py' or path == 'testsuite/tools/check.py' + or path.startswith('testsuite/harness/')): + enable(selected, 'valgrind') + if path == 'testsuite/harness/interoperability.py': + enable(selected, 'interoperability') + + +def select(paths): + if not paths: + raise ValueError('No changed paths') + selected = empty() + for raw in paths: + path = raw.replace('\\', '/') + parsed = PurePosixPath(path) + if not path or parsed.is_absolute() or '..' in parsed.parts: + raise ValueError('Invalid changed path') + if path == 'rsync.1.md': + return full() + if path in ARCHIVE_DOCUMENTS: + enable(selected, 'archive', 'diagnostics') + elif documentation(path): + continue + elif path.startswith('testsuite/'): + testsuite(selected, path) + elif path.startswith('old_versions/'): + enable(selected, 'versions', 'diagnostics') + elif path.startswith('.github/workflows/'): + if not workflow(selected, parsed.name): + return full() + elif path.startswith('.github/scripts/'): + if not script(selected, parsed.name): + return full() + elif path.startswith('.github/actions/interoperability/'): + enable(selected, 'interoperability', 'diagnostics') + elif path.startswith('.github/actions/'): + return full() + else: + return full() + return selected + + +def changed_paths(event): + request = event.get('pull_request') + if not isinstance(request, dict): + raise ValueError('Pull request event has no request') + base = request.get('base', {}).get('sha', '') + head = request.get('head', {}).get('sha', '') + if not re.fullmatch(r'[0-9a-f]{40}', base) or not re.fullmatch(r'[0-9a-f]{40}', head): + raise ValueError('Invalid pull request revisions') + result = subprocess.run(['git', 'diff', '--name-only', '--no-renames', '-z', '--diff-filter=ACDMRTUXB', + base + '...' + head], capture_output=True, timeout=30) + if result.returncode or len(result.stdout) > 1024 * 1024: + raise ValueError('Cannot determine changed paths') + paths = [os.fsdecode(value) for value in result.stdout.split(b'\0') if value] + if len(paths) > 10000: + raise ValueError('Changed path count exceeds limit') + return paths + + +def main(): + event_name = os.environ.get('GITHUB_EVENT_NAME') + if event_name == 'pull_request': + event = json.loads(Path(os.environ['GITHUB_EVENT_PATH']).read_text()) + selected = select(changed_paths(event)) + else: + selected = full() + output = Path(os.environ['GITHUB_OUTPUT']) + with output.open('a') as stream: + for name in OUTPUTS: + stream.write(f'{name}={str(selected[name]).lower()}\n') + return 0 + + +if __name__ == '__main__': + try: + raise SystemExit(main()) + except (KeyError, OSError, TypeError, ValueError, subprocess.SubprocessError) as error: + print(f'Change selection failed: {error}', file=sys.stderr) + raise SystemExit(1) diff --git a/.github/scripts/check-platform.py b/.github/scripts/check-platform.py new file mode 100644 index 000000000..85ff566f2 --- /dev/null +++ b/.github/scripts/check-platform.py @@ -0,0 +1,44 @@ +import argparse +from pathlib import Path +import platform +import shlex +import sys + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('system', choices=['ubuntu', 'almalinux', 'macos', 'cygwin']) + parser.add_argument('--release') + parser.add_argument('--arch', required=True) + args = parser.parse_args() + print(platform.platform()) + if platform.machine() != args.arch: + raise ValueError(f'Expected {args.arch}, got {platform.machine()}') + if args.system in ('ubuntu', 'almalinux'): + values = {} + for line in Path('/etc/os-release').read_text().splitlines(): + name, separator, value = line.partition('=') + if separator and name in ('ID', 'VERSION_ID'): + parts = shlex.split(value) + if len(parts) != 1: + raise ValueError(f'Invalid {name} in os-release') + values[name] = parts[0] + release = values['VERSION_ID'] + if args.system == 'almalinux': + release = release.split('.')[0] + if values['ID'] != args.system or release != args.release: + raise ValueError(f'Unexpected Linux identity: {values}') + elif args.system == 'macos': + if platform.system() != 'Darwin' or platform.mac_ver()[0].split('.')[0] != args.release: + raise ValueError('Unexpected macOS identity') + elif not platform.system().startswith('CYGWIN'): + raise ValueError('Expected Cygwin') + return 0 + + +if __name__ == '__main__': + try: + raise SystemExit(main()) + except (KeyError, OSError, ValueError) as error: + print(f'Platform check failed: {error}', file=sys.stderr) + raise SystemExit(1) diff --git a/.github/scripts/check-results.py b/.github/scripts/check-results.py new file mode 100644 index 000000000..5b35a0c98 --- /dev/null +++ b/.github/scripts/check-results.py @@ -0,0 +1,74 @@ +import json +import os +import sys + + +def check(data, expected, selections, scheduled=False): + def require(condition, message): + if not condition: + raise ValueError(message) + + require(isinstance(data, dict), 'Invalid job results') + changes = data.get('changes') + require(isinstance(changes, dict) and changes.get('result') == 'success', + 'Change selection was not successful') + require(set(data) == set(expected) | {'changes'}, + 'Missing or unexpected job families') + require(expected and all(value in ('true', 'false') for value in expected.values()), + 'Invalid job selection') + require(isinstance(selections, dict) + and set(selections) == set(expected) | {'asan', 'coverage', 'clang'} + and all(value in ('true', 'false') for value in selections.values()), + 'Invalid change selection') + selected_families = {name: selections[name] for name in expected} + if scheduled: + require(all(value == 'true' for value in selections.values()), + 'Scheduled run did not select every check') + else: + require(selected_families == expected, 'Job selection differs from changed paths') + messages = [] + for family in expected: + group = data[family] + require(isinstance(group, dict), f'{family}: invalid job results') + result = group.get('result') + if expected[family] == 'false': + require(result == 'skipped', f'{family}: expected skipped') + messages.append(f'{family}: skipped') + continue + require(result == 'success', f'{family}: not successful') + outputs = group.get('outputs', {}) + require(isinstance(outputs, dict), f'{family}: invalid outputs') + jobs = json.loads(outputs.get('jobs', 'null')) + inputs = json.loads(outputs.get('inputs', '{}')) + require(isinstance(jobs, dict) and jobs, f'{family}: missing job results') + require(isinstance(inputs, dict) and all(type(value) is bool for value in inputs.values()), + f'{family}: invalid job selections') + require(inputs.keys() <= jobs.keys(), f'{family}: unknown job selection') + require(any(inputs.get(name) is not False for name in jobs), f'{family}: no jobs selected') + for name, job in jobs.items(): + require(isinstance(job, dict), f'{family}/{name}: invalid job result') + result = job.get('result') + if inputs.get(name) is False: + require(result == 'skipped', f'{family}/{name}: expected skipped') + else: + require(result == 'success', f'{family}/{name}: {result or "missing result"}') + messages.append(f'{family}/{name}: {result}') + return messages + + +def main(): + try: + expected = {name[len('CI_EXPECT_'):].lower(): value for name, value in os.environ.items() + if name.startswith('CI_EXPECT_')} + messages = check(json.loads(os.environ['CI_RESULTS']), expected, + json.loads(os.environ['CI_SELECTIONS']), + os.environ.get('GITHUB_EVENT_NAME') == 'schedule') + except (KeyError, TypeError, ValueError) as error: + print(f'CI result check failed: {error}', file=sys.stderr) + return 1 + print('\n'.join(messages)) + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/.github/scripts/diagnostics.py b/.github/scripts/diagnostics.py new file mode 100644 index 000000000..1065a79b6 --- /dev/null +++ b/.github/scripts/diagnostics.py @@ -0,0 +1,147 @@ +import argparse +import hashlib +import io +import os +from pathlib import Path +import re +import shutil +import signal +import subprocess +import sys +import tarfile +import time +import urllib.error +import urllib.request + + +def logged(command, path): + with path.open('w') as log, subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + text=True, errors='replace') as proc: + def cancel(signum, frame): + signal.signal(signal.SIGTERM, signal.SIG_IGN) + signal.signal(signal.SIGINT, signal.SIG_IGN) + proc.terminate() + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait(timeout=5) + raise SystemExit(128 + signum) + previous = {sig: signal.signal(sig, cancel) for sig in (signal.SIGTERM, signal.SIGINT)} + try: + for line in proc.stdout: + log.write(line) + print(line, end='', flush=True) + return proc.wait() + finally: + proc.stdout.close() + for sig, handler in previous.items(): + signal.signal(sig, handler) + + +def summary(title, path, pattern): + lines = [] + if path.is_file(): + with path.open(errors='replace') as stream: + lines = [line.rstrip() for line in stream if re.search(pattern, line)] + text = f'## {title}\n\n```text\n' + '\n'.join(lines or ['No summary produced']) + '\n```\n' + print(text) + if os.environ.get('GITHUB_STEP_SUMMARY'): + with open(os.environ['GITHUB_STEP_SUMMARY'], 'a') as stream: + stream.write(text) + + +def install_actionlint(): + name = 'actionlint_1.7.12_linux_amd64.tar.gz' + url = f'https://github.com/rhysd/actionlint/releases/download/v1.7.12/{name}' + for attempt in range(5): + try: + with urllib.request.urlopen(url, timeout=60) as response: + data = response.read(32 * 1024 * 1024 + 1) + break + except (OSError, urllib.error.URLError): + if attempt == 4: + raise + time.sleep(3 * (attempt + 1)) + if hashlib.sha256(data).hexdigest() != '8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8': + raise ValueError('actionlint SHA-256 mismatch') + with tarfile.open(fileobj=io.BytesIO(data), mode='r:gz') as archive: + member = archive.getmember('actionlint') + if not member.isfile(): + raise ValueError('actionlint archive has no regular executable') + with archive.extractfile(member) as source, Path('actionlint').open('wb') as destination: + shutil.copyfileobj(source, destination) + Path('actionlint').chmod(0o755) + with open(os.environ['GITHUB_PATH'], 'a') as stream: + stream.write(str(Path.cwd()) + '\n') + + +def main(): + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest='task', required=True) + sub.add_parser('install-actionlint') + coverage = sub.add_parser('coverage') + coverage.add_argument('transport', choices=['pipe', 'tcp']) + sub.add_parser('coverage-summary') + sub.add_parser('scan') + for name in ('valgrind', 'valgrind-results'): + command = sub.add_parser(name) + command.add_argument('--privilege', choices=['root', 'non-root'], required=True) + command.add_argument('--transport', choices=['pipe', 'tcp'], required=True) + command.add_argument('--profiles') + args = parser.parse_args() + if args.task.startswith('valgrind') and args.privilege == 'root' and os.geteuid() != 0: + return subprocess.call(['sudo', '-E', sys.executable, '-S', __file__, *sys.argv[1:]]) + if args.task == 'install-actionlint': + install_actionlint() + return 0 + if args.task == 'coverage': + target = 'coverage' if args.transport == 'pipe' else 'coverage-tcp' + return logged(['make', target], Path(f'cov-{args.transport}.log')) + if args.task == 'coverage-summary': + for transport in ('pipe', 'tcp'): + summary(f'Coverage ({transport})', Path(f'cov-{transport}.log'), r'^(lines|functions|branches|decisions):') + return 0 + if args.task == 'scan': + command = ['scan-build-21', '--status-bugs', '-o', str(Path('scan-report').resolve()), + 'make', 'check-progs', f'-j{len(os.sched_getaffinity(0))}'] + result = logged(command, Path('scan-build.out')) + summary('Clang 21', Path('scan-build.out'), r'scan-build: .* bugs? found|scan-build: No bugs found') + return result + output = Path(os.environ['VALGRIND_SCRATCH']) + evidence = Path('test-results') + evidence.mkdir(exist_ok=True) + if args.task == 'valgrind': + if not args.profiles: + parser.error('--profiles is required for Valgrind tests') + options = ['--use-tcp', '--daemon-tests-only'] if args.transport == 'tcp' else [] + command = [sys.executable, '-S', 'testsuite/tools/check.py', '.', './rsync', args.profiles, str(output), + '--valgrind', '--valgrind-opts=--leak-check=no --error-exitcode=0', + *options, '-j2', '--timeout=600', '--preserve-scratch'] + result = logged(command, evidence / f'valgrind-{args.privilege}-{args.transport}.log') + if (output / 'receipt.json').is_file(): + shutil.copyfile(output / 'receipt.json', evidence / f'valgrind-{args.privilege}-{args.transport}.json') + return result + logs = sorted((output / 'scratch' / 'testtmp' / 'valgrind-logs').glob('valgrind.*.log')) + if not logs: + raise ValueError('No Valgrind logs were produced') + destination = evidence / 'valgrind-logs' + destination.mkdir(exist_ok=True) + failed = [] + for path in logs: + shutil.copyfile(path, destination / path.name) + with path.open(errors='replace') as stream: + if any(re.search(r'ERROR SUMMARY: [1-9][0-9]* errors', line) for line in stream): + failed.append(path.name) + print(f'Scanned {len(logs)} Valgrind logs') + if failed: + raise ValueError('Valgrind reported errors: ' + ', '.join(failed)) + return 0 + + +if __name__ == '__main__': + try: + raise SystemExit(main()) + except (OSError, ValueError, subprocess.SubprocessError) as error: + print(f'Diagnostic check failed: {error}', file=sys.stderr) + raise SystemExit(1) diff --git a/.github/scripts/guest.py b/.github/scripts/guest.py new file mode 100644 index 000000000..1cd818f3e --- /dev/null +++ b/.github/scripts/guest.py @@ -0,0 +1,203 @@ +import argparse +import json +import os +from pathlib import Path +import platform +import pwd +import shutil +import signal +import struct +import subprocess +import sys +import tempfile + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('platform', choices=['freebsd', 'netbsd', 'openbsd', 'solaris', 'alpine', 'check']) + parser.add_argument('--release') + parser.add_argument('--output', type=Path, default=Path('test-results')) + args = parser.parse_args() + if args.platform == 'check': + try: + with (args.output / 'status').open('rb') as stream: + status = stream.read(3) + except OSError as error: + print(f'Guest results unavailable: {error.strerror}', file=sys.stderr) + return 1 + if status not in (b'0', b'0\n'): + print('Guest checks failed or produced an invalid status', file=sys.stderr) + return 1 + return 0 + + os.umask(0o022) + output = args.output.resolve() + output.mkdir(parents=True, exist_ok=True) + status = 1 + active = None + cross_device = None + mounted = False + source = None + env = {**os.environ, 'PYTHONDONTWRITEBYTECODE': '1'} + + def cancel(signum, frame): + signal.signal(signal.SIGTERM, signal.SIG_IGN) + signal.signal(signal.SIGINT, signal.SIG_IGN) + if active is not None and active.poll() is None: + active.terminate() + try: + active.wait(timeout=10) + except subprocess.TimeoutExpired: + active.kill() + active.wait(timeout=5) + raise SystemExit(128 + signum) + + def command(label, argv, *, required=True, destination=None, account=None): + nonlocal active + print(f'::group::{label}', flush=True) + options = {} + current_env = env.copy() + if account: + options = dict(user=account.pw_uid, group=account.pw_gid, + extra_groups=os.getgrouplist(account.pw_name, account.pw_gid)) + current_env['HOME'] = account.pw_dir + stream = None + try: + stream = destination.open('w') if destination else None + active = subprocess.Popen(argv, env=current_env, stdout=stream, **options) + result = active.wait() + if result and required: + raise subprocess.CalledProcessError(result, argv) + return result + except OSError as error: + if required: + raise + print(f'::error::{label}: {error}', file=sys.stderr) + return 2 + finally: + active = None + if stream: + stream.close() + print('::endgroup::', flush=True) + + previous = {sig: signal.signal(sig, cancel) for sig in (signal.SIGTERM, signal.SIGINT)} + try: + if args.platform == 'alpine': + shutil.copytree('/source', '/work', symlinks=True) + os.chdir('/work') + source = Path.cwd() + expected = {'freebsd': 'FreeBSD', 'netbsd': 'NetBSD', 'openbsd': 'OpenBSD', + 'solaris': 'SunOS', 'alpine': 'Linux'}[args.platform] + command('Platform', ['uname', '-a']) + if platform.system() != expected: + raise ValueError(f'Expected {expected}, got {platform.system()}') + if args.platform in ('freebsd', 'netbsd', 'openbsd'): + actual = platform.release() + if platform.machine() != 'amd64' or not args.release or not ( + actual == args.release or args.platform == 'freebsd' and actual.startswith(args.release + '-RELEASE')): + raise ValueError(f'Unexpected guest identity: {actual} {platform.machine()}') + if args.platform == 'alpine': + actual = subprocess.check_output(['apk', '--print-arch'], text=True).strip() + if actual != env['APK_ARCH'] or struct.calcsize('P') * 8 != int(env['BITS']): + raise ValueError('Unexpected Alpine userspace architecture') + + flags = ['--with-rrsync'] + features = [] + profiles = f'non-asan,self-peer,{args.platform}' + if args.platform == 'freebsd': + env.update(CPPFLAGS='-I/usr/local/include', LDFLAGS='-L/usr/local/lib') + flags += ['--disable-md2man', '--enable-iconv', '--enable-acl-support', '--enable-xattr-support', + '--enable-roll-simd', '--enable-roll-asm', '--enable-md5-asm'] + features = ['zstd', 'lz4', 'xxh128', 'xxh3', 'xxh64', 'iconv', 'IDN', 'ACLs', 'xattrs'] + cross_device = Path(tempfile.mkdtemp(prefix='rsync-cross-device.')) + command('Cross-device filesystem', ['mount', '-t', 'tmpfs', '-o', 'size=256m,mode=1777,nosuid', + 'tmpfs', str(cross_device)]) + mounted = True + env['RSYNC_TEST_TMP'] = str(cross_device) + elif args.platform in ('netbsd', 'solaris'): + flags += ['--disable-zstd', '--disable-md2man', '--disable-xxhash', '--disable-lz4', '--disable-idn'] + profiles += ',no-idn,no-xxhash,no-zstd-threads' + elif args.platform == 'openbsd': + env.update(AUTOCONF_VERSION='2.71', AUTOMAKE_VERSION='1.16', + CPPFLAGS='-I/usr/local/include -DXXH_INLINE_ALL=1', LDFLAGS='-L/usr/local/lib', + ac_cv_search_XXH64_createState='', + RSYNC_EXCLUDE='acl-symlink-race,sender-symlink-target-race,sender-remove-source-secure') + flags += ['--disable-md2man', '--enable-iconv'] + features = ['zstd', 'lz4', 'xxh128', 'xxh3', 'xxh64', 'iconv', 'IDN'] + else: + flags += ['--enable-iconv', '--enable-acl-support', '--enable-xattr-support'] + features = ['zstd', 'lz4', 'xxh128', 'xxh3', 'xxh64', 'iconv', 'IDN', 'ACLs', 'xattrs'] + profiles = env['PROFILES'] + command('Configure', ['./configure', *flags]) + command('Build', ['make', *(['-j4'] if args.platform == 'alpine' else []), 'check-progs']) + status = int(bool(command('Version', ['./rsync', '--version'], required=False))) + if args.platform == 'alpine': + binary = (source / 'rsync').read_bytes() + version = json.loads(subprocess.check_output(['./rsync', '-VV'], text=True)) + if binary[:4] != b'\x7fELF' or binary[4] != int(env['BITS']) // 32 or version['capabilities']['file_bits'] != 64: + raise ValueError('Unexpected Alpine binary or file-offset width') + if features: + status |= bool(command('Features', [sys.executable, '-S', 'testsuite/tools/check-features.py', + str(source / 'rsync'), *features], required=False, + destination=output / 'features.json')) + if args.platform == 'alpine': + for argv in (['addgroup', '-S', 'rsynctest'], ['addgroup', '-S', 'rsynctest-extra'], + ['adduser', '-S', '-D', '-h', '/home/rsynctest', '-G', 'rsynctest', 'rsynctest'], + ['addgroup', 'rsynctest', 'rsynctest-extra']): + command('Test account', argv) + account = None if args.platform == 'solaris' else pwd.getpwnam('rsynctest') + if account: + (output / 'nonroot').mkdir() + os.chown(output / 'nonroot', account.pw_uid, account.pw_gid) + if args.platform != 'alpine': + status |= bool(command('Build directory probe', ['make', 'dirfd-probe'], required=False)) + if account: + status |= bool(command('Directory descriptors', [str(source / 'dirfd-probe'), str(output / 'nonroot')], + required=False, account=account, + destination=output / 'nonroot' / 'dirfd-probe.tsv')) + passes = [('root', 'pipe', None), ('root', 'tcp', None)] + if args.platform == 'alpine': + passes[1:1] = [('root', 'protocol-30', 30), ('root', 'protocol-29', 29)] + if account: + passes += [('nonroot', 'pipe', None), ('nonroot', 'tcp', None)] + for role, transport, protocol in passes: + selected = f'{role},{profiles}' + ('' if transport == 'tcp' else ',pipe') + options = [] + if transport == 'tcp': + options += ['--use-tcp', '--daemon-tests-only'] + if protocol: + selected += f',protocol-{protocol}' + options += [f'--protocol={protocol}'] + parallel = 4 if args.platform == 'alpine' else 2 if args.platform == 'openbsd' and transport == 'tcp' else 8 + label = f'protocol {protocol}' if protocol else 'TCP' if transport == 'tcp' else 'pipe' + status |= bool(command(f'Tests ({"non-root" if role == "nonroot" else role} {label})', + [sys.executable, '-S', 'testsuite/tools/check.py', str(source), str(source / 'rsync'), + selected, str(output / role / transport), *options, f'-j{parallel}'], + required=False, account=account if role == 'nonroot' else None)) + except (KeyError, OSError, ValueError, subprocess.SubprocessError, SystemExit) as error: + print(f'::error::Guest checks failed: {error}', file=sys.stderr) + status = 1 + finally: + if cross_device is not None: + try: + if mounted: + subprocess.run(['umount', str(cross_device)], check=True) + cross_device.rmdir() + except (OSError, subprocess.SubprocessError) as error: + print(f'::error::Cross-device cleanup failed: {error}', file=sys.stderr) + status = 1 + if args.platform == 'alpine' and source is not None: + (output / 'bin').mkdir(exist_ok=True) + for name in ('rsync', 'rsync-ssl', 'rsync.1', 'rsync-ssl.1', 'rsyncd.conf.5', 'rrsync.1', 'rrsync'): + if (source / name).is_file(): + shutil.copy2(source / name, output / 'bin' / name) + if (source / 'config.log').is_file(): + shutil.copyfile(source / 'config.log', output / 'config.log') + (output / 'status').write_text(f'{int(status)}\n') + for sig, handler in previous.items(): + signal.signal(sig, handler) + return int(status) if args.platform == 'alpine' else 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/.github/scripts/install-cygwin.ps1 b/.github/scripts/install-cygwin.ps1 new file mode 100644 index 000000000..38a957e1e --- /dev/null +++ b/.github/scripts/install-cygwin.ps1 @@ -0,0 +1,33 @@ +param([switch]$OpenSSH) + +$ErrorActionPreference = 'Stop' +if ($Env:GITHUB_ACTIONS -ne 'true' -or $Env:RUNNER_ENVIRONMENT -ne 'github-hosted') { + throw 'Cygwin installation is restricted to disposable hosted runners' +} +$setup = Join-Path $Env:RUNNER_TEMP 'setup-x86_64.exe' +$sums = Join-Path $Env:RUNNER_TEMP 'cygwin-sha512.sum' +$packages = 'make,autoconf,automake,gcc-core,git,attr,libattr-devel,python39,python39-pip,libzstd-devel,liblz4-devel,libssl-devel,libxxhash0,libxxhash-devel,libidn2-devel' +if ($OpenSSH) { $packages += ',openssh,openssl' } + +Invoke-WebRequest https://cygwin.com/setup-x86_64.exe -OutFile $setup +Invoke-WebRequest https://cygwin.com/sha512.sum -OutFile $sums +$sum = Select-String -LiteralPath $sums -Pattern '^[0-9a-fA-F]{128}\s+\*?setup-x86_64\.exe$' | Select-Object -First 1 +if (-not $sum) { throw 'setup-x86_64.exe is missing from Cygwin sha512.sum' } +$expected = ($sum.Line -split '\s+')[0] +$actual = (Get-FileHash -LiteralPath $setup -Algorithm SHA512).Hash +if ($actual -ine $expected) { throw 'Cygwin setup SHA-512 mismatch' } + +$arguments = @( + '--quiet-mode', '--no-desktop', '--no-startmenu', '--no-shortcuts', + '--root', 'C:\tools\cygwin', + '--local-package-dir', (Join-Path $Env:RUNNER_TEMP 'cygwin-packages'), + '--site', 'https://mirrors.kernel.org/sourceware/cygwin/', + '--packages', $packages +) +$install = Start-Process -FilePath $setup -ArgumentList $arguments -Wait -PassThru -WindowStyle Hidden +if ($install.ExitCode -ne 0) { exit $install.ExitCode } +$bash = 'C:\tools\cygwin\bin\bash.exe' +if (-not (Test-Path -LiteralPath $bash)) { throw 'Cygwin setup did not install bash' } +& $bash -lc 'command -v make aclocal gcc /usr/bin/git python3 >/dev/null' +if ($LASTEXITCODE -ne 0) { throw 'Cygwin setup did not install all required build tools' } +'C:/tools/cygwin/bin' >>$Env:GITHUB_PATH diff --git a/.github/scripts/installcheck.py b/.github/scripts/installcheck.py new file mode 100644 index 000000000..5685a4ba1 --- /dev/null +++ b/.github/scripts/installcheck.py @@ -0,0 +1,30 @@ +from pathlib import Path +import subprocess +import sys +import tempfile + + +def main(): + with tempfile.TemporaryDirectory(prefix='rsync-install-') as directory: + root = Path(directory) + subprocess.run(['make', 'install-all', f'DESTDIR={root}'], check=True) + required = ['usr/local/bin/rsync', 'usr/local/bin/rsync-ssl', 'usr/local/bin/rrsync', + 'usr/local/share/man/man1/rsync.1', 'usr/local/share/man/man1/rsync-ssl.1', + 'usr/local/share/man/man1/rrsync.1', 'usr/local/share/man/man5/rsyncd.conf.5', + 'etc/stunnel/rsyncd.conf'] + missing = [name for name in required if not (root / name).exists()] + if missing: + raise ValueError('Missing installed files: ' + ', '.join(missing)) + subprocess.run(['make', 'uninstall-all', f'DESTDIR={root}'], check=True) + leftover = [str(path.relative_to(root)) for path in root.rglob('*') if path.is_file() or path.is_symlink()] + if leftover: + raise ValueError('Uninstall left files: ' + ', '.join(leftover)) + return 0 + + +if __name__ == '__main__': + try: + raise SystemExit(main()) + except (OSError, ValueError, subprocess.SubprocessError) as error: + print(f'Installation check failed: {error}', file=sys.stderr) + raise SystemExit(1) diff --git a/.github/scripts/interop-broker.py b/.github/scripts/interop-broker.py index 81acb4c9c..8ba7cf01a 100644 --- a/.github/scripts/interop-broker.py +++ b/.github/scripts/interop-broker.py @@ -6,11 +6,14 @@ import subprocess import sys -from interop_ci import ROLES, NotSelected, PeerUnavailable, bind, check_ci, find_controller, identity, records, source, write +from interop_ci import ROLES, STARTUP_TIMEOUT, NotSelected, PeerUnavailable, bind, check_ci, find_controller, identity, records, source, write from interop_enrol import seal from interop_network import NetworkError, Tailnet, download_records from interop_policy import TAILNET, admission, check_policy, update_policy +sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'testsuite')) +from harness.interop_metadata import FEATURES, metadata_cases, supported + def load(root, kind): path = root / (kind + '.json') @@ -51,6 +54,30 @@ def verify_cases(receipts, source_sha): collected.update(assigned) if commands != expected or received != expected: raise NetworkError('Directed transfer coverage is incomplete') + verify_metadata(receipts) + + +def verify_metadata(receipts): + expected = set(metadata_cases()) + for role, record in receipts.items(): + acl, extended = FEATURES[role] + actual = record['identity'].get('metadata') + if (not isinstance(actual, dict) or type(actual.get('xattrs')) is not bool + or actual != dict(acl=acl, xattrs=extended)): + raise NetworkError('Endpoint did not qualify its required metadata features') + for field in ('metadata_commands', 'metadata_updates', 'metadata_initial_verified', 'metadata_verified', 'metadata_restored'): + if field in ('metadata_initial_verified', 'metadata_verified'): + assigned = {case for case in expected if case[1] == role} + elif field == 'metadata_restored': + assigned = {case for case in expected if case[0] == role and case[-1] == 'backup'} + else: + assigned = {case for case in expected if (case[0] if case[2] == 'push' else case[1]) == role + and (field == 'metadata_commands' or case[-1] != 'backup' and supported(case))} + cases = record.get(field) + if (not isinstance(cases, list) or len(cases) != len(assigned) + or any(not isinstance(case, list) or len(case) != 5 or any(not isinstance(value, str) for value in case) for case in cases) + or {tuple(case) for case in cases} != assigned): + raise NetworkError('Missing, duplicate or incorrectly assigned metadata case') def prepare(root, api): @@ -232,7 +259,7 @@ def main(): api = Tailnet() if args.phase == 'broker': check_policy(api.policy()[0]) - recipients, keys = records('public', timeout=1800), {} + recipients, keys = records('public', timeout=STARTUP_TIMEOUT), {} if len({record.get('public_key') for record in recipients.values()}) != len(ROLES): raise NetworkError('Endpoints share an enrolment recipient') encrypted = {} diff --git a/.github/scripts/interop-client.py b/.github/scripts/interop-client.py new file mode 100644 index 000000000..c9b86ae90 --- /dev/null +++ b/.github/scripts/interop-client.py @@ -0,0 +1,162 @@ +import argparse +import hashlib +import json +import os +from pathlib import Path +import shutil +import subprocess +import sys +import tarfile +import time +import urllib.request + +from interop_network import NetworkError, NoRedirect + +VERSION = '1.104.4' +LINUX_SHA = '624ea093d47d62cf56f17b8332d74b8decda4a1faa610ebc60a1de334b2b3dc9' +WINDOWS_SHA = '1d12cd8b1d6dbb447da8579b1a5268ae2fc050967bebfaa02e32d2bb645ef6ee' +SOURCE = '27047635f237d1f3e0b08c0f487674757f30d2f7' + + +def run(argv, timeout=60, check=True, **options): + try: + result = subprocess.run(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout, **options) + except FileNotFoundError: + raise NetworkError('Required command not found: ' + Path(argv[0]).name) from None + if check and result.returncode: + raise NetworkError(Path(argv[0]).name + ' failed with exit code ' + str(result.returncode)) + return result + + +def download(url, path, wanted): + digest, size = hashlib.sha256(), 0 + with urllib.request.build_opener(NoRedirect()).open(url, timeout=30) as stream, path.open('xb') as output: + while True: + data = stream.read(65536) + if not data: + break + size += len(data) + if size > 150 * 1024 * 1024: + raise NetworkError('Tailscale package exceeds limit') + digest.update(data) + output.write(data) + if digest.hexdigest() != wanted: + raise NetworkError('Tailscale package checksum differs') + + +def install(root): + if os.environ.get('GITHUB_ACTIONS') != 'true' or os.environ.get('RUNNER_ENVIRONMENT') != 'github-hosted': + raise NetworkError('Client installation is restricted to disposable hosted runners') + binary = root / 'bin' + binary.mkdir() + if sys.platform.startswith('cygwin'): + package = root / 'tailscale.msi' + download(f'https://pkgs.tailscale.com/stable/tailscale-setup-{VERSION}-amd64.msi', package, WINDOWS_SHA) + native = run(['cygpath', '-w', str(package)]).stdout.decode().strip() + run(['msiexec.exe', '/quiet', '/norestart', '/i', native], timeout=180) + return dict(command=['/cygdrive/c/Program Files/Tailscale/tailscale.exe'], windows=True) + if sys.platform == 'darwin': + env = dict(os.environ, GOBIN=str(binary), GOENV='off', GOWORK='off', GOFLAGS='', + GOPROXY='https://proxy.golang.org', GOSUMDB='sum.golang.org', GONOSUMDB='', GOPRIVATE='') + run(['go', 'install', 'tailscale.com/cmd/tailscale@' + SOURCE, + 'tailscale.com/cmd/tailscaled@' + SOURCE], env=env, timeout=900) + else: + package = root / 'tailscale.tgz' + download(f'https://pkgs.tailscale.com/stable/tailscale_{VERSION}_amd64.tgz', package, LINUX_SHA) + with tarfile.open(str(package), 'r:gz') as archive: + for name in ('tailscale', 'tailscaled'): + members = [member for member in archive.getmembers() if member.name == f'tailscale_{VERSION}_amd64/{name}'] + if len(members) != 1 or not members[0].isreg() or members[0].size > 100 * 1024 * 1024: + raise NetworkError('Unexpected client package member') + with archive.extractfile(members[0]) as source, (binary / name).open('xb') as output: + shutil.copyfileobj(source, output) + (binary / name).chmod(0o755) + return dict(command=['sudo', str(binary / 'tailscale'), '--socket=' + str(root / 'tailscaled.sock')], windows=False) + + +def daemon(root): + env = {key: os.environ[key] for key in ('PATH', 'RUNNER_TRACKING_ID') if key in os.environ} + with (root / 'tailscaled.log').open('wb') as log: + with subprocess.Popen([str(root / 'bin' / 'tailscaled'), '--state=mem:', '--socket=' + str(root / 'tailscaled.sock')], + env=env, stdin=subprocess.DEVNULL, stdout=log, stderr=log) as child: + try: + deadline = time.monotonic() + 3000 + while not (root / 'daemon-stop').exists() and time.monotonic() < deadline: + if child.poll() is not None: + raise NetworkError('Tailscale daemon exited') + time.sleep(0.1) + finally: + if child.poll() is None: + child.terminate() + try: + child.wait(timeout=10) + except subprocess.TimeoutExpired: + child.kill() + child.wait(timeout=5) + (root / 'daemon-stopped').touch() + + +def connect(root, key, hostname): + client = json.loads((root / 'client.json').read_text()) + if not client['windows']: + env = {name: os.environ[name] for name in ('PATH', 'RUNNER_TRACKING_ID') if name in os.environ} + subprocess.Popen(['sudo', '--preserve-env=RUNNER_TRACKING_ID', sys.executable, '-B', '-S', + str(Path(__file__).resolve()), 'daemon', '--root', str(root)], + env=env, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, start_new_session=True) + (root / 'daemon-started').touch() + deadline = time.monotonic() + 30 + while True: + result = run([*client['command'], 'status', '--json'], check=False) + if result.stdout.startswith(b'{'): + break + if time.monotonic() >= deadline: + raise NetworkError('Tailscale daemon did not start') + time.sleep(0.2) + path = str(key) + if client['windows']: + path = run(['cygpath', '-w', path]).stdout.decode().strip() + run([*client['command'], 'up', '--auth-key=file:' + path, '--hostname=' + hostname, + '--accept-routes=false', '--accept-dns=false', '--timeout=60s'], timeout=75) + prefs = json.loads(run([*client['command'], 'debug', 'prefs']).stdout) + if any(prefs.get(key) is not False for key in ('RouteAll', 'CorpDNS')): + raise NetworkError('Route or DNS acceptance is still enabled') + return json.loads(run([*client['command'], 'status', '--json']).stdout)['Self'] + + +def stop(root): + if not (root / 'client.json').exists(): + return + client = json.loads((root / 'client.json').read_text()) + if not client['windows'] and not (root / 'daemon-started').exists(): + return + try: + # Controller cleanup owns device removal and may revoke it first + run([*client['command'], 'logout'], check=False) + finally: + if client['windows']: + run(['sc.exe', 'stop', 'Tailscale'], check=False) + else: + (root / 'daemon-stop').touch() + deadline = time.monotonic() + 15 + while not (root / 'daemon-stopped').exists(): + if time.monotonic() >= deadline: + raise NetworkError('Tailscale daemon did not stop') + time.sleep(0.1) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('phase', choices=('daemon', 'stop')) + parser.add_argument('--root', type=Path, required=True) + args = parser.parse_args() + try: + daemon(args.root) if args.phase == 'daemon' else stop(args.root) + except (OSError, ValueError, subprocess.SubprocessError) as error: + print('Tailscale client failed: ' + type(error).__name__, file=sys.stderr) + return 1 + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/.github/scripts/interop-endpoint.py b/.github/scripts/interop-endpoint.py new file mode 100644 index 000000000..cb4f29107 --- /dev/null +++ b/.github/scripts/interop-endpoint.py @@ -0,0 +1,396 @@ +import argparse +from contextlib import ExitStack +import ipaddress +import json +import os +from pathlib import Path +import pwd +import re +import signal +import subprocess +import sys +import tempfile +import time + +sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'testsuite')) +from harness.interoperability import PLATFORMS, endpoint_identity, seed, snapshot, transfer_cases, update +from harness.interop_metadata import (FEATURES, metadata_cases, metadata_filesystem, metadata_snapshot, + qualify, seed_metadata, supported) +from harness import xattrs +from harness.process import quote_command +from harness.ssh import ProcessServer, SSHServer, client_command, command, public_key + +SSH_PORT, DAEMON_PORT = 18731, 18733 + + +def name(case): + if tuple(case) not in transfer_cases() + metadata_cases(): + raise ValueError('Unknown transfer case') + return '-'.join(case) + + +def publish(root, filename, data): + temporary = root / (filename + '.tmp') + temporary.write_text(json.dumps(data, sort_keys=True) + '\n') + temporary.replace(root / (filename + '.json')) + + +def read(root, filename): + with (root / (filename + '.json')).open('rb') as stream: + data = stream.read(65537) + if len(data) > 65536: + raise ValueError('Endpoint record exceeds limit') + value = json.loads(data) + if not isinstance(value, dict): + raise ValueError('Invalid endpoint record') + return value + + +def local_cases(role, field): + return [case for case in transfer_cases() if case[field] == role] + + +def serve(root, role, binary, rrsync, address='127.0.0.1', revision=None): + storage = ExitStack() + try: + identity = endpoint_identity(binary, role) + if revision: + repository = Path(__file__).resolve().parents[2] + try: + actual = subprocess.check_output(['git', '-c', 'safe.directory=' + str(repository), '-C', str(repository), + 'rev-parse', 'HEAD'], stderr=subprocess.PIPE, text=True, timeout=10).strip() + except (OSError, subprocess.SubprocessError): + raise ValueError('Native endpoint revision check failed') from None + if actual != revision: + raise ValueError('Native endpoint has a different candidate revision') + identity['source_sha'] = actual + data = storage.enter_context(metadata_filesystem(root, role)) + (data / 'source').mkdir(parents=True) + (data / 'destination').mkdir() + for case in local_cases(role, 0): + seed(data / 'source' / name(case)) + for case in local_cases(role, 1): + (data / 'destination' / name(case)).mkdir() + seed(root / 'reference') + prepare_metadata(data, role, binary, identity) + env = {key: os.environ[key] for key in ('PATH', 'RUNNER_TRACKING_ID') if key in os.environ} + env.update(HOME=str(root), LC_ALL='C') + conf = root / 'rsyncd.conf' + conf.write_text(f'use chroot = no\nuid = {os.geteuid()}\ngid = {os.getegid()}\n' + f'log file = {root / "rsyncd.log"}\n' + f'[data]\npath = {data}\nread only = no\nmunge symlinks = no\n' + f'[backup]\npath = {data}\nread only = no\nmunge symlinks = no\nfake super = yes\n') + daemon = root / 'daemon' + daemon.mkdir() + with ExitStack() as resources: + # Cases have disjoint trees, so the per-directory convenience lock is unnecessary. + overlay = address != '127.0.0.1' + ssh = resources.enter_context(SSHServer(root / 'ssh', data, binary, rrsync, address=address, + port=SSH_PORT if overlay else 0, parallel=True)) + daemon = resources.enter_context(ProcessServer([str(binary), '--daemon', '--no-detach', '--config=' + str(conf)], + env, daemon, address, DAEMON_PORT if overlay else 0)) + if overlay: + resources.enter_context(ProcessServer([ssh.sshd, '-i', '-e', '-f', str(ssh.config)], + ssh.env, ssh.directory, address, 18732)) + keys = {key: public_key(' '.join((root / 'ssh' / (key + '.pub')).read_text().split()[:2])) + for key in ('client', 'host')} + publish(root, 'local', dict(identity=identity, ssh_user=ssh.user, host_key=keys['host'], + client_key=keys['client'], ssh_port=ssh.port, daemon_port=daemon.port)) + deadline = time.monotonic() + 3000 + while not (root / 'stop').exists(): + if time.monotonic() >= deadline: + raise ValueError('Endpoint service deadline expired') + time.sleep(0.1) + except BaseException as error: + publish(root, 'error', dict(error=str(error) if isinstance(error, ValueError) else type(error).__name__)) + raise + finally: + try: + storage.close() + except BaseException: + if not (root / 'error.json').exists(): + publish(root, 'error', dict(error='Metadata filesystem cleanup failed')) + raise + finally: + for filename in ('client', 'host'): + path = root / 'ssh' / filename + if path.exists(): + path.unlink() + (root / 'stopped').touch() + + +def prepare(role, binary, rrsync, address='127.0.0.1', revision=None): + root = Path(tempfile.mkdtemp(prefix='rsync-interop-', dir=pwd.getpwuid(os.geteuid()).pw_dir)) + env = {key: os.environ[key] for key in ('PATH', 'RUNNER_TRACKING_ID') if key in os.environ} + with (root / 'agent.log').open('wb') as log: + subprocess.Popen([sys.executable, '-B', '-S', str(Path(__file__).resolve()), 'serve', role, '--root', str(root), + '--binary', str(binary), '--rrsync', str(rrsync), '--address', address, + *(['--source-sha', revision] if revision else [])], env=env, stdin=subprocess.DEVNULL, + stdout=log, stderr=subprocess.STDOUT, start_new_session=True) + deadline = time.monotonic() + 90 + while not (root / 'local.json').exists(): + if (root / 'error.json').exists(): + raise ValueError(read(root, 'error')['error']) + if time.monotonic() >= deadline: + (root / 'stop').touch() + raise ValueError('Endpoint services did not start') + time.sleep(0.1) + return dict(read(root, 'local'), root=str(root), python=sys.executable, script=str(Path(__file__).resolve())) + + +def configure(root, role, nodes): + if not isinstance(nodes, dict) or set(nodes) != set(PLATFORMS): + raise ValueError('Incomplete endpoint set') + hosts, clients, addresses, host_keys = [], [], set(), set() + for peer, record in nodes.items(): + if not isinstance(record, dict) or not isinstance(record.get('identity'), dict): + raise ValueError('Invalid native endpoint record') + native = record['identity'] + system = native.get('system', '') + if (native.get('platform') != peer or not isinstance(system, str) + or not (system.startswith('CYGWIN_') if peer == 'cygwin' else system == PLATFORMS[peer]) + or not isinstance(native.get('binary_sha256'), str) or not re.fullmatch(r'[0-9a-f]{64}', native['binary_sha256']) + or not isinstance(record.get('ssh_user'), str) + or not re.fullmatch(r'[A-Za-z_][A-Za-z0-9_-]*\$?', record['ssh_user'])): + raise ValueError('Invalid native endpoint identity') + address = record['address'] + if not isinstance(address, str) or address in addresses or ipaddress.ip_address(address) not in ipaddress.ip_network('100.64.0.0/10'): + raise ValueError('Invalid or repeated overlay address') + addresses.add(address) + host_key = public_key(record['host_key']) + hosts.append(f'[{address}]:{SSH_PORT} ' + host_key) + host_keys.add(host_key) + clients.append(public_key(record['client_key'])) + own = read(root, 'local') + if (nodes[role]['identity'] != own['identity'] or nodes[role]['host_key'] != own['host_key'] + or nodes[role]['client_key'] != own['client_key'] or len(set(clients)) != len(PLATFORMS) + or len(host_keys) != len(PLATFORMS) or (root / 'peers.json').exists()): + raise ValueError('Topology replaced an endpoint identity or reused a key') + (root / 'ssh' / 'known_hosts').write_text('\n'.join(hosts) + '\n') + keys = root / 'ssh' / 'authorized_keys.tmp' + keys.write_text('\n'.join(clients) + '\n') + keys.chmod(0o600) + keys.replace(root / 'ssh' / 'authorized_keys') + publish(root, 'peers', nodes) + + +def copy(root, binary, peer, transport, push, local, remote_path, options, fake_remote=False): + if transport == 'ssh': + remote = f'{peer["ssh_user"]}@{peer["address"]}:{remote_path}' + protocol = ['--old-args', '--rsync-path=rsync', '-e', + quote_command(client_command(root / 'ssh', SSH_PORT, attempts=3))] + if fake_remote: + protocol.append('-M--fake-super') + else: + module = 'backup' if fake_remote else 'data' + remote = f'rsync://{peer["address"]}:{DAEMON_PORT}/{module}/{remote_path}' + protocol = [] + operands = [str(local) + '/', remote] if push else [remote, str(local) + '/'] + env = {key: os.environ[key] for key in ('PATH', 'RUNNER_TRACKING_ID') if key in os.environ} + return command([str(binary), *options, *protocol, *operands], env, timeout=60) + + +def prepare_metadata(data, role, binary, identity): + base = data / 'metadata' + for part in ('source', 'destination', 'opaque', 'restored'): + (base / part).mkdir(parents=True) + identity['metadata'] = qualify(binary, base / 'source/backup', role) + for kind in ('acl', 'xattr'): + seed_metadata(base / 'source' / kind, role, kind) + if FEATURES[role][1] and role != 'cygwin': + seed_metadata(base / 'source/empty-xattr', role, 'empty-xattr') + for case in metadata_cases(): + if case[1] == role: + (base / 'destination' / name(case)).mkdir() + + +def metadata_transfers(root, role, binary, nodes, phase): + commands, restored = [], [] + for case in metadata_cases(): + source, destination, client, transport, kind = case + if (source if client == 'push' else destination) != role: + continue + if phase == 2 and (kind == 'backup' or not supported(case)): + continue + source_path = 'metadata/source/' + kind + '/' + destination_path = 'metadata/destination/' + name(case) + '/' + peer = nodes[destination if client == 'push' else source] + local = root / 'data' / (source_path if client == 'push' else destination_path) + options = ['-rltp', '--numeric-ids', '-A' if kind in ('acl', 'backup') else '-X'] + if kind == 'backup': + if FEATURES[source][1]: + options.append('-X') + if client == 'pull': + options.append('--fake-super') + result = copy(root, binary, peer, transport, client == 'push', local, + destination_path if client == 'push' else source_path, options, + fake_remote=kind == 'backup' and client == 'push') + if supported(case): + if result.returncode: + raise ValueError(name(case) + ': metadata transfer failed: ' + result.stderr.decode(errors='replace')[-2048:]) + elif kind == 'empty-xattr' and destination == 'cygwin': + if result.returncode != 23 or b'Invalid argument' not in result.stderr: + raise ValueError(name(case) + ': empty NTFS xattr was not explicitly refused') + elif (result.returncode == 0 + or (b'ACLs are not supported' if kind == 'acl' else b'extended attributes are not supported') not in result.stderr): + raise ValueError(name(case) + ': unsupported metadata was not explicitly refused') + commands.append(list(case)) + if phase == 2: + for case in metadata_cases(): + if case[0] != role or case[-1] != 'backup': + continue + source, destination, client, transport, kind = case + target = root / 'data' / 'metadata/restored' / name(case) + opaque = root / 'data' / 'metadata/opaque' / name(case) + options = ['-rltpA', '--numeric-ids', *(['-X'] if FEATURES[source][1] else [])] + # Saved ACLs belong to the originating platform. The storage host + # must return their bytes without decoding them. + result = copy(root, binary, nodes[destination], transport, False, opaque, + 'metadata/destination/' + name(case) + '/', ['-rltXX', '--numeric-ids']) + if result.returncode: + raise ValueError(name(case) + ': opaque backup retrieval failed: ' + result.stderr.decode(errors='replace')[-2048:]) + env = {key: os.environ[key] for key in ('PATH', 'RUNNER_TRACKING_ID') if key in os.environ} + result = command([str(binary), *options, '--fake-super', '-M--super', + str(opaque) + '/', str(target) + '/'], env, timeout=60) + expected = root / 'data' / 'metadata/source/backup' + if result.returncode: + raise ValueError(name(case) + ': native backup restoration failed: ' + result.stderr.decode(errors='replace')[-2048:]) + try: + wanted = metadata_snapshot(expected, role, kind) + actual = metadata_snapshot(target, role, kind) + except ValueError as error: + raise ValueError(name(case) + ': ' + str(error)) from error + if wanted != actual: + raise ValueError(name(case) + ': restored metadata differs: ' + json.dumps(dict(expected=wanted, actual=actual), sort_keys=True)) + restored.append(list(case)) + return dict(metadata_commands=commands, metadata_restored=restored) + + +def transfers(root, role, binary, phase): + nodes = read(root, 'peers') + commands = [] + for case in transfer_cases(): + source, destination, client, transport = case + if (source if client == 'push' else destination) != role: + continue + source_path, destination_path = 'source/' + name(case) + '/', 'destination/' + name(case) + '/' + peer = nodes[destination if client == 'push' else source] + result = copy(root, binary, peer, transport, client == 'push', + root / 'data' / (source_path if client == 'push' else destination_path), + destination_path if client == 'push' else source_path, + ['-rltpH', *(['--checksum', '--delete'] if phase == 2 else [])]) + if result.returncode: + raise ValueError(f'{name(case)}: rsync exited {result.returncode}: ' + + result.stderr.decode(errors='replace')[-2048:]) + commands.append(list(case)) + result = dict(commands=commands, **metadata_transfers(root, role, binary, nodes, phase)) + publish(root, 'transferred-' + str(phase), result) + return result + + +def verify_metadata(root, role, phase): + base = root / 'data' / 'metadata' + expected = {kind: metadata_snapshot(base / 'source' / kind, role, kind, phase) + for kind, available in (('acl', FEATURES[role][0]), ('xattr', FEATURES[role][1]), + ('empty-xattr', FEATURES[role][1] and role != 'cygwin')) if available} + verified = [] + for case in metadata_cases(): + if case[1] != role: + continue + target, kind = base / 'destination' / name(case), case[-1] + if not supported(case): + if kind == 'empty-xattr' and role == 'cygwin': + if ((target / 'directory/file').read_bytes() != b'metadata transfer\n' + or 'rsync-ci-empty' in xattrs.dump_values(target / 'directory', target / 'directory/file')): + raise ValueError(name(case) + ': empty NTFS xattr refusal had an unexpected result') + elif any(target.iterdir()): + raise ValueError(name(case) + ': refused metadata transfer wrote destination files') + elif kind == 'backup': + for path in (target / 'directory', target / 'directory/file'): + if not xattrs.get_value('rsync.%aacl', path): + raise ValueError(name(case) + ': backup has no saved ACL') + if (target / 'directory/file').read_bytes() != b'metadata transfer\n': + raise ValueError(name(case) + ': backup file contents differ') + elif expected[kind] != metadata_snapshot(target, role, kind, phase): + raise ValueError(name(case) + ': destination ACLs or xattrs differ') + verified.append(list(case)) + if phase == 1: + for kind in ('acl', 'xattr'): + seed_metadata(base / 'source' / kind, role, kind, 2) + return verified + + +def verify(root, role, phase): + expected = snapshot(root / 'reference') + verified = [] + for case in local_cases(role, 1): + if snapshot(root / 'data' / 'destination' / name(case)) != expected: + raise ValueError(name(case) + ': transferred data or metadata differ') + verified.append(list(case)) + if phase == 1: + for case in local_cases(role, 0): + update(root / 'data' / 'source' / name(case)) + update(root / 'reference') + result = dict(verified=verified, metadata_verified=verify_metadata(root, role, phase)) + publish(root, 'verified-' + str(phase), result) + return result + + +def stop(root): + (root / 'stop').touch() + deadline = time.monotonic() + 15 + while not (root / 'stopped').exists(): + if (root / 'error.json').exists(): + raise ValueError(read(root, 'error')['error']) + if time.monotonic() >= deadline: + raise ValueError('Endpoint services did not stop') + time.sleep(0.1) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('phase', choices=('prepare', 'serve', 'configure', 'transfer1', 'next', 'transfer2', 'verify', 'stop')) + parser.add_argument('platform', choices=PLATFORMS) + parser.add_argument('--root', type=Path) + parser.add_argument('--binary', type=Path) + parser.add_argument('--rrsync', type=Path) + parser.add_argument('--address', default='127.0.0.1') + parser.add_argument('--source-sha') + args = parser.parse_args() + if (args.phase in ('prepare', 'serve') and (args.binary is None or args.rrsync is None) + or args.phase != 'prepare' and args.root is None): + parser.error('The phase is missing its endpoint paths') + if args.source_sha is not None and not re.fullmatch(r'[0-9a-f]{40}', args.source_sha): + parser.error('Invalid source revision') + def interrupted(signum, frame): + signal.signal(signal.SIGTERM, signal.SIG_IGN) + raise ValueError('Endpoint interrupted') + signal.signal(signal.SIGTERM, interrupted) + try: + result = {} + if args.phase in ('prepare', 'serve'): + binary, rrsync = args.binary.resolve(strict=True), args.rrsync.resolve(strict=True) + if args.phase == 'prepare': + result = prepare(args.platform, binary, rrsync, args.address, args.source_sha) + else: + serve(args.root, args.platform, binary, rrsync, args.address, args.source_sha) + elif args.phase == 'configure': + data = sys.stdin.buffer.read(65537) + if len(data) > 65536: + raise ValueError('Peer manifest exceeds limit') + configure(args.root, args.platform, json.loads(data)) + elif args.phase in ('transfer1', 'transfer2'): + result = transfers(args.root, args.platform, args.binary.resolve(strict=True), int(args.phase[-1])) + elif args.phase in ('next', 'verify'): + result = verify(args.root, args.platform, 1 if args.phase == 'next' else 2) + else: + stop(args.root) + print(json.dumps(result, sort_keys=True)) + except (OSError, ValueError, subprocess.SubprocessError) as error: + print('Endpoint failed: ' + str(error), file=sys.stderr) + return 1 + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/.github/scripts/interop-host.py b/.github/scripts/interop-host.py new file mode 100644 index 000000000..ab4232cf7 --- /dev/null +++ b/.github/scripts/interop-host.py @@ -0,0 +1,288 @@ +import argparse +import ipaddress +import json +import os +from pathlib import Path +import runpy +import shlex +import socket +import subprocess +import sys +import time + +from interop_ci import ROLES, STARTUP_TIMEOUT, PeerUnavailable, identity, published, records, write +from interop_enrol import decrypt, forget, generate +from interop_network import NetworkError, require_no_oidc + +client = runpy.run_path(str(Path(__file__).with_name('interop-client.py'))) + + +def path(value): + if sys.platform.startswith('cygwin') and len(value) > 2 and value[1] == ':': + value = subprocess.check_output(['cygpath', '-u', value], text=True, timeout=10).strip() + return Path(value) + + +def call(root, role, phase, data=None): + guest = os.environ.get('INTEROP_GUEST') == 'true' + workspace = path(os.environ['GITHUB_WORKSPACE']) + args = ['python3', '-B', '-S', str(workspace / '.github/scripts/interop-endpoint.py'), phase, role] + if phase == 'prepare': + args += ['--binary', str(workspace / ('rsync.exe' if role == 'cygwin' else 'rsync')), + '--rrsync', str(workspace / 'support/rrsync'), '--source-sha', identity()['source_sha']] + if not guest: + args += ['--address', json.loads((root / 'node.json').read_text())['address']] + else: + endpoint = json.loads((root / 'endpoint.json').read_text()) + args += ['--root', endpoint['root']] + if phase in ('transfer1', 'transfer2'): + args += ['--binary', str(workspace / ('rsync.exe' if role == 'cygwin' else 'rsync'))] + if guest: + remote = ['env', 'PATH=/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin:/usr/gnu/bin:/usr/bin:/bin:/usr/sbin:/sbin', *args] + args = ['ssh', '-oBatchMode=yes', '-oConnectTimeout=10', role, ' '.join(map(shlex.quote, remote))] + else: + args[0] = sys.executable + if not sys.platform.startswith('cygwin'): + args = ['sudo', '--preserve-env=RUNNER_TRACKING_ID', *args] + result = subprocess.run(args, input=None if data is None else json.dumps(data).encode(), + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + timeout=1600 if phase.startswith('transfer') else (120 if phase in ('prepare', 'next', 'verify') else 45)) + if result.returncode: + raise NetworkError(f'{role} {phase} failed: ' + result.stderr.decode(errors='replace')[-2048:]) + if len(result.stdout) > 65536: + raise NetworkError('Native endpoint output exceeds limit') + return json.loads(result.stdout) + + +def forward(root, role): + node = json.loads((root / 'node.json').read_text()) + endpoint = json.loads((root / 'endpoint.json').read_text()) + argv = ['ssh', '-N', '-oBatchMode=yes', '-oExitOnForwardFailure=yes', '-oConnectTimeout=10'] + for port, target in ((18731, endpoint['ssh_port']), (18732, endpoint['ssh_port']), (18733, endpoint['daemon_port'])): + argv += ['-L', f'{node["address"]}:{port}:127.0.0.1:{target}'] + argv.append(role) + with (root / 'forward.log').open('wb') as log, subprocess.Popen(argv, stdout=log, stderr=log) as process: + try: + deadline = time.monotonic() + 3000 + while not (root / 'forward-stop').exists() and time.monotonic() < deadline: + if process.poll() is not None: + raise NetworkError('Guest fixture forwarding exited') + time.sleep(0.1) + finally: + if process.poll() is None: + process.terminate() + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=5) + (root / 'forward-stopped').touch() + + +def connect(address, port): + try: + with socket.create_connection((address, port), timeout=2): + return True + except (socket.timeout, ConnectionRefusedError, ConnectionResetError, PermissionError): + return False + except OSError as error: + import errno + if error.errno in (errno.EHOSTUNREACH, errno.ENETUNREACH, errno.ETIMEDOUT): + return False + raise + + +def healthy(node): + if not all(connect(node['address'], port) for port in (18731, 18732, 18733)): + raise NetworkError('Local fixture services are not listening') + + +def network(root, role, phase): + own = json.loads((root / 'ready.json').read_text()) + topology = published('topology')['nodes'] + if set(topology) != set(ROLES) or topology[role] != own: + raise NetworkError('Network topology changed the local endpoint') + healthy(own) + if phase == 'quarantine': + if any(connect(peer['address'], 18731) for name, peer in topology.items() if name != role): + raise NetworkError('A quarantined peer is reachable') + elif phase == 'admit': + admission = published('admission') + if admission['nodes'] != topology: + raise NetworkError('Admission replaced the endpoint set') + deadline = time.monotonic() + 60 + for name, peer in topology.items(): + if name == role: + continue + while not all(connect(peer['address'], port) for port in (18731, 18733)): + if time.monotonic() >= deadline: + raise NetworkError('An admitted endpoint is unreachable') + time.sleep(1) + if connect(peer['address'], 18732): + raise NetworkError('A forbidden fixture port is reachable') + call(root, role, 'configure', topology) + else: + if published('revocation').get('revoked') is not True: + raise NetworkError('Network revocation was not recorded') + deadline = time.monotonic() + 60 + for name, peer in topology.items(): + if name == role: + continue + while any(connect(peer['address'], port) for port in (18731, 18733)): + if time.monotonic() >= deadline: + raise NetworkError('A revoked endpoint remains reachable') + time.sleep(1) + healthy(own) + if phase != 'admit': + write(root, phase, role=role, node_id=own['node_id'], passed=True) + else: + write(root, 'admitted', role=role, node_id=own['node_id'], passed=True) + + +def stop(root, role): + errors = [] + if (root / 'endpoint.json').exists(): + try: + call(root, role, 'stop') + except (OSError, ValueError, subprocess.SubprocessError) as error: + errors.append(type(error).__name__) + if (root / 'forward.started').exists(): + (root / 'forward-stop').touch() + deadline = time.monotonic() + 15 + while not (root / 'forward-stopped').exists(): + if time.monotonic() >= deadline: + errors.append('Guest forwarding did not stop') + break + time.sleep(0.1) + try: + client['stop'](root) + finally: + forget(root) + if errors: + raise NetworkError('; '.join(errors)) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('phase', choices=('install', 'recipient', 'connect', 'prepare', 'forward', 'quarantine', 'admit', + 'transfer1', 'wait1', 'next', 'transfer2', 'wait2', 'verify', + 'revoked', 'stop', 'result')) + parser.add_argument('platform', choices=(*ROLES, 'all')) + parser.add_argument('--root', type=Path, required=True) + args = parser.parse_args() + root = path(str(args.root)).resolve() + root.mkdir(mode=0o700, parents=True, exist_ok=True) + try: + require_no_oidc() + role, phase = args.platform, args.phase + if phase == 'stop': + stop(root, role) + return 0 + if phase == 'forward': + forward(root, role) + return 0 + if phase == 'install': + (root / 'client.json').write_text(json.dumps(client['install'](root))) + return 0 + state = identity() + if phase == 'recipient': + generate(root, role, state) + elif phase == 'connect': + enrolment = published('enrolment', timeout=STARTUP_TIMEOUT)['recipients'][role] + decrypt(root, role, enrolment, state) + try: + node = client['connect'](root, root / 'enrol-key', f'ci-{state["run_id"]}-{state["run_attempt"]}-{role}') + finally: + forget(root) + addresses = [value for value in node['TailscaleIPs'] if ':' not in value] + if len(addresses) != 1 or ipaddress.ip_address(addresses[0]) not in ipaddress.ip_network('100.64.0.0/10'): + raise NetworkError('No unique Tailscale IPv4 address') + deadline = time.monotonic() + 15 + while True: + try: + with socket.socket() as listener: + listener.bind((addresses[0], 0)) + break + except OSError: + if time.monotonic() >= deadline: + raise NetworkError('The allocated overlay address is not available locally') from None + time.sleep(0.1) + (root / 'node.json').write_text(json.dumps(dict(node_id=node['ID'], address=addresses[0], key_id=enrolment['key_id']))) + elif phase == 'prepare': + endpoint = call(root, role, 'prepare') + if endpoint.get('identity', {}).get('platform') != role: + raise NetworkError('The host substituted for the native endpoint') + (root / 'endpoint.json').write_text(json.dumps(endpoint)) + if os.environ.get('INTEROP_GUEST') == 'true': + env = {key: os.environ[key] for key in ('PATH', 'RUNNER_TRACKING_ID') if key in os.environ} + subprocess.Popen([sys.executable, '-B', '-S', str(Path(__file__).resolve()), 'forward', role, + '--root', str(root)], env=env, stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True) + (root / 'forward.started').touch() + own = json.loads((root / 'node.json').read_text()) + deadline = time.monotonic() + 15 + while not connect(own['address'], 18731): + if time.monotonic() >= deadline: + raise NetworkError('Guest fixture forwarding did not start') + time.sleep(0.1) + public = {key: endpoint[key] for key in ('identity', 'ssh_user', 'host_key', 'client_key')} + write(root, 'ready', **public, **json.loads((root / 'node.json').read_text()), role=role, oidc_available=False) + elif phase in ('quarantine', 'admit', 'revoked'): + network(root, role, phase) + elif phase in ('transfer1', 'transfer2'): + records('admitted' if phase == 'transfer1' else 'next', timeout=600) + try: + result = call(root, role, phase) + except (OSError, ValueError, subprocess.SubprocessError): + write(root, phase, failed=True) + raise + write(root, phase, **result) + elif phase in ('wait1', 'wait2'): + transfer = 'transfer' + phase[-1] + completed = records(transfer, timeout=1800) + failed = [name for name, record in completed.items() if record.get('failed') is True] + if failed: + raise PeerUnavailable('Failed transfer endpoints: ' + ', '.join(failed)) + elif phase in ('next', 'verify'): + records('transfer1' if phase == 'next' else 'transfer2', timeout=1800) + result = call(root, role, phase) + write(root, phase, **result) + if phase == 'verify': + own = json.loads((root / 'ready.json').read_text()) + initial = json.loads((root / 'transfer1.json').read_text()) + initial_check = json.loads((root / 'next.json').read_text()) + updated = json.loads((root / 'transfer2.json').read_text()) + write(root, 'result', identity=own['identity'], node_id=own['node_id'], passed=True, + commands=updated['commands'], verified=result['verified'], + metadata_commands=initial['metadata_commands'], metadata_updates=updated['metadata_commands'], + metadata_initial_verified=initial_check['metadata_verified'], + metadata_verified=result['metadata_verified'], metadata_restored=updated['metadata_restored']) + else: + jobs = json.loads(os.environ['INTEROP_JOBS']) + if set(jobs) != set(ROLES): + raise NetworkError('Incomplete endpoint job set') + incomplete = [role for role, job in jobs.items() + if job.get('result') != 'success' or job.get('outputs', {}).get('blocked') != 'false'] + if incomplete: + raise NetworkError('Failed or blocked endpoints: ' + ', '.join(incomplete)) + broker = runpy.run_path(str(Path(__file__).with_name('interop-broker.py'))) + broker['verify_cases'](records('result'), state['source_sha']) + proof = published('result') + if proof.get('passed') is not True or proof.get('cleaned') is not True or proof.get('roles') != list(ROLES): + raise NetworkError('Missing verified network cleanup') + print('Verified directed transfers, metadata restoration and network cleanup') + except (OSError, ValueError, KeyError, subprocess.SubprocessError) as error: + if isinstance(error, PeerUnavailable) and args.phase not in ('result', 'stop'): + write(root, 'blocked', role=args.platform, phase=args.phase, passed=False, reason=str(error)) + with open(os.environ['GITHUB_ENV'], 'a') as stream: + stream.write('INTEROP_BLOCKED=true\n') + print('Interoperability blocked: ' + str(error)) + return 0 + detail = str(error) if isinstance(error, NetworkError) else type(error).__name__ + print('Interoperability failed: ' + detail, file=sys.stderr) + return 1 + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/.github/scripts/interop_ci.py b/.github/scripts/interop_ci.py index c8c179767..060d85585 100644 --- a/.github/scripts/interop_ci.py +++ b/.github/scripts/interop_ci.py @@ -4,11 +4,13 @@ import re import time +from changes import select as select_changes from interop_network import NetworkError, decode, download_records, fetch from interop_policy import ROLES API = 'https://api.github.com/repos/RsyncProject/rsync' WORKFLOW = 'RsyncProject/rsync/.github/workflows/interoperability-tests.yml@' +STARTUP_TIMEOUT = 1800 TIMESTAMP = r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z' bound = None api_cache = {} @@ -22,8 +24,9 @@ class PeerUnavailable(NetworkError): pass -def api(path): - cached = api_cache.get(path) +def api(path, array=False): + key = path, array + cached = api_cache.get(key) try: body, headers = fetch(API + path, os.environ['GH_TOKEN'], headers={'If-None-Match': cached[0]} if cached else {}, operation='Read CI identity') @@ -31,12 +34,78 @@ def api(path): if error.code == 304 and cached: return cached[1] raise - value = decode(body) + if array: + try: + value = json.loads(body) + except ValueError: + raise NetworkError('Invalid JSON response') from None + if not isinstance(value, list): + raise NetworkError('Expected a JSON array') + else: + value = decode(body) if headers.get('ETag'): - api_cache[path] = headers['ETag'], value + api_cache[key] = headers['ETag'], value return value +def pull_paths(run): + requests = run.get('pull_requests') + if not isinstance(requests, list) or len(requests) != 1: + return None + request = requests[0] + if not isinstance(request, dict): + return None + number = request.get('number') + base = request.get('base') + head = request.get('head') + if (type(number) is not int or number < 1 + or not isinstance(base, dict) or not isinstance(head, dict) + or not isinstance(base.get('repo'), dict) + or base.get('repo', {}).get('id') != 266877886 + or not re.fullmatch(r'[0-9a-f]{40}', base.get('sha', '')) + or not re.fullmatch(r'[0-9a-f]{40}', head.get('sha', ''))): + return None + current = api('/pulls/' + str(number)) + if (not isinstance(current, dict) or current.get('number') != number + or not isinstance(current.get('base'), dict) + or not isinstance(current.get('head'), dict) + or not isinstance(current.get('base', {}).get('repo'), dict) + or current.get('base', {}).get('repo', {}).get('id') != 266877886 + or current.get('base', {}).get('sha') != base['sha'] + or current.get('head', {}).get('sha') != head['sha']): + return None + paths = [] + statuses = {'added', 'removed', 'modified', 'renamed', 'copied', 'changed', 'unchanged'} + for page in range(1, 31): + files = api('/pulls/' + str(number) + '/files?per_page=100&page=' + str(page), array=True) + if not isinstance(files, list) or len(files) > 100: + return None + for item in files: + if (not isinstance(item, dict) or item.get('status') not in statuses + or not isinstance(item.get('filename'), str)): + return None + if item['status'] == 'renamed': + if not isinstance(item.get('previous_filename'), str): + return None + paths.append(item['previous_filename']) + paths.append(item['filename']) + if len(files) < 100: + return paths or None + return None + + +def selected(run): + if run['event'] != 'pull_request': + return True + paths = pull_paths(run) + if paths is None: + return True + try: + return select_changes(paths)['interoperability'] is True + except (TypeError, ValueError): + return True + + @lru_cache(maxsize=1) def source(): if os.environ.get('GITHUB_REPOSITORY_ID') != '266877886': @@ -62,8 +131,11 @@ def source(): if (run.get('path') != '.github/workflows/ci.yml' or run.get('name') != 'CI' or run.get('id') != run_id or run.get('repository', {}).get('id') != 266877886 or run.get('run_attempt') != attempt or run.get('event') not in ('pull_request', 'push', 'schedule', 'workflow_dispatch') - or not re.fullmatch(TIMESTAMP, run.get('created_at', ''))): + or not re.fullmatch(TIMESTAMP, run.get('created_at', '')) + or not isinstance(run.get('display_title'), str) or not run['display_title']): raise NetworkError('CI run does not match its event') + if not selected(run): + raise NotSelected('CI has not selected interoperability') references = [item for item in run.get('referenced_workflows', []) if item.get('path', '').startswith(WORKFLOW)] deadline = time.monotonic() + 180 while not references and run.get('status') in ('queued', 'in_progress') and time.monotonic() < deadline: @@ -79,7 +151,7 @@ def source(): if run.get('status') == 'completed': raise NotSelected('CI has already completed') return dict(run_id=str(run_id), run_attempt=str(attempt), source_sha=references[0]['sha'], - head_sha=run['head_sha'], created_at=run['created_at']) + head_sha=run['head_sha'], display_title=run['display_title'], created_at=run['created_at']) def identity(): @@ -197,7 +269,7 @@ def find_controller(exclude=None): def controller(): - deadline = time.monotonic() + 1800 + deadline = time.monotonic() + STARTUP_TIMEOUT while time.monotonic() < deadline: run = find_controller() if run is not None: diff --git a/.github/scripts/interop_enrol.py b/.github/scripts/interop_enrol.py index 2bbbb7218..01195d74a 100644 --- a/.github/scripts/interop_enrol.py +++ b/.github/scripts/interop_enrol.py @@ -2,7 +2,6 @@ import hashlib import json import os -from pathlib import Path import re import subprocess diff --git a/.github/scripts/termux.py b/.github/scripts/termux.py new file mode 100644 index 000000000..edb17325f --- /dev/null +++ b/.github/scripts/termux.py @@ -0,0 +1,225 @@ +from contextlib import redirect_stdout +import base64 +import hashlib +import json +from pathlib import Path, PurePosixPath +import re +import subprocess +import sys +import tempfile +import time +import urllib.request + +from android import check_transfer as transfer, quote_command, run + +PACKAGE = 'com.termux' +VERSION = '0.118.3' +APK_SHA256 = '3550e61f4d9eb49b712fd1bd9519dc37085a4d8eb597c57a340f0a64859b7144' +PREFIX = '/data/data/com.termux/files/usr' +HOME = '/data/data/com.termux/files/home' + + +def download_apk(path): + url = f'https://github.com/termux/termux-app/releases/download/v{VERSION}/termux-app_v{VERSION}%2Bgithub-debug_x86_64.apk' + digest, size = hashlib.sha256(), 0 + with urllib.request.urlopen(url, timeout=30) as source, path.open('xb') as output: + while True: + data = source.read(65536) + if not data: + break + size += len(data) + if size > 64 * 1024 * 1024: + raise ValueError('Termux APK exceeds the size limit') + digest.update(data) + output.write(data) + if digest.hexdigest() != APK_SHA256: + raise ValueError('Termux APK checksum differs') + + +def check_identity(uid, context, expected_uid): + if not 10000 <= expected_uid < 20000 or uid != str(expected_uid): + raise ValueError('Termux command did not use the application UID') + if not re.fullmatch(r'u:r:untrusted_app_27:s0:c\d+(?:,c\d+)*', context): + raise ValueError('Termux command did not use the application SELinux domain') + + +def wait_for_bootstrap(adb, timeout=180): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + pid = adb('shell', 'pidof', '-s', PACKAGE, capture_output=True, check=False).stdout.strip() + if re.fullmatch(r'\d+', pid): + log = adb('logcat', '-d', '--pid=' + pid, '-v', 'brief', '-t', '1000', + capture_output=True, encoding='utf-8', errors='replace').stdout + if 'Bootstrap Error:' in log: + raise ValueError('Termux bootstrap failed') + if 'Bootstrap packages installed successfully.' in log: + return + time.sleep(1) + raise TimeoutError('Termux bootstrap did not finish') + + +class Termux: + def __init__(self, adb, root): + if not re.fullmatch(re.escape(HOME) + r'/rsync-ci\.[A-Za-z0-9]{8}', root): + raise ValueError('Invalid Termux scratch directory') + self.adb, self.root, self.sequence = adb, root, 0 + + def run_as(self, *args, **kwargs): + return self.adb('shell', '-T', '-e', 'none', quote_command(['run-as', PACKAGE, *args]), **kwargs) + + def path(self, path): + if not path.startswith(self.root + '/') or '..' in PurePosixPath(path).parts or '\0' in path: + raise ValueError('Path is outside the Termux fixture') + return path + + def write_file(self, path, data): + if len(data) > 16 * 1024 * 1024: + raise ValueError('Termux fixture input exceeds the size limit') + self.run_as('sh', '-c', 'base64 -d >' + quote_command([self.path(path)]), input=base64.b64encode(data), + text=False, capture_output=True) + + def read_file(self, path, optional=False): + self.path(path) + if optional: + exists = self.run_as('test', '-f', path, check=False, capture_output=True) + if exists.returncode == 1: + return b'' + exists.check_returncode() + encoded = self.run_as('sh', '-c', 'set -o pipefail; head -c 1048577 ' + quote_command([path]) + ' | base64', + text=False, capture_output=True).stdout + data = base64.b64decode(b''.join(encoded.split()), validate=True) + if len(data) > 1048576: + raise ValueError('Termux fixture output exceeds the size limit') + return data + + def command(self, *args, capture_output=False, check=True, text=True, timeout=30): + self.sequence += 1 + script = f'{self.root}/command-{self.sequence}' + result = f'{self.root}/result-{self.sequence}' + self.write_file(script, ('#!' + PREFIX + '/bin/sh\nexec ' + quote_command(args) + '\n').encode()) + self.run_as('chmod', '700', script) + self.run_as(PREFIX + '/bin/am', 'startservice', '-n', PACKAGE + '/.app.RunCommandService', + '-a', PACKAGE + '.RUN_COMMAND', '--es', PACKAGE + '.RUN_COMMAND_PATH', script, + '--es', PACKAGE + '.RUN_COMMAND_WORKDIR', self.root, + '--ez', PACKAGE + '.RUN_COMMAND_BACKGROUND', 'true', + '--es', PACKAGE + '.RUN_COMMAND_RESULT_DIRECTORY', result, capture_output=True) + deadline = time.monotonic() + timeout + # Termux writes err atomically after the other result files. + while self.run_as('test', '-f', result + '/err', check=False, capture_output=True).returncode: + if time.monotonic() >= deadline: + raise TimeoutError('Termux command did not produce a result') + time.sleep(0.2) + error = self.read_file(result + '/err').decode().strip() + if error != '-1': + raise ValueError('Termux rejected the command: ' + error) + code = self.read_file(result + '/exit_code').decode().strip() + if not re.fullmatch(r'-?\d+', code): + raise ValueError('Invalid Termux command exit status') + stdout = self.read_file(result + '/stdout', optional=True) + stderr = self.read_file(result + '/stderr', optional=True) + value = subprocess.CompletedProcess(args, int(code), stdout.decode() if text else stdout, + stderr.decode() if text else stderr) + if check: + value.check_returncode() + if not capture_output: + sys.stdout.write(stdout.decode(errors='replace')) + sys.stderr.write(stderr.decode(errors='replace')) + return value + + +def check_transfer(adb, binary, output, device): + def shell(*args, **kwargs): + return adb('shell', '-T', quote_command(args), **kwargs) + packages = shell('pm', 'list', 'packages', PACKAGE, capture_output=True).stdout.splitlines() + if 'package:' + PACKAGE in packages: + raise ValueError('Termux is already installed on the test device') + owned = False + try: + with tempfile.TemporaryDirectory(prefix='rsync-termux-apk-') as directory: + apk = Path(directory) / 'termux.apk' + download_apk(apk) + owned = True + adb('install', str(apk), timeout=120, capture_output=True) + def run_as(*args, **kwargs): + return shell('run-as', PACKAGE, *args, **kwargs) + run_as('mkdir', '-p', 'files/home/.termux') + run_as('sh', '-c', 'cat >files/home/.termux/termux.properties', input='allow-external-apps=true\n', capture_output=True) + shell('am', 'start', '-n', PACKAGE + '/.app.TermuxActivity', capture_output=True) + wait_for_bootstrap(adb) + run_as('test', '-x', 'files/usr/bin/am', capture_output=True) + root = run_as('mktemp', '-d', HOME + '/rsync-ci.XXXXXXXX', capture_output=True).stdout.strip() + session = Termux(adb, root) + package = shell('pm', 'list', 'packages', '-U', '--user', '0', PACKAGE, capture_output=True).stdout + match = re.fullmatch(r'package:com\.termux uid:(\d+)\s*', package) + if not match: + raise ValueError('Cannot identify the Termux application UID') + uid = session.command('/system/bin/id', '-u', capture_output=True).stdout.strip() + context = session.command('/system/bin/id', '-Z', capture_output=True).stdout.strip() + check_identity(uid, context, int(match[1])) + candidate = root + '/rsync' + data = binary.read_bytes() + digest = hashlib.sha256(data).hexdigest() + session.write_file(candidate, data) + session.run_as('chmod', '700', candidate) + actual = session.command('sha256sum', candidate, capture_output=True).stdout.split()[0] + if actual != digest: + size = session.run_as('stat', '-c', '%s', candidate, capture_output=True).stdout.strip() + raise ValueError(f'Termux binary hash differs from the candidate: {actual}, {size} bytes') + with (output / 'transfer.log').open('w') as log: + with redirect_stdout(log): + transfer(session.command, candidate, root, hardlinks=False, + write_file=session.write_file, read_file=session.read_file) + private = root + '/source/private' + session.write_file(private, b'must not be copied\n') + session.command('chmod', '000', private) + refused = session.command(candidate, '-r', root + '/source/', root + '/refused/', + capture_output=True, check=False) + (output / 'permissions.log').write_text(refused.stdout + refused.stderr) + if refused.returncode != 23 or 'Permission denied' not in refused.stderr: + raise ValueError('Unreadable Termux source was not rejected') + session.command('test', '!', '-e', root + '/refused/private') + shell('test', '-d', '/data/system', capture_output=True) + session.command('test', '!', '-r', '/data/system') + receipt = dict(device=device, app=dict(package=PACKAGE, version=VERSION, apk_sha256=APK_SHA256, + uid=uid, selinux_context=context), binary_sha256=digest, + source=run('git', 'rev-parse', 'HEAD', capture_output=True).stdout.strip()) + except (OSError, ValueError, subprocess.SubprocessError): + if owned: + try: + pid = shell('pidof', '-s', PACKAGE, capture_output=True).stdout.strip() + if re.fullmatch(r'\d+', pid): + log = adb('logcat', '-d', '--pid=' + pid, '-v', 'brief', '-t', '200', + capture_output=True, encoding='utf-8', errors='replace') + (output / 'termux.log').write_text(log.stdout + log.stderr) + except (OSError, subprocess.SubprocessError): + print('Termux log capture failed', file=sys.stderr) + raise + finally: + if owned: + failed = sys.exc_info()[0] is not None + errors = [] + for command in (('shell', 'am', 'force-stop', PACKAGE), ('uninstall', PACKAGE)): + try: + value = adb(*command, timeout=30, capture_output=True, check=False) + if value.returncode: + errors.append(command[0]) + except (OSError, subprocess.SubprocessError): + errors.append(command[0]) + try: + remaining = shell('pm', 'list', 'packages', PACKAGE, capture_output=True).stdout.splitlines() + if 'package:' + PACKAGE in remaining: + errors.append('installation remains') + running = shell('pidof', '-s', PACKAGE, capture_output=True, check=False) + if running.returncode not in (0, 1): + errors.append('process check') + elif running.stdout.strip() or running.returncode == 0: + errors.append('application remains running') + except (OSError, subprocess.SubprocessError): + errors.append('cleanup verification') + if errors: + message = 'Termux cleanup failed: ' + ', '.join(errors) + if not failed: + raise ValueError(message) + print(message, file=sys.stderr) + receipt['app']['cleaned'] = True + (output / 'receipt.json').write_text(json.dumps(receipt, indent=2) + '\n') diff --git a/.github/scripts/test-android.py b/.github/scripts/test-android.py new file mode 100644 index 000000000..8d7cbb094 --- /dev/null +++ b/.github/scripts/test-android.py @@ -0,0 +1,198 @@ +from contextlib import redirect_stderr, redirect_stdout +import hashlib +from io import StringIO +import os +from pathlib import Path +import runpy +import subprocess +import sys +import tempfile +from types import SimpleNamespace +import unittest +from unittest.mock import Mock, patch + + +android = runpy.run_path(str(Path(__file__).with_name('android-emulator.py'))) +restore_binary = runpy.run_path(str(Path(__file__).with_name('android.py')))['restore_binary'] + + +class Android(unittest.TestCase): + def test_context_selection(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + binary = root / 'rsync' + binary.write_bytes(b'candidate') + image = root / 'system-images/android-36/google_apis/x86_64/source.properties' + image.parent.mkdir(parents=True) + image.write_text('Pkg.Revision=7\n') + def checked(adb, binary, output, *device): + (output / 'transfer.log').write_text('passed\n') + for app in (False, True): + shell, termux = Mock(side_effect=checked), Mock(side_effect=checked) + args = ['android-emulator.py', '--binary', str(binary), '--output', str(root / str(app))] + with self.subTest(termux=app), patch.object(sys, 'argv', args + (['--termux'] if app else [])), \ + patch.dict(os.environ, ANDROID_HOME=str(root)), patch('shutil.which', return_value='/emulator'), \ + patch('subprocess.Popen'), redirect_stdout(StringIO()), \ + patch.dict(android['main'].__globals__, run=Mock(return_value=SimpleNamespace(returncode=0, stdout='', stderr='')), + wait_for_boot=Mock(), stop_emulator=Mock(), check_device=Mock(return_value={}), + check_transfer=shell, check_termux=termux): + android['main']() + self.assertEqual(termux.call_count, int(app)) + self.assertEqual(shell.call_count, int(not app)) + + def test_restore(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + binary = root / 'rsync-3.5.2-android-x86_64' + binary.write_bytes(b'candidate') + digest = hashlib.sha256(binary.read_bytes()).hexdigest() + checksum = root / f'{binary.name}.sha256' + destination = root / 'rsync' + for value in ('not a checksum', f'{"0" * 64} {binary.name}\n', f'{digest} wrong-name\n'): + checksum.write_text(value) + with self.subTest(checksum=value), self.assertRaises(ValueError): + restore_binary(root, 'x86_64', destination) + self.assertFalse(destination.exists()) + checksum.write_text(f'{digest} {binary.name}\n') + restore_binary(root, 'x86_64', destination) + self.assertEqual(destination.read_bytes(), binary.read_bytes()) + with self.assertRaises(FileExistsError): + restore_binary(root, 'x86_64', destination) + binary.with_name('rsync-3.5.1-android-x86_64').write_bytes(b'other') + with self.assertRaises(ValueError): + restore_binary(root, 'x86_64', root / 'other') + with self.assertRaises(ValueError): + restore_binary(root, 'arm64-v8a', root / 'other') + + def device(self, changes=None): + values = {'getprop ro.build.version.sdk': '36', 'getprop ro.product.cpu.abi': 'x86_64', + 'getprop ro.kernel.qemu': '1', 'getenforce': 'Enforcing', 'id -u': '2000', + 'uname -r': '6.6.30-android15', 'getprop ro.build.fingerprint': 'test-device'} + values.update(changes or {}) + return lambda *args, **kwargs: SimpleNamespace(stdout=values[args[1]] + '\n') + + def test_device(self): + with redirect_stdout(StringIO()): + self.assertEqual(android['check_device'](self.device())['uid'], '2000') + for command, value in {'getprop ro.build.version.sdk': '35', + 'getprop ro.product.cpu.abi': 'aarch64', + 'getprop ro.kernel.qemu': '', 'getenforce': 'Permissive', + 'id -u': '0'}.items(): + with self.subTest(command=command), self.assertRaises(ValueError): + android['check_device'](self.device({command: value})) + + def test_boot(self): + process = SimpleNamespace(poll=lambda: None) + with patch('time.sleep'), patch('time.monotonic', side_effect=[0, 1]): + android['wait_for_boot'](lambda *a, **kw: SimpleNamespace(stdout='1\n'), process) + with self.assertRaises(RuntimeError): + android['wait_for_boot'](self.device(), SimpleNamespace(poll=lambda: 1)) + for failure in (SimpleNamespace(stdout=''), subprocess.CalledProcessError(1, 'adb'), + subprocess.TimeoutExpired('adb', 10)): + with patch('time.sleep'), patch('time.monotonic', side_effect=[0, 1, 301]), \ + self.assertRaises(TimeoutError): + def adb(*args, **kwargs): + if isinstance(failure, Exception): + raise failure + return failure + android['wait_for_boot'](adb, process) + + def test_permissions(self): + with tempfile.TemporaryDirectory() as directory: + for code, stderr in ((0, ''), (23, 'other error'), (12, 'Permission denied'), + (23, 'Permission denied')): + calls = [] + def adb(*args, **kwargs): + calls.append(args) + return SimpleNamespace(returncode=code, stdout='', stderr=stderr) + with self.subTest(code=code, stderr=stderr): + args = (adb, Path('rsync'), '/data/local/tmp/rsync-ci.12345678', Path(directory)) + if code == 23 and 'Permission denied' in stderr: + android['check_permissions'](*args) + self.assertIn(('shell', 'test ! -r /data/system'), calls) + self.assertTrue(any('test' in call[-1] and '/refused/private' in call[-1] + for call in calls)) + else: + with self.assertRaises(ValueError): + android['check_permissions'](*args) + + def test_stop(self): + adb = Mock() + process = Mock() + process.poll.return_value = 0 + android['stop_emulator'](adb, process) + adb.assert_not_called() + process.poll.return_value = None + android['stop_emulator'](adb, process) + adb.assert_called_once_with('emu', 'kill', timeout=10, capture_output=True) + process.terminate.assert_not_called() + process.reset_mock() + adb.side_effect = subprocess.CalledProcessError(1, 'adb') + process.wait.side_effect = [subprocess.TimeoutExpired('emulator', 10), 0] + android['stop_emulator'](adb, process) + process.terminate.assert_called_once() + process.kill.assert_called_once() + self.assertEqual(process.wait.call_count, 2) + + def test_transfer(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + binary = root / 'candidate rsync' + binary.write_bytes(b'candidate') + digest = hashlib.sha256(binary.read_bytes()).hexdigest() + for mismatch, failed, cleanup_error in ((False, False, False), (True, False, False), + (False, True, False), (True, False, True)): + try: + (root / 'receipt.json').unlink() + except FileNotFoundError: + pass + calls = [] + def adb(*args, **kwargs): + calls.append(args) + command = args[-1] + if cleanup_error and command.startswith('rm -rf'): + raise subprocess.CalledProcessError(1, 'adb cleanup') + if args[0] == 'pull': + Path(args[2]).write_bytes(b'raw\r\n\0data') + if command.startswith('mktemp'): + return SimpleNamespace(stdout='/data/local/tmp/rsync-ci.12345678\n') + if command.startswith('sha256sum'): + return SimpleNamespace(stdout=f'{"0" * 64 if mismatch else digest} rsync\n') + return SimpleNamespace(stdout='') + def fixture(command, *args, **kwargs): + if failed: + raise subprocess.CalledProcessError(1, 'rsync') + command('cat', 'data', capture_output=True, text=False) + kwargs['write_file']('sample', b'raw\r\n\0data') + self.assertEqual(Path(calls[-1][1]).read_bytes(), b'raw\r\n\0data') + self.assertEqual(kwargs['read_file']('sample'), b'raw\r\n\0data') + with patch.dict(android['check_transfer'].__globals__, + check_device=lambda adb: {}, check_permissions=lambda *a: None, + transfer=fixture), \ + patch('subprocess.run', return_value=SimpleNamespace(stdout='revision\n')), \ + redirect_stderr(StringIO()): + if mismatch or failed: + with self.assertRaises(ValueError if mismatch else subprocess.CalledProcessError): + android['check_transfer'](adb, binary, root) + self.assertFalse((root / 'receipt.json').exists()) + else: + android['check_transfer'](adb, binary, root) + self.assertTrue((root / 'receipt.json').is_file()) + self.assertIn(('shell', '-T', 'cat data'), calls) + self.assertEqual(calls[-1], ('shell', 'rm -rf /data/local/tmp/rsync-ci.12345678')) + + def test_scratch(self): + for work in ('', '/', '/data/local/tmp/rsync-ci.x/../../escape.12345678', + '/data/local/tmp/rsync-ci.12345678;echo bad'): + calls = [] + def adb(*args, **kwargs): + calls.append(args) + return SimpleNamespace(stdout=work) + with patch.dict(android['check_transfer'].__globals__, check_device=lambda adb: {}), \ + self.subTest(work=work), self.assertRaises(ValueError): + android['check_transfer'](adb, Path('rsync'), Path('.')) + self.assertEqual(len(calls), 1) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/test-archive.py b/.github/scripts/test-archive.py new file mode 100644 index 000000000..bf23d19b0 --- /dev/null +++ b/.github/scripts/test-archive.py @@ -0,0 +1,53 @@ +import io +from pathlib import Path +import tempfile +import tarfile +import unittest + +from archive import GEN_FILES, qualify + + +class Archive(unittest.TestCase): + def reject(self, change): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + archive = root / 'source.tar.gz' + entries = {f'rsync-3.5.2/{name}': (0o644, 0, b'generated\n') for name in GEN_FILES} + for name in ('configure', 'configure.sh', 'rsync-ssl', 'support/rrsync'): + entries[f'rsync-3.5.2/{name}'] = (0o755, 0, b'executable\n') + change(entries) + with tarfile.open(archive, 'w:gz') as output: + for name, (mode, uid, content) in entries.items(): + entry = tarfile.TarInfo(name) + entry.mode, entry.uid = mode, uid + if isinstance(content, tuple): + entry.type, entry.linkname = tarfile.SYMTYPE, content[0] + output.addfile(entry) + else: + entry.size = len(content) + output.addfile(entry, io.BytesIO(content)) + with self.assertRaises(ValueError): + qualify(archive, root) + self.assertFalse((root / 'qualification.json').exists()) + + def test_missing_generated(self): + self.reject(lambda entries: entries.pop('rsync-3.5.2/rsync.1')) + + def test_executable_mode(self): + self.reject(lambda entries: entries.update({'rsync-3.5.2/configure': (0o644, 0, b'configure')})) + + def test_ownership(self): + self.reject(lambda entries: entries.update({'rsync-3.5.2/file': (0o644, 1000, b'file')})) + + def test_paths(self): + for path in ('../outside', 'rsync-3.5.2/../outside', 'rsync-3.5.2/.git/config', + 'rsync-3.5.2/.github/workflows/ci.yml'): + with self.subTest(path=path): + self.reject(lambda entries: entries.update({path: (0o644, 0, b'file')})) + + def test_link_escape(self): + self.reject(lambda entries: entries.update({'rsync-3.5.2/link': (0o777, 0, ('../../outside',))})) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/test-changes.py b/.github/scripts/test-changes.py new file mode 100644 index 000000000..0cec8e3d7 --- /dev/null +++ b/.github/scripts/test-changes.py @@ -0,0 +1,129 @@ +import json +import os +from pathlib import Path +import runpy +import subprocess +import tempfile +import unittest +from unittest.mock import patch + + +changes = runpy.run_path(str(Path(__file__).with_name('changes.py'))) +select = changes['select'] +families = changes['FAMILIES'] + + +class Changes(unittest.TestCase): + def enabled(self, paths): + return {name for name, value in select(paths).items() if value} + + def test_documentation_selects_no_test_family(self): + for paths in (['README.md'], ['SECURITY.md', '.github/INTEROPERABILITY.md'], + ['INSTALL.md', 'NEWS.md']): + with self.subTest(paths=paths): + self.assertEqual(self.enabled(paths), set()) + + def test_generated_document_inputs_select_build_qualification(self): + self.assertEqual(self.enabled(['rsync.1.md']), {*families, 'asan', 'coverage', 'clang'}) + for path in ('rsync-ssl.1.md', 'rsyncd.conf.5.md', 'support/rrsync.1.md', + 'rsync.1', 'rsync.1.html'): + with self.subTest(path=path): + self.assertEqual(self.enabled([path]), {'archive', 'diagnostics'}) + + def test_production_and_unknown_paths_fail_closed(self): + for path in ('receiver.c', 'Makefile.in', 'packaging/release.py', + 'README.c', + '.github/scripts/new-helper.py', '.github/workflows/new.yml', + '.github/actions/new/action.yml'): + with self.subTest(path=path): + self.assertEqual(self.enabled([path]), {*families, 'asan', 'coverage', 'clang'}) + + def test_testsuite_avoids_unrelated_product_lanes(self): + expected = {'linux', 'bsd', 'macos', 'other', 'versions', 'diagnostics', 'asan'} + self.assertEqual(self.enabled(['testsuite/tests/transfer/exclude.py']), expected) + self.assertEqual(self.enabled(['testsuite/harness/runner.py']), expected | {'valgrind'}) + self.assertEqual(self.enabled(['testsuite/harness/interop_metadata.py']), + {'diagnostics', 'interoperability'}) + self.assertEqual(self.enabled(['testsuite/profiles/valgrind.json']), + {'valgrind', 'diagnostics'}) + self.assertEqual(self.enabled(['testsuite/profiles/archive.json']), + {'archive', 'diagnostics'}) + self.assertEqual(self.enabled(['testsuite/profiles/asan.json']), + {'asan', 'diagnostics'}) + self.assertEqual(self.enabled(['testsuite/tools/check-storage.py']), + {'linux', 'diagnostics'}) + + def test_ci_files_select_their_owner(self): + cases = { + '.github/workflows/linux.yml': {'linux', 'diagnostics'}, + '.github/workflows/ci.yml': {*families, 'asan', 'coverage', 'clang'}, + '.github/workflows/diagnostics.yml': {'diagnostics', 'asan', 'coverage', 'clang'}, + '.github/workflows/interoperability.yml': {'interoperability', 'diagnostics'}, + '.github/scripts/android-emulator.py': {'android', 'diagnostics'}, + '.github/scripts/test-archive.py': {'archive', 'diagnostics'}, + '.github/scripts/guest.py': {'linux', 'bsd', 'other', 'diagnostics'}, + '.github/scripts/diagnostics.py': {'diagnostics', 'valgrind', 'asan', 'coverage', 'clang'}, + '.github/scripts/test-diagnostics.py': {'diagnostics'}, + '.github/scripts/check-results.py': {*families, 'asan', 'coverage', 'clang'}, + '.github/scripts/test-results.py': {'diagnostics'}, + '.github/scripts/changes.py': {*families, 'asan', 'coverage', 'clang'}, + '.github/scripts/test-changes.py': {'diagnostics'}, + '.github/scripts/check-platform.py': {'linux', 'macos', 'other', 'diagnostics'}, + '.github/actions/interoperability/action.yml': {'interoperability', 'diagnostics'}, + } + for path, expected in cases.items(): + with self.subTest(path=path): + self.assertEqual(self.enabled([path]), expected) + + def test_multiple_paths_union_their_owners(self): + self.assertEqual(self.enabled(['README.md', '.github/scripts/android.py', + 'testsuite/tests/transfer/exclude.py']), + {'linux', 'bsd', 'macos', 'other', 'versions', + 'android', 'diagnostics', 'asan'}) + + def test_invalid_or_empty_paths_are_rejected(self): + for paths in ([], ['../receiver.c'], ['/receiver.c']): + with self.subTest(paths=paths), self.assertRaises(ValueError): + select(paths) + + def test_pull_request_revisions_and_git_result_are_validated(self): + event = {'pull_request': {'base': {'sha': 'a' * 40}, 'head': {'sha': 'b' * 40}}} + result = subprocess.CompletedProcess([], 0, b'README.md\0receiver.c\0', b'') + with patch.object(changes['subprocess'], 'run', return_value=result) as run: + self.assertEqual(changes['changed_paths'](event), ['README.md', 'receiver.c']) + self.assertEqual(run.call_args[0][0][-1], 'a' * 40 + '...' + 'b' * 40) + self.assertIn('--no-renames', run.call_args[0][0]) + self.assertIn('--diff-filter=ACDMRTUXB', run.call_args[0][0]) + for mutation in ({}, {'pull_request': {}}, + {'pull_request': {'base': {'sha': 'x'}, 'head': {'sha': 'b' * 40}}}): + with self.subTest(event=mutation), self.assertRaises(ValueError): + changes['changed_paths'](mutation) + with patch.object(changes['subprocess'], 'run', return_value=subprocess.CompletedProcess([], 1, b'', b'')), \ + self.assertRaises(ValueError): + changes['changed_paths'](event) + + def test_main_writes_every_output(self): + with tempfile.TemporaryDirectory() as directory: + output = Path(directory) / 'output' + event = Path(directory) / 'event.json' + event.write_text(json.dumps({'pull_request': {'base': {'sha': 'a' * 40}, + 'head': {'sha': 'b' * 40}}})) + git = subprocess.CompletedProcess([], 0, b'README.md\0', b'') + with patch.dict(os.environ, GITHUB_EVENT_NAME='pull_request', GITHUB_EVENT_PATH=str(event), + GITHUB_OUTPUT=str(output)), patch.object(changes['subprocess'], 'run', return_value=git): + self.assertEqual(changes['main'](), 0) + values = dict(line.split('=', 1) for line in output.read_text().splitlines()) + self.assertEqual(set(values), set(changes['OUTPUTS'])) + self.assertEqual(set(values.values()), {'false'}) + + def test_non_pull_request_selects_every_family(self): + with tempfile.TemporaryDirectory() as directory: + output = Path(directory) / 'output' + with patch.dict(os.environ, GITHUB_EVENT_NAME='push', GITHUB_OUTPUT=str(output)): + self.assertEqual(changes['main'](), 0) + values = dict(line.split('=', 1) for line in output.read_text().splitlines()) + self.assertEqual(set(values.values()), {'true'}) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/test-diagnostics.py b/.github/scripts/test-diagnostics.py new file mode 100644 index 000000000..f92c78259 --- /dev/null +++ b/.github/scripts/test-diagnostics.py @@ -0,0 +1,77 @@ +from contextlib import redirect_stdout +from io import StringIO +import os +from pathlib import Path +import runpy +import sys +import tempfile +import unittest +from unittest.mock import patch + + +diagnostics = runpy.run_path(str(Path(__file__).with_name('diagnostics.py'))) + + +class Diagnostics(unittest.TestCase): + def test_logged_failure(self): + with tempfile.TemporaryDirectory(prefix='rsync-diagnostics-') as directory: + log = Path(directory) / 'run.log' + output = StringIO() + with redirect_stdout(output): + status = diagnostics['logged']([sys.executable, '-S', '-c', + 'print("retained output"); raise SystemExit(17)'], log) + self.assertEqual(status, 17) + self.assertEqual(log.read_text(), 'retained output\n') + self.assertEqual(output.getvalue(), log.read_text()) + + def test_valgrind_results(self): + with tempfile.TemporaryDirectory(prefix='rsync-diagnostics-') as directory: + root = Path(directory) + output = root / 'scratch' / 'testtmp' / 'valgrind-logs' + output.mkdir(parents=True) + previous = Path.cwd() + os.chdir(root) + try: + argv = ['diagnostics.py', 'valgrind-results', '--privilege', 'non-root', '--transport', 'pipe'] + with patch.dict(os.environ, {'VALGRIND_SCRATCH': str(root)}), \ + patch.object(sys, 'argv', argv), redirect_stdout(StringIO()): + with self.assertRaisesRegex(ValueError, 'No Valgrind logs'): + diagnostics['main']() + path = output / 'valgrind.1.log' + path.write_text('ERROR SUMMARY: 0 errors\n') + self.assertEqual(diagnostics['main'](), 0) + self.assertEqual((root / 'test-results/valgrind-logs' / path.name).read_text(), path.read_text()) + path.write_text('ERROR SUMMARY: 2 errors\n') + with self.assertRaisesRegex(ValueError, 'Valgrind reported errors'): + diagnostics['main']() + finally: + os.chdir(previous) + + def test_summary(self): + with tempfile.TemporaryDirectory(prefix='rsync-diagnostics-') as directory: + root = Path(directory) + log = root / 'build.log' + log.write_text('compiler output\nlines: 50%\n') + summary = root / 'summary.md' + with patch.dict(os.environ, {'GITHUB_STEP_SUMMARY': str(summary)}), redirect_stdout(StringIO()): + diagnostics['summary']('Coverage', log, r'^lines:') + self.assertIn('lines: 50%', summary.read_text()) + self.assertNotIn('compiler output', summary.read_text()) + + def test_scan_flags_and_exit_status(self): + for status in (0, 17): + with self.subTest(status=status): + calls = [] + def logged(command, path): + calls.append(command) + return status + argv = ['diagnostics.py', 'scan'] + with patch.object(sys, 'argv', argv), patch.dict(diagnostics['main'].__globals__, + {'logged': logged, 'summary': lambda *args: None}): + self.assertEqual(diagnostics['main'](), status) + self.assertEqual(calls[0][0], 'scan-build-21') + self.assertIn('--status-bugs', calls[0]) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/test-guest.py b/.github/scripts/test-guest.py new file mode 100644 index 000000000..7b0abbfb7 --- /dev/null +++ b/.github/scripts/test-guest.py @@ -0,0 +1,94 @@ +from contextlib import redirect_stderr, redirect_stdout +from io import StringIO +from pathlib import Path +import runpy +import sys +import tempfile +from types import SimpleNamespace +import unittest +from unittest.mock import patch + + +guest = runpy.run_path(str(Path(__file__).with_name('guest.py'))) + + +class Guest(unittest.TestCase): + def run_guest(self, platform, failure=None): + with tempfile.TemporaryDirectory(prefix='rsync-guest-') as directory: + output = Path(directory) / 'results' + calls = [] + log = StringIO() + account = SimpleNamespace(pw_name='rsynctest', pw_uid=1000, pw_gid=1000, pw_dir='/home/rsynctest') + def process(command, **kwargs): + code = 1 if failure and failure(command) else 0 + calls.append((command, kwargs)) + return SimpleNamespace(wait=lambda: code, poll=lambda: code) + argv = ['guest.py', platform, '--release', '11.0', '--output', str(output)] + system = {'freebsd': 'FreeBSD', 'netbsd': 'NetBSD', 'openbsd': 'OpenBSD', 'solaris': 'SunOS'}[platform] + with patch.object(sys, 'argv', argv), patch('platform.system', return_value=system), \ + patch('platform.release', return_value='11.0'), patch('platform.machine', return_value='amd64'), \ + patch('pwd.getpwnam', return_value=account), patch('os.getgrouplist', return_value=[1000, 1001]), \ + patch('os.chown'), patch('subprocess.Popen', side_effect=process), \ + patch('subprocess.run', return_value=SimpleNamespace(returncode=0)), \ + redirect_stdout(log), redirect_stderr(StringIO()): + result = guest['main']() + return result, (output / 'status').read_text().strip(), calls, log.getvalue() + + def test_native_passes(self): + for platform in ('freebsd', 'netbsd', 'openbsd', 'solaris'): + with self.subTest(platform=platform): + result, status, calls, log = self.run_guest(platform) + self.assertEqual((result, status), (0, '0')) + tests = [(command, options) for command, options in calls if 'testsuite/tools/check.py' in command] + self.assertEqual(len(tests), 2 if platform == 'solaris' else 4) + for role in ('root',) if platform == 'solaris' else ('root', 'non-root'): + for transport in ('pipe', 'TCP'): + self.assertIn(f'::group::Tests ({role} {transport})\n', log) + for command, options in tests: + if 'nonroot' in command[5].split(','): + self.assertEqual(options['user'], 1000) + self.assertEqual(options['group'], 1000) + self.assertEqual(options['extra_groups'], [1000, 1001]) + self.assertEqual(options['env']['HOME'], '/home/rsynctest') + else: + self.assertNotIn('user', options) + if platform == 'openbsd': + self.assertTrue(all('-j2' in command for command, _ in tests if '--use-tcp' in command)) + if platform in ('netbsd', 'solaris'): + configure = next(command for command, _ in calls if command[0] == './configure') + self.assertIn('--disable-zstd', configure) + self.assertIn('--disable-idn', configure) + + def test_failed_pass_does_not_hide_remaining_results(self): + result, status, calls, _ = self.run_guest('netbsd', lambda command: 'testsuite/tools/check.py' in command) + self.assertEqual((result, status), (0, '1')) + self.assertEqual(sum('testsuite/tools/check.py' in command for command, _ in calls), 4) + + def test_failed_build_stops_tests(self): + result, status, calls, _ = self.run_guest('netbsd', lambda command: command[:2] == ['make', 'check-progs']) + self.assertEqual((result, status), (0, '1')) + self.assertFalse(any('testsuite/tools/check.py' in command for command, _ in calls)) + + def test_missing_probe_does_not_stop_tests(self): + def missing(command): + if command[0].endswith('/dirfd-probe'): + raise FileNotFoundError(command[0]) + return False + result, status, calls, _ = self.run_guest('netbsd', missing) + self.assertEqual((result, status), (0, '1')) + self.assertEqual(sum('testsuite/tools/check.py' in command for command, _ in calls), 4) + + def test_host_status(self): + with tempfile.TemporaryDirectory(prefix='rsync-guest-') as directory: + path = Path(directory) / 'status' + for status in ('0', '0\n', '1', '', '00', 'false', '0\n1', ' 0', '0 ', '0\r\n'): + path.write_text(status) + with patch.object(sys, 'argv', ['guest.py', 'check', '--output', directory]), redirect_stderr(StringIO()): + self.assertEqual(guest['main'](), 0 if status in ('0', '0\n') else 1) + path.unlink() + with patch.object(sys, 'argv', ['guest.py', 'check', '--output', directory]), redirect_stderr(StringIO()): + self.assertEqual(guest['main'](), 1) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/test-interop-metadata.py b/.github/scripts/test-interop-metadata.py new file mode 100644 index 000000000..2f73bbe8b --- /dev/null +++ b/.github/scripts/test-interop-metadata.py @@ -0,0 +1,217 @@ +from collections import Counter +import json +from pathlib import Path +import runpy +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import Mock, patch + +sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'testsuite')) +from harness import interop_metadata as metadata +from harness.interoperability import PLATFORMS + +endpoint = runpy.run_path(str(Path(__file__).with_name('interop-endpoint.py'))) + + +class Attributes(unittest.TestCase): + def test_command_failures_identify_the_tool(self): + failures = [FileNotFoundError(2, 'missing'), + subprocess.CalledProcessError(1, ['newfs'], stderr=b'cannot format device\n'), + subprocess.TimeoutExpired(['newfs'], 60)] + for failure in failures: + with self.subTest(failure=failure), patch.object(metadata.subprocess, 'run', side_effect=failure), \ + self.assertRaisesRegex(ValueError, 'newfs'): + metadata.command('newfs', '/dev/fixture') + with patch.object(metadata.subprocess, 'run', side_effect=failures[1]), \ + self.assertRaisesRegex(ValueError, 'cannot format device'): + metadata.command('newfs', '/dev/fixture') + + def test_freebsd_default_acls_are_separate(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) + for phase in (1, 2): + with self.subTest(phase=phase), patch.object(metadata, 'command') as command: + metadata.set_acl(path, 'freebsd', phase) + calls = [call[0] for call in command.call_args_list] + self.assertEqual(calls[:2], [('setfacl', '-b', path), ('setfacl', '-k', path)]) + self.assertEqual(calls[2][:2], ('setfacl', '-m')) + self.assertEqual(calls[3][:3], ('setfacl', '-d', '-m')) + self.assertEqual(calls[2][2], calls[3][3]) + self.assertNotIn('default:', calls[2][2]) + self.assertEqual('group:544:' in calls[2][2], phase == 1) + access = b'user::rwx\ngroup::r-x\nother::---\n' + default = b'user::rwx\ngroup::---\nother::---\n' + with patch.object(metadata, 'command', side_effect=[access, default]) as command: + result = metadata.read_acl(path, 'freebsd') + self.assertEqual(result['group:'], 'r-x') + self.assertEqual(result['default:group:'], '---') + self.assertEqual(command.call_args_list[1][0], ('getfacl', '-n', '-d', path)) + path.joinpath('file').touch() + with patch.object(metadata, 'command', return_value=access) as command: + metadata.read_acl(path / 'file', 'freebsd') + self.assertEqual(command.call_count, 1) + + def test_posix_backups_use_the_portable_group_form(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) + for role in ('linux', 'cygwin'): + with self.subTest(role=role), patch.object(metadata, 'command') as command: + metadata.set_acl(path, role, 1) + normal = command.call_args[0][2] + metadata.set_acl(path, role, 1, backup=True) + backup = command.call_args[0][2] + self.assertIn('group::---', normal) + self.assertIn('group::r-x', backup) + self.assertIn('mask::r-x', backup) + directory_acl = { + 'user:': 'rwx', 'group:': 'r-x', 'group:545': 'r-x', 'mask:': 'r-x', 'other:': '---', + 'group:544': '---', 'default:user:': 'rwx', 'default:group:': 'r-x', + 'default:group:545': 'r-x', 'default:mask:': 'r-x', 'default:other:': '---', + 'default:group:544': '---'} + file_acl = {'user:': 'rw-', 'group:': 'r--', 'group:545': 'r--', 'mask:': 'r--', + 'other:': '---', 'group:544': '---'} + (path / 'directory').mkdir() + (path / 'directory/file').write_text('metadata transfer\n') + values = {'rsync-ci': b'first', 'rsync-ci-large': b'A' * 512, + 'rsync-ci-remove': b'remove on update'} + with patch.object(metadata, 'read_acl', side_effect=[directory_acl, file_acl]), \ + patch.object(metadata.xattrs, 'get_value', side_effect=lambda key, _: values[key]): + snapshot = metadata.metadata_snapshot(path, 'linux', 'backup') + self.assertEqual(snapshot['entries']['directory']['acl'], directory_acl) + + def test_solaris_uses_current_acl_tools(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) + with patch.object(metadata, 'command') as command: + metadata.set_acl(path, 'solaris', 1) + self.assertEqual(command.call_args[0][0], '/usr/bin/chmod') + self.assertTrue(command.call_args[0][1].startswith('A=user::rwx,')) + self.assertIn('group:545:r-x', command.call_args[0][1]) + self.assertIn('default:group:545:r-x', command.call_args[0][1]) + self.assertEqual(command.call_args[0][2], path) + listing = b'''drwx------+ 2 root root 512 Nov 13 13:35 fixture + 0:user::rwx + 1:group::---\t\t#effective:--- + 2:group:544:---\t\t#effective:--- + 3:group:545:r-x\t\t#effective:r-x + 4:mask:r-x + 5:other:--- + 6:default:user::rwx + 7:default:group::--- + 8:default:group:544:--- + 9:default:group:545:r-x + 10:default:mask:r-x + 11:default:other:--- +''' + with patch.object(metadata, 'command', return_value=listing) as command: + result = metadata.read_acl(path, 'solaris') + self.assertEqual(result['group:545'], 'r-x') + self.assertEqual(result['default:group:545'], 'r-x') + self.assertEqual(command.call_args[0], ('/usr/bin/ls', '-Vd', path)) + + def test_case_contract(self): + self.assertEqual(set(metadata.FEATURES), set(PLATFORMS)) + cases = metadata.metadata_cases() + self.assertEqual(len(cases), len(set(cases))) + self.assertEqual(Counter(case[-1] for case in cases), {'xattr': 168, 'empty-xattr': 96, 'acl': 136, 'backup': 80}) + expected = {(source, destination, client, transport, 'xattr') + for source in PLATFORMS for destination in PLATFORMS if source != destination + for client in ('push', 'pull') for transport in ('ssh', 'daemon')} + self.assertEqual({case for case in cases if case[-1] == 'xattr'}, expected) + for source, destination, client, transport, kind in cases: + self.assertNotEqual(source, destination) + self.assertIn(client, ('push', 'pull')) + self.assertIn(transport, ('ssh', 'daemon')) + if kind == 'backup': + self.assertTrue(metadata.FEATURES[source][0]) + self.assertTrue(metadata.FEATURES[destination][1]) + elif kind == 'acl' and metadata.FEATURES[source][0] and metadata.FEATURES[destination][0]: + self.assertEqual(metadata.FEATURES[source][0], metadata.FEATURES[destination][0]) + self.assertIn(('macos', 'linux', 'push', 'ssh', 'backup'), cases) + self.assertIn(('netbsd', 'solaris', 'pull', 'daemon', 'acl'), cases) + self.assertIn(('openbsd', 'cygwin', 'push', 'ssh', 'xattr'), cases) + + def test_backups_are_decoded_only_on_the_origin(self): + case = ('linux', 'freebsd', 'push', 'ssh', 'backup') + copy, command = Mock(return_value=subprocess.CompletedProcess([], 0, b'', b'')), Mock(return_value=subprocess.CompletedProcess([], 0, b'', b'')) + with patch.dict(endpoint['metadata_transfers'].__globals__, metadata_cases=lambda: [case], copy=copy, + command=command, metadata_snapshot=lambda *args: {'fixture': True}): + result = endpoint['metadata_transfers'](Path('/fixture'), 'linux', Path('/rsync'), {'freebsd': {}}, 2) + self.assertEqual(copy.call_args[0][-1], ['-rltXX', '--numeric-ids']) + self.assertFalse(copy.call_args[1].get('fake_remote', False)) + self.assertIn('--fake-super', command.call_args[0][0]) + self.assertIn('-M--super', command.call_args[0][0]) + self.assertEqual(result['metadata_restored'], [list(case)]) + + def test_posix_acl_formats(self): + source = '# file: fixture\nuser::rw-\ngroup::---\ngroup:545:r-- #effective:r--\nmask::r--\nother::---\n' + value = metadata.parse_acl(source) + self.assertEqual(value['group:545'], 'r--') + self.assertEqual(metadata.parse_acl(source.replace('mask::', 'mask:').replace('other::', 'other:')), value) + with patch.object(metadata.grp, 'getgrnam', return_value=Mock(gr_gid=545)): + self.assertEqual(metadata.parse_acl(source.replace('group:545:', 'group:users:')), value) + for text in ('user::rw-\n', 'default\n', source + 'group:545:rw-\n', source + 'everyone@:rwx:allow\n', source.replace('r--', 'read')): + with self.subTest(text=text), self.assertRaises(ValueError): + metadata.parse_acl(text) + + def test_missing_binary_feature_is_not_unsupported(self): + for capabilities in ({'ACLs': False, 'xattrs': True}, {'ACLs': True, 'xattrs': False}): + with self.subTest(capabilities=capabilities), patch.object(metadata, 'command', return_value=json.dumps(dict(capabilities=capabilities)).encode()), \ + patch.object(metadata, 'seed_metadata') as seed, self.assertRaises(ValueError): + metadata.qualify(Path('/rsync'), Path('/fixture'), 'linux') + seed.assert_not_called() + + def test_fake_super_is_only_applied_to_the_backup_side(self): + root = Path('/fixture') + peer = dict(ssh_user='fixture', address='100.64.0.2') + for transport in ('ssh', 'daemon'): + command = Mock() + with self.subTest(transport=transport), patch.dict(endpoint['copy'].__globals__, command=command): + endpoint['copy'](root, Path('/rsync'), peer, transport, True, Path('/source'), 'metadata/destination/case/', + ['-rltpA'], fake_remote=True) + args = command.call_args[0][0] + if transport == 'ssh': + self.assertIn('-M--fake-super', args) + self.assertNotIn('--fake-super', args) + else: + self.assertTrue(args[-1].startswith('rsync://100.64.0.2:18733/backup/')) + self.assertNotIn('-M--fake-super', args) + + def test_disks_are_detached_when_a_mount_fails(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + calls = [] + def command(*args, **kwargs): + calls.append(args) + if args[0] == 'mdconfig' and '-a' in args: + return b'md8\n' + if args[0] == 'mount': + raise subprocess.CalledProcessError(1, args) + return b'' + with patch.object(metadata, 'command', side_effect=command), self.assertRaises(subprocess.CalledProcessError): + with metadata.metadata_filesystem(root, 'freebsd'): + self.fail('Unsuccessful mount was used') + self.assertIn(('mdconfig', '-d', '-u', '8'), calls) + self.assertFalse(any(call[0] == 'umount' for call in calls)) + + def test_incorrect_acl_and_xattr_values_fail(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / 'directory').mkdir() + (root / 'directory/file').write_text('metadata transfer\n') + with patch.object(metadata, 'read_acl', return_value={'user:': 'rw-', 'group:': '---', 'other:': '---'}), \ + self.assertRaises(ValueError): + metadata.metadata_snapshot(root, 'linux', 'acl') + with patch.object(metadata.xattrs, 'get_value', return_value=b'wrong'), self.assertRaises(ValueError): + metadata.metadata_snapshot(root, 'linux', 'xattr') + values = {'rsync-ci': b'second', 'rsync-ci-large': b'B' * 512} + with patch.object(metadata.xattrs, 'get_value', side_effect=lambda key, _: values[key]), \ + patch.object(metadata.xattrs, 'dump_values', return_value='user.rsync-ci-remove=stale'), \ + self.assertRaises(ValueError): + metadata.metadata_snapshot(root, 'linux', 'xattr', 2) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/test-interop.py b/.github/scripts/test-interop.py index b377d6fd5..8bccd21bf 100644 --- a/.github/scripts/test-interop.py +++ b/.github/scripts/test-interop.py @@ -76,10 +76,10 @@ def absent(): self.assertEqual(output.read_text(), 'enabled=false\n') self.assertFalse((self.root / 'context.json').exists()) - def test_recovery_uses_workflow_path_not_the_dynamic_run_title(self): + def test_recovery_uses_workflow_path_not_the_run_title(self): event = copy.deepcopy(self.event) event.update(action='completed', workflow=dict(path='.github/workflows/interoperability.yml')) - event['workflow_run'].update(name='Network CI 123 attempt 2', event='workflow_run') + event['workflow_run'].update(name='arbitrary title', event='workflow_run') (self.root / 'event.json').write_text(json.dumps(event)) network.workflow_event() event['workflow_run']['event'] = 'pull_request' @@ -93,25 +93,99 @@ def test_ci_identity_uses_the_merge_commit_and_accepts_forks(self): self.addCleanup(ci.source.cache_clear) run = dict(id=123, run_attempt=2, path='.github/workflows/ci.yml', name='CI', event='pull_request', repository=dict(id=266877886), head_repository=dict(id=999), head_sha='b' * 40, - created_at='2026-11-14T00:00:00Z', + display_title='testsuite, ci: rework tests', created_at='2026-11-14T00:00:00Z', + pull_requests=[dict(number=1, base=dict(sha='c' * 40, repo=dict(id=266877886)), + head=dict(sha='d' * 40, repo=dict(id=999)))], referenced_workflows=[dict(path=ci.WORKFLOW + 'a' * 40, sha='a' * 40)]) - with patch.object(ci, 'api', return_value=run): + request = dict(number=1, base=dict(sha='c' * 40, repo=dict(id=266877886)), + head=dict(sha='d' * 40, repo=dict(id=999))) + def production(path, array=False): + if path == '/actions/runs/123': + return run + if path == '/pulls/1': + return request + if path == '/pulls/1/files?per_page=100&page=1': + self.assertTrue(array) + return [dict(status='modified', filename='receiver.c')] + self.fail('Unexpected API request: ' + path) + with patch.object(ci, 'api', side_effect=production): self.assertEqual(ci.identity(), self.state) ci.source.cache_clear() - with patch.object(ci, 'api', return_value=dict(run, referenced_workflows=[])), self.assertRaises(ci.NotSelected): + docs = dict(run, referenced_workflows=[]) + def documentation(path, array=False): + if path == '/actions/runs/123': + return docs + if path == '/pulls/1': + return request + if path == '/pulls/1/files?per_page=100&page=1': + self.assertTrue(array) + return [dict(status='modified', filename='README.md')] + self.fail('Unexpected API request: ' + path) + with patch.object(ci, 'api', side_effect=documentation), \ + patch.object(time, 'sleep', side_effect=AssertionError('Unselected CI waited')), \ + self.assertRaises(ci.NotSelected): ci.source() + ci.source.cache_clear() + without_reference = dict(run, event='push', pull_requests=[], referenced_workflows=[]) + with patch.object(ci, 'api', return_value=without_reference), self.assertRaises(ci.NotSelected): + ci.source() + ci.source.cache_clear() + with patch.object(ci, 'api', return_value=dict(without_reference, display_title='')), self.assertRaises(network.NetworkError): + ci.source() + + def test_controller_uses_trusted_change_selection(self): + run = dict(event='pull_request', pull_requests=[dict( + number=1, base=dict(sha='a' * 40, repo=dict(id=266877886)), + head=dict(sha='b' * 40, repo=dict(id=999)))]) + request = dict(number=1, base=dict(sha='a' * 40, repo=dict(id=266877886)), + head=dict(sha='b' * 40, repo=dict(id=999))) + def files(items, current=request): + def response(path, array=False): + if path == '/pulls/1': + return current + if path == '/pulls/1/files?per_page=100&page=1': + self.assertTrue(array) + return items + self.fail('Unexpected API request: ' + path) + return response + with patch.object(ci, 'api', side_effect=files([dict(status='modified', filename='README.md')])): + self.assertFalse(ci.selected(run)) + with patch.object(ci, 'api', side_effect=files([dict(status='modified', filename='receiver.c')])): + self.assertTrue(ci.selected(run)) + renamed = [dict(status='renamed', filename='README.md', previous_filename='receiver.c')] + with patch.object(ci, 'api', side_effect=files(renamed)): + self.assertTrue(ci.selected(run)) + stale = dict(request, head=dict(sha='c' * 40, repo=dict(id=999))) + with patch.object(ci, 'api', side_effect=files([], stale)): + self.assertTrue(ci.selected(run)) + malformed = [dict(status='renamed', filename='README.md')] + with patch.object(ci, 'api', side_effect=files(malformed)): + self.assertTrue(ci.selected(run)) + + def test_api_preserves_the_expected_json_shape(self): + ci.api_cache.clear() + self.addCleanup(ci.api_cache.clear) + expected = [dict(status='modified', filename='README.md')] + with patch.object(ci, 'fetch', return_value=(json.dumps(expected).encode(), {})): + self.assertEqual(ci.api('/pulls/1/files', array=True), expected) + with patch.object(ci, 'fetch', return_value=(b'[]', {})), self.assertRaisesRegex( + network.NetworkError, 'JSON object'): + ci.api('/actions/runs/1') def test_controller_discovery_requires_trusted_code_and_exact_attempt(self): - origin = dict(self.state, head_sha='b' * 40, created_at='2026-11-14T00:00:00Z') - run = dict(id=456, run_attempt=1, path='.github/workflows/interoperability.yml', head_branch='master', - event='workflow_run', display_title='Network CI 123 attempt 2', repository=dict(id=266877886), - head_sha='c' * 40, status='in_progress', created_at='2026-11-14T00:00:01Z') + origin = dict(self.state, head_sha='b' * 40, display_title='testsuite, ci: rework tests', + created_at='2026-11-14T00:00:00Z') metadata = dict(self.state, control_run_id='456', control_attempt='1', control_sha='c' * 40) - with patch.object(ci, 'source', return_value=origin), \ - patch.object(ci, 'api', side_effect=[dict(total_count=1, workflow_runs=[run]), - dict(total_count=1, artifacts=[dict(name='interoperability-context-1', expired=False)])]), \ - patch.object(ci, 'download_records', return_value={None: (metadata, b'{}')}): - self.assertEqual(ci.controller(), '456') + titles = ('Network controller', 'Network CI 123 attempt 2') + for title in titles: + run = dict(id=456, run_attempt=1, path='.github/workflows/interoperability.yml', head_branch='master', + event='workflow_run', display_title=title, repository=dict(id=266877886), + head_sha='c' * 40, status='in_progress', created_at='2026-11-14T00:00:01Z') + with self.subTest(title=title), patch.object(ci, 'source', return_value=origin), \ + patch.object(ci, 'api', side_effect=[dict(total_count=1, workflow_runs=[run]), + dict(total_count=1, artifacts=[dict(name='interoperability-context-1', expired=False)])]), \ + patch.object(ci, 'download_records', return_value={None: (metadata, b'{}')}): + self.assertEqual(ci.controller(), '456') for field, value in dict(head_branch='untrusted', event='pull_request', repository=dict(id=1)).items(): with self.subTest(field=field), patch.object(ci, 'source', return_value=origin), \ patch.object(ci, 'api', return_value=dict( @@ -119,9 +193,11 @@ def test_controller_discovery_requires_trusted_code_and_exact_attempt(self): ci.controller() def test_first_controller_context_survives_a_later_duplicate(self): - origin = dict(self.state, head_sha='b' * 40, created_at='2026-11-14T00:00:00Z') + origin = dict(self.state, head_sha='b' * 40, display_title='testsuite, ci: rework tests', + created_at='2026-11-14T00:00:00Z') run = dict(id=456, run_attempt=1, path='.github/workflows/interoperability.yml', head_branch='master', - event='workflow_run', display_title='Network CI 123 attempt 2', repository=dict(id=266877886), + event='workflow_run', display_title='Network controller', + repository=dict(id=266877886), head_sha='c' * 40, status='completed', conclusion='success', created_at='2026-11-14T00:00:01Z') duplicate = dict(run, id=457, conclusion='failure') @@ -132,44 +208,45 @@ def test_first_controller_context_survives_a_later_duplicate(self): patch.object(ci, 'download_records', return_value={None: (metadata, b'{}')}): self.assertEqual(ci.controller(), '456') - def test_controller_listing_is_paged_and_stops_before_the_ci_run(self): - origin = dict(self.state, head_sha='b' * 40, created_at='2026-11-14T00:00:20Z') + def test_controller_listing_is_paged_and_bounded_to_the_ci_run(self): + origin = dict(self.state, head_sha='b' * 40, display_title='current', + created_at='2026-11-14T00:00:20Z') unrelated = [dict(id=500 + index, created_at='2026-11-14T00:00:30Z', display_title='other') for index in range(20)] current = dict(id=456, run_attempt=1, path='.github/workflows/interoperability.yml', - head_branch='master', event='workflow_run', display_title='Network CI 123 attempt 2', + head_branch='master', event='workflow_run', display_title='Network controller', repository=dict(id=266877886), head_sha='c' * 40, status='in_progress', created_at='2026-11-14T00:00:21Z') older = dict(current, id=455, created_at='2026-11-14T00:00:19Z') + artifact = dict(total_count=1, artifacts=[dict(name='interoperability-context-1', expired=False)]) metadata = dict(self.state, control_run_id='456', control_attempt='1', control_sha='c' * 40) - def listing(path): - if '/actions/workflows/' in path: - self.assertIn('per_page=20&page=', path) - if path.endswith('page=1'): - return dict(total_count=22, workflow_runs=unrelated) - self.assertTrue(path.endswith('page=2')) - return dict(total_count=22, workflow_runs=[current, older]) - self.assertEqual(path, '/actions/runs/456/artifacts?per_page=100') - return dict(total_count=1, artifacts=[dict(name='interoperability-context-1', expired=False)]) - with patch.object(ci, 'source', return_value=origin), patch.object(ci, 'api', side_effect=listing), \ + with patch.object(ci, 'source', return_value=origin), patch.object(ci, 'api', side_effect=[ + dict(total_count=22, workflow_runs=unrelated), + dict(total_count=22, workflow_runs=[current, older]), artifact]) as api, \ patch.object(ci, 'download_records', return_value={None: (metadata, b'{}')}): self.assertEqual(ci.controller(), '456') + self.assertIn('per_page=20&page=1', api.call_args_list[0][0][0]) + self.assertIn('per_page=20&page=2', api.call_args_list[1][0][0]) - def test_normal_controller_title_uses_the_exact_context_identity(self): - origin = dict(self.state, head_sha='b' * 40, created_at='2026-11-14T00:00:00Z') + def test_controller_title_collision_uses_the_context_identity(self): + origin = dict(self.state, head_sha='b' * 40, display_title='same title', + created_at='2026-11-14T00:00:00Z') run = dict(run_attempt=1, path='.github/workflows/interoperability.yml', head_branch='master', - event='workflow_run', display_title='Network controller', repository=dict(id=266877886), - head_sha='c' * 40, status='in_progress', created_at='2026-11-14T00:00:01Z') - foreign, current = dict(run, id=455), dict(run, id=456) - foreign_context = dict(self.state, run_id='122', control_run_id='455', control_attempt='1', control_sha='c' * 40) - current_context = dict(self.state, control_run_id='456', control_attempt='1', control_sha='c' * 40) + event='workflow_run', display_title='Network controller', + repository=dict(id=266877886), head_sha='c' * 40, status='in_progress', + created_at='2026-11-14T00:00:01Z') + stale, current = dict(run, id=455), dict(run, id=456) + stale_metadata = dict(self.state, run_id='122', control_run_id='455', + control_attempt='1', control_sha='c' * 40) + current_metadata = dict(self.state, control_run_id='456', control_attempt='1', control_sha='c' * 40) for name in ('interoperability-context-attempt-1', 'interoperability-context-1'): artifact = dict(total_count=1, artifacts=[dict(name=name, expired=False)]) with self.subTest(name=name), patch.object(ci, 'source', return_value=origin), \ patch.object(ci, 'api', side_effect=[ - dict(total_count=2, workflow_runs=[current, foreign]), artifact, artifact]), \ + dict(total_count=2, workflow_runs=[current, stale]), artifact, artifact]), \ patch.object(ci, 'download_records', side_effect=[ - {None: (foreign_context, b'{}')}, {None: (current_context, b'{}')}]) as download: + {None: (stale_metadata, b'{}')}, {None: (current_metadata, b'{}')}]) as download, \ + patch.object(time, 'sleep'): self.assertEqual(ci.controller(), '456') self.assertEqual(download.call_args[0][:2], ('456', {name: None})) @@ -200,19 +277,7 @@ def test_duplicate_controller_never_issues_another_enrolment(self): self.assertEqual(output.read_text(), 'enabled=false\n') self.assertFalse((self.root / 'context.json').exists()) - def test_controller_requires_all_directed_transfer_cases(self): - cases = [(a, b, client, transport) for a in ROLES for b in ROLES if a != b - for client in ('push', 'pull') for transport in ('ssh', 'daemon')] - self.assertEqual(len(cases), 168) - results = {role: dict(identity=dict(platform=role, source_sha=self.state['source_sha']), passed=True, - commands=[list(case) for case in cases if (case[0] if case[2] == 'push' else case[1]) == role], - verified=[list(case) for case in cases if case[1] == role]) for role in ROLES} - broker['verify_cases'](results, self.state['source_sha']) - for field in ('commands', 'verified'): - broken = copy.deepcopy(results) - broken['linux'][field].pop() - with self.subTest(field=field), self.assertRaises(network.NetworkError): - broker['verify_cases'](broken, self.state['source_sha']) + def test_controller_event_is_bound_to_repository_and_workflow(self): for field in ('path', 'owner', 'repository'): event = copy.deepcopy(self.event) if field == 'path': @@ -459,6 +524,31 @@ def test_key_cleanup_attempts_every_recorded_key(self): self.assertEqual(api.delete_key.call_count, 2) self.assertFalse((self.root / 'result.json').exists()) + def test_first_liveness_check_does_not_assume_runner_uptime(self): + def downloading(*args, **kwargs): + kwargs['progress']() + self.fail('Failed endpoint was ignored on the first poll') + jobs = dict(total_count=1, jobs=[dict(name='Interoperability / FreeBSD', conclusion='failure')]) + with patch.object(ci, 'api', side_effect=[dict(run_attempt=2, status='in_progress'), jobs]), \ + patch.object(ci, 'download_records', side_effect=downloading), \ + patch.object(time, 'monotonic', return_value=1), \ + self.assertRaisesRegex(ci.PeerUnavailable, 'FreeBSD'): + ci.records('public') + + def test_stopped_controller_ends_waits_but_result_remains_readable(self): + def downloading(*args, **kwargs): + kwargs['progress']() + return {None: (self.state, b'{}')} + responses = [dict(run_attempt=2, status='in_progress'), dict(total_count=0, jobs=[]), + dict(status='completed')] + with patch.object(ci, 'api', side_effect=responses), patch.object(time, 'monotonic', return_value=1), \ + patch.object(ci, 'download_records', side_effect=downloading), \ + self.assertRaisesRegex(ci.PeerUnavailable, 'controller has stopped'): + ci.records('enrolment', (None,), '456') + with patch.object(ci, 'api', side_effect=responses[:2]), patch.object(time, 'monotonic', return_value=1), \ + patch.object(ci, 'download_records', side_effect=downloading): + self.assertEqual(ci.records('result', (None,), '456'), {None: self.state}) + def test_recovery_handles_completed_and_unstarted_controllers(self): run = dict(id=456, run_attempt=1, path='.github/workflows/interoperability.yml', head_branch='master', event='workflow_run', head_sha='c' * 40, repository=dict(id=266877886), @@ -555,31 +645,6 @@ def downloading(*args, **kwargs): self.assertRaisesRegex(ci.PeerUnavailable, 'NetBSD'): ci.records('ready') - def test_first_liveness_check_does_not_assume_runner_uptime(self): - def downloading(*args, **kwargs): - kwargs['progress']() - self.fail('Failed endpoint was ignored on the first poll') - jobs = dict(total_count=1, jobs=[dict(name='Interoperability / FreeBSD', conclusion='failure')]) - with patch.object(ci, 'api', side_effect=[dict(run_attempt=2, status='in_progress'), jobs]), \ - patch.object(ci, 'download_records', side_effect=downloading), \ - patch.object(time, 'monotonic', return_value=1), \ - self.assertRaisesRegex(ci.PeerUnavailable, 'FreeBSD'): - ci.records('public') - - def test_stopped_controller_ends_waits_but_result_remains_readable(self): - def downloading(*args, **kwargs): - kwargs['progress']() - return {None: (self.state, b'{}')} - responses = [dict(run_attempt=2, status='in_progress'), dict(total_count=0, jobs=[]), - dict(status='completed')] - with patch.object(ci, 'api', side_effect=responses), patch.object(time, 'monotonic', return_value=1), \ - patch.object(ci, 'download_records', side_effect=downloading), \ - self.assertRaisesRegex(ci.PeerUnavailable, 'controller has stopped'): - ci.records('enrolment', (None,), '456') - with patch.object(ci, 'api', side_effect=responses[:2]), patch.object(time, 'monotonic', return_value=1), \ - patch.object(ci, 'download_records', side_effect=downloading): - self.assertEqual(ci.records('result', (None,), '456'), {None: self.state}) - def test_failed_controller_blocks_its_dependants(self): for conclusion in ('failure', 'cancelled', 'timed_out'): with self.subTest(conclusion=conclusion), patch.object(ci, 'find_controller', return_value=dict(conclusion=conclusion)), \ diff --git a/.github/scripts/test-results.py b/.github/scripts/test-results.py new file mode 100644 index 000000000..63c3a905f --- /dev/null +++ b/.github/scripts/test-results.py @@ -0,0 +1,244 @@ +import copy +import json +import os +from pathlib import Path +import runpy +import subprocess +import sys +import unittest + + +checker = Path(__file__).with_name('check-results.py') +check = runpy.run_path(str(checker))['check'] + + +class Results(unittest.TestCase): + def setUp(self): + self.expected = dict(linux='true', diagnostics='true', bsd='false', macos='true', + other='true', versions='true', android='true', archive='true', valgrind='true', interoperability='true') + self.selections = dict(self.expected, asan='false', coverage='false', clang='false') + self.data = { + 'changes': {'result': 'success', 'outputs': {}}, + 'interoperability': {'result': 'success', 'outputs': {'jobs': json.dumps({name: {'result': 'success'} + for name in ('linux', 'freebsd', 'netbsd', 'openbsd', 'macos', 'solaris', 'cygwin', 'result')})}}, + 'archive': {'result': 'success', 'outputs': {'jobs': json.dumps({'archive': {'result': 'success'}})}}, + 'linux': {'result': 'success', 'outputs': { + 'jobs': json.dumps({'ubuntu': {'result': 'success'}, 'almalinux': {'result': 'skipped'}}), + 'inputs': json.dumps({'ubuntu': True, 'almalinux': False})}}, + 'diagnostics': {'result': 'success', 'outputs': { + 'jobs': json.dumps({'clang_21': {'result': 'success'}, + 'asan': {'result': 'skipped'}, 'python_37': {'result': 'success'}}), + 'inputs': json.dumps({'clang_21': True, 'asan': False, 'python_37': True})}}, + 'bsd': {'result': 'skipped', 'outputs': {}}, + 'macos': {'result': 'success', 'outputs': { + 'jobs': json.dumps({'macos': {'result': 'success'}})}}, + 'other': {'result': 'success', 'outputs': { + 'jobs': json.dumps({'cygwin': {'result': 'success'}, 'solaris': {'result': 'skipped'}}), + 'inputs': json.dumps({'cygwin': True, 'solaris': False})}}, + 'versions': {'result': 'success', 'outputs': { + 'jobs': json.dumps({'mix': {'result': 'success'}})}}, + 'android': {'result': 'success', 'outputs': { + 'jobs': json.dumps({name: {'result': 'success'} for name in ('bionic', 'x86_64', 'termux', 'emulator')}), + 'inputs': json.dumps(dict(bionic=True, x86_64=True, termux=True, emulator=True))}}, + 'valgrind': {'result': 'success', 'outputs': { + 'jobs': json.dumps({'test': {'result': 'success'}})}}, + } + + def reject(self): + with self.assertRaises((TypeError, ValueError)): + check(self.data, self.expected, self.selections) + + def test_success(self): + self.assertTrue(check(self.data, self.expected, self.selections)) + + def test_families(self): + original = copy.deepcopy(self.data) + for name in original: + with self.subTest(missing=name): + self.data = {key: value for key, value in original.items() if key != name} + self.reject() + self.data = {**original, 'extra': {'result': 'success', 'outputs': {}}} + self.reject() + self.data = {} + self.reject() + + def test_family_results(self): + for name in self.data: + expected = self.data[name]['result'] + for result in ('success', 'failure', 'cancelled', 'skipped', None): + if result == expected: + continue + with self.subTest(family=name, result=result): + self.data[name]['result'] = result + self.reject() + self.data[name]['result'] = expected + + def test_selections(self): + for name in self.expected: + with self.subTest(disabled=name): + original = self.data[name] + self.expected[name] = 'false' + self.selections[name] = 'false' + self.data[name] = {'result': 'skipped', 'outputs': {}} + self.assertTrue(check(self.data, self.expected, self.selections)) + self.data[name] = {'result': 'success', 'outputs': {}} + self.reject() + self.data[name] = original + self.expected[name] = 'false' if name == 'bsd' else 'true' + self.selections[name] = self.expected[name] + self.expected = {name: 'false' for name in self.expected} + self.selections = {**self.selections, **self.expected} + self.data = {name: ({'result': 'success', 'outputs': {}} if name == 'changes' + else {'result': 'skipped', 'outputs': {}}) for name in self.data} + self.assertTrue(check(self.data, self.expected, self.selections)) + for value in ('TRUE', '', True, 1): + self.expected['linux'] = value + self.reject() + + def test_nested_results(self): + outputs = self.data['linux']['outputs'] + original = copy.deepcopy(outputs) + for value in ('{}', '[]', 'null', 'not JSON', None): + with self.subTest(jobs=value): + outputs['jobs'] = value + self.reject() + outputs.clear() + self.reject() + outputs.update(original) + for field in ('inputs', 'jobs'): + for value in ('[]', 'not JSON', 'null'): + with self.subTest(field=field, value=value): + outputs[field] = value + self.reject() + outputs.update(original) + jobs = json.loads(original['jobs']) + for value in ('failure', 'cancelled', 'skipped', None): + outputs['jobs'] = json.dumps({**jobs, 'ubuntu': {'result': value}}) + self.reject() + outputs['jobs'] = json.dumps({'almalinux': jobs['almalinux']}) + self.reject() + outputs['jobs'] = json.dumps({**jobs, 'almalinux': {'result': 'success'}}) + self.reject() + outputs.update(original) + for inputs in ({'unknown': True}, {'ubuntu': 'true'}, {'ubuntu': 1}, {'ubuntu': False}, + {'ubuntu': False, 'almalinux': False}): + outputs['inputs'] = json.dumps(inputs) + self.reject() + + def test_diagnostics_jobs(self): + outputs = self.data['diagnostics']['outputs'] + jobs = json.loads(outputs['jobs']) + for result in ('failure', 'cancelled', 'skipped', None): + with self.subTest(result=result): + outputs['jobs'] = json.dumps({**jobs, 'clang_21': {'result': result}}) + self.reject() + outputs['jobs'] = json.dumps({key: value for key, value in jobs.items() if key != 'clang_21'}) + self.reject() + outputs['jobs'] = json.dumps({**jobs, 'clang_21': {'result': 'skipped'}}) + outputs['inputs'] = json.dumps({'clang_21': False, 'asan': False, 'python_37': True}) + self.assertTrue(check(self.data, self.expected, self.selections)) + + def test_change_selections_are_complete_booleans(self): + original = self.selections + for value in ({}, {'unknown': 'true'}, {**original, 'linux': ''}, + {**original, 'linux': True}): + with self.subTest(value=value): + self.selections = value + self.reject() + self.selections = original + + def test_selected_families_must_run(self): + self.selections['linux'] = 'false' + self.reject() + self.selections = {name: 'true' for name in self.selections} + self.assertTrue(check(self.data, self.expected, self.selections, scheduled=True)) + self.selections['asan'] = 'false' + with self.assertRaises(ValueError): + check(self.data, self.expected, self.selections, scheduled=True) + + def test_failed_job_cannot_be_advisory(self): + outputs = self.data['diagnostics']['outputs'] + jobs = json.loads(outputs['jobs']) + jobs['clang_21']['result'] = 'failure' + outputs['jobs'] = json.dumps(jobs) + outputs['advisory'] = 'clang_21' + self.reject() + + def test_every_family_requires_nested_results(self): + for name in ('linux', 'diagnostics', 'macos', 'other', 'versions', 'android', 'archive', 'valgrind', 'interoperability'): + original = self.data[name]['outputs'] + with self.subTest(family=name): + self.data[name]['outputs'] = {} + self.reject() + self.data[name]['outputs'] = original + outputs = self.data['macos']['outputs'] + for result in ('failure', 'cancelled', 'skipped', None): + outputs['jobs'] = json.dumps({'macos': {'result': result}}) + self.reject() + + def test_change_selection_is_required(self): + original = self.data['changes'] + for value in ('failure', 'cancelled', 'skipped', None): + with self.subTest(result=value): + self.data['changes'] = {'result': value, 'outputs': {}} + self.reject() + self.data.pop('changes') + self.reject() + self.data['changes'] = original + + def test_other_selections(self): + names = ('cygwin', 'solaris') + for bits in range(1, 4): + inputs = {name: bool(bits & (1 << i)) for i, name in enumerate(names)} + jobs = {name: {'result': 'success' if inputs[name] else 'skipped'} for name in names} + self.data['other']['outputs'] = {'jobs': json.dumps(jobs), 'inputs': json.dumps(inputs)} + self.assertTrue(check(self.data, self.expected, self.selections)) + for name in names: + original = jobs[name]['result'] + invalid = ('failure', 'cancelled', 'skipped', None) if inputs[name] else ('success', 'failure') + for result in invalid: + with self.subTest(selections=bits, job=name, result=result): + jobs[name]['result'] = result + self.data['other']['outputs']['jobs'] = json.dumps(jobs) + self.reject() + jobs[name]['result'] = original + self.data['other']['outputs']['jobs'] = json.dumps(jobs) + self.data['other']['outputs'] = { + 'jobs': json.dumps({name: {'result': 'skipped'} for name in names}), + 'inputs': json.dumps({name: False for name in names})} + self.reject() + + def test_android_jobs(self): + outputs = self.data['android']['outputs'] + jobs = json.loads(outputs['jobs']) + for name in jobs: + for result in ('failure', 'cancelled', 'skipped', None): + with self.subTest(job=name, result=result): + outputs['jobs'] = json.dumps({**jobs, name: {'result': result}}) + self.reject() + outputs['jobs'] = json.dumps({key: value for key, value in jobs.items() if key != name}) + self.reject() + + def test_command(self): + env = {name: value for name, value in os.environ.items() if not name.startswith('CI_EXPECT_')} + env.update({f'CI_EXPECT_{name.upper()}': value for name, value in self.expected.items()}) + env['CI_RESULTS'] = json.dumps(self.data) + env['CI_SELECTIONS'] = json.dumps(self.selections) + env['GITHUB_EVENT_NAME'] = 'pull_request' + command = [sys.executable, '-S', str(checker)] + result = subprocess.run(command, env=env, capture_output=True, text=True, timeout=10) + self.assertEqual(result.returncode, 0, result.stderr) + outputs = self.data['diagnostics']['outputs'] + jobs = json.loads(outputs['jobs']) + outputs['jobs'] = json.dumps({**jobs, 'clang_21': {'result': 'failure'}}) + result = subprocess.run(command, env={**env, 'CI_RESULTS': json.dumps(self.data)}, + capture_output=True, text=True, timeout=10) + self.assertNotEqual(result.returncode, 0) + self.assertIn('diagnostics/clang_21: failure', result.stderr) + result = subprocess.run(command, env={**env, 'CI_RESULTS': 'not JSON'}, + capture_output=True, text=True, timeout=10) + self.assertNotEqual(result.returncode, 0) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/test-storage.py b/.github/scripts/test-storage.py new file mode 100644 index 000000000..80f17e7bb --- /dev/null +++ b/.github/scripts/test-storage.py @@ -0,0 +1,218 @@ +import io +import json +from pathlib import Path, PurePosixPath +import runpy +import subprocess +import sys +import tempfile +from types import SimpleNamespace +import unittest +from unittest.mock import Mock, patch + +storage = runpy.run_path(str(Path(__file__).resolve().parents[2] / 'testsuite/tools/check-storage.py')) + + +class Storage(unittest.TestCase): + def test_refusal_checks_status_and_cause(self): + for code, message in ((0, b'No space left on device'), (11, b'Permission denied'), (23, b'No space left on device')): + with self.subTest(code=code, message=message), self.assertRaises(ValueError): + storage['refused'](subprocess.CompletedProcess([], code, b'', message), 11, 'No space left on device') + storage['refused'](subprocess.CompletedProcess([], 11, b'', b'No space left on device'), 11, 'No space left on device') + + def test_mount_must_be_the_requested_filesystem(self): + path = PurePosixPath('/fixture') + run = Mock(return_value=Mock(returncode=0, stdout=json.dumps(dict(filesystems=[dict(target='/fixture', fstype='ext4', options='ro,noexec')])).encode())) + with patch.dict(storage['mount_options'].__globals__, run=run): + self.assertEqual(storage['mount_options'](path), ('ext4', {'ro', 'noexec'})) + run.return_value.stdout = b'{"filesystems": [{"target": "/", "fstype": "ext4", "options": "rw"}]}' + with self.assertRaises(ValueError): + storage['mount_options'](path) + run.return_value.returncode = 1 + self.assertIsNone(storage['mount_options'](path, required=False)) + run.return_value.returncode = 2 + with self.assertRaises(ValueError): + storage['mount_options'](path, required=False) + + def test_cleanup_closes_a_partially_opened_mapping(self): + volume = object.__new__(storage['EncryptedVolume']) + volume.name, volume.mount = 'fixture', Path('/fixture') + volume.device = Mock() + volume.device.exists.return_value = False + run = Mock(side_effect=[Mock(returncode=0), Mock(), Mock(returncode=4)]) + with patch.dict(storage['EncryptedVolume'].close.__globals__, run=run, mount_options=Mock(return_value=None)): + volume.close() + self.assertIn(unittest.mock.call('cryptsetup', 'close', 'fixture'), run.call_args_list) + + def test_command_does_not_log_key_input(self): + with patch('sys.stdout', new_callable=io.StringIO) as output, patch('subprocess.Popen') as started: + child = started.return_value.__enter__.return_value + child.communicate.return_value, child.returncode = (b'', b''), 0 + storage['run']('cryptsetup', 'open', '--key-file=-', 'fixture', input=b'private fixture key') + self.assertNotIn('private fixture key', output.getvalue()) + child.communicate.assert_called_once_with(b'private fixture key', timeout=90) + self.assertTrue(started.call_args[1]['start_new_session']) + + def test_cleanup_finds_an_interrupted_loop_allocation(self): + image = Path('/fixture/image').resolve() + device = {'name': '/dev/loop8', 'back-file': str(image)} + run = Mock(side_effect=[Mock(stdout=json.dumps(dict(loopdevices=[device])).encode()), Mock(), + Mock(stdout=b'{"loopdevices": []}')]) + with patch.dict(storage['detach_loops'].__globals__, run=run): + storage['detach_loops'](image) + self.assertIn(unittest.mock.call('losetup', '--detach', '/dev/loop8'), run.call_args_list) + + def test_loop_identity_and_access_mode(self): + image, device = Path('/fixture/image').resolve(), PurePosixPath('/dev/loop8') + record = {'name': str(device), 'back-file': str(image), 'ro': False, 'offset': 0, 'sizelimit': 0} + run = Mock(return_value=Mock(stdout=json.dumps(dict(loopdevices=[record])).encode())) + with patch.dict(storage['check_loop'].__globals__, run=run): + storage['check_loop'](device, image) + for field, value in (('name', '/dev/sda'), ('back-file', '/other/image'), ('ro', True), ('ro', 0), + ('offset', 512), ('sizelimit', 4096)): + run.return_value.stdout = json.dumps(dict(loopdevices=[dict(record, **{field: value})])).encode() + with self.subTest(field=field, value=value), self.assertRaises(ValueError): + storage['check_loop'](device, image) + run.reset_mock() + with self.assertRaises(ValueError): + storage['check_loop'](PurePosixPath('/dev/sda'), image) + run.assert_not_called() + + def test_failed_loop_validation_never_opens_the_device(self): + with tempfile.TemporaryDirectory() as directory: + image = Path(directory) / 'image' + image.write_bytes(b'fixture') + detach = Mock() + with patch.dict(storage['loop_device'].__wrapped__.__globals__, run=Mock(return_value=Mock(stdout=b'/dev/loop8\n')), + check_loop=Mock(side_effect=ValueError('foreign image')), detach_loops=detach), \ + patch('os.open') as opened, self.assertRaises(ValueError): + with storage['loop_device'](image): + self.fail('Unverified device was used') + opened.assert_not_called() + detach.assert_called_once_with(image) + + def test_cleanup_does_not_detach_foreign_images(self): + image = Path('/fixture/image').resolve() + run = Mock(return_value=Mock(stdout=b'{"loopdevices": [{"name": "/dev/loop8", "back-file": "/other/image"}]}')) + with patch.dict(storage['detach_loops'].__globals__, run=run), self.assertRaises(ValueError): + storage['detach_loops'](image) + self.assertEqual(run.call_count, 1) + + def test_cleanup_waits_for_lazy_loop_release(self): + image = Path('/fixture/image').resolve() + attached = json.dumps(dict(loopdevices=[{'name': '/dev/loop8', 'back-file': str(image)}])).encode() + run = Mock(side_effect=[Mock(stdout=attached), Mock(), Mock(stdout=attached), Mock(stdout=b'{"loopdevices": []}')]) + with patch.dict(storage['detach_loops'].__globals__, run=run), patch('time.monotonic', side_effect=[0, 1]), \ + patch('time.sleep') as sleep: + storage['detach_loops'](image) + sleep.assert_called_once_with(0.05) + self.assertEqual(run.call_args_list.count(unittest.mock.call('losetup', '--detach', '/dev/loop8')), 1) + + def test_loop_release_has_a_deadline(self): + image = Path('/fixture/image').resolve() + attached = json.dumps(dict(loopdevices=[{'name': '/dev/loop8', 'back-file': str(image)}])).encode() + run = Mock(return_value=Mock(stdout=attached)) + with patch.dict(storage['detach_loops'].__globals__, run=run), patch('time.monotonic', side_effect=[0, 6]), \ + self.assertRaises(ValueError): + storage['detach_loops'](image) + + def test_bind_mount_cleanup_uses_the_mount_table(self): + path = Path('/fixture') + run = Mock() + with patch.dict(storage['mounted'].__wrapped__.__globals__, run=run, mount_options=Mock(return_value=('ext4', {'rw'}))), \ + patch('os.path.ismount', return_value=False): + with storage['mounted'](path, '--bind', '/source'): + pass + self.assertIn(unittest.mock.call('umount', path), run.call_args_list) + + def test_command_timeout_kills_its_process_group(self): + failure = subprocess.TimeoutExpired(['fixture'], 90) + with patch('sys.stdout', new_callable=io.StringIO), patch('subprocess.Popen') as started, \ + patch('os.killpg') as kill, self.assertRaises(subprocess.TimeoutExpired): + child = started.return_value.__enter__.return_value + child.pid = 1234 + child.communicate.side_effect = [failure, (b'', b'')] + storage['run']('fixture') + kill.assert_called_once_with(1234, storage['signal'].SIGKILL) + self.assertEqual(child.communicate.call_args, unittest.mock.call(timeout=5)) + + def test_copy_as_requires_real_supplementary_group_coverage(self): + root = SimpleNamespace(pw_name='root', pw_uid=0, pw_gid=0) + account = SimpleNamespace(pw_name='fixture', pw_uid=1000, pw_gid=1000) + with patch.dict('os.environ', SUDO_USER='root'), patch.object(storage['pwd'], 'getpwnam', return_value=root), \ + patch.object(storage['pwd'], 'getpwall', return_value=[root, account]), \ + patch('os.getgrouplist', return_value=[0, 1000, 27]): + self.assertEqual(storage['copy_account'](), (account, 27)) + with patch.dict('os.environ', SUDO_USER=''), patch.object(storage['pwd'], 'getpwall', return_value=[account]), \ + patch('os.getgrouplist', return_value=[1000]), self.assertRaises(ValueError): + storage['copy_account']() + + def test_device_writer_rejects_nonprivate_or_foreign_nodes(self): + image = Path('/fixture/image') + device = Mock() + device.stat.return_value = SimpleNamespace(st_rdev=8) + run, validate = Mock(), Mock() + with patch.dict(storage['write_device'].__globals__, run=run, check_loop=validate, + device_identity=Mock(return_value=(1, 9))): + with self.assertRaises(ValueError): + storage['write_device'](Path('/rsync'), Path('/source'), Path('/dev/loop8'), device, image) + validate.assert_not_called() + with self.assertRaises(ValueError): + storage['write_device'](Path('/rsync'), Path('/source'), Path('/fixture/alias'), device, image) + run.assert_not_called() + + def test_device_writer_requires_the_same_device_after_copy(self): + device = Mock() + device.stat.return_value = SimpleNamespace(st_rdev=8) + with patch.dict(storage['write_device'].__globals__, run=Mock(), check_loop=Mock(), + device_identity=Mock(side_effect=[(1, 8), (2, 8)])), self.assertRaises(ValueError): + storage['write_device'](Path('/rsync'), Path('/source'), Path('/fixture/alias'), device, Path('/fixture/image')) + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / 'regular' + path.write_bytes(b'data') + with self.assertRaises(ValueError): + storage['device_identity'](path) + + def test_copy_as_checks_data_and_both_identity_components(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / 'file' + path.write_bytes(b'contents') + info = path.stat() + storage['copied_as'](path, b'contents', info.st_uid, info.st_gid) + for contents, uid, gid in ((b'wrong', info.st_uid, info.st_gid), + (b'contents', info.st_uid + 1, info.st_gid), + (b'contents', info.st_uid, info.st_gid + 1)): + with self.subTest(contents=contents, uid=uid, gid=gid), self.assertRaises(ValueError): + storage['copied_as'](path, contents, uid, gid) + + def test_cli_runs_every_storage_group_before_publishing(self): + groups = ('encrypted', 'filesystem_boundaries', 'device_transfers', 'copy_as', 'restricted', 'full_filesystem') + with tempfile.TemporaryDirectory() as directory: + base = Path(directory) + binary = base / 'rsync' + binary.write_bytes(b'candidate') + def checked(name): + return lambda binary, root, checks: checks.append(name) + for fail in (False, True): + output = base / str(fail) + callbacks = {name: checked(name) for name in groups} + if fail: + callbacks['copy_as'] = Mock(side_effect=ValueError('identity check failed')) + args = ['check-storage.py', '--binary', str(binary), '--output', str(output), '--parent-namespace=old'] + readlink = storage['os'].readlink + with patch.object(sys, 'argv', args), patch('os.geteuid', return_value=0), \ + patch('os.readlink', side_effect=lambda path, *args, **kwargs: 'new' if path == '/proc/self/ns/mnt' else readlink(path, *args, **kwargs)), \ + patch('signal.signal'), \ + patch.dict(storage['main'].__globals__, callbacks), patch('sys.stdout', new_callable=io.StringIO): + if fail: + with self.assertRaises(ValueError): + storage['main']() + self.assertFalse((output / 'receipt.json').exists()) + else: + self.assertEqual(storage['main'](), 0) + receipt = json.loads((output / 'receipt.json').read_text()) + self.assertEqual(receipt['checks'], list(groups)) + self.assertIs(receipt['cleaned'], True) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/test-termux.py b/.github/scripts/test-termux.py new file mode 100644 index 000000000..a38512e79 --- /dev/null +++ b/.github/scripts/test-termux.py @@ -0,0 +1,202 @@ +from contextlib import redirect_stderr +import base64 +import hashlib +from io import BytesIO, StringIO +import json +from pathlib import Path +import runpy +import subprocess +import tempfile +from types import SimpleNamespace +import unittest +from unittest.mock import Mock, patch + +termux = runpy.run_path(str(Path(__file__).with_name('termux.py'))) +Termux = termux['Termux'] +ROOT = termux['HOME'] + '/rsync-ci.12345678' + + +class Sandbox(unittest.TestCase): + def test_identity(self): + context = 'u:r:untrusted_app_27:s0:c216,c256,c512,c768' + termux['check_identity']('10216', context, 10216) + for uid, domain, expected in (('0', context, 10216), ('2000', context, 10216), + ('10217', context, 10216), ('10216', context, 2000), + ('10216', 'u:r:runas_app:s0:c216,c256,c512,c768', 10216), + ('10216', 'u:r:shell:s0', 10216), + ('10216', 'u:r:untrusted_app_27:s0', 10216), + ('10216', 'u:r:su:s0', 10216)): + with self.subTest(uid=uid, domain=domain, expected=expected), self.assertRaises(ValueError): + termux['check_identity'](uid, domain, expected) + + def test_download(self): + payload = b'APK fixture' + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / 'termux.apk' + with patch('urllib.request.urlopen', return_value=BytesIO(payload)), self.assertRaises(ValueError): + termux['download_apk'](path) + path.unlink() + with patch('urllib.request.urlopen', return_value=BytesIO(payload)), \ + patch.dict(termux['download_apk'].__globals__, APK_SHA256=hashlib.sha256(payload).hexdigest()): + termux['download_apk'](path) + self.assertEqual(path.read_bytes(), payload) + with self.assertRaises(FileExistsError), patch('urllib.request.urlopen', return_value=BytesIO(payload)): + termux['download_apk'](path) + + def test_paths(self): + for root in ('', '/', '/data/local/tmp/rsync-ci.12345678', ROOT + '/..', ROOT + ';echo bad'): + with self.subTest(root=root), self.assertRaises(ValueError): + Termux(Mock(), root) + session = Termux(Mock(), ROOT) + for path in ('/data/system/secret', ROOT + '/../escape', ROOT + '\0'): + with self.subTest(path=path), self.assertRaises(ValueError): + session.write_file(path, b'data') + session.adb.assert_not_called() + + def test_private_bytes(self): + payload = b'raw\r\n\0data\x1a' + bytes(range(256)) + adb = Mock(return_value=SimpleNamespace(stdout=base64.b64encode(payload))) + session = Termux(adb, ROOT) + session.write_file(ROOT + '/two words', payload) + self.assertEqual(base64.b64decode(adb.call_args[1]['input']), payload) + self.assertFalse(adb.call_args[1]['text']) + self.assertIn("'" + ROOT + "/two words'", adb.call_args[0][-1]) + self.assertEqual(session.read_file(ROOT + '/two words'), payload) + self.assertIn('head -c 1048577', adb.call_args[0][-1]) + self.assertIn('set -o pipefail', adb.call_args[0][-1]) + adb.return_value = SimpleNamespace(stdout=base64.b64encode(b'x' * 1048577)) + with self.assertRaises(ValueError): + session.read_file(ROOT + '/large') + adb.return_value = SimpleNamespace(stdout=b'not base64') + with self.assertRaises(ValueError): + session.read_file(ROOT + '/malformed') + adb.return_value = subprocess.CompletedProcess([], 1, b'') + self.assertEqual(session.read_file(ROOT + '/missing', optional=True), b'') + adb.return_value = subprocess.CompletedProcess([], 2, b'') + with self.assertRaises(subprocess.CalledProcessError): + session.read_file(ROOT + '/error', optional=True) + + def command(self, *, error=b'-1', code=b'0', stdout=b'result', stderr=b''): + adb = Mock(return_value=SimpleNamespace(returncode=0, stdout='service started')) + session = Termux(adb, ROOT) + values = {'err': error, 'exit_code': code, 'stdout': stdout, 'stderr': stderr} + def read(path, optional=False): + return values[path.rsplit('/', 1)[-1]] + return session, adb, read + + def test_command_uses_the_service(self): + session, adb, read = self.command(stdout=b'10216\n') + with patch.object(Termux, 'read_file', side_effect=read), patch.object(Termux, 'write_file') as write: + result = session.command('/system/bin/id', '-u', capture_output=True) + self.assertEqual(result.stdout, '10216\n') + self.assertEqual(result.returncode, 0) + self.assertIn(b'exec /system/bin/id -u\n', write.call_args[0][1]) + calls = [call[0][-1] for call in adb.call_args_list] + self.assertTrue(any('com.termux/.app.RunCommandService' in call and 'com.termux.RUN_COMMAND_RESULT_DIRECTORY' in call for call in calls)) + self.assertTrue(any('/result-1/err' in call for call in calls)) + self.assertFalse(any('run-as com.termux /system/bin/id' in call for call in calls)) + + def test_failed_and_malformed_results(self): + for error, code in ((b'0', b'0'), (b'-1', b''), (b'-1', b'not an exit status')): + session, adb, read = self.command(error=error, code=code) + with self.subTest(error=error, code=code), patch.object(Termux, 'read_file', side_effect=read), \ + patch.object(Termux, 'write_file'), self.assertRaises(ValueError): + session.command('test', '!', '-e', ROOT + '/missing', capture_output=True) + session, adb, read = self.command(code=b'23', stderr=b'Permission denied') + with patch.object(Termux, 'read_file', side_effect=read), patch.object(Termux, 'write_file'): + with self.assertRaises(subprocess.CalledProcessError): + session.command('rsync', capture_output=True) + result = session.command('rsync', capture_output=True, check=False) + self.assertEqual(result.returncode, 23) + self.assertEqual(result.stderr, 'Permission denied') + + def test_timeout(self): + session = Termux(Mock(return_value=SimpleNamespace(returncode=1, stdout='')), ROOT) + with patch.object(Termux, 'write_file'), patch('time.monotonic', side_effect=[0, 31]), \ + self.assertRaises(TimeoutError): + session.command('sleep', '300', capture_output=True) + + def test_bootstrap(self): + for message, expected in (('Bootstrap packages installed successfully.', None), + ('Bootstrap Error: failed', ValueError), ('extracting packages', TimeoutError)): + def adb(*args, **kwargs): + if args[0] == 'shell': + return SimpleNamespace(stdout='2221\n') + self.assertIn('--pid=2221', args) + return SimpleNamespace(stdout=message) + with self.subTest(message=message), patch('time.sleep'), \ + patch('time.monotonic', side_effect=[0, 1, 181]): + if expected: + with self.assertRaises(expected): + termux['wait_for_bootstrap'](adb) + else: + termux['wait_for_bootstrap'](adb) + + def test_refuses_an_existing_installation(self): + adb = Mock(return_value=SimpleNamespace(stdout='package:com.termux\n')) + with patch.dict(termux['check_transfer'].__globals__, download_apk=Mock(side_effect=AssertionError('Downloaded over an existing install'))), \ + self.assertRaises(ValueError): + termux['check_transfer'](adb, Path('rsync'), Path('.'), {}) + self.assertEqual(adb.call_count, 1) + + def test_cleanup_and_receipt(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + binary = root / 'rsync' + binary.write_bytes(b'candidate') + digest = hashlib.sha256(binary.read_bytes()).hexdigest() + for failure in ('install', 'identity', 'transfer', 'cleanup', 'remaining', 'running', 'process-check', None): + output = root / str(failure) + output.mkdir() + calls = [] + primary = subprocess.CalledProcessError(1, 'adb install') + def adb(*args, **kwargs): + calls.append(args) + if args[0] == 'install' and failure == 'install': + raise primary + stdout = '' + if args[-1].startswith('pm list packages -U'): + stdout = 'package:com.termux uid:10216\n' + if 'mktemp -d' in args[-1]: + stdout = ROOT + '\n' + if any(call[0] == 'uninstall' for call in calls): + if args[-1] == 'pm list packages com.termux' and failure == 'remaining': + stdout = 'package:com.termux\n' + if args[-1] == 'pidof -s com.termux' and failure == 'running': + stdout = '2221\n' + code = 1 if args[0] == 'uninstall' and failure in ('install', 'cleanup') else 0 + if args[-1] == 'pidof -s com.termux': + code = 2 if failure == 'process-check' else (0 if stdout else 1) + return SimpleNamespace(returncode=code, stdout=stdout, stderr='') + def command(*args, **kwargs): + stdout, code, stderr = '', 0, '' + if args[:2] == ('/system/bin/id', '-u'): + stdout = '2000' if failure == 'identity' else '10216' + elif args[:2] == ('/system/bin/id', '-Z'): + stdout = 'u:r:untrusted_app_27:s0:c216,c256,c512,c768' + elif args[0] == 'sha256sum': + stdout = digest + ' rsync\n' + elif args[0] == ROOT + '/rsync': + code, stderr = 23, 'Permission denied' + return SimpleNamespace(returncode=code, stdout=stdout, stderr=stderr) + session = SimpleNamespace(command=command, run_as=Mock(), write_file=Mock(), read_file=Mock()) + fixture = Mock(side_effect=ValueError('transfer failed') if failure == 'transfer' else None) + with self.subTest(failure=failure), patch.dict(termux['check_transfer'].__globals__, + download_apk=Mock(), Termux=Mock(return_value=session), transfer=fixture, + wait_for_bootstrap=Mock(), + run=Mock(return_value=SimpleNamespace(stdout='revision\n'))), redirect_stderr(StringIO()): + if failure: + with self.assertRaises(subprocess.CalledProcessError if failure == 'install' else ValueError) as caught: + termux['check_transfer'](adb, binary, output, {}) + if failure == 'install': + self.assertIs(caught.exception, primary) + self.assertFalse((output / 'receipt.json').exists()) + else: + termux['check_transfer'](adb, binary, output, {}) + self.assertIs(json.loads((output / 'receipt.json').read_text())['app']['cleaned'], True) + self.assertIn(('shell', 'am', 'force-stop', 'com.termux'), calls) + self.assertIn(('uninstall', 'com.termux'), calls) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/test-transfers.py b/.github/scripts/test-transfers.py new file mode 100644 index 000000000..790ac6e21 --- /dev/null +++ b/.github/scripts/test-transfers.py @@ -0,0 +1,428 @@ +import io +import json +import os +from pathlib import Path +import runpy +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import patch + +import interop_ci as ci +import interop_enrol as enrol +from interop_network import NetworkError + +broker = runpy.run_path(str(Path(__file__).with_name('interop-broker.py'))) +endpoint = runpy.run_path(str(Path(__file__).with_name('interop-endpoint.py'))) +client = runpy.run_path(str(Path(__file__).with_name('interop-client.py'))) +host = runpy.run_path(str(Path(__file__).with_name('interop-host.py'))) +from harness.interoperability import PLATFORMS, seed, transfer_cases, update +from harness.interop_metadata import FEATURES, metadata_cases, supported +from harness.ssh import public_key + + +class Transfers(unittest.TestCase): + def setUp(self): + self.directory = tempfile.TemporaryDirectory() + self.addCleanup(self.directory.cleanup) + self.root = Path(self.directory.name) + self.state = dict(run_id='123', run_attempt='2', source_sha='a' * 40) + self.nodes = {role: dict(self.state, role=role, node_id='n' + str(index), address='100.64.0.' + str(index)) + for index, role in enumerate(ci.ROLES, 1)} + self.run = dict(id=123, run_attempt=2, path='.github/workflows/ci.yml', name='CI', + repository=dict(id=266877886), event='pull_request', head_sha='b' * 40, + display_title='testsuite, ci: rework tests', created_at='2026-11-14T00:00:00Z', + head_repository=dict(id=999, full_name='contributor/rsync'), + referenced_workflows=[dict(path=ci.WORKFLOW + 'a' * 40, sha='a' * 40)]) + self.env = patch.dict(os.environ, dict(GITHUB_REPOSITORY_ID='266877886', GITHUB_EVENT_NAME='pull_request', + GITHUB_RUN_ID='123', GITHUB_RUN_ATTEMPT='2', GH_TOKEN='fixture'), clear=True) + self.env.start() + self.addCleanup(self.env.stop) + ci.source.cache_clear() + ci.api_cache.clear() + ci.bound = None + self.addCleanup(ci.source.cache_clear) + self.addCleanup(setattr, ci, 'bound', None) + + def receipts(self): + cases = transfer_cases() + result = {role: dict(platform=role, identity=dict(platform=role, source_sha=self.state['source_sha'], + metadata=dict(acl=FEATURES[role][0], xattrs=FEATURES[role][1])), passed=True, + commands=[list(case) for case in cases if (case[0] if case[2] == 'push' else case[1]) == role], + verified=[list(case) for case in cases if case[1] == role]) for role in ci.ROLES} + for role, record in result.items(): + cases = metadata_cases() + record.update(metadata_commands=[list(case) for case in cases if (case[0] if case[2] == 'push' else case[1]) == role], + metadata_updates=[list(case) for case in cases if (case[0] if case[2] == 'push' else case[1]) == role + and case[-1] != 'backup' and supported(case)], + metadata_verified=[list(case) for case in cases if case[1] == role], + metadata_initial_verified=[list(case) for case in cases if case[1] == role], + metadata_restored=[list(case) for case in cases if case[0] == role and case[-1] == 'backup']) + return result + + def test_complete_native_case_set(self): + self.assertEqual(set(ci.ROLES), set(PLATFORMS)) + cases = transfer_cases() + self.assertEqual(len(cases), 168) + self.assertEqual(len({(a, b) for a, b, _, _ in cases if 'linux' not in (a, b)}), 30) + broker['verify_cases'](self.receipts(), self.state['source_sha']) + + def test_missing_duplicate_foreign_and_misassigned_cases_fail(self): + for role in ci.ROLES: + for field in ('commands', 'verified'): + for mutation in ('missing', 'duplicate', 'foreign', 'wrong role'): + values = self.receipts() + cases = values[role][field] + if mutation == 'missing': + cases.pop() + elif mutation == 'duplicate': + cases[0] = cases[1] + elif mutation == 'foreign': + cases[0] = ['unknown', *cases[0][1:]] + else: + other = next(value for value in ci.ROLES if value != role) + cases[0] = values[other][field][0] + with self.subTest(role=role, field=field, mutation=mutation), self.assertRaises(NetworkError): + broker['verify_cases'](values, self.state['source_sha']) + values = self.receipts() + del values[role] + with self.subTest(missing=role), self.assertRaises(NetworkError): + broker['verify_cases'](values, self.state['source_sha']) + + def test_metadata_evidence_cannot_be_omitted_or_substituted(self): + for role in ci.ROLES: + for field in ('metadata_commands', 'metadata_updates', 'metadata_initial_verified', 'metadata_verified', 'metadata_restored'): + values = self.receipts() + del values[role][field] + with self.subTest(role=role, missing=field), self.assertRaises(NetworkError): + broker['verify_cases'](values, self.state['source_sha']) + values = self.receipts() + cases = values[role][field] + if cases: + cases[0] = ['unknown', *cases[0][1:]] + with self.subTest(role=role, substituted=field), self.assertRaises(NetworkError): + broker['verify_cases'](values, self.state['source_sha']) + values = self.receipts() + values[role][field].append(values[role][field][0]) + with self.subTest(role=role, duplicate=field), self.assertRaises(NetworkError): + broker['verify_cases'](values, self.state['source_sha']) + values = self.receipts() + values[role]['identity']['metadata']['xattrs'] = not FEATURES[role][1] + with self.subTest(role=role, feature='xattrs'), self.assertRaises(NetworkError): + broker['verify_cases'](values, self.state['source_sha']) + values = self.receipts() + values[role]['identity']['platform'] = 'host' + with self.subTest(substitute=role), self.assertRaises(NetworkError): + broker['verify_cases'](values, self.state['source_sha']) + + def test_merge_revision_not_head_is_tested(self): + with patch.object(ci, 'api', return_value=self.run): + self.assertEqual(ci.identity(), self.state) + self.assertNotEqual(ci.identity()['source_sha'], self.run['head_sha']) + + def test_native_revision_errors_are_identified_without_command_output(self): + failures = (FileNotFoundError('fixture-private-path'), subprocess.TimeoutExpired('git', 10), + subprocess.CalledProcessError(128, 'git', stderr=b'fixture-private-output')) + for error in failures: + with self.subTest(error=type(error).__name__), \ + patch.dict(endpoint['serve'].__globals__, endpoint_identity=lambda *_: {}), \ + patch.object(subprocess, 'check_output', side_effect=error), \ + self.assertRaisesRegex(ValueError, '^Native endpoint revision check failed$'): + endpoint['serve'](self.root, 'linux', self.root / 'rsync', self.root / 'rrsync', + revision=self.state['source_sha']) + self.assertEqual(json.loads((self.root / 'error.json').read_text()), + dict(error='Native endpoint revision check failed')) + self.assertFalse((self.root / 'data').exists()) + + def test_native_revision_must_match_the_candidate(self): + with patch.dict(endpoint['serve'].__globals__, endpoint_identity=lambda *_: {}), \ + patch.object(subprocess, 'check_output', return_value='b' * 40 + '\n'), \ + self.assertRaisesRegex(ValueError, 'Native endpoint has a different candidate revision'): + endpoint['serve'](self.root, 'linux', self.root / 'rsync', self.root / 'rrsync', + revision=self.state['source_sha']) + self.assertFalse((self.root / 'data').exists()) + + def test_fork_needs_no_oidc(self): + with patch.object(ci, 'api', return_value=self.run): + self.assertEqual(ci.identity(), self.state) + self.assertNotIn('ACTIONS_ID_TOKEN_REQUEST_TOKEN', os.environ) + + def test_unknown_ci_or_missing_workflow_is_refused(self): + for field, value in dict(path='.github/workflows/other.yml', run_attempt=3, status='completed', + repository=dict(id=1), referenced_workflows=[]).items(): + ci.source.cache_clear() + with self.subTest(field=field), patch.object(ci, 'api', return_value=dict(self.run, **{field: value})), self.assertRaises(NetworkError): + ci.identity() + with patch.dict(os.environ, GITHUB_EVENT_NAME='pull_request_target'), self.assertRaises(NetworkError): + ci.identity() + + def test_recovery_keeps_the_original_ci_attempt(self): + ci.bind(self.state) + with patch.object(ci, 'api', side_effect=AssertionError('Recovery read a newer CI attempt')): + self.assertEqual(ci.identity(), self.state) + for key, value in dict(run_id='../1', run_attempt='0', source_sha='not a hash').items(): + with self.subTest(key=key), self.assertRaises(NetworkError): + ci.bind(dict(self.state, **{key: value})) + + def test_stale_record_is_refused(self): + for key, value in dict(run_id='124', run_attempt='3', source_sha='b' * 40).items(): + with self.subTest(key=key), self.assertRaises(NetworkError): + ci.validate(dict(self.state, **{key: value}), self.state) + + def test_enrolment_is_private_and_bound_to_the_tested_revision(self): + enrol.generate(self.root, 'linux', self.state) + recipient = json.loads((self.root / 'public-key.json').read_text()) + record = enrol.seal(self.root, 'linux', recipient, 'key1', 'tskey-auth-fixture', self.state) + self.assertNotIn('tskey-auth-fixture', json.dumps(record)) + enrol.decrypt(self.root, 'linux', record, self.state) + self.assertEqual((self.root / 'enrol-key').read_text(), 'tskey-auth-fixture') + enrol.forget(self.root) + self.assertFalse((self.root / 'enrol-key').exists()) + with self.assertRaises(NetworkError): + enrol.seal(self.root, 'linux', recipient, 'key1', 'tskey-auth-fixture', dict(self.state, run_attempt='3')) + + def test_enrolment_wait_matches_the_controller_startup_window(self): + published = unittest.mock.Mock(return_value={'recipients': {'linux': {'key_id': 'key1'}}}) + node = {'ID': 'n1', 'TailscaleIPs': ['100.64.0.1']} + with patch.object(sys, 'argv', ['interop-host.py', 'connect', 'linux', '--root', str(self.root)]), \ + patch.dict(host['main'].__globals__, identity=lambda: self.state, published=published, + decrypt=unittest.mock.Mock(), forget=unittest.mock.Mock()), \ + patch.dict(host['client'], connect=unittest.mock.Mock(return_value=node)), \ + patch.object(host['socket'], 'socket'): + self.assertEqual(host['main'](), 0) + published.assert_called_once_with('enrolment', timeout=ci.STARTUP_TIMEOUT) + + def test_bad_public_keys_are_refused(self): + for value in ('', 'ssh-rsa AAAA', 'ssh-ed25519 AAAA', 'ssh-ed25519 AAAA\ncommand=evil'): + with self.subTest(value=value), self.assertRaises(ValueError): + public_key(value) + + def test_case_names_cannot_escape_the_root(self): + for case in (['../outside', 'linux', 'push', 'ssh'], ['linux', 'linux', 'push', 'ssh'], ['linux']): + with self.subTest(case=case), self.assertRaises(ValueError): + endpoint['name'](case) + + def test_interoperability_ssh_retries_connection_setup(self): + command = unittest.mock.Mock(return_value=subprocess.CompletedProcess([], 0, b'', b'')) + peer = {'ssh_user': 'user', 'address': '100.64.0.2'} + with patch.dict(endpoint['copy'].__globals__, command=command): + endpoint['copy'](self.root, self.root / 'rsync', peer, 'ssh', True, + self.root / 'source', 'destination', []) + remote_shell = command.call_args[0][0] + self.assertIn('ConnectionAttempts=3', remote_shell[remote_shell.index('-e') + 1]) + + def test_native_verification_checks_initial_and_updated_data(self): + root = self.root + (root / 'data' / 'source').mkdir(parents=True) + (root / 'data' / 'destination').mkdir() + seed(root / 'reference') + for case in transfer_cases(): + if case[0] == 'linux': + seed(root / 'data' / 'source' / endpoint['name'](case)) + if case[1] == 'linux': + seed(root / 'data' / 'destination' / endpoint['name'](case)) + with patch.dict(endpoint['verify'].__globals__, verify_metadata=lambda *_: []): + result = endpoint['verify'](root, 'linux', 1) + self.assertEqual(len(result['verified']), 24) + with self.assertRaises(ValueError): + endpoint['verify'](root, 'linux', 2) + for case in transfer_cases(): + if case[1] == 'linux': + update(root / 'data' / 'destination' / endpoint['name'](case)) + self.assertEqual(len(endpoint['verify'](root, 'linux', 2)['verified']), 24) + first = next(case for case in transfer_cases() if case[1] == 'linux') + (root / 'data' / 'destination' / endpoint['name'](first) / 'space name').write_text('corrupt') + with self.assertRaises(ValueError): + endpoint['verify'](root, 'linux', 2) + + def test_client_packages_are_verified_before_execution(self): + opener = type('Opener', (), {'open': lambda *_args, **_kwargs: io.BytesIO(b'not the expected package')})() + with patch('urllib.request.build_opener', return_value=opener), self.assertRaises(NetworkError): + client['download']('https://pkgs.tailscale.com/stable/example', self.root / 'package', '0' * 64) + + def test_missing_client_tool_is_identified_without_arguments(self): + with patch.object(subprocess, 'run', side_effect=FileNotFoundError(2, 'not found', 'go')): + with self.assertRaisesRegex(NetworkError, '^Required command not found: go$'): + client['run'](['go', 'fixture-not-an-access-token']) + + def test_client_stop_tolerates_prior_controller_revocation(self): + (self.root / 'client.json').write_text(json.dumps(dict(command=['sudo', '/tailscale'], windows=False))) + (self.root / 'daemon-started').touch() + (self.root / 'daemon-stopped').touch() + result = subprocess.CompletedProcess([], 1) + stop = unittest.mock.Mock(return_value=result) + with patch.dict(client['stop'].__globals__, run=stop): + client['stop'](self.root) + stop.assert_called_once_with(['sudo', '/tailscale', 'logout'], check=False) + self.assertTrue((self.root / 'daemon-stop').is_file()) + + + def test_barriers_precede_copy_and_do_not_wait_for_their_own_upload(self): + calls = [] + def native(root, role, phase): + calls.append(phase) + return {} + def barrier(kind, **kwargs): + calls.append(kind) + return {} + shared = dict(identity=lambda: self.state, call=native, records=barrier, write=lambda *args, **kwargs: None) + for phase, expected in (('transfer1', ['admitted', 'transfer1']), ('wait1', ['transfer1']), + ('transfer2', ['next', 'transfer2']), ('wait2', ['transfer2']), + ('next', ['transfer1', 'next'])): + calls.clear() + with self.subTest(phase=phase), patch.object(sys, 'argv', ['interop-host.py', phase, 'linux', '--root', str(self.root)]), \ + patch.dict(host['main'].__globals__, shared): + self.assertEqual(host['main'](), 0) + self.assertEqual(calls, expected) + + def test_result_retains_both_metadata_verification_phases(self): + record = self.receipts()['linux'] + initial = list(reversed(record['metadata_verified'])) + (self.root / 'ready.json').write_text(json.dumps(dict(identity=record['identity'], node_id='n1'))) + (self.root / 'transfer1.json').write_text(json.dumps(dict(metadata_commands=record['metadata_commands']))) + (self.root / 'next.json').write_text(json.dumps(dict(metadata_verified=initial))) + (self.root / 'transfer2.json').write_text(json.dumps(dict(commands=record['commands'], + metadata_commands=record['metadata_updates'], metadata_restored=record['metadata_restored']))) + publish = unittest.mock.Mock() + final = dict(verified=record['verified'], metadata_verified=record['metadata_verified']) + with patch.object(sys, 'argv', ['interop-host.py', 'verify', 'linux', '--root', str(self.root)]), \ + patch.dict(host['main'].__globals__, identity=lambda: self.state, records=lambda *args, **kwargs: {}, + call=lambda *args: final, write=publish): + self.assertEqual(host['main'](), 0) + self.assertEqual(publish.call_args[0][1], 'result') + self.assertEqual(publish.call_args[1]['metadata_initial_verified'], initial) + self.assertEqual(publish.call_args[1]['metadata_verified'], record['metadata_verified']) + self.assertEqual(publish.call_args[1]['metadata_commands'], record['metadata_commands']) + self.assertEqual(publish.call_args[1]['metadata_updates'], record['metadata_updates']) + + def test_peer_failure_records_a_blocked_phase(self): + environment = self.root / 'github-env' + ci.bind(self.state) + with patch.object(sys, 'argv', ['interop-host.py', 'transfer1', 'linux', '--root', str(self.root)]), \ + patch.dict(os.environ, GITHUB_ENV=str(environment)), \ + patch.dict(host['main'].__globals__, identity=lambda: self.state, + records=unittest.mock.Mock(side_effect=ci.PeerUnavailable('Unavailable endpoints: NetBSD')), + call=lambda *args: self.fail('A blocked transfer ran')): + self.assertEqual(host['main'](), 0) + self.assertEqual(environment.read_text(), 'INTEROP_BLOCKED=true\n') + record = json.loads((self.root / 'blocked.json').read_text()) + self.assertEqual(record['role'], 'linux') + self.assertEqual(record['phase'], 'transfer1') + self.assertIs(record['passed'], False) + self.assertFalse((self.root / 'transfer1.json').exists()) + + def test_local_failures_are_not_blocked_results(self): + environment = self.root / 'github-env' + with patch.object(sys, 'argv', ['interop-host.py', 'transfer1', 'linux', '--root', str(self.root)]), \ + patch.dict(os.environ, GITHUB_ENV=str(environment)), \ + patch.dict(host['main'].__globals__, identity=lambda: self.state, + records=unittest.mock.Mock(side_effect=NetworkError('Invalid receipt'))), \ + patch('sys.stderr', new_callable=io.StringIO): + self.assertEqual(host['main'](), 1) + self.assertFalse(environment.exists()) + self.assertFalse((self.root / 'blocked.json').exists()) + + def test_failed_transfer_is_published_before_the_phase_barrier(self): + ci.bind(self.state) + with patch.object(sys, 'argv', ['interop-host.py', 'transfer2', 'linux', '--root', str(self.root)]), \ + patch.dict(host['main'].__globals__, identity=lambda: self.state, records=lambda *args, **kwargs: {}, + call=unittest.mock.Mock(side_effect=NetworkError('transfer failed'))), \ + patch('sys.stderr', new_callable=io.StringIO): + self.assertEqual(host['main'](), 1) + self.assertIs(json.loads((self.root / 'transfer2.json').read_text())['failed'], True) + + def test_transfer_barrier_blocks_peers_without_stopping_their_services_early(self): + ci.bind(self.state) + environment = self.root / 'github-env' + completed = {role: dict(self.state, failed=role == 'cygwin') for role in ci.ROLES} + with patch.object(sys, 'argv', ['interop-host.py', 'wait2', 'linux', '--root', str(self.root)]), \ + patch.dict(os.environ, GITHUB_ENV=str(environment)), \ + patch.dict(host['main'].__globals__, identity=lambda: self.state, + records=unittest.mock.Mock(return_value=completed)): + self.assertEqual(host['main'](), 0) + self.assertEqual(environment.read_text(), 'INTEROP_BLOCKED=true\n') + blocked = json.loads((self.root / 'blocked.json').read_text()) + self.assertEqual((blocked['phase'], blocked['passed']), ('wait2', False)) + + def test_result_rejects_blocked_failed_and_missing_outcomes(self): + for role, job in (('netbsd', dict(result='success', outputs=dict(blocked='true'))), + ('cygwin', dict(result='failure', outputs={})), + ('linux', dict(result='success', outputs={}))): + jobs = {name: dict(result='success', outputs=dict(blocked='false')) for name in ci.ROLES} + jobs[role] = job + with self.subTest(role=role), patch.dict(os.environ, INTEROP_JOBS=json.dumps(jobs)), \ + patch.object(sys, 'argv', ['interop-host.py', 'result', 'all', '--root', str(self.root)]), \ + patch.dict(host['main'].__globals__, identity=lambda: self.state, + records=lambda *args: self.fail('Incomplete endpoints reached receipt validation')), \ + patch('sys.stderr', new_callable=io.StringIO) as stderr: + self.assertEqual(host['main'](), 1) + self.assertIn(role, stderr.getvalue()) + self.assertFalse((self.root / 'blocked.json').exists()) + + def test_result_requires_transfers_and_verified_cleanup(self): + jobs = {name: dict(result='success', outputs=dict(blocked='false')) for name in ci.ROLES} + valid = dict(passed=True, cleaned=True, roles=list(ci.ROLES)) + for proof, expected in ((valid, 0), (dict(valid, passed=False), 1), (dict(valid, cleaned=False), 1), + (dict(valid, roles=list(ci.ROLES)[:-1]), 1)): + with self.subTest(proof=proof), patch.dict(os.environ, INTEROP_JOBS=json.dumps(jobs)), \ + patch.object(sys, 'argv', ['interop-host.py', 'result', 'all', '--root', str(self.root)]), \ + patch.dict(host['main'].__globals__, identity=lambda: self.state, + records=lambda *args: self.receipts(), published=lambda *args: proof), \ + patch('sys.stderr', new_callable=io.StringIO), patch('sys.stdout', new_callable=io.StringIO): + self.assertEqual(host['main'](), expected) + + def test_controller_failure_still_fails_result(self): + jobs = {name: dict(result='success', outputs=dict(blocked='false')) for name in ci.ROLES} + with patch.dict(os.environ, INTEROP_JOBS=json.dumps(jobs)), \ + patch.object(sys, 'argv', ['interop-host.py', 'result', 'all', '--root', str(self.root)]), \ + patch.dict(host['main'].__globals__, identity=lambda: self.state, records=lambda *args: self.receipts(), + published=unittest.mock.Mock(side_effect=ci.PeerUnavailable('Network controller failed'))), \ + patch('sys.stderr', new_callable=io.StringIO): + self.assertEqual(host['main'](), 1) + self.assertFalse((self.root / 'blocked.json').exists()) + + def test_denial_has_healthy_controls_before_and_after(self): + ci.bind(self.state) + (self.root / 'ready.json').write_text(json.dumps(self.nodes['linux'])) + calls = [] + def connected(address, port): + calls.append((address, port)) + return address == self.nodes['linux']['address'] + shared = dict(published=lambda *args: dict(nodes=self.nodes), connect=connected) + with patch.dict(host['network'].__globals__, shared): + host['network'](self.root, 'linux', 'quarantine') + for port in (18731, 18732, 18733): + self.assertEqual(calls.count((self.nodes['linux']['address'], port)), 2) + for role, node in self.nodes.items(): + if role != 'linux': + self.assertIn((node['address'], 18731), calls) + self.assertTrue(json.loads((self.root / 'quarantine.json').read_text())['passed']) + + def test_a_dead_service_cannot_prove_network_denial(self): + (self.root / 'ready.json').write_text(json.dumps(self.nodes['linux'])) + with patch.dict(host['network'].__globals__, published=lambda *args: dict(nodes=self.nodes), connect=lambda *args: False): + with self.assertRaisesRegex(NetworkError, 'not listening'): + host['network'](self.root, 'linux', 'quarantine') + self.assertFalse((self.root / 'quarantine.json').exists()) + + def test_client_checks_active_preferences_without_retaining_private_data(self): + (self.root / 'client.json').write_text(json.dumps(dict(command=['tailscale'], windows=False))) + prefs = dict(RouteAll=False, CorpDNS=False, Persist=dict(PrivateNodeKey='fixture-not-a-key')) + responses = [subprocess.CompletedProcess([], 0, b'{}'), subprocess.CompletedProcess([], 0, b''), + subprocess.CompletedProcess([], 0, json.dumps(prefs).encode()), + subprocess.CompletedProcess([], 0, b'{"Self":{"ID":"n1"}}')] + with patch.dict(client['connect'].__globals__, run=unittest.mock.Mock(side_effect=responses)), patch.object(subprocess, 'Popen'): + self.assertEqual(client['connect'](self.root, self.root / 'enrol-key', 'fixture')['ID'], 'n1') + self.assertNotIn('fixture-not-a-key', (self.root / 'client.json').read_text()) + for value in (True, None, 0, 'false'): + broken = json.dumps(dict(prefs, CorpDNS=value)).encode() + results = [subprocess.CompletedProcess([], 0, b'{}'), subprocess.CompletedProcess([], 0, b''), + subprocess.CompletedProcess([], 0, broken)] + with self.subTest(value=value), patch.dict(client['connect'].__globals__, run=unittest.mock.Mock(side_effect=results)), patch.object(subprocess, 'Popen'): + with self.assertRaisesRegex(NetworkError, 'still enabled'): + client['connect'](self.root, self.root / 'enrol-key', 'fixture') + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/versions.py b/.github/scripts/versions.py new file mode 100644 index 000000000..1de837c97 --- /dev/null +++ b/.github/scripts/versions.py @@ -0,0 +1,79 @@ +import argparse +import os +from pathlib import Path +import signal +import subprocess +import sys + +sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'testsuite')) +from harness.profile import load_binaries + + +def main(): + parser = argparse.ArgumentParser(description='Run historical-peer checks') + parser.add_argument('--role', choices=('primary', 'secondary'), required=True) + parser.add_argument('--current', default='rsync', help='Current rsync executable') + parser.add_argument('--output', default='test-results/version-mix', help='Result directory') + args = parser.parse_args() + output = Path(args.output) + output.mkdir(parents=True, exist_ok=True) + archive = Path('old_versions') + records = load_binaries(archive) + current = Path(args.current).resolve(strict=True) + status = 0 + active = None + + def cancel(signum, frame): + signal.signal(signal.SIGTERM, signal.SIG_IGN) + signal.signal(signal.SIGINT, signal.SIG_IGN) + if active is not None and active.poll() is None: + active.terminate() + try: + active.wait(timeout=10) + except subprocess.TimeoutExpired: + active.kill() + active.wait(timeout=5) + raise SystemExit(128 + signum) + + previous = {sig: signal.signal(sig, cancel) for sig in (signal.SIGTERM, signal.SIGINT)} + try: + for name in sorted(records, key=lambda value: tuple( + int(part) for part in value[len('rsync_'):].split('.'))): + peer = archive / name + peer.chmod(peer.stat().st_mode | 0o111) + version = peer.name[len('rsync_'):] + profile = 'peer-' + version + primary, secondary = ((peer.resolve(), current) if args.role == 'primary' + else (current, peer.resolve())) + for transport in ('pipe', 'tcp'): + label = f'{peer.name} ({args.role}, {transport})' + profiles = ['non-asan', 'nonroot', 'linux', profile] + if transport == 'pipe': + profiles.insert(0, 'pipe') + print(f'::group::{label}', flush=True) + try: + command = [sys.executable, '-S', 'testsuite/tools/check.py', '.', str(primary), + ','.join(profiles), + str(output / f'{peer.name}-{args.role}-{transport}'), + f'--rsync-bin2={secondary}', f'--archived-role={args.role}', '-j8'] + if transport == 'tcp': + command += ['--use-tcp', '--daemon-tests-only'] + active = subprocess.Popen(command, env={**os.environ, 'PYTHONDONTWRITEBYTECODE': '1'}) + if active.wait(): + print(f'::error::Version mix failed: {label}', file=sys.stderr) + status = 1 + finally: + active = None + print('::endgroup::', flush=True) + finally: + for sig, handler in previous.items(): + signal.signal(sig, handler) + return status + + +if __name__ == '__main__': + try: + raise SystemExit(main()) + except (OSError, ValueError, subprocess.SubprocessError) as error: + print(f'Version mix failed: {error}', file=sys.stderr) + raise SystemExit(1) diff --git a/.github/workflows/actionlint.yml b/.github/workflows/actionlint.yml deleted file mode 100644 index 8d30e4096..000000000 --- a/.github/workflows/actionlint.yml +++ /dev/null @@ -1,47 +0,0 @@ -name: Lint GitHub Actions workflows - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -# Static-check the workflow YAML with rhysd/actionlint. Catches missing -# secrets, bad expressions, expression-type errors, unsupported runner -# images, and (via embedded shellcheck) common pitfalls in `run:` scripts. -# Trigger only on changes under .github/workflows/ so the rest of the -# matrix isn't billed when nothing here moves. - -on: - push: - branches: [ master ] - paths: - - '.github/workflows/*.yml' - - '.github/actionlint.yaml' - - '.github/actionlint.yml' - pull_request: - types: [opened, synchronize, reopened] - paths: - - '.github/workflows/*.yml' - - '.github/actionlint.yaml' - - '.github/actionlint.yml' - -permissions: - contents: read - -jobs: - actionlint: - runs-on: ubuntu-latest - name: actionlint - steps: - - uses: actions/checkout@v4 - - name: install actionlint - # Pin a version so this job is reproducible; bump deliberately. - # The download script verifies a SHA256 of the release tarball. - run: | - bash <(curl --proto '=https' --tlsv1.2 -fsSL \ - https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash) \ - 1.7.12 - echo "$PWD" >>"$GITHUB_PATH" - - name: actionlint --version - run: actionlint -version - - name: actionlint .github/workflows/*.yml - run: actionlint -color diff --git a/.github/workflows/almalinux-8-build.yml b/.github/workflows/almalinux-8-build.yml deleted file mode 100644 index 8eb4598c3..000000000 --- a/.github/workflows/almalinux-8-build.yml +++ /dev/null @@ -1,88 +0,0 @@ -name: Test rsync on AlmaLinux 8 - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -# Older-LTS coverage on the Fedora/RHEL family to help with backporting -# security fixes. AlmaLinux 8 is the RHEL 8 rebuild and is the oldest -# active LTS in this family (RHEL 8 full support runs to 2029). -# GitHub Actions has no native runner for this family, so the job runs -# inside an almalinux:8 container hosted on ubuntu-latest. - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/almalinux-8-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/almalinux-8-build.yml' - schedule: - - cron: '42 8 * * *' - -jobs: - test: - runs-on: ubuntu-latest - container: - image: almalinux:8 - name: Test rsync on AlmaLinux 8 - steps: - - name: install git - # actions/checkout needs git in the container before the checkout step. - run: dnf -y install git - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - # PowerTools is needed for libzstd-devel etc; xxhash and lz4 dev - # headers live in EPEL on RHEL 8. The default python3 on RHEL 8 - # is 3.6, which is too old for runtests.py (uses capture_output= - # / text= introduced in 3.7), so install python39 and point - # /usr/bin/python3 at it. - run: | - dnf -y install epel-release - dnf config-manager --set-enabled powertools - dnf -y install gcc gcc-c++ make autoconf automake m4 \ - python39 python39-pip diffutils \ - openssl openssl-devel \ - attr libattr-devel acl libacl-devel \ - zstd libzstd-devel \ - lz4 lz4-devel \ - xxhash xxhash-devel \ - libidn2 libidn2-devel - alternatives --set python3 /usr/bin/python3.9 - pip3 install commonmark - - name: configure - run: ./configure --with-rrsync - - name: make - run: make - - name: info - run: ./rsync --version - - name: check - # In the container we already run as root, so no sudo. The - # crtimes-not-supported skip matches the other Linux jobs; - # daemon-chroot-acl and proxy-response-line-too-long skip because - # the default (secure) transport opens no listening socket. - run: RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt make check - - name: check (TCP daemon transport) - # Second run exercising the real loopback-TCP daemon path. - run: ./runtests.py --rsync-bin="$PWD/rsync" --use-tcp -j 8 - - name: ssl file list - run: ./rsync-ssl --no-motd download.samba.org::rsyncftp/ || true - - name: save artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: almalinux-8-bin - path: | - rsync - rsync-ssl - rsync.1 - rsync-ssl.1 - rsyncd.conf.5 - rrsync.1 - rrsync diff --git a/.github/workflows/android-static-build.yml b/.github/workflows/android-static-build.yml deleted file mode 100644 index 14593a6dc..000000000 --- a/.github/workflows/android-static-build.yml +++ /dev/null @@ -1,125 +0,0 @@ -name: Build static rsync for Android - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -# Cross-compiles statically-linked rsync binaries with the Android NDK, -# suitable for dropping onto a phone (adb push / Termux) with no shared -# libraries. arm64-v8a covers all modern phones; armeabi-v7a covers older -# 32-bit devices. The binaries are uploaded as workflow artifacts. -# -# These are cross-compiled, so the test suite can't run here; we sanity -# check that each binary is the right architecture, is static, and that -# it executes (`--version`) under qemu-user. - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/android-static-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/android-static-build.yml' - schedule: - - cron: '42 8 * * 1' - workflow_dispatch: - -env: - # Minimum supported API level. 24 (Android 7.0) runs on every modern - # phone while keeping broad reach; bump if you need newer Bionic APIs. - ANDROID_API: 24 - -jobs: - build: - runs-on: ubuntu-latest - name: ${{ matrix.abi }} - strategy: - fail-fast: false - matrix: - include: - - abi: arm64-v8a # modern phones - triple: aarch64-linux-android - qemu: qemu-aarch64-static - - abi: armeabi-v7a # older 32-bit phones - triple: armv7a-linux-androideabi - qemu: qemu-arm-static - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - name: Install build prerequisites - run: sudo apt-get update && sudo apt-get install -y autoconf automake gawk qemu-user-static - - - name: Configure and build (${{ matrix.abi }}) - shell: bash - run: | - set -euo pipefail - NDK="${ANDROID_NDK_LATEST_HOME:-$ANDROID_NDK_ROOT}" - TC="$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin" - export CC="$TC/${{ matrix.triple }}${ANDROID_API}-clang" - export AR="$TC/llvm-ar" RANLIB="$TC/llvm-ranlib" STRIP="$TC/llvm-strip" - export CFLAGS="-O2" LDFLAGS="-static" - - # Bionic doesn't declare lchmod()/lutimes() until API 36, but the - # symbols link, so configure mis-detects them -- force them off so - # rsync uses its fallbacks. The other cache vars restore values - # that configure can't probe when cross-compiling (Android runs a - # normal Linux kernel, so these match the native Linux result). - export ac_cv_func_lchmod=no ac_cv_func_lutimes=no \ - rsync_cv_HAVE_SOCKETPAIR=yes \ - rsync_cv_MKNOD_CREATES_FIFOS=yes \ - rsync_cv_MKNOD_CREATES_SOCKETS=yes - - # Self-contained build: drop optional external libraries so the - # static binary needs nothing at runtime. rsync keeps md5/md4 - # checksums and its bundled zlib. - ./configure --host=${{ matrix.triple }} --build=x86_64-pc-linux-gnu \ - --enable-ipv6 \ - --disable-zstd --disable-lz4 --disable-xxhash --disable-openssl --disable-idn \ - --disable-iconv --disable-iconv-open \ - --disable-acl-support --disable-xattr-support \ - --disable-md2man --disable-roll-simd \ - --with-included-popt --with-included-zlib - - # Generate the awk-built headers serially first so the parallel - # build can't race on proto.h <- daemon-parm.h. - make proto.h - make -j"$(nproc)" rsync - "$STRIP" rsync - - - name: Verify binary - shell: bash - run: | - set -euo pipefail - file rsync - # Gate: must be a statically-linked executable (no interpreter). - file rsync | grep -q "statically linked" - if file rsync | grep -q "dynamically linked"; then - echo "ERROR: binary is not static" >&2; exit 1 - fi - # Best-effort: confirm it actually runs under qemu-user. - ${{ matrix.qemu }} ./rsync --version | head -3 || \ - echo "WARNING: qemu smoke test did not run cleanly (check on a real device)" - - - name: Package - shell: bash - run: | - set -euo pipefail - VER=$(sed -n 's/.*RSYNC_VERSION "\([^"]*\)".*/\1/p' version.h) - out="rsync-${VER}-android-${{ matrix.abi }}" - mkdir -p dist - cp rsync "dist/$out" - ( cd dist && sha256sum "$out" > "$out.sha256" ) - echo "ARTIFACT_NAME=rsync-android-${{ matrix.abi }}" >>"$GITHUB_ENV" - - - name: Upload artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: ${{ env.ARTIFACT_NAME }} - path: dist/ diff --git a/.github/workflows/android.yml b/.github/workflows/android.yml new file mode 100644 index 000000000..10c939162 --- /dev/null +++ b/.github/workflows/android.yml @@ -0,0 +1,179 @@ +name: Android + +on: + workflow_call: + outputs: + jobs: + description: Job results + value: ${{ toJSON(jobs) }} + inputs: + description: Job selections + value: '{"bionic": true, "x86_64": true, "termux": true, "emulator": true}' + +permissions: + contents: read + +defaults: + run: + shell: sh + +env: + ANDROID_API: 24 + +jobs: + bionic: + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: ${{ matrix.name }} + env: + ANDROID_TRIPLE: ${{ matrix.triple }} + strategy: + fail-fast: false + matrix: + include: + - name: Bionic API 24 ARM64 + abi: arm64-v8a + triple: aarch64-linux-android + executor: qemu-aarch64-static + packages: qemu-user-static + - name: Bionic API 24 ARMv7 + abi: armeabi-v7a + triple: armv7a-linux-androideabi + executor: qemu-arm-static + packages: qemu-user-static + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Install dependencies + run: sudo apt-get update && sudo apt-get install -y autoconf automake gawk ${{ matrix.packages }} + + - &build + name: Build + run: | + set -eu + NDK="${ANDROID_NDK_LATEST_HOME:-$ANDROID_NDK_ROOT}" + TC="$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin" + export CC="$TC/${ANDROID_TRIPLE}${ANDROID_API}-clang" + export CXX="$TC/${ANDROID_TRIPLE}${ANDROID_API}-clang++" + export AR="$TC/llvm-ar" RANLIB="$TC/llvm-ranlib" STRIP="$TC/llvm-strip" + export CFLAGS="-O2" LDFLAGS="-static" + + # Bionic hides lchmod and lutimes before API 36; cross-builds cannot run probes + export ac_cv_func_lchmod=no ac_cv_func_lutimes=no \ + rsync_cv_HAVE_SOCKETPAIR=yes \ + rsync_cv_MKNOD_CREATES_FIFOS=yes \ + rsync_cv_MKNOD_CREATES_SOCKETS=yes + + ./configure --host="$ANDROID_TRIPLE" --build=x86_64-pc-linux-gnu \ + --enable-ipv6 \ + --disable-zstd --disable-lz4 --disable-xxhash --disable-openssl --disable-idn \ + --disable-iconv --disable-iconv-open \ + --disable-acl-support --disable-xattr-support \ + --disable-md2man --disable-roll-simd \ + --with-included-popt --with-included-zlib + + # Generate shared headers before the parallel build + make proto.h + make -j"$(nproc)" rsync + "$STRIP" rsync + + - name: Tests + run: python3 -S .github/scripts/android.py test --executor "${{ matrix.executor }}" + + - name: Package + run: python3 -S .github/scripts/android.py package --abi "${{ matrix.abi }}" + + - &upload + name: Upload binaries + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: ${{ env.ARTIFACT_NAME }} + if-no-files-found: error + path: dist/ + + x86_64: + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: Bionic API 24 X86-64 + env: + ANDROID_TRIPLE: x86_64-linux-android + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Install dependencies + run: sudo apt-get update && sudo apt-get install -y autoconf automake gawk + - *build + - name: Tests + run: python3 -S .github/scripts/android.py test --executor direct + - name: Package + run: python3 -S .github/scripts/android.py package --abi x86_64 + - *upload + + termux: + needs: x86_64 + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: Termux API 36 X86-64 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - &emulator-dependencies + name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install -y acl libpulse0 + sudo setfacl -m "u:$(id -un):rw" /dev/kvm + "$ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager" 'system-images;android-36;google_apis;x86_64' emulator + - &download + name: Download binary + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: rsync-android-x86_64 + path: dist/ + - &restore + name: Verify binary + run: python3 -S .github/scripts/android.py restore --abi x86_64 + - name: Tests + id: test + run: python3 -S .github/scripts/android-emulator.py --termux --output test-results/termux + - name: Upload results + if: ${{ !cancelled() && (steps.test.outcome == 'success' || steps.test.outcome == 'failure') }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: android-termux-results + if-no-files-found: error + path: test-results/termux/ + + emulator: + needs: x86_64 + runs-on: ubuntu-24.04 + timeout-minutes: 30 + name: Emulator API 36 X86-64 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - *emulator-dependencies + - *download + - *restore + - name: Tests + id: test + run: python3 -S .github/scripts/android-emulator.py + - name: Upload results + if: ${{ !cancelled() && (steps.test.outcome == 'success' || steps.test.outcome == 'failure') }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: android-emulator-results + if-no-files-found: error + path: test-results/android-emulator/ diff --git a/.github/workflows/archive.yml b/.github/workflows/archive.yml new file mode 100644 index 000000000..4d3aac098 --- /dev/null +++ b/.github/workflows/archive.yml @@ -0,0 +1,50 @@ +name: Archive + +on: + workflow_call: + outputs: + jobs: + description: Job results + value: ${{ toJSON(jobs) }} + +permissions: + contents: read + +defaults: + run: + shell: sh + +jobs: + archive: + name: Ubuntu 24.04 X86-64 + runs-on: ubuntu-24.04 + timeout-minutes: 45 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install -y autoconf automake gawk fakeroot acl attr libacl1-dev libattr1-dev libxxhash-dev libzstd-dev liblz4-dev libidn2-dev libpopt-dev libssl-dev python3-cmarkgfm openssl + - name: Prepare source + run: ./prepare-source + - name: Configure + run: ./configure --with-rrsync + - name: Build + run: | + make -j2 + make gen + - name: Create archive + run: python3 -B -S .github/scripts/archive.py create --archive test-results/source.tar.gz + - name: Qualify archive + run: python3 -B -S .github/scripts/archive.py test --archive test-results/source.tar.gz --output test-results/archive + - name: Upload results + if: ${{ !cancelled() }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: source-archive-results + retention-days: 45 + if-no-files-found: error + path: test-results/ diff --git a/.github/workflows/asan-build.yml b/.github/workflows/asan-build.yml deleted file mode 100644 index 246a31f9d..000000000 --- a/.github/workflows/asan-build.yml +++ /dev/null @@ -1,76 +0,0 @@ -name: rsync ASan+UBSan (clang) - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/asan-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/asan-build.yml' - schedule: - # Weekly (Mon 09:42 UTC): catch breakage from a moving ubuntu-latest/clang - # toolchain (a new clang can add a UBSan check, or change ASan behaviour) - # that no code push would otherwise trigger. Push/PR already gate every - # code change, so daily would just re-run an unchanged tree. - - cron: '42 9 * * 1' - workflow_dispatch: - -jobs: - asan: - runs-on: ubuntu-latest - name: rsync ASan+UBSan (clang) - env: - # rsync intentionally leaks small allocations at process exit, so leak - # detection would be all noise; chase only memory-safety errors. - ASAN_OPTIONS: detect_leaks=0:abort_on_error=1 - # UBSan is a gate: -fno-sanitize-recover=undefined (below) aborts on the - # first finding and halt_on_error=1 makes that fatal, so any undefined - # behaviour fails the run. This needs the tree to be UBSan-clean: the - # remaining findings are fixed in code (hashtable/mdfour shifts, xattrs, - # and log.c's file_struct, kept aligned via rounding.h); only byteorder.h's - # intentional unaligned accessors are suppressed, with no_sanitize. - UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1 - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - sudo apt-get update - sudo apt-get install -y clang acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev openssl - echo "/usr/local/bin" >>"$GITHUB_PATH" - - name: configure - # -DNDEBUG builds as a shipped release does (assert() compiled out), so - # AddressSanitizer catches the over-reads/over-writes that an "assert() - # instead of a real bounds check" bug would cause in a production build. - # UBSan rides along on the same build; -fno-sanitize-recover=undefined - # makes any undefined behaviour abort (and thus fail the run) instead of - # merely printing it. - run: | - CC=clang \ - CFLAGS="-fsanitize=address,undefined -fno-sanitize-recover=undefined -fno-omit-frame-pointer -g -O1 -DNDEBUG" \ - LDFLAGS="-fsanitize=address,undefined" \ - ./configure --with-rrsync --disable-md2man - - name: make - # check-progs builds rsync plus the test helper programs (tls, trimslash, - # t_unsafe, ...) that runtests.py requires; plain "make" builds only rsync - # and runtests aborts on the missing helpers. - run: make check-progs - - name: info - run: ./rsync --version - - name: check (stdio-pipe transport) - # ASan+UBSan-instrumented coverage of the transfer, daemon, sender, - # receiver and metadata paths over the default stdio-pipe transport. - run: ./runtests.py --rsync-bin="$PWD/rsync" -j8 - - name: check (TCP daemon transport) - # --use-tcp also exercises the loopback rsyncd listener and the client's - # TCP connection path. - run: ./runtests.py --rsync-bin="$PWD/rsync" --use-tcp -j8 diff --git a/.github/workflows/bsd.yml b/.github/workflows/bsd.yml new file mode 100644 index 000000000..1b5b6aea3 --- /dev/null +++ b/.github/workflows/bsd.yml @@ -0,0 +1,214 @@ +name: BSD + +on: + workflow_call: + outputs: + jobs: + description: Job results + value: ${{ toJSON(jobs) }} + +permissions: + contents: read + +defaults: + run: + shell: sh + +jobs: + freebsd: + strategy: + fail-fast: false + matrix: + release: ['15.1'] + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: FreeBSD ${{ matrix.release }} X86-64 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Prepare VM host + timeout-minutes: 10 + run: | + set -- qemu-utils qemu-system-x86 ovmf + sudo apt-get install -y -q -o Acquire::Retries=3 --no-install-recommends "$@" || { + sudo apt-get update -q + sudo apt-get install -y -q -o Acquire::Retries=3 --no-install-recommends "$@" + } + - name: Run VM and collect results + uses: vmactions/freebsd-vm@c46abacb49f09938ca4e1702d15d836285d694cc # v1.5.9 + with: + release: ${{ matrix.release }} + cache-after-prepare: true + prepare: | + pkg install -y bash autotools m4 xxhash zstd liblz4 libidn2 libiconv python3 git + /usr/sbin/pw groupadd rsynctest + /usr/sbin/pw groupadd rsynctest-extra + /usr/sbin/pw useradd rsynctest -g rsynctest -G rsynctest-extra -m -d /home/rsynctest -s /bin/sh -w no + run: python3 -S .github/scripts/guest.py freebsd --release "${{ matrix.release }}" + - name: Verify guest results + run: python3 -S .github/scripts/guest.py check + - name: Upload binaries + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: freebsd-${{ matrix.release }}-amd64-bin + if-no-files-found: error + path: | + rsync + rsync-ssl + rsync.1 + rsync-ssl.1 + rsyncd.conf.5 + rrsync.1 + rrsync + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: freebsd-${{ matrix.release }}-amd64-results + if-no-files-found: error + path: | + config.log + test-results/status + test-results/features.json + test-results/nonroot/dirfd-probe.tsv + test-results/*/*/receipt.json + test-results/*/*/run.log + + netbsd: + strategy: + fail-fast: false + matrix: + release: ['11.1'] + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: NetBSD ${{ matrix.release }} X86-64 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Prepare VM host + timeout-minutes: 10 + run: | + set -- autoconf automake qemu-utils qemu-system-x86 ovmf + sudo apt-get install -y -q -o Acquire::Retries=3 --no-install-recommends "$@" || { + sudo apt-get update -q + sudo apt-get install -y -q -o Acquire::Retries=3 --no-install-recommends "$@" + } + - name: Prepare source + run: ./prepare-source + - name: Run VM and collect results + uses: vmactions/netbsd-vm@5957cf330aef2e6ddc63c604364c99ce2e02b91d # v1.5.3 + with: + release: ${{ matrix.release }} + sync: tar + cache-after-prepare: true + prepare: | + export PKG_PATH="https://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/$(uname -p)/$(uname -r)/All" + for attempt in 1 2 3; do + PATH=/usr/sbin:$PATH pkg_add -U git-base python312 && break + test "$attempt" -lt 3 || exit 1 + sleep 5 + done + ln -sf /usr/pkg/bin/python3.12 /usr/pkg/bin/python3 + /usr/sbin/groupadd rsynctest-extra + /usr/sbin/useradd -m -d /home/rsynctest -g =uid -G rsynctest-extra -s /bin/sh rsynctest + run: python3 -S .github/scripts/guest.py netbsd --release "${{ matrix.release }}" + - name: Verify guest results + run: python3 -S .github/scripts/guest.py check + - name: Upload binaries + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: netbsd-${{ matrix.release }}-amd64-bin + if-no-files-found: error + path: | + rsync + rsync-ssl + rsync.1 + rsync-ssl.1 + rsyncd.conf.5 + rrsync.1 + rrsync + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: netbsd-${{ matrix.release }}-amd64-results + if-no-files-found: error + path: | + config.log + test-results/status + test-results/nonroot/dirfd-probe.tsv + test-results/*/*/receipt.json + test-results/*/*/run.log + + openbsd: + strategy: + fail-fast: false + matrix: + release: ['8.0'] + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: OpenBSD ${{ matrix.release }} X86-64 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Prepare VM host + timeout-minutes: 10 + run: | + set -- qemu-utils qemu-system-x86 ovmf + sudo apt-get install -y -q -o Acquire::Retries=3 --no-install-recommends "$@" || { + sudo apt-get update -q + sudo apt-get install -y -q -o Acquire::Retries=3 --no-install-recommends "$@" + } + - name: Run VM and collect results + uses: vmactions/openbsd-vm@f1ca8bc8ecc5b6ef7218e5b914836e067e1546d3 # v1.4.9 + with: + release: ${{ matrix.release }} + cache-after-prepare: true + prepare: | + pkg_add -I bash autoconf%2.71 automake%1.16 xxhash zstd lz4 libidn2 libiconv git + /usr/sbin/groupadd rsynctest-extra + /usr/sbin/useradd -m -d /home/rsynctest -g =uid -G rsynctest-extra -s /bin/sh rsynctest + run: python3 -S .github/scripts/guest.py openbsd --release "${{ matrix.release }}" + - name: Verify guest results + run: python3 -S .github/scripts/guest.py check + - name: Upload binaries + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: openbsd-${{ matrix.release }}-amd64-bin + if-no-files-found: error + path: | + rsync + rsync-ssl + rsync.1 + rsync-ssl.1 + rsyncd.conf.5 + rrsync.1 + rrsync + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: openbsd-${{ matrix.release }}-amd64-results + if-no-files-found: error + path: | + config.log + test-results/status + test-results/features.json + test-results/nonroot/dirfd-probe.tsv + test-results/*/*/receipt.json + test-results/*/*/run.log diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 000000000..18a6d220c --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,156 @@ +name: CI + +on: + push: + branches: [master] + pull_request: + types: [opened, synchronize, reopened] + workflow_dispatch: + schedule: + - cron: '42 8 * * 0,2-6' + - cron: '42 8 * * 1' + - cron: '52 8 * * 1' + - cron: '42 9 * * 1' + - cron: '17 4 * * *' + +permissions: + contents: read + +defaults: + run: + shell: sh + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} + cancel-in-progress: true + +jobs: + changes: + name: Changes + runs-on: ubuntu-24.04 + timeout-minutes: 5 + outputs: + linux: ${{ steps.select.outputs.linux }} + bsd: ${{ steps.select.outputs.bsd }} + macos: ${{ steps.select.outputs.macos }} + other: ${{ steps.select.outputs.other }} + versions: ${{ steps.select.outputs.versions }} + android: ${{ steps.select.outputs.android }} + archive: ${{ steps.select.outputs.archive }} + diagnostics: ${{ steps.select.outputs.diagnostics }} + valgrind: ${{ steps.select.outputs.valgrind }} + interoperability: ${{ steps.select.outputs.interoperability }} + asan: ${{ steps.select.outputs.asan }} + coverage: ${{ steps.select.outputs.coverage }} + clang: ${{ steps.select.outputs.clang }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Select checks + id: select + run: python3 -B -S .github/scripts/changes.py + + linux: + name: Linux + needs: changes + if: &linux_selected ${{ needs.changes.outputs.linux == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 8 * * 0,2-6' || github.event.schedule == '42 8 * * 1') }} + uses: ./.github/workflows/linux.yml + with: + ubuntu: &weekly_schedule ${{ github.event_name != 'schedule' || github.event.schedule == '42 8 * * 1' }} + almalinux: &daily_schedule ${{ github.event_name != 'schedule' || github.event.schedule == '42 8 * * 0,2-6' || github.event.schedule == '42 8 * * 1' }} + alpine: *weekly_schedule + + bsd: + name: BSD + needs: changes + if: &bsd_selected ${{ needs.changes.outputs.bsd == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 8 * * 1') }} + uses: ./.github/workflows/bsd.yml + + macos: + name: macOS + needs: changes + if: &macos_selected ${{ needs.changes.outputs.macos == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 8 * * 0,2-6' || github.event.schedule == '42 8 * * 1') }} + uses: ./.github/workflows/macos.yml + + other: + name: Other + needs: changes + if: &other_selected ${{ needs.changes.outputs.other == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 8 * * 0,2-6' || github.event.schedule == '42 8 * * 1') }} + uses: ./.github/workflows/other.yml + with: + cygwin: *daily_schedule + solaris: *weekly_schedule + + versions: + name: Versions + needs: changes + if: &versions_selected ${{ needs.changes.outputs.versions == 'true' && (github.event_name != 'schedule' || github.event.schedule == '52 8 * * 1') }} + uses: ./.github/workflows/versions.yml + + android: + name: Android + needs: changes + if: &android_selected ${{ needs.changes.outputs.android == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 8 * * 1') }} + uses: ./.github/workflows/android.yml + + archive: + name: Archive + needs: changes + if: &archive_selected ${{ needs.changes.outputs.archive == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 8 * * 1') }} + uses: ./.github/workflows/archive.yml + + diagnostics: + name: Diagnostics + needs: changes + if: &diagnostics_selected ${{ needs.changes.outputs.diagnostics == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 9 * * 1') }} + uses: ./.github/workflows/diagnostics.yml + with: + lint: ®ular ${{ github.event_name != 'schedule' }} + python_37: *regular + asan: ${{ needs.changes.outputs.asan == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 9 * * 1') }} + coverage: ${{ needs.changes.outputs.coverage == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 9 * * 1') }} + clang_21: ${{ needs.changes.outputs.clang == 'true' && (github.event_name != 'schedule' || github.event.schedule == '42 9 * * 1') }} + + valgrind: + name: Valgrind + needs: changes + if: &valgrind_selected ${{ needs.changes.outputs.valgrind == 'true' && (github.event_name != 'schedule' || github.event.schedule == '17 4 * * *') }} + uses: ./.github/workflows/valgrind.yml + + interoperability: + name: Interoperability + needs: changes + if: &interoperability_selected ${{ needs.changes.outputs.interoperability == 'true' && github.event_name != 'schedule' }} + uses: ./.github/workflows/interoperability-tests.yml + permissions: + contents: read + actions: read + + required: + name: Required + if: always() + needs: [changes, linux, bsd, macos, other, versions, android, archive, diagnostics, valgrind, interoperability] + runs-on: ubuntu-24.04 + timeout-minutes: 5 + env: + CI_EXPECT_LINUX: *linux_selected + CI_EXPECT_BSD: *bsd_selected + CI_EXPECT_MACOS: *macos_selected + CI_EXPECT_OTHER: *other_selected + CI_EXPECT_VERSIONS: *versions_selected + CI_EXPECT_ANDROID: *android_selected + CI_EXPECT_ARCHIVE: *archive_selected + CI_EXPECT_DIAGNOSTICS: *diagnostics_selected + CI_EXPECT_VALGRIND: *valgrind_selected + CI_EXPECT_INTEROPERABILITY: *interoperability_selected + CI_SELECTIONS: ${{ toJSON(needs.changes.outputs) }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Check results + env: + CI_RESULTS: ${{ toJSON(needs) }} + run: python3 -S .github/scripts/check-results.py diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml deleted file mode 100644 index 6e0eccaff..000000000 --- a/.github/workflows/coverage.yml +++ /dev/null @@ -1,76 +0,0 @@ -name: Coverage (Ubuntu) - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/coverage.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/coverage.yml' - schedule: - - cron: '42 9 * * 1' - workflow_dispatch: - -jobs: - coverage: - runs-on: ubuntu-latest - name: gcov coverage - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - sudo apt-get update - sudo apt-get install -y acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev python3-cmarkgfm openssl gcovr - echo "/usr/local/bin" >>"$GITHUB_PATH" - - name: configure - run: ./configure --enable-coverage --with-rrsync - - name: make - run: make - - name: info - run: rsync --version - # Two coverage runs: the default pipe transport, then a second pass over a - # real loopback rsyncd (--use-tcp) which also exercises the require_tcp-only - # tests. gcovr's --print-summary line/branch/decision totals go to the step - # log (and the job summary below), so the numbers are visible in CI. - # `make coverage` exits with the suite's status, so a regression fails CI. - - name: coverage (pipe transport) - run: | - set -o pipefail - sudo make coverage 2>&1 | tee cov-pipe.log - - name: coverage (TCP transport) - run: | - set -o pipefail - sudo make coverage-tcp 2>&1 | tee cov-tcp.log - - name: coverage summary - if: always() - run: | - { - echo "## gcov coverage" - echo "### Pipe transport (\`make coverage\`)" - echo '```' - grep -E '^(lines|functions|branches|decisions):' cov-pipe.log || echo '(no summary -- see step log)' - echo '```' - echo "### TCP transport (\`make coverage-tcp\`)" - echo '```' - grep -E '^(lines|functions|branches|decisions):' cov-tcp.log || echo '(no summary -- see step log)' - echo '```' - } >> "$GITHUB_STEP_SUMMARY" - - name: upload HTML reports - if: always() - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: coverage-html - path: | - coverage - coverage-tcp diff --git a/.github/workflows/cygwin-build.yml b/.github/workflows/cygwin-build.yml deleted file mode 100644 index b82e18b12..000000000 --- a/.github/workflows/cygwin-build.yml +++ /dev/null @@ -1,109 +0,0 @@ -name: Test rsync on Cygwin - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/cygwin-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/cygwin-build.yml' - schedule: - - cron: '42 8 * * *' - -jobs: - test: - runs-on: windows-2022 - name: Test rsync on Cygwin - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: cygwin - run: | - $setup = Join-Path $Env:RUNNER_TEMP 'setup-x86_64.exe' - $sums = Join-Path $Env:RUNNER_TEMP 'cygwin-sha512.sum' - $packages = 'make,autoconf,automake,gcc-core,attr,libattr-devel,python39,python39-pip,libzstd-devel,liblz4-devel,libssl-devel,libxxhash0,libxxhash-devel,libidn2-devel' - - Invoke-WebRequest https://cygwin.com/setup-x86_64.exe -OutFile $setup - Invoke-WebRequest https://cygwin.com/sha512.sum -OutFile $sums - - $sum = Select-String -LiteralPath $sums -Pattern '^[0-9a-fA-F]{128}\s+\*?setup-x86_64\.exe$' | Select-Object -First 1 - if (-not $sum) { - throw 'setup-x86_64.exe is missing from Cygwin sha512.sum' - } - $expected = ($sum.Line -split '\s+')[0] - $actual = (Get-FileHash -LiteralPath $setup -Algorithm SHA512).Hash - if ($actual -ine $expected) { - throw 'Cygwin setup SHA-512 mismatch' - } - - $arguments = @( - '--quiet-mode', - '--no-desktop', - '--no-startmenu', - '--no-shortcuts', - '--root', 'C:\tools\cygwin', - '--local-package-dir', (Join-Path $Env:RUNNER_TEMP 'cygwin-packages'), - '--site', 'https://mirrors.kernel.org/sourceware/cygwin/', - '--packages', $packages - ) - $install = Start-Process -FilePath $setup -ArgumentList $arguments -Wait -PassThru -NoNewWindow - if ($install.ExitCode -ne 0) { - exit $install.ExitCode - } - - $bash = 'C:\tools\cygwin\bin\bash.exe' - if (-not (Test-Path -LiteralPath $bash)) { - throw 'Cygwin setup did not install bash' - } - & $bash -lc 'command -v make aclocal gcc python3 >/dev/null' - if ($LASTEXITCODE -ne 0) { - throw 'Cygwin setup did not install all required build tools' - } - - echo 'C:/tools/cygwin/bin' >>$Env:GITHUB_PATH - - name: commonmark - run: bash -c 'python3 -mpip install --user commonmark' - - name: configure - run: bash -c './configure --with-rrsync' - - name: make - run: bash -c 'make' - - name: install - run: bash -c 'make install' - - name: info - run: bash -c '/usr/local/bin/rsync --version' - - name: check - # chown-fake / devices-fake / xattrs / xattrs-hlink now RUN on Cygwin - # (rsyncfns.py drives xattrs via getfattr/setfattr from the `attr` - # package installed above), verified on a real Cygwin host. The real - # chown/devices tests still skip (need root/mknod), as do the - # RESOLVE_BENEATH symlink-race tests. Cygwin runs non-root, so the - # namecvt empty-response regression can run despite its common root skip. - run: bash -c 'RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/common.txt,@testsuite/skiplist/cygwin.txt,-daemon-namecvt-empty-response make check' - - name: check (TCP daemon transport) - # Second run with daemon tests over a real loopback rsyncd; the default - # 'make check' above uses the secure stdio-pipe transport. - run: bash -c './runtests.py --rsync-bin=`pwd`/rsync.exe --use-tcp -j 8' - - name: ssl file list - run: bash -c 'PATH="/usr/local/bin:$PATH" rsync-ssl --no-motd download.samba.org::rsyncftp/ || true' - - name: save artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: cygwin-bin - path: | - rsync.exe - rsync-ssl - rsync.1 - rsync-ssl.1 - rsyncd.conf.5 - rrsync.1 - rrsync diff --git a/.github/workflows/diagnostics.yml b/.github/workflows/diagnostics.yml new file mode 100644 index 000000000..c35709eef --- /dev/null +++ b/.github/workflows/diagnostics.yml @@ -0,0 +1,243 @@ +name: Diagnostics + +on: + workflow_call: + inputs: + lint: + required: true + type: boolean + python_37: + required: true + type: boolean + asan: + required: true + type: boolean + coverage: + required: true + type: boolean + clang_21: + required: true + type: boolean + outputs: + jobs: + description: Job results + value: ${{ toJSON(jobs) }} + inputs: + description: Job selections + value: ${{ toJSON(inputs) }} + +permissions: + contents: read + +defaults: + run: + shell: sh + +jobs: + python_37: + if: ${{ inputs.python_37 }} + runs-on: ubuntu-24.04 + timeout-minutes: 20 + name: Python 3.7 + container: python:3.7.17-bullseye@sha256:f36f6fe9ddb2622aab9803944cbe7e2bc50be75d624a081f7895e60e7c6b7f0a + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Configure + run: ./configure --disable-md2man --disable-openssl --disable-xxhash --disable-zstd --disable-lz4 --disable-idn --disable-acl-support --disable-xattr-support --with-included-popt + - name: Build + run: make -j2 check-progs + - name: Check Python sources + run: | + python3 -B -S - <<'PY' + import os + from pathlib import Path + import subprocess + + subprocess.run(['git', 'config', '--global', '--add', + 'safe.directory', str(Path.cwd())], check=True) + files = subprocess.check_output( + ['git', 'ls-files', '-z', '*.py'] + ).split(b'\0') + for name in filter(None, files): + path = Path(os.fsdecode(name)) + compile(path.read_bytes(), str(path), 'exec') + PY + - name: Check helpers + run: | + python3 -B -S .github/scripts/test-android.py + python3 -B -S .github/scripts/test-termux.py + python3 -B -S .github/scripts/test-storage.py + python3 -B -S .github/scripts/test-interop-metadata.py + python3 -B -S .github/scripts/test-changes.py + python3 -B -S .github/scripts/test-results.py + python3 -B -S .github/scripts/test-guest.py + python3 -B -S .github/scripts/test-diagnostics.py + python3 -B -S .github/scripts/test-archive.py + python3 -B -S .github/scripts/test-interop-policy.py + python3 -B -S .github/scripts/test-interop.py + python3 -B -S .github/scripts/test-transfers.py + - name: Tests + id: test + run: python3 -B -S testsuite/tools/check.py . ./rsync pipe,root,self-peer,linux,non-asan,no-userns test-results/python37 metadata profile process cli-transfers runner check-recipe -j2 + - name: Upload results + if: ${{ !cancelled() && (steps.test.outcome == 'success' || steps.test.outcome == 'failure') }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: python37-results + retention-days: 45 + if-no-files-found: error + path: test-results/python37/ + + lint: + if: ${{ inputs.lint }} + runs-on: ubuntu-24.04 + timeout-minutes: 15 + name: Workflow lint + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Install actionlint + run: python3 -S .github/scripts/diagnostics.py install-actionlint + - name: Version + run: actionlint -version + - name: Check workflows + run: actionlint -color -ignore '^label "ubuntu-26\.04(-arm)?" is unknown\.' + - name: Check result gate + run: python3 -S .github/scripts/test-results.py + - name: Check helpers + run: | + python3 -S .github/scripts/test-changes.py + python3 -S .github/scripts/test-guest.py + python3 -S .github/scripts/test-diagnostics.py + python3 -S .github/scripts/test-android.py + python3 -S .github/scripts/test-termux.py + python3 -S .github/scripts/test-storage.py + python3 -S .github/scripts/test-interop-metadata.py + python3 -S .github/scripts/test-archive.py + python3 -S .github/scripts/test-interop-policy.py + python3 -S .github/scripts/test-interop.py + python3 -S .github/scripts/test-transfers.py + - name: Check Python runtime + run: python3 -S testsuite/runtests.py --describe-tests > /dev/null + + asan: + if: ${{ inputs.asan }} + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: ASan + UBSan + env: + # Process-exit leaks are intentional; memory errors remain fatal + ASAN_OPTIONS: detect_leaks=0:abort_on_error=1 + UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install -y clang acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev openssl + echo "/usr/local/bin" >>"$GITHUB_PATH" + - name: Configure + # Exercise release behaviour with assertions disabled + run: | + CC=clang \ + CFLAGS="-fsanitize=address,undefined -fno-sanitize-recover=undefined -fno-omit-frame-pointer -g -O1 -DNDEBUG" \ + LDFLAGS="-fsanitize=address,undefined" \ + ./configure --with-rrsync --disable-md2man + - name: Build + id: build + run: make check-progs + - name: Version + run: ./rsync --version + - name: Tests (non-root pipe) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S testsuite/tools/check.py . ./rsync pipe,nonroot,self-peer,linux,asan,no-userns test-results/nonroot/pipe -j8 + - name: Tests (non-root TCP) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S testsuite/tools/check.py . ./rsync nonroot,self-peer,linux,asan,no-userns test-results/nonroot/tcp --use-tcp --daemon-tests-only -j8 + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: asan-results + if-no-files-found: error + path: | + test-results/*/*/receipt.json + test-results/*/*/run.log + + coverage: + if: ${{ inputs.coverage }} + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: Coverage + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install -y acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev python3-cmarkgfm openssl gcovr + echo "/usr/local/bin" >>"$GITHUB_PATH" + - name: Configure + run: ./configure --enable-coverage --with-rrsync + - name: Build + id: build + run: make + - name: Version + run: ./rsync --version + - name: Coverage (root pipe) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: sudo env RSYNC_TEST_PROFILES=pipe,non-asan,root,self-peer,linux,no-userns python3 -S .github/scripts/diagnostics.py coverage pipe + - name: Coverage (root TCP) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: sudo env RSYNC_TEST_PROFILES=non-asan,root,self-peer,linux,no-userns python3 -S .github/scripts/diagnostics.py coverage tcp + - name: Coverage summary + if: always() + run: python3 -S .github/scripts/diagnostics.py coverage-summary + - name: Upload reports + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: coverage-html + if-no-files-found: error + path: | + coverage + coverage-tcp + + clang_21: + if: ${{ inputs.clang_21 }} + runs-on: ubuntu-26.04 + timeout-minutes: 45 + name: Clang 21 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install -y clang-21 clang-tools-21 acl libacl1-dev attr libattr1-dev liblz4-dev libidn2-dev libzstd-dev libxxhash-dev libpopt-dev openssl + - name: Compiler version + run: clang-21 --version + - name: Configure + run: scan-build-21 ./configure --with-rrsync --disable-md2man + - name: Analyse + run: python3 -S .github/scripts/diagnostics.py scan + - name: Upload report + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: scan-build-report-clang21 + path: scan-report + if-no-files-found: ignore diff --git a/.github/workflows/fleettest.yml b/.github/workflows/fleettest.yml deleted file mode 100644 index 40f6f8fc6..000000000 --- a/.github/workflows/fleettest.yml +++ /dev/null @@ -1,74 +0,0 @@ -name: Test fleettest harness - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -# Bitrot check for testsuite/fleettest.py (the developer fleet CI harness). -# fleettest is meant to be run by developers on a modern Ubuntu box, so this -# job runs only on ubuntu-latest: it stands up a one-host "fleet" of two -# targets that both ssh to localhost and runs a real fleettest pass against it. -# It does not run on the BSD/Solaris/macOS/Cygwin matrix. - -on: - push: - branches: [ master ] - paths: - - 'testsuite/fleettest.py' - - '.github/workflows/fleettest.yml' - - 'runtests.py' - - 'testsuite/skiplist/**' - - 'testsuite/skiplist-spec_test.py' - pull_request: - types: [opened, synchronize, reopened] - paths: - - 'testsuite/fleettest.py' - - '.github/workflows/fleettest.yml' - - 'runtests.py' - - 'testsuite/skiplist/**' - - 'testsuite/skiplist-spec_test.py' - workflow_dispatch: - schedule: - - cron: '17 7 * * 1' - -jobs: - fleettest: - runs-on: ubuntu-latest - name: fleettest against localhost - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - sudo apt-get update - sudo apt-get install -y gcc g++ gawk autoconf automake \ - acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev \ - python3-cmarkgfm openssl rsync openssh-server - - name: set up ssh to localhost - run: | - mkdir -p ~/.ssh && chmod 700 ~/.ssh - ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519 - cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys - chmod 600 ~/.ssh/authorized_keys - sudo systemctl start ssh || sudo service ssh start - # fleettest connects with `ssh -o BatchMode=yes localhost`, which won't - # answer a host-key prompt -- so pre-trust localhost in known_hosts. - ssh-keyscan -H localhost 127.0.0.1 >> ~/.ssh/known_hosts 2>/dev/null - ssh -o BatchMode=yes -o ConnectTimeout=15 localhost 'echo ssh-to-localhost-ok' - - name: write localhost fleet config - run: | - cat > fleettest-ci.json <<'EOF' - { "targets": [ - { "name": "local-a", "ssh_host": "localhost", "workflow": "none.yml", - "configure_flags": [], "builddir": "rsync-citest-a", "privilege": "sudo" }, - { "name": "local-b", "ssh_host": "localhost", "workflow": "none.yml", - "configure_flags": [], "builddir": "rsync-citest-b", "privilege": "sudo" } - ] } - EOF - - name: fleettest --list (config sanity) - run: python3 testsuite/fleettest.py --fleet fleettest-ci.json --list - - name: run fleettest against localhost - # Two targets both on localhost exercise the parallel multi-target path - # and the per-run dir / port isolation; exit 0 iff every cell is OK. - run: python3 testsuite/fleettest.py --fleet fleettest-ci.json --timing diff --git a/.github/workflows/freebsd-build.yml b/.github/workflows/freebsd-build.yml deleted file mode 100644 index fe87913f5..000000000 --- a/.github/workflows/freebsd-build.yml +++ /dev/null @@ -1,58 +0,0 @@ -name: Test rsync on FreeBSD - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/freebsd-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/freebsd-build.yml' - schedule: - - cron: '42 8 * * 1' - -jobs: - test: - runs-on: ubuntu-latest - name: Test rsync on FreeBSD - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: Test in FreeBSD VM - id: test - uses: vmactions/freebsd-vm@v1 - with: - usesh: true - prepare: | - pkg install -y bash autotools m4 devel/xxhash zstd liblz4 python3 archivers/liblz4 git - run: | - freebsd-version - ./configure --with-rrsync \ - -disable-zstd --disable-md2man --disable-xxhash --disable-lz4 --disable-idn \ - --enable-roll-simd --enable-roll-asm --enable-md5-asm - make - ./rsync --version - make check - ./runtests.py --rsync-bin=`pwd`/rsync --use-tcp -j 8 - ./rsync-ssl --no-motd download.samba.org::rsyncftp/ || true - - name: save artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: freebsd-bin - path: | - rsync - rsync-ssl - rsync.1 - rsync-ssl.1 - rsyncd.conf.5 - rrsync.1 - rrsync diff --git a/.github/workflows/interoperability-tests.yml b/.github/workflows/interoperability-tests.yml new file mode 100644 index 000000000..0f6405476 --- /dev/null +++ b/.github/workflows/interoperability-tests.yml @@ -0,0 +1,244 @@ +name: Interoperability + +on: + workflow_call: + outputs: + jobs: + description: Native endpoint and result jobs + value: ${{ toJSON(jobs) }} + +permissions: + contents: read + actions: read + +jobs: + linux: + name: Ubuntu 24.04 X86-64 + outputs: + blocked: ${{ steps.transfers.outputs.blocked }} + runs-on: ubuntu-24.04 + timeout-minutes: 75 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install -y gcc make openssl openssh-server acl attr libacl1-dev libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev + sudo install -d -o root -g root -m 0755 /run/sshd + - name: Configure + run: ./configure --disable-md2man + - name: Build + run: make -j2 + - name: Transfers + id: transfers + uses: ./.github/actions/interoperability + with: + platform: linux + + freebsd: + name: FreeBSD 15.1 X86-64 + outputs: + blocked: ${{ steps.transfers.outputs.blocked }} + runs-on: ubuntu-24.04 + timeout-minutes: 75 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Prepare VM host + run: sudo apt-get update && sudo apt-get install -y qemu-utils qemu-system-x86 ovmf + - name: Configure and build + uses: vmactions/freebsd-vm@c46abacb49f09938ca4e1702d15d836285d694cc # v1.5.9 + with: + release: '15.1' + copyback: 'false' + cache-after-prepare: true + prepare: pkg install -y bash autotools m4 xxhash zstd liblz4 libidn2 libiconv python3 git + run: CPPFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib ./configure --disable-md2man --enable-acl-support --enable-xattr-support && make -j2 + - name: Transfers + id: transfers + uses: ./.github/actions/interoperability + with: + platform: freebsd + guest: 'true' + + netbsd: + name: NetBSD 11.1 X86-64 + outputs: + blocked: ${{ steps.transfers.outputs.blocked }} + runs-on: ubuntu-24.04 + timeout-minutes: 75 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Prepare VM host + run: sudo apt-get update && sudo apt-get install -y autoconf automake qemu-utils qemu-system-x86 ovmf + - name: Prepare source + run: ./prepare-source + - name: Configure and build + uses: vmactions/netbsd-vm@5957cf330aef2e6ddc63c604364c99ce2e02b91d # v1.5.3 + with: + release: '11.1' + sync: tar + copyback: 'false' + cache-after-prepare: true + prepare: | + export PKG_PATH="https://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/$(uname -p)/$(uname -r)/All" + for attempt in 1 2 3; do + PATH=/usr/sbin:$PATH pkg_add -U git-base python312 && break + test "$attempt" -lt 3 || exit 1 + sleep 5 + done + ln -sf /usr/pkg/bin/python3.12 /usr/pkg/bin/python3 + run: | + ./configure --disable-md2man --disable-zstd --disable-xxhash --disable-lz4 --disable-idn \ + --disable-acl-support --disable-xattr-support + make -j2 + - name: Transfers + id: transfers + uses: ./.github/actions/interoperability + with: + platform: netbsd + guest: 'true' + + openbsd: + name: OpenBSD 8.0 X86-64 + outputs: + blocked: ${{ steps.transfers.outputs.blocked }} + runs-on: ubuntu-24.04 + timeout-minutes: 75 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Prepare VM host + run: sudo apt-get update && sudo apt-get install -y qemu-utils qemu-system-x86 ovmf + - name: Configure and build + uses: vmactions/openbsd-vm@f1ca8bc8ecc5b6ef7218e5b914836e067e1546d3 # v1.4.9 + with: + release: '8.0' + copyback: 'false' + cache-after-prepare: true + prepare: pkg_add -I bash autoconf%2.71 automake%1.16 xxhash zstd lz4 libidn2 libiconv git + run: | + export AUTOCONF_VERSION=2.71 AUTOMAKE_VERSION=1.16 + export CPPFLAGS='-I/usr/local/include -DXXH_INLINE_ALL=1' LDFLAGS=-L/usr/local/lib ac_cv_search_XXH64_createState='' + ./configure --disable-md2man + make -j2 + - name: Transfers + id: transfers + uses: ./.github/actions/interoperability + with: + platform: openbsd + guest: 'true' + + macos: + name: macOS 26 ARM64 + outputs: + blocked: ${{ steps.transfers.outputs.blocked }} + runs-on: macos-26 + timeout-minutes: 75 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: '1.27.4' + cache: false + - name: Install dependencies + run: | + brew install automake openssl xxhash zstd lz4 libidn2 + echo "$(brew --prefix openssl)/bin" >>"$GITHUB_PATH" + - name: Configure + run: | + prefix=$(brew --prefix) + openssl=$(brew --prefix openssl) + CPPFLAGS="-I$prefix/include -I$openssl/include" LDFLAGS="-L$prefix/lib -L$openssl/lib" ./configure --disable-md2man + - name: Build + run: make -j2 + - name: Transfers + id: transfers + uses: ./.github/actions/interoperability + with: + platform: macos + + solaris: + name: Solaris 11.4 X86-64 + outputs: + blocked: ${{ steps.transfers.outputs.blocked }} + runs-on: ubuntu-24.04 + timeout-minutes: 75 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Prepare VM host + run: sudo apt-get update && sudo apt-get install -y qemu-utils qemu-system-x86 ovmf + - name: Configure and build + uses: vmactions/solaris-vm@87d3c436511cef802cd1637f86f73b1a97715c8c # v1.4.1 + with: + release: '11.4' + copyback: 'false' + cache-after-prepare: true + prepare: pkg install bash automake gnu-m4 pkg://solaris/runtime/python-35 autoconf gcc git system/file-system/ufs + run: ./configure --disable-md2man --disable-zstd --disable-xxhash --disable-lz4 --disable-idn && make + - name: Transfers + id: transfers + uses: ./.github/actions/interoperability + with: + platform: solaris + guest: 'true' + + cygwin: + name: Cygwin X86-64 + outputs: + blocked: ${{ steps.transfers.outputs.blocked }} + runs-on: windows-2025 + timeout-minutes: 75 + defaults: + run: + shell: pwsh + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Install Cygwin + run: .github/scripts/install-cygwin.ps1 -OpenSSH + - name: Configure + run: bash -c './configure --disable-md2man' + - name: Build + run: bash -c 'make -j2' + - name: Transfers + id: transfers + uses: ./.github/actions/interoperability + with: + platform: cygwin + - name: Restore Git + if: always() + run: | + $directory = 'C:\Program Files\Git\cmd' + if (-not (Test-Path -LiteralPath (Join-Path $directory 'git.exe'))) { + throw 'Runner Git is unavailable' + } + $directory >>$Env:GITHUB_PATH + + result: + name: Result + if: always() + needs: [linux, freebsd, netbsd, openbsd, macos, solaris, cygwin] + runs-on: ubuntu-24.04 + timeout-minutes: 15 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Verify all endpoint results and cleanup + env: + GH_TOKEN: ${{ github.token }} + INTEROP_JOBS: ${{ toJSON(needs) }} + run: python3 -B -S .github/scripts/interop-host.py result all --root "$RUNNER_TEMP/interoperability" diff --git a/.github/workflows/linux.yml b/.github/workflows/linux.yml new file mode 100644 index 000000000..127b535b6 --- /dev/null +++ b/.github/workflows/linux.yml @@ -0,0 +1,287 @@ +name: Linux + +on: + workflow_call: + inputs: + ubuntu: + required: true + type: boolean + almalinux: + required: true + type: boolean + alpine: + required: true + type: boolean + outputs: + jobs: + description: Job results + value: ${{ toJSON(jobs) }} + inputs: + description: Job selections + value: ${{ toJSON(inputs) }} + +permissions: + contents: read + +defaults: + run: + shell: sh + +jobs: + ubuntu: + if: ${{ inputs.ubuntu }} + strategy: + fail-fast: false + matrix: + include: + - release: '26.04' + arch: amd64 + machine: x86_64 + runner: ubuntu-26.04 + profiles: non-asan,self-peer,linux,no-userns + - release: '24.04' + arch: amd64 + machine: x86_64 + runner: ubuntu-24.04 + profiles: non-asan,self-peer,linux,no-userns + - release: '22.04' + arch: amd64 + machine: x86_64 + runner: ubuntu-22.04 + profiles: non-asan,self-peer,linux,no-zstd-threads + - release: '26.04' + arch: arm64 + machine: aarch64 + runner: ubuntu-26.04-arm + profiles: non-asan,self-peer,linux,no-simd,no-userns + - release: '24.04' + arch: arm64 + machine: aarch64 + runner: ubuntu-24.04-arm + profiles: non-asan,self-peer,linux,no-simd,no-userns + runs-on: ${{ matrix.runner }} + timeout-minutes: 45 + name: Ubuntu ${{ matrix.release }} ${{ matrix.arch == 'amd64' && 'X86-64' || 'ARM64' }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Platform + run: python3 -S .github/scripts/check-platform.py ubuntu --release "${{ matrix.release }}" --arch "${{ matrix.machine }}" + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev python3-cmarkgfm openssl openssh-server + sudo install -d -o root -g root -m 0755 /run/sshd + sudo sysctl -w fs.protected_regular=2 + echo "/usr/local/bin" >>"$GITHUB_PATH" + mkdir -p test-results + - name: Configure + run: ./configure --with-rrsync + - name: Build + id: build + run: make check-progs + - name: Installation check + run: python3 -S .github/scripts/installcheck.py + - name: Install + run: sudo make install + - name: Version + run: ./rsync --version + - name: Verify installed rsync-ssl + run: | + python3 -S testsuite/tools/check-rsync-ssl.py \ + --rsync /usr/local/bin/rsync \ + --rsync-ssl /usr/local/bin/rsync-ssl \ + --openssl /usr/bin/openssl + - name: Verify restricted SSH + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: sudo python3 -S testsuite/tools/check-ssh.py --rsync /usr/local/bin/rsync --rrsync support/rrsync + - name: Install storage dependencies + if: ${{ !cancelled() && steps.build.outcome == 'success' && matrix.release == '24.04' && matrix.arch == 'amd64' }} + run: sudo apt-get install -y cryptsetup-bin e2fsprogs + - name: Storage + if: ${{ !cancelled() && steps.build.outcome == 'success' && matrix.release == '24.04' && matrix.arch == 'amd64' }} + run: sudo python3 -B -S testsuite/tools/check-storage.py --binary "$PWD/rsync" --output "$PWD/test-results/root/storage" + - name: Tests (root pipe) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: sudo python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "pipe,root,${{ matrix.profiles }}" "$PWD/test-results/root/pipe" -j8 + - name: Tests (root protocol 30) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: sudo python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "pipe,root,${{ matrix.profiles }},protocol-30" "$PWD/test-results/root/protocol-30" --protocol=30 -j8 + - name: Tests (root protocol 29) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: sudo python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "pipe,root,${{ matrix.profiles }},protocol-29" "$PWD/test-results/root/protocol-29" --protocol=29 -j8 + - name: Tests (root TCP) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: sudo python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "root,${{ matrix.profiles }}" "$PWD/test-results/root/tcp" --use-tcp --daemon-tests-only -j8 + - name: Tests (non-root pipe) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "pipe,nonroot,${{ matrix.profiles }}" "$PWD/test-results/nonroot/pipe" -j8 + - name: Tests (non-root TCP) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "nonroot,${{ matrix.profiles }}" "$PWD/test-results/nonroot/tcp" --use-tcp --daemon-tests-only -j8 + - name: Upload binaries + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: ubuntu-${{ matrix.release }}-${{ matrix.arch }}-bin + if-no-files-found: error + path: | + rsync + rsync-ssl + rsync.1 + rsync-ssl.1 + rsyncd.conf.5 + rrsync.1 + rrsync + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: ubuntu-${{ matrix.release }}-${{ matrix.arch }}-results + if-no-files-found: error + path: | + config.log + test-results/*/*/receipt.json + test-results/*/*/run.log + + almalinux: + if: ${{ inputs.almalinux }} + runs-on: ubuntu-24.04 + timeout-minutes: 45 + container: + image: almalinux:8 + name: AlmaLinux 8 X86-64 + steps: + - name: Install Git + run: dnf -y install git + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Install dependencies + run: | + # RHEL 8's default Python is too old for the test runner + dnf -y install epel-release + dnf config-manager --set-enabled powertools + dnf -y install gcc gcc-c++ make autoconf automake m4 \ + python39 python39-pip diffutils \ + openssl openssl-devel \ + attr libattr-devel acl libacl-devel \ + zstd libzstd-devel \ + lz4 lz4-devel \ + xxhash xxhash-devel \ + libidn2 libidn2-devel + alternatives --set python3 /usr/bin/python3.9 + pip3 install commonmark + - name: Platform + run: python3 -S .github/scripts/check-platform.py almalinux --release 8 --arch x86_64 + - name: Configure + run: ./configure --with-rrsync + - name: Build + id: build + run: make check-progs + - name: Version + run: ./rsync --version + - name: Tests (root pipe) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" pipe,non-asan,root,self-peer,linux,no-zstd-threads,no-userns "$PWD/test-results/root/pipe" -j8 + - name: Tests (root TCP) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" non-asan,root,self-peer,linux,no-zstd-threads,no-userns "$PWD/test-results/root/tcp" --use-tcp --daemon-tests-only -j8 + - name: Upload binaries + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: almalinux-8-amd64-bin + if-no-files-found: error + path: | + rsync + rsync-ssl + rsync.1 + rsync-ssl.1 + rsyncd.conf.5 + rrsync.1 + rrsync + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: almalinux-8-amd64-results + if-no-files-found: error + path: | + config.log + test-results/*/*/run.log + test-results/*/*/receipt.json + + alpine: + if: ${{ inputs.alpine }} + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: Alpine 3.24.4 ${{ matrix.arch == 'amd64' && 'X86-64' || 'X86-32' }} + strategy: + fail-fast: false + matrix: + include: + - arch: amd64 + platform: linux/amd64 + apk_arch: x86_64 + bits: 64 + image: alpine@sha256:0f49f331d9212aff11b932863be4eb01cdf962969aa86dab7d1ee21ac6f5f52b + profiles: non-asan,self-peer,linux,no-simd,no-userns + - arch: x86 + platform: linux/386 + apk_arch: x86 + bits: 32 + image: alpine@sha256:55d68c85222dd92297b0f8885354739d89f0dc17fc2f903c49dc1d9c129c3d6a + profiles: non-asan,self-peer,linux,no-simd,no-userns + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Prepare host + run: | + sudo sysctl -w fs.protected_regular=2 + mkdir -p "$RUNNER_TEMP/rsync-alpine-${{ matrix.arch }}" + docker pull --platform "${{ matrix.platform }}" "${{ matrix.image }}" + - name: Run Alpine and collect results + env: + APK_ARCH: ${{ matrix.apk_arch }} + BITS: ${{ matrix.bits }} + PROFILES: ${{ matrix.profiles }} + run: | + docker run --rm -i --platform "${{ matrix.platform }}" \ + -e APK_ARCH -e BITS -e PROFILES -e RSYNC_TEST_TMP=/dev/shm \ + -v "$PWD:/source:ro" \ + -v "$RUNNER_TEMP/rsync-alpine-${{ matrix.arch }}:/results" \ + "${{ matrix.image }}" sh -ec ' + apk add --no-cache acl acl-dev attr attr-dev autoconf automake bash build-base diffutils gawk git \ + libidn2-dev linux-headers lz4-dev musl-locales openssl-dev py3-commonmark python3 setpriv xxhash-dev zstd-dev + exec python3 -S /source/.github/scripts/guest.py alpine --output /results + ' + - name: Upload binaries + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: alpine-3.24.4-${{ matrix.arch }}-bin + if-no-files-found: error + path: ${{ runner.temp }}/rsync-alpine-${{ matrix.arch }}/bin + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: alpine-3.24.4-${{ matrix.arch }}-results + if-no-files-found: error + path: | + ${{ runner.temp }}/rsync-alpine-${{ matrix.arch }}/config.log + ${{ runner.temp }}/rsync-alpine-${{ matrix.arch }}/features.json + ${{ runner.temp }}/rsync-alpine-${{ matrix.arch }}/*/*/receipt.json + ${{ runner.temp }}/rsync-alpine-${{ matrix.arch }}/*/*/run.log diff --git a/.github/workflows/macos-build.yml b/.github/workflows/macos-build.yml deleted file mode 100644 index e66f5822c..000000000 --- a/.github/workflows/macos-build.yml +++ /dev/null @@ -1,74 +0,0 @@ -name: Test rsync on macOS - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/macos-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/macos-build.yml' - schedule: - - cron: '42 8 * * *' - -jobs: - test: - runs-on: macos-latest - name: Test rsync on macOS - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - brew install automake openssl xxhash zstd lz4 libidn2 - pip3 install --user --break-system-packages commonmark - echo "$(brew --prefix)/bin" >>"$GITHUB_PATH" - - name: configure - run: | - BREW_PREFIX=$(brew --prefix) - OPENSSL_PREFIX=$(brew --prefix openssl) - CPPFLAGS="-I${BREW_PREFIX}/include -I${OPENSSL_PREFIX}/include" \ - LDFLAGS="-L${BREW_PREFIX}/lib -L${OPENSSL_PREFIX}/lib" \ - ./configure --with-rrsync - - name: make - run: make check-progs - - name: install - run: sudo make install - - name: info - run: rsync --version - - name: check search-only directories - run: | - ./runtests.py --rsync-bin="$PWD/rsync" \ - search-only-destination search-only-held-dirfd - - name: check - # chown-fake / devices-fake / xattrs / xattrs-hlink now RUN on macOS - # (rsyncfns.py drives xattrs via the `xattr` command), verified on a - # real macOS host, so they're no longer in the skip set. - run: sudo RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/common.txt,@testsuite/skiplist/macos.txt make check - - name: check (TCP daemon transport) - # Second run with daemon tests over a real loopback rsyncd; the default - # 'make check' above uses the secure stdio-pipe transport. - run: sudo ./runtests.py --rsync-bin="$PWD/rsync" --use-tcp -j 8 - - name: ssl file list - run: rsync-ssl --no-motd download.samba.org::rsyncftp/ || true - - name: save artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: macos-bin - path: | - rsync - rsync-ssl - rsync.1 - rsync-ssl.1 - rsyncd.conf.5 - rrsync.1 - rrsync diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml new file mode 100644 index 000000000..38a91457f --- /dev/null +++ b/.github/workflows/macos.yml @@ -0,0 +1,106 @@ +name: macOS + +on: + workflow_call: + outputs: + jobs: + description: Job results + value: ${{ toJSON(jobs) }} + +permissions: + contents: read + +defaults: + run: + shell: sh + +jobs: + macos: + strategy: + fail-fast: false + matrix: + include: + - release: '26' + arch: arm64 + runner: macos-26 + - release: '26' + arch: x86_64 + runner: macos-26-intel + runs-on: ${{ matrix.runner }} + timeout-minutes: 45 + name: ${{ matrix.release }} ${{ matrix.arch == 'x86_64' && 'X86-64' || 'ARM64' }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Platform + run: python3 -S .github/scripts/check-platform.py macos --release "${{ matrix.release }}" --arch "${{ matrix.arch }}" + - name: Install dependencies + run: | + brew install automake openssl xxhash zstd lz4 libidn2 + pip3 install --user --break-system-packages commonmark + echo "$(brew --prefix)/bin" >>"$GITHUB_PATH" + mkdir -p test-results + - name: Configure + run: | + BREW_PREFIX=$(brew --prefix) + OPENSSL_PREFIX=$(brew --prefix openssl) + CPPFLAGS="-I${BREW_PREFIX}/include -I${OPENSSL_PREFIX}/include" \ + LDFLAGS="-L${BREW_PREFIX}/lib -L${OPENSSL_PREFIX}/lib" \ + ./configure --with-rrsync --enable-iconv --enable-acl-support --enable-xattr-support + - name: Build + id: build + run: make check-progs + - name: Install + run: sudo make install + - name: Version + run: /usr/local/bin/rsync --version + - name: Features + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S testsuite/tools/check-features.py "$PWD/rsync" zstd lz4 xxh128 xxh3 xxh64 iconv IDN ACLs xattrs > "$PWD/test-results/features.json" + - name: Directory descriptors + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: | + make dirfd-probe + "$PWD/dirfd-probe" "$PWD/test-results" > "$PWD/test-results/dirfd-probe.tsv" + - name: Tests (non-root pipe) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "pipe,non-asan,nonroot,self-peer,macos" "$PWD/test-results/nonroot/pipe" -j8 + - name: Tests (non-root TCP) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "non-asan,nonroot,self-peer,macos" "$PWD/test-results/nonroot/tcp" --use-tcp --daemon-tests-only -j8 + - name: Tests (root pipe) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: sudo python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "pipe,non-asan,root,self-peer,macos" "$PWD/test-results/root/pipe" -j8 + - name: Tests (root TCP) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: sudo python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync" "non-asan,root,self-peer,macos" "$PWD/test-results/root/tcp" --use-tcp --daemon-tests-only -j8 + - name: Upload binaries + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: macos-${{ matrix.release }}-${{ matrix.arch }}-bin + if-no-files-found: error + path: | + rsync + rsync-ssl + rsync.1 + rsync-ssl.1 + rsyncd.conf.5 + rrsync.1 + rrsync + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: macos-${{ matrix.release }}-${{ matrix.arch }}-results + if-no-files-found: error + path: | + config.log + test-results/features.json + test-results/dirfd-probe.tsv + test-results/*/*/receipt.json + test-results/*/*/run.log diff --git a/.github/workflows/netbsd-build.yml b/.github/workflows/netbsd-build.yml deleted file mode 100644 index 4d2463b78..000000000 --- a/.github/workflows/netbsd-build.yml +++ /dev/null @@ -1,57 +0,0 @@ -name: Test rsync on NetBSD - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/netbsd-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/netbsd-build.yml' - schedule: - - cron: '42 8 * * 1' - -jobs: - test: - runs-on: ubuntu-latest - name: Test rsync on NetBSD - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: Test in NetBSD VM - id: test - uses: vmactions/netbsd-vm@v1 - with: - usesh: true - prepare: | - PATH=/usr/sbin:$PATH pkg_add autoconf automake python312 - ln -sf /usr/pkg/bin/python3.12 /usr/pkg/bin/python3 - run: | - uname -a - ./configure --with-rrsync --disable-zstd --disable-md2man --disable-xxhash --disable-lz4 --disable-idn - make - ./rsync --version - make check - ./runtests.py --rsync-bin=`pwd`/rsync --use-tcp -j 8 - ./rsync-ssl --no-motd download.samba.org::rsyncftp/ || true - - name: save artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: netbsd-bin - path: | - rsync - rsync-ssl - rsync.1 - rsync-ssl.1 - rsyncd.conf.5 - rrsync.1 - rrsync diff --git a/.github/workflows/openbsd-build.yml b/.github/workflows/openbsd-build.yml deleted file mode 100644 index 6218b9a24..000000000 --- a/.github/workflows/openbsd-build.yml +++ /dev/null @@ -1,73 +0,0 @@ -name: Test rsync on OpenBSD - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/openbsd-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/openbsd-build.yml' - schedule: - - cron: '42 8 * * 1' - -jobs: - test: - runs-on: ubuntu-latest - name: Test rsync on OpenBSD - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: Test in OpenBSD VM - id: test - uses: vmactions/openbsd-vm@v1 - with: - usesh: true - prepare: | - pkg_add -I bash autoconf%2.71 automake%1.16 - run: | - uname -a - export AUTOCONF_VERSION=2.71 - export AUTOMAKE_VERSION=1.16 - ./configure --with-rrsync --disable-zstd --disable-md2man --disable-xxhash --disable-lz4 --disable-idn - make - ./rsync --version - # The flipper (symlink-race) tests are excluded on OpenBSD, as on the - # fleet's OpenBSD box: this kernel has a connect()-under-rename-load - # lost-wakeup and an FFS rename-storm corruption that hang them to - # the 300s timeout for non-rsync reasons (see - # dev-notes/openbsd-connect-lost-wakeup-report.txt); the protections - # they exercise are verified on the Linux/BSD boxes. - export RSYNC_EXCLUDE=acl-symlink-race,sender-readlink-atfd,sender-remove-source-secure - make check - # The --use-tcp daemon tests run at -j2 here (vs -j8 elsewhere): this - # job runs inside a nested VM, and at -j8 the many concurrent loopback - # daemons occasionally lose a connection-handshake timing race under - # that resource pressure, hanging one test to the 300s timeout. It is - # an environment artifact, not an rsync bug (the handshake is - # deadlock-free and unreproducible elsewhere, even pinned to 1 CPU at - # -j8); -j2 keeps the VM from over-subscribing. The pipe `make check` - # above stays at the default parallelism. - ./runtests.py --rsync-bin=`pwd`/rsync --use-tcp -j 2 - ./rsync-ssl --no-motd download.samba.org::rsyncftp/ || true - - name: save artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: openbsd-bin - path: | - rsync - rsync-ssl - rsync.1 - rsync-ssl.1 - rsyncd.conf.5 - rrsync.1 - rrsync diff --git a/.github/workflows/other.yml b/.github/workflows/other.yml new file mode 100644 index 000000000..37ddf23c6 --- /dev/null +++ b/.github/workflows/other.yml @@ -0,0 +1,150 @@ +name: Other + +on: + workflow_call: + inputs: + cygwin: + required: true + type: boolean + solaris: + required: true + type: boolean + outputs: + jobs: + description: Job results + value: ${{ toJSON(jobs) }} + inputs: + description: Job selections + value: ${{ toJSON(inputs) }} + +permissions: + contents: read + +defaults: + run: + shell: sh + +jobs: + cygwin: + if: ${{ inputs.cygwin }} + runs-on: windows-2025 + timeout-minutes: 60 + name: Cygwin X86-64 + defaults: + run: + shell: pwsh + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Install Cygwin + run: .github/scripts/install-cygwin.ps1 + - name: Install CommonMark + run: bash -c 'python3 -mpip install --user commonmark' + - name: Platform + run: bash -c 'python3 -S .github/scripts/check-platform.py cygwin --arch x86_64' + - name: Configure + run: bash -c './configure --with-rrsync' + - name: Build + id: build + run: bash -c 'make check-progs' + - name: Install + run: bash -c 'make install' + - name: Version + run: bash -c '/usr/local/bin/rsync --version' + - name: Tests (non-root pipe) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: bash -c 'python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync.exe" pipe,non-asan,nonroot,self-peer,cygwin "$PWD/test-results/nonroot/pipe" -j8' + - name: Tests (non-root TCP) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: bash -c 'python3 -S testsuite/tools/check.py "$PWD" "$PWD/rsync.exe" non-asan,nonroot,self-peer,cygwin "$PWD/test-results/nonroot/tcp" --use-tcp --daemon-tests-only -j8' + - name: Upload binaries + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: cygwin-amd64-bin + if-no-files-found: error + path: | + rsync.exe + rsync-ssl + rsync.1 + rsync-ssl.1 + rsyncd.conf.5 + rrsync.1 + rrsync + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: cygwin-amd64-results + if-no-files-found: error + path: | + test-results/*/*/run.log + test-results/*/*/receipt.json + - name: Restore Git + if: always() + run: | + $directory = 'C:\Program Files\Git\cmd' + if (-not (Test-Path -LiteralPath (Join-Path $directory 'git.exe'))) { + throw 'Runner Git is unavailable' + } + $directory >>$Env:GITHUB_PATH + + solaris: + if: ${{ inputs.solaris }} + runs-on: ubuntu-24.04 + timeout-minutes: 45 + name: Solaris 11.4 X86-64 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Prepare VM host + timeout-minutes: 10 + run: | + set -- qemu-utils qemu-system-x86 ovmf + sudo apt-get install -y -q -o Acquire::Retries=3 --no-install-recommends "$@" || { + sudo apt-get update -q + sudo apt-get install -y -q -o Acquire::Retries=3 --no-install-recommends "$@" + } + - name: Run VM and collect results + uses: vmactions/solaris-vm@87d3c436511cef802cd1637f86f73b1a97715c8c # v1.4.1 + with: + release: '11.4' + cache-after-prepare: true + prepare: | + pkg install bash automake gnu-m4 pkg://solaris/runtime/python-35 autoconf gcc git + run: python3 -S .github/scripts/guest.py solaris + - name: Verify guest results + run: python3 -S .github/scripts/guest.py check + - name: Upload binaries + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: solaris-11.4-amd64-bin + if-no-files-found: error + path: | + rsync + rsync-ssl + rsync.1 + rsync-ssl.1 + rsyncd.conf.5 + rrsync.1 + rrsync + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: solaris-11.4-amd64-results + if-no-files-found: error + path: | + config.log + test-results/status + test-results/*/*/receipt.json + test-results/*/*/run.log diff --git a/.github/workflows/scan-build.yml b/.github/workflows/scan-build.yml deleted file mode 100644 index 6aafaa12e..000000000 --- a/.github/workflows/scan-build.yml +++ /dev/null @@ -1,92 +0,0 @@ -name: rsync scan-build (clang analyzer) - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/scan-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/scan-build.yml' - workflow_dispatch: - -jobs: - # GATING run: pinned clang-18 on a pinned runner so the checker set -- and - # thus the expected zero -- is deterministic. The tree is kept clean for - # clang-18, so --status-bugs (non-zero exit on any report) fails the build - # when a new finding appears. Pin both the analyzer (clang-18/clang-tools-18) - # and the runner (ubuntu-24.04, whose apt repos carry those packages). - gate-clang18: - runs-on: ubuntu-24.04 - name: scan-build gate (clang-18, pinned) - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - sudo apt-get update - sudo apt-get install -y clang-18 clang-tools-18 acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev libpopt-dev openssl - - name: configure (under scan-build) - # Run configure under scan-build so its analyzer compiler-wrapper is baked - # into the Makefile's $(CC); --disable-md2man avoids the doc toolchain. - run: scan-build-18 ./configure --with-rrsync --disable-md2man - - name: scan-build (gating) - # --status-bugs makes scan-build exit non-zero if it finds ANY report. - # pipefail + 'exit $status' propagate that through the tee so the job goes - # red while still printing the summary; the report uploads for triage. - run: | - set -o pipefail - status=0 - scan-build-18 --status-bugs -o "$PWD/scan-report" make check-progs -j"$(nproc)" 2>&1 | tee scan-build.out || status=$? - echo '## scan-build gate (clang-18)' >>"$GITHUB_STEP_SUMMARY" - grep -E 'scan-build: .* bugs? found|scan-build: No bugs found' scan-build.out >>"$GITHUB_STEP_SUMMARY" || true - exit $status - - name: upload report - if: always() - uses: actions/upload-artifact@v4 - with: - name: scan-build-report-clang18 - path: scan-report - if-no-files-found: ignore - - # INFORMATIONAL run: whatever clang ubuntu-latest currently ships. Newer - # clang releases enable extra, FP-heavy checkers (e.g. unix.Chroot - # "no chdir after chroot", alpha.unix.Stream) that the gate deliberately - # avoids, so this is NOT a gate (no --status-bugs). It surfaces what the - # newest analyzer sees -- useful for spotting genuine new findings before a - # gate bump -- without blocking merges. continue-on-error keeps a noisy or - # broken run from affecting the workflow's required status. - informational-latest: - runs-on: ubuntu-latest - name: scan-build (latest clang, informational) - continue-on-error: true - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - sudo apt-get update - sudo apt-get install -y clang clang-tools acl libacl1-dev attr libattr1-dev liblz4-dev libidn2-dev libzstd-dev libxxhash-dev libpopt-dev openssl - - name: configure (under scan-build) - run: scan-build ./configure --with-rrsync --disable-md2man - - name: scan-build (informational) - run: | - scan-build -o "$PWD/scan-report" make check-progs -j"$(nproc)" 2>&1 | tee scan-build.out - echo '## scan-build informational (latest clang)' >>"$GITHUB_STEP_SUMMARY" - grep -E 'scan-build: .* bugs? found|scan-build: No bugs found' scan-build.out >>"$GITHUB_STEP_SUMMARY" || true - - name: upload report - if: always() - uses: actions/upload-artifact@v4 - with: - name: scan-build-report-latest - path: scan-report - if-no-files-found: ignore diff --git a/.github/workflows/solaris-build.yml b/.github/workflows/solaris-build.yml deleted file mode 100644 index 1b328e4a1..000000000 --- a/.github/workflows/solaris-build.yml +++ /dev/null @@ -1,56 +0,0 @@ -name: Test rsync on Solaris - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/solaris-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/solaris-build.yml' - schedule: - - cron: '42 8 * * 1' - -jobs: - test: - runs-on: ubuntu-latest - name: Test rsync on Solaris - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: Test in Solaris VM - id: test - uses: vmactions/solaris-vm@v1 - with: - usesh: true - prepare: | - pkg install bash automake gnu-m4 pkg://solaris/runtime/python-35 autoconf gcc git - run: | - uname -a - ./configure --with-rrsync -disable-zstd --disable-md2man --disable-xxhash --disable-lz4 --disable-idn - make - ./rsync --version - make check - ./runtests.py --rsync-bin=`pwd`/rsync --use-tcp -j 8 - ./rsync-ssl --no-motd download.samba.org::rsyncftp/ || true - - name: save artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: solaris-bin - path: | - rsync - rsync-ssl - rsync.1 - rsync-ssl.1 - rsyncd.conf.5 - rrsync.1 - rrsync diff --git a/.github/workflows/ubuntu-22.04-build.yml b/.github/workflows/ubuntu-22.04-build.yml deleted file mode 100644 index ce05cf1a2..000000000 --- a/.github/workflows/ubuntu-22.04-build.yml +++ /dev/null @@ -1,70 +0,0 @@ -name: Test rsync on Ubuntu 22.04 - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -# Older-LTS coverage to help with backporting security fixes. ubuntu-22.04 -# is currently the oldest GitHub Actions runner image (20.04 was retired -# in April 2025). - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/ubuntu-22.04-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/ubuntu-22.04-build.yml' - schedule: - - cron: '42 8 * * *' - -jobs: - test: - runs-on: ubuntu-22.04 - name: Test rsync on Ubuntu 22.04 - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - sudo apt-get update - sudo apt-get install acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev python3-cmarkgfm openssl - echo "/usr/local/bin" >>"$GITHUB_PATH" - - name: configure - run: ./configure --with-rrsync - - name: make - run: make - - name: install - run: sudo make install - - name: info - run: rsync --version - - name: check - run: sudo RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt make check - - name: check30 - run: sudo RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt,@testsuite/skiplist/proto30.txt make check30 - - name: check29 - run: sudo RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt,@testsuite/skiplist/proto29.txt make check29 - - name: check (TCP daemon transport) - # Second run with daemon tests over a real loopback rsyncd; the default - # 'make check' above uses the secure stdio-pipe transport. - run: sudo ./runtests.py --rsync-bin="$PWD/rsync" --use-tcp -j 8 - - name: ssl file list - run: rsync-ssl --no-motd download.samba.org::rsyncftp/ || true - - name: save artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: ubuntu-22.04-bin - path: | - rsync - rsync-ssl - rsync.1 - rsync-ssl.1 - rsyncd.conf.5 - rrsync.1 - rrsync diff --git a/.github/workflows/ubuntu-build.yml b/.github/workflows/ubuntu-build.yml deleted file mode 100644 index 3adab91e6..000000000 --- a/.github/workflows/ubuntu-build.yml +++ /dev/null @@ -1,84 +0,0 @@ -name: Test rsync on Ubuntu - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/ubuntu-build.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/ubuntu-build.yml' - schedule: - - cron: '42 8 * * *' - -jobs: - test: - runs-on: ubuntu-latest - name: Test rsync on Ubuntu - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - sudo apt-get update - sudo apt-get install acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev python3-cmarkgfm openssl - echo "/usr/local/bin" >>"$GITHUB_PATH" - - name: configure - run: ./configure --with-rrsync - - name: make - run: make - - name: install - run: sudo make install - - name: info - run: rsync --version - - name: check - run: sudo RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt make check - - name: check30 - run: sudo RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt,@testsuite/skiplist/proto30.txt make check30 - - name: check29 - run: sudo RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt,@testsuite/skiplist/proto29.txt make check29 - - name: check (TCP daemon transport) - # Second run with daemon tests over a real loopback rsyncd. The default - # 'make check' above uses the secure stdio-pipe transport (no listening - # sockets); this run exercises the real TCP accept/auth path. Skip-set - # is env-dependent here (chroot-acl), so leave RSYNC_EXPECT_SKIPPED unset. - run: sudo ./runtests.py --rsync-bin="$PWD/rsync" --use-tcp -j 8 - - name: check (non-root, targeted) - # Every run above is root (sudo), so privilege-sensitive tests never hit - # their non-root path. Run those here as the unprivileged 'runner' user - # (NO sudo). Explicit test names make runtests.py full_run False, so - # RSYNC_EXPECT_SKIPPED is bypassed -- no per-platform skip list needed. - # daemon-namecvt-empty-response -- REQUIRES non-root (skips as root by - # design); the only test with no other CI coverage (Benjamin #2). - # ownership-depth -- non-root takes the group-only remap path. - # daemon -- non-root takes the default-config path. - # CONVENTION: a new test that requires/meaningfully exercises a non-root - # path must be added to the list below (kept in sync with the fleet - # harness's nonroot_tests). - run: | - sudo rm -rf testtmp # prior root steps left it root-owned - ./runtests.py --rsync-bin="$PWD/rsync" \ - daemon-namecvt-empty-response ownership-depth daemon - - name: ssl file list - run: rsync-ssl --no-motd download.samba.org::rsyncftp/ || true - - name: save artifact - uses: actions/upload-artifact@v4 - with: - retention-days: 45 - name: ubuntu-bin - path: | - rsync - rsync-ssl - rsync.1 - rsync-ssl.1 - rsyncd.conf.5 - rrsync.1 - rrsync diff --git a/.github/workflows/ubuntu-version-mix.yml b/.github/workflows/ubuntu-version-mix.yml deleted file mode 100644 index d16dd09dd..000000000 --- a/.github/workflows/ubuntu-version-mix.yml +++ /dev/null @@ -1,82 +0,0 @@ -name: Test rsync version mixing on Ubuntu - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true - -# Runs the CURRENT test suite with two different rsync binaries: the freshly -# built ./rsync as the client/driver, and a committed OLD static binary -# (old_versions/rsync_) as the daemon / remote-shell peer. This exercises -# real version mixing over the wire -- more convincing than --protocol forcing, -# which only makes the current binary speak an old protocol. -# -# Direction is fixed: the current binary always drives (only it understands the -# new test scripts); the old binary is only ever the server/daemon side. The -# reverse (old client driving new scripts) is not possible -- but one test, -# reverse-daemon-delta, swaps the roles internally (current build as the daemon, -# old binary as the client) to cover the backward-compat direction: a current -# daemon serving the installed base of old clients. -# -# The per-version manifest testsuite/expect/rsync_.expect lists exactly -# which tests run and each one's expected outcome (pass/skip/fail/xfail), so an -# old peer's known feature gaps are recorded rather than treated as breakage. -# -# All peers run in a SINGLE job (looped, not a matrix) so the PR shows one check -# line rather than one per version. Each peer/transport is a foldable ::group:: -# in the log, and a failure annotates which peer/transport broke. - -on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/ubuntu-version-mix.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/ubuntu-version-mix.yml' - schedule: - - cron: '52 8 * * 1' - -jobs: - version-mix: - runs-on: ubuntu-latest - name: rsync version-mix - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - sudo apt-get update - sudo apt-get install acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev python3-cmarkgfm openssl - echo "/usr/local/bin" >>"$GITHUB_PATH" - - name: configure - run: ./configure --with-rrsync - - name: make - # check-progs builds rsync AND the test helper programs (tls, trimslash, - # t_unsafe, ...) that runtests.py requires; plain `make` does not. - run: make check-progs - - name: info - run: ./rsync --version | head -1 - - name: version mixing (all peers, pipe + TCP transports) - run: | - rc=0 - for peer in old_versions/rsync_*; do - chmod +x "$peer" - name=$(basename "$peer") - expect="testsuite/expect/$name.expect" - for transport in pipe tcp; do - tcp=() - [ "$transport" = tcp ] && tcp=(--use-tcp) - echo "::group::$name ($transport): $("$peer" --version | head -1)" - if ! ./runtests.py --rsync-bin="$PWD/rsync" --rsync-bin2="$PWD/$peer" \ - --expect-result "$expect" "${tcp[@]}" -j 8; then - echo "::error::version-mix failed: $name ($transport)" - rc=1 - fi - echo "::endgroup::" - done - done - exit $rc diff --git a/.github/workflows/valgrind.yml b/.github/workflows/valgrind.yml index 33be5f102..1e82705c5 100644 --- a/.github/workflows/valgrind.yml +++ b/.github/workflows/valgrind.yml @@ -1,100 +1,79 @@ -name: Valgrind memcheck - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} - cancel-in-progress: true +name: Valgrind on: - push: - branches: [ master ] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/valgrind.yml' - pull_request: - types: [opened, synchronize, reopened] - paths-ignore: - - '.github/workflows/*.yml' - - '!.github/workflows/valgrind.yml' - schedule: - - cron: '17 4 * * *' - workflow_dispatch: + workflow_call: + outputs: + jobs: + description: Job results + value: ${{ toJSON(jobs) }} + +permissions: + contents: read + +defaults: + run: + shell: sh jobs: - memcheck: - runs-on: ubuntu-latest - timeout-minutes: 120 + test: + runs-on: ubuntu-24.04 + timeout-minutes: 60 strategy: fail-fast: false matrix: - privilege: [ user, root ] - transport: [ pipe, tcp ] - name: memcheck (${{ matrix.privilege }}, ${{ matrix.transport }}) + include: + - name: Root pipe + privilege: root + transport: pipe + profiles: pipe,non-asan,root,self-peer,linux,valgrind,no-userns + - name: Root TCP + privilege: root + transport: tcp + profiles: non-asan,root,self-peer,linux,valgrind,no-userns + - name: Non-root pipe + privilege: non-root + transport: pipe + profiles: pipe,non-asan,nonroot,self-peer,linux,valgrind,no-userns + - name: Non-root TCP + privilege: non-root + transport: tcp + profiles: non-asan,nonroot,self-peer,linux,valgrind,no-userns + name: ${{ matrix.name }} + env: + VALGRIND_SCRATCH: /tmp/rsync-valgrind-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.privilege }}-${{ matrix.transport }} steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - name: prep - run: | - sudo apt-get update - sudo apt-get install -y valgrind acl libacl1-dev attr libattr1-dev \ - liblz4-dev libzstd-dev libxxhash-dev libidn2-dev python3-cmarkgfm openssl - echo "/usr/local/bin" >>"$GITHUB_PATH" - - name: configure - run: ./configure --with-rrsync --enable-debug - - name: make - run: make check-progs # builds rsync + the test helper programs runtests.py needs - - name: info - run: ./rsync --version - - # Run the whole suite under valgrind. We gate on memory *errors* (uninit - # reads, invalid read/write, bad frees, uninit syscall params), not leaks: - # rsync deliberately leaves file-list/socket/option memory unfreed at exit - # (short-lived process; the OS reclaims), so --leak-check=no avoids a sea of - # by-design "definitely lost" reports. Functional pass/fail is covered by - # the other workflows, so the suite is allowed to finish regardless of - # per-test results; the scan step below is the gate. --error-exitcode=0 - # keeps valgrind from perturbing test exit codes; the bundled - # testsuite/valgrind.supp silences known-benign reports. - - name: run testsuite under valgrind - run: | - SUDO= - [ "${{ matrix.privilege }}" = root ] && SUDO="sudo -E" - TCP= - [ "${{ matrix.transport }}" = tcp ] && TCP="--use-tcp" - $SUDO ./runtests.py --valgrind \ - --valgrind-opts="--leak-check=no --error-exitcode=0" \ - $TCP -j8 --preserve-scratch || true - - - name: scan for unsuppressed valgrind errors - run: | - sudo chown -R "$USER" testtmp 2>/dev/null || true - mapfile -t logs < <(find testtmp -name 'valgrind.*.log' 2>/dev/null) - if [ "${#logs[@]}" -eq 0 ]; then - echo "::error::no valgrind logs were produced -- the suite did not run" - exit 1 - fi - echo "scanned ${#logs[@]} valgrind log(s)" - bad=() - for f in "${logs[@]}"; do - grep -qE 'ERROR SUMMARY: [1-9][0-9]* errors' "$f" && bad+=("$f") - done - if [ "${#bad[@]}" -ne 0 ]; then - echo "::error::valgrind reported unsuppressed errors in ${#bad[@]} run(s)" - for f in "${bad[@]}"; do - echo "===== $f =====" - sed 's/==[0-9]*== //' "$f" | grep -A18 \ - -E 'depends on uninitialised|points to uninitialised|Invalid (read|write|free)|lost in loss record|Mismatched free' \ - | head -60 - done - exit 1 - fi - echo "valgrind clean: no unsuppressed errors" + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install -y valgrind acl libacl1-dev attr libattr1-dev \ + liblz4-dev libzstd-dev libxxhash-dev libidn2-dev python3-cmarkgfm openssl + echo "/usr/local/bin" >>"$GITHUB_PATH" + - name: Configure + run: ./configure --with-rrsync --enable-debug + - name: Build + id: build + run: make check-progs + - name: Version + run: ./rsync --version - - name: upload valgrind logs on failure - if: failure() - uses: actions/upload-artifact@v4 - with: - name: valgrind-logs-${{ matrix.privilege }}-${{ matrix.transport }} - path: testtmp/**/valgrind.*.log - if-no-files-found: ignore - retention-days: 7 + # Gate memory errors rather than intentional process-exit leaks + - name: Tests + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S .github/scripts/diagnostics.py valgrind --privilege "${{ matrix.privilege }}" --transport "${{ matrix.transport }}" --profiles "${{ matrix.profiles }}" + - name: Check memory errors + if: always() + run: python3 -S .github/scripts/diagnostics.py valgrind-results --privilege "${{ matrix.privilege }}" --transport "${{ matrix.transport }}" + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: valgrind-logs-${{ matrix.privilege }}-${{ matrix.transport }} + path: | + test-results/valgrind-logs/*.log + test-results/*.json + if-no-files-found: error + retention-days: 45 diff --git a/.github/workflows/versions.yml b/.github/workflows/versions.yml new file mode 100644 index 000000000..ff779673d --- /dev/null +++ b/.github/workflows/versions.yml @@ -0,0 +1,59 @@ +name: Versions + +on: + workflow_call: + outputs: + jobs: + description: Job results + value: ${{ toJSON(jobs) }} + +permissions: + contents: read + +defaults: + run: + shell: sh + +jobs: + mix: + name: ${{ matrix.name }} + runs-on: ubuntu-24.04 + timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + include: + - name: Archived primary + role: primary + - name: Archived secondary + role: secondary + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev libidn2-dev python3-cmarkgfm openssl + echo "/usr/local/bin" >>"$GITHUB_PATH" + - name: Configure + run: ./configure --with-rrsync + - name: Build + id: build + run: make check-progs + - name: Version + run: ./rsync --version + - name: Validate profiles + run: python3 -S testsuite/runtests.py profile --rsync-bin="$PWD/rsync" + - name: Tests (non-root pipe and TCP) + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + run: python3 -S .github/scripts/versions.py --role "${{ matrix.role }}" + - name: Upload results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + retention-days: 45 + name: version-mix-${{ matrix.role }}-receipts + if-no-files-found: error + path: test-results/version-mix diff --git a/.gitignore b/.gitignore index 7a44021da..370e78d02 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ *.[oa] *~ +__pycache__/ dummy ID Makefile @@ -13,7 +14,6 @@ config.h.in.old config.log config.status aclocal.m4 -__pycache__ /proto.h /proto.h-tstamp /rsync*.[15] @@ -40,12 +40,14 @@ __pycache__ /testdir /tests-dont-exist /testtmp +/test-results/ /tls /testrun /trimslash /t_unsafe /wildtest /getfsdev +/dirfd-probe /simdtest /t_acl /t_chmod_secure @@ -61,11 +63,6 @@ __pycache__ /doc/rsync.pdf /doc/rsync.ps /support/savetransfer -/testsuite/chown-fake.test -/testsuite/devices-fake.test -/testsuite/xattrs-hlink.test -/testsuite/fleettest.json -/fleettest-logs /patches /patches.gen /build diff --git a/Makefile.in b/Makefile.in index 3a5d5098b..4531edb0b 100644 --- a/Makefile.in +++ b/Makefile.in @@ -62,8 +62,10 @@ CHECK_PROGS = rsync$(EXEEXT) tls$(EXEEXT) getgroups$(EXEEXT) getfsdev$(EXEEXT) \ testrun$(EXEEXT) trimslash$(EXEEXT) t_unsafe$(EXEEXT) t_chmod_secure$(EXEEXT) \ t_rename_secure$(EXEEXT) t_symlink_secure$(EXEEXT) t_secure_relpath$(EXEEXT) t_acl$(EXEEXT) t_hashtable_overflow$(EXEEXT) t_iwildmatch$(EXEEXT) t_clean_fname$(EXEEXT) t_safe_arg$(EXEEXT) wildtest$(EXEEXT) simdtest$(EXEEXT) -CHECK_SYMLINKS = testsuite/chown-fake_test.py testsuite/devices-fake_test.py \ - testsuite/xattrs-hlink_test.py testsuite/exclude-lsh_test.py +CHECK_SYMLINKS = testsuite/tests/metadata/chown-fake.py \ + testsuite/tests/metadata/devices-fake.py \ + testsuite/tests/metadata/xattrs-hard-links.py \ + testsuite/tests/transfer/exclude-remote-shell.py # Objects for CHECK_PROGS to clean CHECK_OBJS=tls.o testrun.o getgroups.o getfsdev.o t_stub.o t_unsafe.o t_chmod_secure.o t_rename_secure.o t_symlink_secure.o t_secure_relpath.o t_acl.o t_hashtable_overflow.o t_iwildmatch.o t_clean_fname.o t_safe_arg.o trimslash.o wildtest.o @@ -71,8 +73,7 @@ CHECK_OBJS=tls.o testrun.o getgroups.o getfsdev.o t_stub.o t_unsafe.o t_chmod_se CHECK_COMPILE_OBJS=syscall-no-at-fdcwd.o syscall-no-o-directory.o \ authenticate-no-o-cloexec.o rsync-no-o-cloexec.o clientserver-no-o-directory.o \ flist-no-o-directory.o generator-no-o-directory.o sender-no-o-directory.o \ - lib/acl-no-o-directory.o -CHECK_COMPILE_OBJS=syscall-no-at-fdcwd.o access-no-inet6.o + lib/acl-no-o-directory.o access-no-inet6.o # note that the -I. is needed to handle config.h when using VPATH .c.o: @@ -292,6 +293,9 @@ T_SECURE_RELPATH_OBJ = t_secure_relpath.o syscall.o util1.o util2.o t_stub.o lib t_secure_relpath$(EXEEXT): $(T_SECURE_RELPATH_OBJ) $(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(T_SECURE_RELPATH_OBJ) $(LIBS) +dirfd-probe$(EXEEXT): $(srcdir)/testsuite/tools/dirfd-probe.c config.h + $(CC) -I. -I$(srcdir) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) -o $@ $(srcdir)/testsuite/tools/dirfd-probe.c + # Unit test for lib/acl.c: compares our fd/at ACL ops against the system libacl # (linked via $(LIBS), which carries -lacl). lib/acl.o references no rsync # globals, so this links with no stubs. Self-skips (exit 77) when built @@ -390,7 +394,7 @@ rrsync.1: support/rrsync.1.md md-convert Makefile .PHONY: clean clean: cleantests rm -f *~ $(OBJS) $(CHECK_PROGS) $(CHECK_OBJS) $(CHECK_COMPILE_OBJS) $(CHECK_SYMLINKS) @MAKE_RRSYNC@ \ - git-version.h rounding rounding.h *.old rsync*.1 rsync*.5 @MAKE_RRSYNC_1@ \ + dirfd-probe$(EXEEXT) git-version.h rounding rounding.h *.old rsync*.1 rsync*.5 @MAKE_RRSYNC_1@ \ *.html daemon-parm.h help-*.h default-*.h proto.h proto.h-tstamp rm -f *.gcno *.gcda lib/*.gcno lib/*.gcda zlib/*.gcno zlib/*.gcda popt/*.gcno popt/*.gcda rm -rf coverage coverage-tcp coverage-all coverage-fallback @@ -430,95 +434,55 @@ test: check # This depends on building rsync; if we need any helper programs it # should depend on them too. -# We try to run the scripts with POSIX mode on, in the hope that will -# catch Bash-isms earlier even if we're running on GNU. Of course, we -# might lose in the future where POSIX diverges from old sh. - -# `make check` runs tests in parallel by default. Override with -# `make check CHECK_J=1` (serial) or any other value. +# Test parallelism; CHECK_J=1 runs serially. CHECK_J = 8 -# Parallelism for `make coverage`. Defaults to the same as CHECK_J: the -# coverage build sets -fprofile-update=atomic (atomic in-memory counters) and -# gcc's libgcov serializes the per-source .gcda read-modify-write merge with a -# file lock, so concurrent rsync processes (incl. the forked sender/generator/ -# receiver) accumulate exactly -- verified by a count-linearity check (a hot -# line accumulates identically at -j1 and -P16). Override with -# `make coverage COVERAGE_J=1` if your libgcov does not lock .gcda merges. +# Coverage uses atomic counters and libgcov file locks. Set COVERAGE_J=1 +# with a toolchain which cannot merge counters from concurrent processes. COVERAGE_J = $(CHECK_J) -# Output directory and extra runtests.py flags for `make coverage`. The -# `coverage-tcp` target reuses the coverage recipe with --use-tcp (real -# loopback rsyncd, which exercises the TCP accept/auth path and the -# require_tcp-only tests) and a separate output directory. +# Report directory and extra runner options for coverage targets. COVERAGE_DIR = coverage COVERAGE_RUNFLAGS = -# Excluded from the coverage report so the percentages reflect rsync's own -# runtime source. Three buckets: -# (1) Bundled third-party code rsync ships but does not own: zlib/, popt/, and -# the named lib/ imports (PostgreSQL getaddrinfo, ISC inet_ntop/inet_pton, -# standalone getpass). The other lib/*.c are rsync's own and stay in. -# (2) Test-helper / build-time programs that link against rsync objects but are -# not the rsync runtime: t_*.c, tls.c, wildtest.c, testrun.c, getgroups.c, -# getfsdev.c, trimslash.c, rounding.c. These have their own main() and are -# either driven directly by a test (counted there) or are configure-time -# probes; counting them as "rsync uncovered" is noise. -# (3) Compile-time-dead fallbacks under this build's config.h: lib/md5.c (the -# reference md5 -- openssl's EVP path is used when HAVE_OPENSSL) and -# lib/snprintf.c (only the #include line survives under -# HAVE_C99_VSNPRINTF). Covering these would mean a separate non-openssl / -# non-C99 build, which is out of scope for this report. +# Exclude bundled dependencies, standalone helpers and fallbacks unused by +# the OpenSSL and C99 coverage build. COVERAGE_EXCLUDE = -e '(^|/)zlib/' -e '(^|/)popt/' \ -e '(^|/)lib/(getaddrinfo|getpass|inet_ntop|inet_pton)\.' \ -e '(^|/)(t_[a-z_]+|tls|wildtest|testrun|getgroups|getfsdev|trimslash|rounding)\.c$$' \ -e '(^|/)lib/(md5|snprintf)\.c$$' -# Build everything the test suite needs (rsync + helper programs + symlinks) -# WITHOUT running it. Used by CI jobs that invoke runtests.py directly with -# custom options (e.g. the version-mix workflow's --rsync-bin2/--expect-result). +# Build test prerequisites without running the suite. .PHONY: check-progs check-progs: all $(CHECK_PROGS) $(CHECK_COMPILE_OBJS) $(CHECK_SYMLINKS) .PHONY: check check: all $(CHECK_PROGS) $(CHECK_COMPILE_OBJS) $(CHECK_SYMLINKS) - "$(srcdir)/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(CHECK_J) + "$(srcdir)/testsuite/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(CHECK_J) .PHONY: check29 check29: all $(CHECK_PROGS) $(CHECK_COMPILE_OBJS) $(CHECK_SYMLINKS) - "$(srcdir)/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(CHECK_J) --protocol=29 + "$(srcdir)/testsuite/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(CHECK_J) --protocol=29 .PHONY: check30 check30: all $(CHECK_PROGS) $(CHECK_COMPILE_OBJS) $(CHECK_SYMLINKS) - "$(srcdir)/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(CHECK_J) --protocol=30 - -# Whole-suite gcov coverage report (HTML, with branch + decision coverage). -# Requires a build configured with --enable-coverage and the `gcovr` tool -# (pip install gcovr). Runs the suite in parallel (COVERAGE_J, default CHECK_J): -# this is safe because the coverage build uses -fprofile-update=atomic and -# libgcov locks the per-source .gcda during its merge, so concurrent rsync -# processes accumulate exactly (see COVERAGE_J above). Use COVERAGE_J=1 if your -# toolchain's libgcov does not lock .gcda merges. + "$(srcdir)/testsuite/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(CHECK_J) --protocol=30 + +# HTML branch and decision coverage; requires --enable-coverage and gcovr. .PHONY: coverage coverage: all $(CHECK_PROGS) $(CHECK_SYMLINKS) @case '$(CFLAGS)' in *--coverage*) ;; \ *) echo "*** not a coverage build; reconfigure with --enable-coverage"; exit 1 ;; esac @command -v gcovr >/dev/null 2>&1 || { echo "*** gcovr not found (pip install gcovr)"; exit 1; } find . -name '*.gcda' -delete - @# Daemon modules with `uid = ` setuid the per-connection child - @# (and so the forked generator/receiver), which then cannot create or - @# merge .gcda files in a root-owned build dir -- silently dropping ALL - @# coverage from those processes. Make every .gcno's directory - @# world-writable so any uid can create the sibling .gcda, and set a - @# default ACL of o::rw so the .gcda are world-mergeable regardless of - @# the creator's umask (every test process resets umask to 022 via - @# rsyncfns.py, so a Makefile-level `umask 0` would not survive). + @# Setuid daemon workers need to create and merge counters despite the + @# harness umask. Only run coverage in a disposable build directory. @find . -name '*.gcno' -printf '%h\n' 2>/dev/null | sort -u | \ while read d; do \ chmod a+rwx "$$d"; \ setfacl -m 'd:u::rwx,d:g::rwx,d:o::rwx' "$$d" 2>/dev/null || true; \ done - @rc=0; "$(srcdir)/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(COVERAGE_J) $(COVERAGE_RUNFLAGS) || rc=$$?; \ + @rc=0; "$(srcdir)/testsuite/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(COVERAGE_J) $(COVERAGE_RUNFLAGS) || rc=$$?; \ rm -rf $(COVERAGE_DIR) && mkdir -p $(COVERAGE_DIR); \ gcovr --root $(srcdir) $(COVERAGE_EXCLUDE) --decisions --print-summary \ --gcov-ignore-parse-errors=negative_hits.warn_once_per_file \ @@ -529,36 +493,28 @@ coverage: all $(CHECK_PROGS) $(CHECK_SYMLINKS) fi; \ exit $$rc -# Same as `make coverage` but with the daemon tests run over a real loopback -# rsyncd (--use-tcp), into a separate report directory. +# TCP daemon coverage in a separate report directory. .PHONY: coverage-tcp coverage-tcp: $(MAKE) coverage COVERAGE_RUNFLAGS=--use-tcp COVERAGE_DIR=coverage-tcp -# Comprehensive single report: run the suite under several configurations, -# accumulating into the shared .gcda counters (NOT cleared between runs), then -# emit one merged, rsync-scoped report. Covers the default (pipe) transport, the -# protocol-29/30 compat branches, and the real-TCP daemon path (which also runs -# the require_tcp-only tests). Run under sudo to additionally cover root-only -# paths (devices, chown, use-chroot, protected-regular). Local target -- CI uses -# the plain `coverage`/`coverage-tcp` targets. +# Combine default-protocol, protocol-29, protocol-30 and TCP coverage. .PHONY: coverage-all coverage-all: all $(CHECK_PROGS) $(CHECK_SYMLINKS) @case '$(CFLAGS)' in *--coverage*) ;; \ *) echo "*** not a coverage build; reconfigure with --enable-coverage"; exit 1 ;; esac @command -v gcovr >/dev/null 2>&1 || { echo "*** gcovr not found (pip install gcovr)"; exit 1; } find . -name '*.gcda' -delete - @# See the `coverage` target above for why: setuid'd daemon children must - @# be able to create/merge .gcda owned by a different uid. + @# Allow setuid workers to write counters as in the coverage target. @find . -name '*.gcno' -printf '%h\n' 2>/dev/null | sort -u | \ while read d; do \ chmod a+rwx "$$d"; \ setfacl -m 'd:u::rwx,d:g::rwx,d:o::rwx' "$$d" 2>/dev/null || true; \ done @rc=0; \ - for cfg in '' '--protocol=30' '--protocol=29' '--use-tcp'; do \ + for cfg in '' '--protocol=30' '--protocol=29' '--use-tcp --daemon-tests-only'; do \ echo "===== coverage-all: runtests.py $$cfg ====="; \ - "$(srcdir)/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(COVERAGE_J) $$cfg || rc=$$?; \ + "$(srcdir)/testsuite/runtests.py" --rsync-bin="`pwd`/rsync$(EXEEXT)" -j $(COVERAGE_J) $$cfg || rc=$$?; \ done; \ rm -rf coverage-all && mkdir -p coverage-all; \ gcovr --root $(srcdir) $(COVERAGE_EXCLUDE) --decisions --print-summary \ @@ -570,9 +526,8 @@ coverage-all: all $(CHECK_PROGS) $(CHECK_SYMLINKS) fi; \ exit $$rc -# Coverage for the portable (non-openat2) resolver tier. Requires a SEPARATE -# build configured with --enable-coverage --disable-openat2: its .gcno differ -# from the openat2 build, so this report cannot be merged with the others. +# Use a separate --enable-coverage --disable-openat2 build; its counters +# cannot be merged with the openat2 build. .PHONY: coverage-fallback coverage-fallback: $(MAKE) coverage COVERAGE_DIR=coverage-fallback @@ -589,17 +544,17 @@ simdtest$(EXEEXT): simd-checksum-x86_64.cpp $(HEADERS) touch $@; \ fi -testsuite/chown-fake_test.py: - ln -s chown_test.py $(srcdir)/testsuite/chown-fake_test.py +testsuite/tests/metadata/chown-fake.py: + ln -s chown.py $(srcdir)/testsuite/tests/metadata/chown-fake.py -testsuite/devices-fake_test.py: - ln -s devices_test.py $(srcdir)/testsuite/devices-fake_test.py +testsuite/tests/metadata/devices-fake.py: + ln -s devices.py $(srcdir)/testsuite/tests/metadata/devices-fake.py -testsuite/xattrs-hlink_test.py: - ln -s xattrs_test.py $(srcdir)/testsuite/xattrs-hlink_test.py +testsuite/tests/metadata/xattrs-hard-links.py: + ln -s xattrs.py $(srcdir)/testsuite/tests/metadata/xattrs-hard-links.py -testsuite/exclude-lsh_test.py: - ln -s exclude_test.py $(srcdir)/testsuite/exclude-lsh_test.py +testsuite/tests/transfer/exclude-remote-shell.py: + ln -s exclude.py $(srcdir)/testsuite/tests/transfer/exclude-remote-shell.py # This does *not* depend on building or installing: you can use it to # check a version installed from a binary or some other source tree, @@ -607,7 +562,7 @@ testsuite/exclude-lsh_test.py: .PHONY: installcheck installcheck: $(CHECK_PROGS) $(CHECK_SYMLINKS) - "$(srcdir)/runtests.py" --rsync-bin="$(bindir)/rsync$(EXEEXT)" --srcdir="$(srcdir)" --tooldir="`pwd`" -j $(CHECK_J) + "$(srcdir)/testsuite/runtests.py" --rsync-bin="$(bindir)/rsync$(EXEEXT)" --srcdir="$(srcdir)" --tooldir="`pwd`" -j $(CHECK_J) # TODO: Add 'dist' target; need to know which files will be included diff --git a/NEWS.md b/NEWS.md index ccb51b80d..ab8f86750 100644 --- a/NEWS.md +++ b/NEWS.md @@ -17,6 +17,17 @@ - Changed to `/` immediately after a module chroot, before selecting an inner module directory. A failed working-directory reset aborts the connection. +### BUILD AND TESTS: +- Reworked test discovery, platform profiles and JSON test results. The runner + now lives under `testsuite/` and uses Python 3.7 or later with no third-party + Python packages. +- Expanded native platform tests and historical-peer checks in both executable + roles. Added Android Bionic and Termux application tests and cross-platform + transfers over SSH and daemon connections. +- Added installed-binary TLS checks, release-archive qualification and Linux + storage tests for encrypted volumes, mount boundaries, space exhaustion, + device transfers and `--copy-as`. + # NEWS for rsync 3.5.2 (4 Nov 2026) ## Changes in this version: diff --git a/README.md b/README.md index 6abb33679..9076263e3 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,11 @@ features in your build. [1]: https://github.com/RsyncProject/rsync/blob/master/INSTALL.md +TESTING +------- + +Run `make check` from a configured build directory. See the [testsuite README](testsuite/README.md) for test selection, profiles and CI. + SETUP ----- diff --git a/SECURITY.md b/SECURITY.md index 842019cad..d8d40e840 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -392,11 +392,9 @@ The following are documented as out of scope for this release: whose parent component is *raced* from a directory to a symlink mid-resolution can still slip past confinement that is anchored at the process CWD (e.g. the sender's content open), because the descriptor is not bound to the original - inode. The parent-component symlink-race tests are therefore not enforced on - Cygwin (see `RSYNC_EXPECT_SKIPPED` in `.github/workflows/cygwin-build.yml` and - the Cygwin-only xfail in `symlink-race-source_test.py`). Cygwin is a - development/interoperability target, not a privilege boundary host, so this is - accepted for this release. + inode. We therefore report `dirfd_pinning` as unsupported on Cygwin and + treat Cygwin as an interoperability target rather than a privilege-boundary + platform. * On a platform with no `mknodat()` at all -- macOS before 13 is the supported example, where `mknod()` and `mkfifo()` exist but neither diff --git a/clientserver.c b/clientserver.c index eafd231fd..0304de820 100644 --- a/clientserver.c +++ b/clientserver.c @@ -1081,12 +1081,8 @@ static int rsync_module(int f_in, int f_out, int i, const char *addr, const char if (use_chroot) { /* Cache timezone data before chroot makes /etc/localtime inaccessible */ tzset(); - /* Flush gcov counters now: after chroot the build-tree .gcda - * paths are unreachable, so everything this child has executed - * so far (the whole rsync_module() pre-chroot path) would - * otherwise be lost. Post-chroot coverage from this child is - * still unrecordable -- accepted, documented in - * testsuite/COVERAGE.md. */ + /* Save gcov counters before chroot makes the build tree unreachable. + * Work after chroot cannot be recorded in those files. */ gcov_flush(); if (chroot(module_chdir)) { rsyserr(FLOG, errno, "chroot(\"%s\") failed", module_chdir); diff --git a/old_versions/README.md b/old_versions/README.md index a5ce3a5b6..bae9e4a15 100644 --- a/old_versions/README.md +++ b/old_versions/README.md @@ -1,87 +1,12 @@ -# Old rsync version archive +The rsync_* files are static Linux X86-64 binaries for mixed-version tests. We run them only in isolated test environments because they contain historical defects. -Static rsync binaries built from historical release tags. Two uses: +binaries.json records each binary's SHA-256, release commit, protocol and build inputs. Peer profiles under testsuite/profiles identify the releases; test metadata determines which checks apply. The runner checks binary hashes, version banners and release tags before using an archived peer. -1. **Cross-version behaviour checks** — confirming whether a behaviour a user - reported on an old release is version-specific or option-driven. -2. **The version-mixing test suite** — `runtests.py --rsync-bin2=...` runs the - current code against one of these as the daemon / remote-shell peer; CI - (`.github/workflows/ubuntu-version-mix.yml`) does this for every binary - here against the per-version manifests in `testsuite/expect/`. - -Binaries are **statically linked** so they run regardless of the host's -shared libraries, and named `rsync_`: - -| Binary | Version | Protocol | Notes | -|----------------|---------|----------|-----------------------------------------| -| `rsync_2.6.0` | 2.6.0 | 27 | 2004; needs autoconf regen (see below) | -| `rsync_3.0.0` | 3.0.0 | 30 | 2008 | -| `rsync_3.1.0` | 3.1.0 | 31 | 2013 | -| `rsync_3.1.3` | 3.1.3 | 31 | Ubuntu 18.04 / Debian buster era (2018) | -| `rsync_3.2.0` | 3.2.0 | 31 | 2020 (zstd/lz4/xxhash negotiation added)| -| `rsync_3.2.7` | 3.2.7 | 31 | 2022 | -| `rsync_3.3.0` | 3.3.0 | 31 | 2024 | -| `rsync_3.4.0` | 3.4.0 | 32 | 2025 | -| `rsync_3.4.1` | 3.4.1 | 32 | 2025 | - -These are every `x.y.0` release from 2.6.0 (2004) onward plus a few point -releases. 2.6.0 is the practical floor: older tags need progressively more -porting to build on a current toolchain. - -All built `--disable-openssl` and with `_FORTIFY_SOURCE` disabled (see below); -xxhash/zstd/lz4 are compiled in where the version supports them. - -## Adding a version - -```bash -./build_static.sh 3.2.7 # uses git tag v3.2.7 -./build_static.sh 3.0.9 v3.0.9 # explicit tag if naming differs +To rebuild a release: +```sh +old_versions/build_static.sh 3.2.7 ``` -The script checks out the tag into a throwaway `git worktree`, applies the -minimal patches needed to compile old sources on a modern toolchain, links -statically, verifies the result is static and reports the requested version, -then installs `rsync_` here and removes the worktree. - -Override the source repo with `RSYNC_REPO=/path/to/rsync ./build_static.sh ...` -(defaults to `../rsync.4`). - -## Why the patches? - -Modern GCC (>= 14, C23 default) and glibc reject things old rsync relied on. -`build_static.sh` handles these, each guarded so it's a no-op when not needed: - -1. **K&R `lseek64()` redeclaration** in `syscall.c` clashes with glibc's real - prototype — removed. -2. **`gettimeofday()`** — glibc only has the 2-arg form; configure misdetects - the 1-arg form, so `HAVE_GETTIMEOFDAY_TZ` is forced on in `config.h`. -3. **C23 `()` == `(void)`** breaks K&R prototypes called with arguments - (`qsort` comparator, `pool->bomb`, etc.) — built with `-std=gnu11`. -4. Assorted modern `-Werror` promotions (incompatible pointer types, implicit - declarations) downgraded to warnings; bundled zlib/popt used to keep the - static link self-contained. - -5. **OpenSSL (3.2+)** is disabled with `--disable-openssl`: linking - `libcrypto.a` statically drags in jitterentropy (`jent_*`) and zlib's - `uncompress` (OpenSSL's COMP module), which don't resolve here. OpenSSL only - provided optional MD4/MD5, which rsync implements natively, so checksum - behaviour is unaffected. - -6. **`_FORTIFY_SOURCE` disabled** (`-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0`): - modern Ubuntu defaults it to `=3`, whose stricter object-size checks turn - latent (historically benign) over-reads in OLD rsync into hard - `*** buffer overflow detected ***` aborts when the binary runs as a - server/daemon — which made e.g. 3.1.3 and 3.2.7 unusable as peers. Disabling - it makes the archival binaries behave as the released versions did. - -7. **Pre-3.0 tags (e.g. 2.6.0)** ship `configure.in`, not a generated - `configure`. The script runs `autoheader`/`autoconf` to generate it, after - neutralizing the `AC_CHECK_FUNCS(fn,,AC_LIBOBJ(lib/...))` fallbacks for - `inet_ntop`/`inet_pton`/`getaddrinfo`/`getnameinfo` — modern autoconf emits - broken shell for those never-taken branches (the funcs exist in glibc). It - also generates `proto.h` (no make rule in that era) and stubs the vendored - `lib/addrinfo.h` the tag dropped (modern glibc supplies `struct addrinfo`). - All guarded so they no-op on 3.x. +The script builds from the release tag and checks architecture, static linkage, local copying and a stdio-daemon transfer before replacing the binary. RSYNC_REPO selects another source checkout and RSYNC_OUTPUT selects a separate output file. -Newer versions may need fewer or different tweaks; if a build fails, the -script prints the first compiler errors from its log. +Build adjustments accommodate older Autoconf inputs and libc declarations. OpenSSL, IDN and _FORTIFY_SOURCE are disabled. When replacing a binary, update its manifest entry and run the tests in both executable roles over pipe and TCP. diff --git a/old_versions/binaries.json b/old_versions/binaries.json new file mode 100644 index 000000000..19b05271e --- /dev/null +++ b/old_versions/binaries.json @@ -0,0 +1,137 @@ +{ + "rsync_2.6.0": { + "source": "c1456d83a72215822732424304b1563a0a2c9d8f", + "protocol": 27, + "sha256": "2c5e454c4e32146a39dd6a4942752805a5862367938bac6310c089104c0c71a8", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.0.0": { + "source": "4cb6197b21d9860736c8e6d81b8eb8cac53beb5a", + "protocol": 30, + "sha256": "6ee0734b8cfe095ae45c32fea4b9b6f52e04983c43ce7f7f82feb984fa923839", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.1.0": { + "source": "bc58313bf7f42f0993f7279299e8137e0f14f01c", + "protocol": 31, + "sha256": "7f90bee892bc9b6b1a5a026ec9356284a33225709c8303bfe8e1b2afa4a66383", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.1.3": { + "source": "d73762eea3f15f2c56bb3fa9394ad1883c25c949", + "protocol": 31, + "sha256": "d305a31a93cd37a73d516c74a4656fa3d369aa28dececaf03f74c123c69a5fb4", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.2.0": { + "source": "c225330aaf9d6032a79317bda1bf669e90ac7212", + "protocol": 31, + "sha256": "3d32b82acc8e81dcf2432a7800779df1abd3e0beb266f7fb51d68a2127bb772f", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.2.7": { + "source": "6b5ae825db985e9d1c98576651b50c8e490ddb97", + "protocol": 31, + "sha256": "6273c0cd9573b2713ea449d13942e7b94db7f03e62f84c39fadc35fe97c4fd65", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.3.0": { + "source": "6c8ca91c731b7bf2b081694bda85b7dadc2b7aff", + "protocol": 31, + "sha256": "1b6bb785cc887a3fddaabedb5f6568f63bc331d3ff3d23a1c62040acddd0ec89", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.4.0": { + "source": "e3ee0e7319eb4250ddb76b716510fd20441da329", + "protocol": 32, + "sha256": "123965453fbcefdadb795b0f6a6a9f02fed4140ed8d2e847db230f2450dd9093", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.4.1": { + "source": "3305a7a063ab0167cab5bf7029da53abaa9fdb6e", + "protocol": 32, + "sha256": "4e6caaa566dcc15a9b902650520213fdaee6a45e9e1af5ad01a0640be2780ef8", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.4.2": { + "source": "de3cc03b03be153092f266aa1117b4706947ed07", + "protocol": 32, + "sha256": "19028fc541c2afab51f65b3f6fe8acdce5550e46de13cbb411d0ab423dd40585", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.4.3": { + "source": "2c7777aaa62abeb1bca192da345e6e8c685e872a", + "protocol": 32, + "sha256": "7c30cc997b14c3c2b3e12060ded7664adcf45f6d42ef2f80ccd49ea425ea0408", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.4.4": { + "source": "f26f747b8017b321d3776becc69f330dd889fa21", + "protocol": 32, + "sha256": "e4819607178c6d10ef57880901ee9eec2d605bc5af9fb23eaf1442e20fcd91c7", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.5.0": { + "source": "471e17dc0d68233db84db11be82c9f62f4661214", + "protocol": 32, + "sha256": "c4e6e97b71e2759906693f2b3938a056ee2a2b8ab1ac9137baf42923d89da92c", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.5.1": { + "source": "04355d27b7386d7de0e6bd5e79c556223210f700", + "protocol": 33, + "sha256": "1f2d75dff7303439b2967c3b539631f838d21db47efd75cfc6d4f685c5243a13", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + }, + "rsync_3.5.2": { + "source": "41bbd895668767e155b2eb6da64f37afa9cb8724", + "protocol": 33, + "sha256": "84b431e7e94eb67b03f20d95bad66257681c2ebe78a2f79893db2a3cb9b6bf27", + "recipe": "c26b0d536613ad8a3a185e4d9274fc6b27374c54fe47396f0fe500c7ed69f873", + "compiler": "gcc (Debian 14.2.0-19) 14.2.0", + "target": "x86_64-linux-gnu", + "libc": "ldd (Debian GLIBC 2.41-12+deb13u4) 2.41" + } +} diff --git a/old_versions/build_static.sh b/old_versions/build_static.sh index 7cabcc0cf..a3f1af9ef 100755 --- a/old_versions/build_static.sh +++ b/old_versions/build_static.sh @@ -1,66 +1,54 @@ #!/bin/bash -# Build a static rsync binary from a historical git tag, for cross-version -# behaviour testing. Produces ./rsync_ in this directory. -# -# Usage: ./build_static.sh [git-tag] -# Example: ./build_static.sh 3.1.3 # uses tag v3.1.3 -# ./build_static.sh 3.2.7 v3.2.7 -# -# Old rsync releases don't compile cleanly on a modern toolchain (GCC >= 14 -# defaults to C23, where an empty () prototype means (void); glibc dropped the -# 1-arg gettimeofday; lseek64 K&R redeclarations clash). This script applies -# the minimal, best-effort workarounds and links statically so the result is -# self-contained and reproducible regardless of the host's shared libraries. -# -# Each workaround is guarded so it's a no-op on versions that don't need it. +# Build a tagged release for the Linux AMD64 peer suite +# Usage: build_static.sh VERSION [TAG] +# RSYNC_OUTPUT selects a separate output file for qualification set -euo pipefail VERSION="${1:?usage: build_static.sh [git-tag]}" TAG="${2:-v$VERSION}" +[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+([a-z]+[0-9]+)?$ ]] || { echo "Invalid version: $VERSION" >&2; exit 2; } +if [ "$(uname -sm)" != 'Linux x86_64' ]; then + echo 'The archived peer build requires Linux AMD64' >&2 + exit 2 +fi ARCHIVE_DIR="$(cd "$(dirname "$0")" && pwd)" -REPO="${RSYNC_REPO:-/home/tridge/project/rsync/rsync.4}" # any rsync worktree -WORKTREE="$(mktemp -d /tmp/rsync-build-XXXXXX)" -OUT="$ARCHIVE_DIR/rsync_$VERSION" +REPO="${RSYNC_REPO:-$(dirname "$ARCHIVE_DIR")}" +SOURCE=$(git -c "safe.directory=$REPO" -C "$REPO" rev-parse --verify "refs/tags/$TAG^{commit}") +OUT="${RSYNC_OUTPUT:-$ARCHIVE_DIR/rsync_$VERSION}" +if [[ -L "$OUT" || ( -e "$OUT" && ! -f "$OUT" ) ]]; then + echo "Output must be a regular file: $OUT" >&2 + exit 2 +fi +pending= +BUILD_DIR="$(mktemp -d /tmp/rsync-build-XXXXXX)" -# C standard restores K&R () semantics; permissive flags downgrade the pile of -# modern -Werror promotions (incompatible pointers, implicit decls) to warnings. -# _FORTIFY_SOURCE is forced OFF: modern Ubuntu defaults it to =3, whose stricter -# object-size checks turn latent (historically benign) over-reads in OLD rsync -# into hard "*** buffer overflow detected ***" aborts when the binary acts as a -# server/daemon. Disabling it makes these archival binaries behave the way the -# released versions did, which is the whole point of the archive. +# Preserve old declarations and unchecked release-era behaviour CFLAGS_OLD="-I. -I./zlib -O2 -g -std=gnu11 -fcommon -DHAVE_CONFIG_H -Wno-error \ -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 \ -Wno-incompatible-pointer-types -Wno-implicit-function-declaration -Wno-int-conversion" cleanup() { - cd "$REPO" - git worktree remove --force "$WORKTREE" 2>/dev/null || true - git worktree prune 2>/dev/null || true + if [ -n "$pending" ]; then rm -f -- "$pending"; fi + case "$BUILD_DIR" in + /tmp/rsync-build-*) + [ "$(realpath "$BUILD_DIR")" = "$BUILD_DIR" ] && rm -rf -- "$BUILD_DIR" ;; + *) echo "Unexpected build directory: $BUILD_DIR" >&2 ;; + esac } trap cleanup EXIT -echo ">>> checking out $TAG into $WORKTREE" -# prefer an exact tag to avoid ambiguity with similarly-named branches -REF="$TAG" -if git -C "$REPO" rev-parse -q --verify "refs/tags/$TAG" >/dev/null; then - REF="refs/tags/$TAG" -fi -git -C "$REPO" worktree add --detach "$WORKTREE" "$REF" -cd "$WORKTREE" +echo "Building $TAG at $SOURCE" +git -c "safe.directory=$REPO" -C "$REPO" archive "$SOURCE" | tar -x -C "$BUILD_DIR" +cd "$BUILD_DIR" -# --- workaround 1: K&R lseek64 redeclaration clashes with glibc's prototype --- +# Modern headers already declare lseek64 if grep -q 'off64_t lseek64();' syscall.c 2>/dev/null; then echo ">>> patching syscall.c lseek64 redeclaration" perl -0pi -e 's/#ifdef HAVE_LSEEK64\n#if !SIZEOF_OFF64_T\n\tOFF_T lseek64\(\);\n#else\n\toff64_t lseek64\(\);\n#endif\n\treturn lseek64/#ifdef HAVE_LSEEK64\n\treturn lseek64/' syscall.c fi -# --- workaround 0: pre-3.0 tags ship configure.in, not a generated configure. -# Generate it. Modern autoconf emits broken shell for their -# AC_CHECK_FUNCS(fn,,AC_LIBOBJ(lib/...)) fallbacks -- but those branches are -# dead on a modern host (glibc has inet_ntop/inet_pton/getaddrinfo/getnameinfo), -# so neutralize the AC_LIBOBJ replacements before regenerating. +# Regenerate the old Autoconf inputs without obsolete libc replacements OLD_TREE=0 if [ ! -f ./configure ] && { [ -f configure.in ] || [ -f configure.ac ]; }; then OLD_TREE=1 @@ -72,27 +60,27 @@ if [ ! -f ./configure ] && { [ -f configure.in ] || [ -f configure.ac ]; }; then fi CONF_ARGS=(--disable-md2man --with-included-zlib=yes --with-included-popt=yes) -# OpenSSL (3.2+) only adds optional MD4/MD5 that rsync already implements, but -# linking libcrypto.a statically drags in jitterentropy + zlib's uncompress, -# which aren't resolvable here. Drop it when the flag exists. -if ./configure --help 2>/dev/null | grep -q -- '--disable-openssl'; then - echo ">>> disabling openssl for self-contained static link" +# Use the bundled hashes for the static peer build +help=$(./configure --help) +if [[ "$help" == *--disable-openssl* ]]; then + echo "Using bundled hashes" CONF_ARGS+=(--disable-openssl) fi +if [[ "$help" == *--disable-idn* ]]; then + CONF_ARGS+=(--disable-idn) +fi echo ">>> configure (bundled zlib + popt, static-friendly)" ./configure "${CONF_ARGS[@]}" \ - >"$WORKTREE/conf.log" 2>&1 || { tail -20 "$WORKTREE/conf.log"; exit 1; } + >"$BUILD_DIR/conf.log" 2>&1 || { tail -20 "$BUILD_DIR/conf.log"; exit 1; } -# --- workaround 2: modern glibc only has the 2-arg gettimeofday --------------- +# Modern glibc requires the second gettimeofday argument if grep -q '/\* #undef HAVE_GETTIMEOFDAY_TZ \*/' config.h; then echo ">>> forcing HAVE_GETTIMEOFDAY_TZ (configure misdetects it)" sed -i 's|/\* #undef HAVE_GETTIMEOFDAY_TZ \*/|#define HAVE_GETTIMEOFDAY_TZ 1|' config.h fi -# --- workaround 4 (old trees only): generate proto.h if the tree has no make -# rule for it, and stub a vendored lib/addrinfo.h that the git tag dropped -# (modern glibc supplies struct addrinfo / sockaddr_storage, so empty is right). +# Old tags need generated prototypes and current libc address declarations if [ "$OLD_TREE" = 1 ]; then if [ ! -f proto.h ] && [ -f mkproto.awk ]; then echo ">>> generating proto.h" @@ -105,24 +93,36 @@ if [ "$OLD_TREE" = 1 ]; then fi echo ">>> building (static)" -make -j"$(nproc)" CFLAGS="$CFLAGS_OLD" LDFLAGS="-static" \ - >"$WORKTREE/make.log" 2>&1 || { grep -E 'error:|\*\*\*' "$WORKTREE/make.log" | head; exit 1; } +make -j8 CFLAGS="$CFLAGS_OLD" LDFLAGS="-static" \ + >"$BUILD_DIR/make.log" 2>&1 || { grep -E 'error:|\*\*\*' "$BUILD_DIR/make.log" | head; exit 1; } -# verify it's actually static before we keep it -if ldd ./rsync 2>&1 | grep -qv 'not a dynamic executable'; then - echo "ERROR: binary is not statically linked:" >&2 - ldd ./rsync >&2 +elf=$(readelf -h -l -d ./rsync) +[[ "$elf" == *'Advanced Micro Devices X86-64'* ]] +if [[ "$elf" == *INTERP* || "$elf" == *NEEDED* ]]; then + echo "Binary is not statically linked" >&2 exit 1 fi - -GOT="$(./rsync --version | head -1 | awk '{print $3}')" +banner=$(timeout -k 2 10 ./rsync --version) +GOT=$(printf '%s\n' "$banner" | awk 'NR == 1 {print $3}') if [ "$GOT" != "$VERSION" ]; then - echo "ERROR: built version '$GOT' != requested '$VERSION'" >&2 + echo "Built version '$GOT' differs from '$VERSION'" >&2 exit 1 fi -cp ./rsync "$OUT" -strip "$OUT" -echo ">>> installed $OUT" -"$OUT" --version | head -1 -file "$OUT" +mkdir smoke-source smoke-local smoke-daemon +printf 'peer transfer\n' > smoke-source/file +timeout -k 2 30 ./rsync -r smoke-source/ smoke-local/ +cmp smoke-source/file smoke-local/file +printf 'use chroot = no\nuid = %s\ngid = %s\nlog file = %s\n[module]\npath = %s\nread only = no\n' \ + "$(id -u)" "$(id -g)" "$BUILD_DIR/smoke.log" "$BUILD_DIR/smoke-daemon" > smoke.conf +RSYNC_CONNECT_PROG="'$BUILD_DIR/rsync' --daemon --config='$BUILD_DIR/smoke.conf'" \ + timeout -k 2 30 ./rsync -r smoke-source/ rsync://localhost/module/ +cmp smoke-source/file smoke-daemon/file + +pending=$(mktemp "$OUT.XXXXXX") +install -m 755 ./rsync "$pending" +strip "$pending" +mv -T -- "$pending" "$OUT" +pending= +printf '%s\n' "$banner" +sha256sum "$OUT" diff --git a/old_versions/rsync_2.6.0 b/old_versions/rsync_2.6.0 index face7a023..a2c49cf47 100755 Binary files a/old_versions/rsync_2.6.0 and b/old_versions/rsync_2.6.0 differ diff --git a/old_versions/rsync_3.0.0 b/old_versions/rsync_3.0.0 index c844826ca..2fa6b8de3 100755 Binary files a/old_versions/rsync_3.0.0 and b/old_versions/rsync_3.0.0 differ diff --git a/old_versions/rsync_3.1.0 b/old_versions/rsync_3.1.0 index ebb876077..ed3170034 100755 Binary files a/old_versions/rsync_3.1.0 and b/old_versions/rsync_3.1.0 differ diff --git a/old_versions/rsync_3.1.3 b/old_versions/rsync_3.1.3 index 3cd1ac4ae..8f6ffd532 100755 Binary files a/old_versions/rsync_3.1.3 and b/old_versions/rsync_3.1.3 differ diff --git a/old_versions/rsync_3.2.0 b/old_versions/rsync_3.2.0 index a28c4058a..913ad585f 100755 Binary files a/old_versions/rsync_3.2.0 and b/old_versions/rsync_3.2.0 differ diff --git a/old_versions/rsync_3.2.7 b/old_versions/rsync_3.2.7 index 7ca168206..329155b62 100755 Binary files a/old_versions/rsync_3.2.7 and b/old_versions/rsync_3.2.7 differ diff --git a/old_versions/rsync_3.3.0 b/old_versions/rsync_3.3.0 index 1169447b2..ae6d05dfb 100755 Binary files a/old_versions/rsync_3.3.0 and b/old_versions/rsync_3.3.0 differ diff --git a/old_versions/rsync_3.4.0 b/old_versions/rsync_3.4.0 index 7432dbd4c..45e59d50f 100755 Binary files a/old_versions/rsync_3.4.0 and b/old_versions/rsync_3.4.0 differ diff --git a/old_versions/rsync_3.4.1 b/old_versions/rsync_3.4.1 index 4c66adddd..e2c5fc14b 100755 Binary files a/old_versions/rsync_3.4.1 and b/old_versions/rsync_3.4.1 differ diff --git a/old_versions/rsync_3.4.2 b/old_versions/rsync_3.4.2 new file mode 100755 index 000000000..52829cc4f Binary files /dev/null and b/old_versions/rsync_3.4.2 differ diff --git a/old_versions/rsync_3.4.3 b/old_versions/rsync_3.4.3 new file mode 100755 index 000000000..fbe1222eb Binary files /dev/null and b/old_versions/rsync_3.4.3 differ diff --git a/old_versions/rsync_3.4.4 b/old_versions/rsync_3.4.4 new file mode 100755 index 000000000..d32be50f3 Binary files /dev/null and b/old_versions/rsync_3.4.4 differ diff --git a/old_versions/rsync_3.5.0 b/old_versions/rsync_3.5.0 new file mode 100755 index 000000000..f88ede367 Binary files /dev/null and b/old_versions/rsync_3.5.0 differ diff --git a/old_versions/rsync_3.5.1 b/old_versions/rsync_3.5.1 new file mode 100755 index 000000000..301651588 Binary files /dev/null and b/old_versions/rsync_3.5.1 differ diff --git a/old_versions/rsync_3.5.2 b/old_versions/rsync_3.5.2 new file mode 100755 index 000000000..c0a943a1e Binary files /dev/null and b/old_versions/rsync_3.5.2 differ diff --git a/packaging/release.py b/packaging/release.py index 186148c96..0c5c9e6ee 100755 --- a/packaging/release.py +++ b/packaging/release.py @@ -22,6 +22,7 @@ from datetime import datetime sys.path = ['packaging'] + sys.path +from source_archive import GEN_FILES, make_archive from pkglib import ( warn, die, cmd_run, cmd_chk, cmd_txt, cmd_txt_chk, cmd_pipe, @@ -63,20 +64,6 @@ def default_release_dir(): FTP_REMOTE_PATH = '/home/ftp/pub/rsync' HTML_REMOTE_PATH = '/home/httpd/html/rsync' -# Files that ./configure + make produce and that the release tarball / diff -# need to bundle alongside the git-tracked source. Mirrors the GENFILES -# definition in Makefile.in (with rrsync.1{,.html} since we always configure -# --with-rrsync during the release build). -GEN_FILES = [ - 'configure.sh', - 'aclocal.m4', - 'config.h.in', - 'rsync.1', 'rsync.1.html', - 'rsync-ssl.1', 'rsync-ssl.1.html', - 'rsyncd.conf.5', 'rsyncd.conf.5.html', - 'rrsync.1', 'rrsync.1.html', -] - # ---------- Phase registry ---------- PHASES = [ @@ -618,11 +605,7 @@ def step_7_tarball(args): rsync_ver_dir = os.path.join(WORK_DIR, rsync_ver) shutil.rmtree(a_dir) os.rename(b_dir, rsync_ver_dir) - cmd_chk(f"git archive --format=tar --prefix={rsync_ver}/ {v_ver} | " - f"tar -C {WORK_DIR} -xf -") - cmd_chk(f"support/git-set-file-times --quiet --prefix={rsync_ver_dir}/") - cmd_chk(['fakeroot', 'tar', '-C', WORK_DIR, '-czf', srctar_file, - '--exclude=.github', rsync_ver]) + make_archive(v_ver, rsync_ver_dir, srctar_file) # Leave staging in place for the FTP refresh. print(f"\nCreated:\n {srctar_file}\n {diff_file}") diff --git a/packaging/source_archive.py b/packaging/source_archive.py new file mode 100644 index 000000000..c521a908f --- /dev/null +++ b/packaging/source_archive.py @@ -0,0 +1,37 @@ +from pathlib import Path +import shutil +import subprocess +import tempfile + + +GEN_FILES = [ + 'configure.sh', 'aclocal.m4', 'config.h.in', + 'rsync.1', 'rsync.1.html', 'rsync-ssl.1', 'rsync-ssl.1.html', + 'rsyncd.conf.5', 'rsyncd.conf.5.html', 'rrsync.1', 'rrsync.1.html', +] + + +def make_archive(revision, stage, output): + stage, output = Path(stage).resolve(), Path(output).resolve() + if not stage.name.startswith('rsync-') or not stage.is_dir(): + raise ValueError('Invalid release staging directory') + if {path.name for path in stage.iterdir()} != set(GEN_FILES): + raise ValueError('Release staging must contain exactly the generated release files') + for name in GEN_FILES: + if not (stage / name).is_file() or (stage / name).is_symlink(): + raise ValueError(f'Missing generated release file: {name}') + head = subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip() + selected = subprocess.check_output(['git', 'rev-parse', revision + '^{commit}'], text=True).strip() + if head != selected: + raise ValueError('Archive timestamps require the selected revision to be checked out') + with tempfile.TemporaryDirectory(prefix='rsync-archive-') as directory: + tracked = Path(directory) / 'tracked.tar' + subprocess.run(['git', 'archive', '--format=tar', '--prefix=' + stage.name + '/', + '--output=' + str(tracked), selected], check=True, timeout=120) + subprocess.run(['tar', '-C', str(stage.parent), '-xf', str(tracked)], check=True, timeout=120) + subprocess.run(['support/git-set-file-times', '--quiet', '--prefix=' + str(stage) + '/'], + check=True, timeout=120) + archive = Path(directory) / 'source.tar.gz' + subprocess.run(['fakeroot', 'tar', '-C', str(stage.parent), '-czf', str(archive), + '--exclude=.github', stage.name], check=True, timeout=120) + shutil.copyfile(archive, output) diff --git a/runtests.py b/runtests.py deleted file mode 100755 index 4f62c12f6..000000000 --- a/runtests.py +++ /dev/null @@ -1,1024 +0,0 @@ -#!/usr/bin/env python3 - -# Copyright (C) 2001, 2002 by Martin Pool -# Copyright (C) 2003-2022 Wayne Davison -# Copyright (C) 2026 Andrew Tridgell -# -# Rewrite of runtests.sh in Python (runtests.sh is now deprecated). -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version -# 2 as published by the Free Software Foundation. - -"""rsync test runner. - -Invokes test scripts from testsuite/ and reports results. -Can be called by 'make check' or directly. - -Usage: - ./runtests.py [options] [TEST ...] - -Each TEST is a test name (e.g. 'delete') or glob pattern (e.g. 'xattr*'). -If no tests are specified, all tests are run. -""" - -import argparse -import concurrent.futures -import fnmatch -import glob -import math -import os -import signal -import subprocess -import sys -import threading -import time - -# Share the test exit-code enum with the test helpers. exitcodes.py lives in -# testsuite/ (next to this script); it has no import-time side effects. -sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), 'testsuite')) -from exitcodes import Exit - - -def _race_seconds(text): - """argparse type for --race-timeout: a finite, strictly positive number. - - A race test loops `while monotonic() < deadline`, so a budget of 0 (or a - negative, or a NaN, which fails every comparison) runs the body ZERO times - and the test reports PASS without ever exercising its oracle -- a silently - disarmed security test, which is worse than a slow one. Infinity would run - until the unrelated per-test timeout. The old max(RACE_TIMEOUT, 10.0) floor - used to make this unreachable; validating here restores that guarantee.""" - try: - secs = float(text) - except ValueError: - raise argparse.ArgumentTypeError(f'not a number: {text!r}') - if not math.isfinite(secs) or secs <= 0: - raise argparse.ArgumentTypeError( - f'must be a finite positive number of seconds, got {text!r}; ' - 'a zero/negative/NaN budget would make every race test pass ' - 'without running its race') - return secs - - -def parse_args(): - p = argparse.ArgumentParser(description='Run rsync test suite') - p.add_argument('tests', nargs='*', metavar='TEST', - help='Test names or patterns to run (default: all)') - p.add_argument('--exclude', default=None, metavar='LIST', - help='Comma-separated test names/globs to skip entirely: ' - 'they are not run and not reported as skipped. Useful ' - 'for tests that cannot work in a given build/CI ' - 'environment (e.g. a restricted buildd chroot). ' - 'Falls back to the RSYNC_EXCLUDE environment variable.') - p.add_argument('-j', '--parallel', type=int, default=1, metavar='N', - help='Run up to N tests in parallel (default: 1)') - p.add_argument('--valgrind', action='store_true', - help='Run rsync under valgrind (logs to per-process files)') - p.add_argument('--valgrind-opts', default='', metavar='OPTS', - help='Extra valgrind options (e.g. "--leak-check=full")') - p.add_argument('--preserve-scratch', action='store_true', - help='Keep scratch directories after tests complete') - p.add_argument('--log-level', type=int, default=1, metavar='N', - help='Verbosity level 1-10 (default: 1)') - p.add_argument('--always-log', action='store_true', - help='Show test logs even for passing tests') - p.add_argument('--stop-on-fail', action='store_true', - help='Stop after first test failure') - p.add_argument('--timing', action='store_true', - help='After the run, report each test\'s wall-clock time, ' - 'slowest first. With -j N the report also shows how ' - 'much of the run the slowest test alone accounts for.') - p.add_argument('--timeout', type=int, default=300, metavar='SECS', - help='Per-test timeout in seconds (default: 300)') - p.add_argument('--race-timeout', type=_race_seconds, default=None, metavar='SECS', - help='Budget (seconds) a TOCTOU symlink-race test may spend ' - 'trying to win its race before concluding. Overrides ' - 'every such test\'s own default (5-15s, the suite\'s ' - 'slowest tests: a race test always spends its whole ' - 'budget). Lowering it speeds the suite up but weakens ' - 'the oracle. Unset: each test keeps its default.') - p.add_argument('--rsync-bin', default=None, metavar='PATH', - help='Path to rsync binary (default: ./rsync)') - p.add_argument('--rsync-bin2', default=None, metavar='PATH', - help='Path to a second ("peer") rsync binary used for the ' - 'daemon side and remote-shell --rsync-path. Lets the ' - 'suite mix two rsync versions over the wire. Default: ' - 'same as --rsync-bin (no version mixing).') - p.add_argument('--tooldir', default=None, metavar='DIR', - help='Tool/build directory (default: cwd)') - p.add_argument('--srcdir', default=None, metavar='DIR', - help='Source directory (default: script directory)') - p.add_argument('--protocol', type=int, default=None, metavar='VER', - help='Force protocol version (adds --protocol=VER to rsync)') - p.add_argument('--expect-skipped', default=None, metavar='LIST', - help='Comma-separated list of expected-skipped tests. An ' - '@FILE entry reads a skip list (one test per line, ' - '"#" comments); relative paths resolve against srcdir ' - 'and several may be composed, e.g. ' - '@testsuite/skiplist/linux.txt,@testsuite/skiplist/proto29.txt. ' - 'A -NAME entry removes a name the rest of the spec ' - 'added, for a host that can really run a test its ' - 'platform list expects to skip.') - p.add_argument('--expect-result', default=None, metavar='FILE', - help='Path to an expected-outcome manifest (one ' - '" " per line). When ' - 'set, ONLY the tests listed in FILE are run, and each ' - "test's actual outcome is compared against its " - 'expected one; any mismatch (including an unexpected ' - 'pass) fails the run. Used for version-mixing CI.') - p.add_argument('--daemon-tests-only', action='store_true', - help='Run only the tests that can reach the daemon ' - 'transport. Intended for a --use-tcp pass that follows ' - 'a full default-transport run: the tests this drops ' - 'never call start_test_daemon(), so they cannot observe ' - '--use-tcp and would just repeat themselves. Disables ' - 'the expected-skip oracle (it describes a full run).') - p.add_argument('--use-tcp', action='store_true', - help='Run daemon tests against a real rsyncd bound to ' - '127.0.0.1 (non-default). The default is the secure ' - 'stdio-pipe transport, which opens no listening ' - 'socket; --use-tcp exposes a loopback port for the ' - 'duration of each daemon test.') - return p.parse_args() - - -def find_setfacl_nodef(scratchbase): - """Determine the setfacl command to remove default ACLs.""" - for cmd in [ - ['setacl', '-k', 'u::7,g::5,o:5', scratchbase], - ['setfacl', '-k', scratchbase], - ['setfacl', '-s', 'u::7,g::5,o:5', scratchbase], - ]: - try: - subprocess.run(cmd, capture_output=True, timeout=5) - return cmd[:2] if cmd[0] == 'setacl' else cmd[:2] - except (FileNotFoundError, subprocess.TimeoutExpired): - continue - try: - r = subprocess.run(['setfacl', '--help'], capture_output=True, text=True, timeout=5) - if '-k,' in r.stdout or '-k,' in r.stderr: - return ['setfacl', '-k'] - except (FileNotFoundError, subprocess.TimeoutExpired): - pass - return None - - -def get_tls_args(config_h): - """Determine TLS_ARGS from config.h.""" - args = '' - try: - with open(config_h) as f: - text = f.read() - if '#define HAVE_LUTIMES 1' in text: - args += ' -l' - if '#undef CHOWN_MODIFIES_SYMLINK' in text: - args += ' -L' - except FileNotFoundError: - pass - return args.strip() - - -def read_shconfig(path): - """Read shell config variables from shconfig.""" - env = {} - try: - with open(path) as f: - for line in f: - line = line.strip() - if line.startswith('#') or line.startswith('export') or not line: - continue - if '=' in line: - k, _, v = line.partition('=') - env[k.strip()] = v.strip().strip('"') - except FileNotFoundError: - pass - return env - - -def get_testuser(): - """Determine the current test user.""" - for cmd in ['/usr/bin/whoami', '/usr/ucb/whoami', '/bin/whoami']: - if os.path.isfile(cmd): - try: - return subprocess.check_output([cmd], text=True).strip() - except subprocess.CalledProcessError: - pass - try: - return subprocess.check_output(['id', '-un'], text=True).strip() - except (FileNotFoundError, subprocess.CalledProcessError): - return os.environ.get('LOGNAME', os.environ.get('USER', 'UNKNOWN')) - - -def _move_aside(path): - """Rename an un-removable directory to a unique sibling so its name is free. - - A rename-storm symlink-race test can corrupt a directory on some filesystems - (OpenBSD FFS soft-updates can leave an "empty" dir that still reports - ENOTEMPTY/EPERM and only fsck clears). `rm -rf` then can't remove it, but - renaming the top dir aside succeeds even with a corrupted descendant, freeing - the original name for a clean scratchdir.""" - n = 0 - while os.path.exists(f"{path}.corrupt.{os.getpid()}.{n}"): - n += 1 - try: - os.rename(path, f"{path}.corrupt.{os.getpid()}.{n}") - except OSError: - pass - - -def prep_scratch(scratchdir, srcdir, tooldir, setfacl_nodef): - """Prepare a scratch directory for a test.""" - if os.path.isdir(scratchdir): - subprocess.run(['chmod', '-R', 'u+rwX', scratchdir], capture_output=True) - subprocess.run(['rm', '-rf', scratchdir], capture_output=True) - if os.path.isdir(scratchdir): - _move_aside(scratchdir) # rm -rf left corrupted debris; don't inherit it - os.makedirs(scratchdir, exist_ok=True) - if setfacl_nodef: - subprocess.run(setfacl_nodef + [scratchdir], capture_output=True) - try: - os.chmod(scratchdir, os.stat(scratchdir).st_mode & ~0o2000) # clear setgid - except OSError: - pass - src_link = os.path.join(scratchdir, 'src') - if not os.path.exists(src_link): - if os.path.isabs(srcdir): - os.symlink(srcdir, src_link) - else: - os.symlink(os.path.join(tooldir, srcdir), src_link) - - -# Python tests are identified by a positive "_test.py" suffix so that -# helper modules (e.g. rsyncfns.py) sit in testsuite/ without being mistaken -# for tests. -_PY_TEST_SUFFIX = '_test.py' - - -def _is_test_path(path): - return os.path.basename(path).endswith(_PY_TEST_SUFFIX) - - -def _testbase(path): - """Strip the test extension to get the canonical test name.""" - base = os.path.basename(path) - if base.endswith(_PY_TEST_SUFFIX): - return base[:-len(_PY_TEST_SUFFIX)] - return base - - -def collect_tests(suitedir, patterns): - """Collect test scripts (_test.py) matching the given patterns.""" - if not patterns: - candidates = glob.glob(os.path.join(suitedir, '*' + _PY_TEST_SUFFIX)) - tests = sorted(p for p in candidates if _is_test_path(p)) - else: - seen = set() - tests = [] - for pat in patterns: - # Accept either bare name ("mkpath"), explicit extension, or glob. - if pat.endswith('.py'): - pats = [pat] - else: - pats = [pat + _PY_TEST_SUFFIX] - for p in pats: - for m in sorted(glob.glob(os.path.join(suitedir, p))): - if _is_test_path(m) and m not in seen: - seen.add(m) - tests.append(m) - return tests - - -# Tokens through which a test can reach the daemon transport. --use-tcp works by -# setting RSYNC_TEST_USE_TCP, which is read in exactly one place (rsyncfns -# USE_TCP) and acted on in exactly one function (start_test_daemon): a test whose -# source mentions none of these never gets there, so it behaves identically with -# and without --use-tcp and running it a second time under TCP buys no coverage. -# -# The list is the closure of every rsyncfns helper that reaches USE_TCP, -# start_rsyncd or claim_ports, plus the helper modules that open a daemon -# connection themselves and the bare literals a test might use directly. It is -# deliberately over-broad: a false positive only costs runtime, while a false -# negative would silently drop real coverage. -_DAEMON_API = ( - 'USE_TCP', 'require_tcp', 'start_test_daemon', 'start_rsyncd', - 'claim_ports', 'claim_free_port', 'setup_chroot_inner', - 'stdio_daemon', 'rsync_proto', 'DaemonClient', - 'rsync://', '--daemon', 'rsyncd', -) - - -def select_daemon_tests(tests): - """Split `tests` into (daemon-transport tests, the rest). - - Used by --daemon-tests-only so a TCP pass need not re-run the whole suite. - A test we cannot read is kept, not dropped -- the failure mode of this - filter must always be "ran too much".""" - keep, dropped = [], [] - for path in tests: - try: - with open(path, errors='replace') as f: - text = f.read() - except OSError: - keep.append(path) - continue - (keep if any(tok in text for tok in _DAEMON_API) else dropped).append(path) - return keep, dropped - - -_VALID_OUTCOMES = ('pass', 'skip', 'fail', 'xfail') - - -def parse_expect_result(path): - """Parse an expected-outcome manifest into {testbase: outcome}. - - One " " entry per line; '#' comments and blank lines - are ignored. outcome is one of pass|skip|fail|xfail. The set of listed - tests doubles as the run set (see main()). Exits 2 on a malformed file. - """ - expect = {} - with open(path) as f: - for lineno, raw in enumerate(f, 1): - line = raw.split('#', 1)[0].strip() - if not line: - continue - fields = line.split() - if len(fields) != 2 or fields[1] not in _VALID_OUTCOMES: - sys.stderr.write( - f"{path}:{lineno}: expected ' " - f"<{'|'.join(_VALID_OUTCOMES)}>', got: {raw.rstrip()}\n" - ) - sys.exit(Exit.ERROR) - expect[fields[0]] = fields[1] - return expect - - -def expand_skip_spec(spec, srcdir, suitedir): - """Expand an RSYNC_EXPECT_SKIPPED spec into a normalised csv. - - The spec is a comma-separated list of test names, '@FILE' skip-list - references, and '-name' removals. A skip-list file holds one test name per line ('#' starts a - comment; blank lines are ignored), which is what keeps two branches from - colliding: adding a test edits one line of one file rather than a shared - 3 KB csv. Several may be composed, e.g. - RSYNC_EXPECT_SKIPPED=@testsuite/skiplist/linux.txt,@.../proto29.txt - Relative paths resolve against srcdir (not the cwd) so out-of-tree builds - and `make installcheck` work. A '-name' entry removes a name the rest of - the spec added, for a host that can genuinely run a test its platform list - expects to skip; it is applied last, and must actually remove something. - - Entries must name a real test, and each file must be non-empty, sorted and - free of duplicates: unsorted files defeat the point (everyone appends to - the same last line), and a stale name would otherwise fail as a skip - mismatch far from its cause. Exits 2 on any of those -- nothing malformed - may quietly shrink the expected set, which would disarm the oracle. - """ - def die(msg): - sys.stderr.write(msg + '\n') - sys.exit(Exit.ERROR) - - # An entirely empty spec is the legitimate "expect no skips at all". An - # empty entry *within* a spec is not: it is what an unset shell variable - # expands to, and silently dropping it would quietly shrink the expected - # set. - if not spec.strip(): - return '' - - names = [] - drop = [] - for tok in (t.strip() for t in spec.split(',')): - if not tok: - die('RSYNC_EXPECT_SKIPPED: empty entry (an unset variable?): ' - f'{spec!r}') - if tok.startswith('-'): - # '-name' removes a name a composed list added, for a host that - # really can run a test its platform list expects to skip (e.g. a - # scratch dir on a second filesystem makes a cross-device copy - # work). Subtraction cannot be done by whoever composes the spec, - # because the name lives inside an @FILE that is only expanded - # here. Applied after every addition, so order does not matter. - drop.append((tok[1:], tok)) - continue - if not tok.startswith('@'): - names.append((tok, 'RSYNC_EXPECT_SKIPPED')) - continue - path = tok[1:] - if not os.path.isabs(path): - path = os.path.join(srcdir, path) - try: - with open(path) as f: - lines = f.readlines() - except (OSError, UnicodeDecodeError) as e: - die(f'{tok}: cannot read skip list: {e}') - prev = None - found = 0 - for lineno, raw in enumerate(lines, 1): - name = raw.split('#', 1)[0].strip() - if not name: - continue - where = f'{path}:{lineno}' - if len(name.split()) != 1: - die(f'{where}: expected one test name per line, got: {raw.rstrip()}') - if prev is not None and name <= prev: - die(f'{where}: skip lists must be sorted and duplicate-free ' - f'({name!r} follows {prev!r})') - prev = name - found += 1 - names.append((name, where)) - # A truncated or emptied list must not read as "expect no skips". - if not found: - die(f'{path}: skip list contains no test names') - - seen = {} - for name, where in names: - if name in seen: - continue - seen[name] = where - # A plain test name: no path, and no comma (which the expanded csv, - # and the summary the fleet parses back, use as the separator). - if ',' in name or '/' in name or os.sep in name or name in ('.', '..'): - die(f'{where}: not a test name: {name!r}') - if not os.path.isfile(os.path.join(suitedir, name + '_test.py')): - die(f'{where}: no such test: {name}') - for name, tok in drop: - # Every removal must remove something. A name the spec never added is - # stale (the list stopped expecting that skip, or it is misspelled), and - # a repeated removal is the same no-op written twice. Shrinking the - # expected set is precisely what must not happen quietly, so neither is - # allowed to sit in a config unnoticed. - if name not in seen: - die(f'RSYNC_EXPECT_SKIPPED: {tok!r} removes a name that nothing ' - f'added: {name}') - del seen[name] - return ','.join(sorted(seen)) - - -def read_backport_exclude(tooldir, suitedir): - """Test names from the BUILD tree's testsuite/skiplist/backport.txt. - - A stable-backport branch runs a newer suite than its own code, and this - file names the tests that base cannot run. It lives in the tree being - built, not the suite tree, so it cannot be an @FILE in the expect-skipped - spec -- those resolve against srcdir. - """ - path = os.path.join(tooldir, 'testsuite', 'skiplist', 'backport.txt') - if not os.path.isfile(path): - return set() - names = set() - with open(path) as f: - for lineno, raw in enumerate(f, 1): - name = raw.split('#', 1)[0].strip() - if not name: - continue - where = f'{path}:{lineno}' - if len(name.split()) != 1: - sys.stderr.write(f'{where}: expected one test name per line\n') - sys.exit(Exit.ERROR) - # A stale name here would silently exclude nothing, which is the - # failure this file exists to prevent. - if not os.path.isfile(os.path.join(suitedir, name + '_test.py')): - sys.stderr.write(f'{where}: no such test: {name}\n') - sys.exit(Exit.ERROR) - names.add(name) - return names - - -_TIMING_TOP = 25 - - -def print_timing_report(durations, outcomes, run_wall, parallel): - """Report per-test wall-clock, slowest first (--timing). - - With -j N the run cannot finish sooner than its slowest single test, so the - tail matters as much as the total: a 300s test pins the whole suite to 300s - no matter how many workers there are. The footer gives both bounds -- the - serial sum (what one worker would take) and that floor -- so it is obvious - whether a slow run wants more parallelism or a faster individual test.""" - if not durations: - return - ranked = sorted(durations.items(), key=lambda kv: kv[1], reverse=True) - total = sum(durations.values()) - print(f'----- slowest tests (of {len(ranked)}, wall-clock each):') - for name, secs in ranked[:_TIMING_TOP]: - print(f' {secs:7.1f}s {name:<32} {outcomes.get(name, "?")}') - print(f' serial sum {total:.0f}s over {len(ranked)} tests; ' - f'run took {run_wall:.0f}s with -j{parallel}') - slowest, slowest_secs = ranked[0] - if parallel > 1: - # The floor: even with unlimited workers the suite cannot beat its - # longest single test. - print(f' floor {slowest_secs:.0f}s ({slowest}) = ' - f'{100.0 * slowest_secs / run_wall:.0f}% of this run; ' - f'ideal at -j{parallel} is {total / parallel:.0f}s') - - -def outcome_of(result): - """Map a per-test exit code to an outcome string.""" - if result == Exit.PASS: - return 'pass' - if result == Exit.SKIP: - return 'skip' - if result == Exit.XFAIL: - return 'xfail' - return 'fail' - - -def build_rsync_cmd(rsync_bin, args, scratchbase): - """Build the RSYNC command string for tests.""" - parts = [] - if args.valgrind: - # Logs go in a world-writable+sticky subdir so that rsync children - # which drop privileges (the setpriv cap-drop in partial_nowrite, a - # daemon dropping to the module's uid) can still create their log file - # even when scratchbase itself is root-owned. - vgdir = os.path.join(scratchbase, 'valgrind-logs') - os.makedirs(vgdir, exist_ok=True) - os.chmod(vgdir, 0o1777) - vlog = os.path.join(vgdir, 'valgrind.%p.log') - vopts = f'--log-file={vlog}' - supp = os.path.join(os.path.dirname(os.path.abspath(__file__)), - 'testsuite', 'valgrind.supp') - if os.path.exists(supp): - vopts += f' --suppressions={supp}' - if args.valgrind_opts: - vopts += ' ' + args.valgrind_opts - parts.append(f'valgrind {vopts}') - parts.append(rsync_bin) - if args.protocol is not None: - parts.append(f'--protocol={args.protocol}') - return ' '.join(parts) - - -class TestResult: - """Result of a single test execution.""" - __slots__ = ('testbase', 'result', 'output', 'skipped_reason', 'duration') - - def __init__(self, testbase, result, output='', skipped_reason='', - duration=0.0): - self.testbase = testbase - self.result = result - self.output = output - self.skipped_reason = skipped_reason - self.duration = duration - - -def run_one_test(testscript, testbase, scratchdir, base_env, timeout, - srcdir, tooldir, setfacl_nodef, always_log): - """Run a single test. Returns a TestResult. - - This function is safe to call from multiple threads — it uses only - per-test state (unique scratchdir, copy of env). - """ - started = time.monotonic() - prep_scratch(scratchdir, srcdir, tooldir, setfacl_nodef) - - env = base_env.copy() - env['scratchdir'] = scratchdir - - # Dispatch by extension: shell tests via /bin/sh -e, Python tests via - # the same python3 that's running this runner. - if testscript.endswith('.py'): - cmd = [sys.executable, testscript] - else: - cmd = ['sh', '-e', testscript] - - logfile = os.path.join(scratchdir, 'test.log') - with open(logfile, 'w') as log: - # start_new_session: run the test driver as its own session/group leader - # so the daemon, clients and flipper it spawns inherit that group. A - # timeout then killpg's the whole tree (not just the driver), and the - # lock-file sweep can reap a SIGKILLed run's stranded group the same way. - proc = subprocess.Popen( - cmd, - stdout=log, stderr=subprocess.STDOUT, - env=env, cwd=env.get('TOOLDIR', '.'), - start_new_session=True, - ) - try: - result = proc.wait(timeout=timeout) - except subprocess.TimeoutExpired: - # Reap the whole session group, but only if the driver really is its - # own group leader (start_new_session took) and that group isn't ours - # -- killpg of our own group would take down the runner. - try: - pgid = os.getpgid(proc.pid) - except OSError: - pgid = -1 - if pgid == proc.pid and pgid != os.getpgrp(): - try: - os.killpg(pgid, signal.SIGKILL) - except OSError: - proc.kill() - else: - proc.kill() - proc.wait() - result = 1 - log.write(f"\nTIMEOUT: test took over {timeout} seconds\n") - - # Build output text - output_parts = [] - - show_log = always_log or (result not in (Exit.PASS, Exit.SKIP, Exit.XFAIL)) - if show_log: - output_parts.append(f'----- {testbase} log follows') - try: - with open(logfile) as f: - output_parts.append(f.read().rstrip()) - except FileNotFoundError: - pass - output_parts.append(f'----- {testbase} log ends') - rsyncd_log = os.path.join(scratchdir, 'rsyncd.log') - if os.path.isfile(rsyncd_log): - output_parts.append(f'----- {testbase} rsyncd.log follows') - with open(rsyncd_log) as f: - output_parts.append(f.read().rstrip()) - output_parts.append(f'----- {testbase} rsyncd.log ends') - - skipped_reason = '' - if result == Exit.PASS: - output_parts.append(f'PASS {testbase}') - elif result == Exit.SKIP: - whyfile = os.path.join(scratchdir, 'whyskipped') - try: - with open(whyfile) as f: - skipped_reason = f.read().strip() - except FileNotFoundError: - pass - output_parts.append(f'SKIP {testbase} ({skipped_reason})') - elif result == Exit.XFAIL: - output_parts.append(f'XFAIL {testbase}') - else: - output_parts.append(f'FAIL {testbase}') - - return TestResult(testbase, result, '\n'.join(output_parts), skipped_reason, - time.monotonic() - started) - - -# Lock for serializing output in parallel mode -_print_lock = threading.Lock() - - -def main(): - args = parse_args() - - # Also accept legacy environment variables - if args.preserve_scratch or os.environ.get('preserve_scratch') == 'yes': - args.preserve_scratch = True - if args.log_level == 1: - args.log_level = int(os.environ.get('loglevel', '1')) - if args.expect_skipped is None: - args.expect_skipped = os.environ.get('RSYNC_EXPECT_SKIPPED', 'IGNORE') - if args.exclude is None: - args.exclude = os.environ.get('RSYNC_EXCLUDE', '') - if os.environ.get('whichtests'): - args.tests = [os.environ['whichtests']] - - # Determine directories - tooldir = args.tooldir or os.environ.get('TOOLDIR') or os.getcwd() - script_path = os.path.dirname(os.path.abspath(__file__)) - srcdir = args.srcdir or script_path - if not srcdir or srcdir == '.': - srcdir = tooldir - rsync_bin = args.rsync_bin or os.environ.get('rsync_bin') or os.path.join(tooldir, 'rsync') - # Absolutize: tests run with subprocess(cwd=TOOLDIR) below, so a relative - # argv[0] would re-resolve against TOOLDIR rather than the runner's - # invocation cwd, breaking --rsync-bin=../foo/rsync forms. abspath() - # captures os.getcwd() now, which is what the operator intended. - if rsync_bin and not os.path.isabs(rsync_bin): - rsync_bin = os.path.abspath(rsync_bin) - - # Optional second ("peer") binary for the daemon / remote-shell side, so a - # run can mix two rsync versions. Defaults to rsync_bin -> no mixing. - rsync_bin2 = args.rsync_bin2 or os.environ.get('rsync_bin2') or rsync_bin - if rsync_bin2 and not os.path.isabs(rsync_bin2): - rsync_bin2 = os.path.abspath(rsync_bin2) - - suitedir = os.path.join(srcdir, 'testsuite') - # A backport tree excludes what its base cannot run. Those tests never - # run, so they must also drop out of the expected-skip set -- otherwise the - # oracle demands a skip from a test that was never started. - backport_excl = read_backport_exclude(tooldir, suitedir) - if backport_excl: - args.exclude = ','.join(x for x in (args.exclude, - ','.join(sorted(backport_excl))) if x) - if args.expect_skipped != 'IGNORE': - args.expect_skipped = expand_skip_spec(args.expect_skipped, srcdir, suitedir) - if backport_excl: - args.expect_skipped = ','.join(n for n in args.expect_skipped.split(',') - if n and n not in backport_excl) - scratchbase = os.path.join(os.environ.get('scratchbase', tooldir), 'testtmp') - os.makedirs(scratchbase, exist_ok=True) - - shconfig = read_shconfig(os.path.join(tooldir, 'shconfig')) - tls_args = get_tls_args(os.path.join(tooldir, 'config.h')) - setfacl_nodef = find_setfacl_nodef(scratchbase) - rsync_cmd = build_rsync_cmd(rsync_bin, args, scratchbase) - rsync_peer_cmd = build_rsync_cmd(rsync_bin2, args, scratchbase) - - if not os.path.isfile(rsync_bin): - sys.stderr.write(f"rsync_bin {rsync_bin} is not a file\n") - sys.exit(Exit.ERROR) - if not os.path.isfile(rsync_bin2): - sys.stderr.write(f"rsync_bin2 {rsync_bin2} is not a file\n") - sys.exit(Exit.ERROR) - if not os.path.isdir(srcdir): - sys.stderr.write(f"srcdir {srcdir} is not a directory\n") - sys.exit(Exit.ERROR) - - # Helper programs the test scripts invoke directly. Missing any of these - # would cause many tests to fail with confusing "not found" errors, so - # check up front and point the user at the make target that builds them. - required_helpers = ['tls', 'trimslash', 't_unsafe', 't_chmod_secure', - 't_secure_relpath', - 'wildtest', 'getgroups', 'getfsdev'] - missing = [h for h in required_helpers - if not os.path.isfile(os.path.join(tooldir, h))] - if missing: - sys.stderr.write( - f"runtests.py: missing test helper program(s) in {tooldir}: " - f"{', '.join(missing)}\n" - f"Build them with: make {' '.join(missing)}\n" - f"or run the full test target: make check\n" - ) - sys.exit(Exit.ERROR) - - testuser = get_testuser() - - # Print header - print('=' * 60) - print(f'{sys.argv[0]} running in {tooldir}') - print(f' rsync_bin={rsync_cmd}') - if rsync_peer_cmd != rsync_cmd: - print(f' rsync_peer={rsync_peer_cmd}') - print(f' srcdir={srcdir}') - print(f' TLS_ARGS={tls_args}') - print(f' testuser={testuser}') - print(f' os={subprocess.check_output(["uname", "-a"], text=True).strip()}') - print(f' preserve_scratch={"yes" if args.preserve_scratch else "no"}') - if args.valgrind: - print(f' valgrind=enabled (logs in valgrind-logs/valgrind.*.log)') - if args.parallel > 1: - print(f' parallel={args.parallel}') - print(f' daemon_transport={"tcp (loopback)" if args.use_tcp else "pipe (secure default)"}') - print(f' scratchbase={scratchbase}') - - # Build base environment for test scripts - path = os.environ.get('PATH', '') - if os.path.isdir('/usr/xpg4/bin'): - path = '/usr/xpg4/bin:' + path - - # Make the testsuite/ directory importable so Python tests can `import rsyncfns`. - pythonpath = suitedir - if os.environ.get('PYTHONPATH'): - pythonpath = suitedir + os.pathsep + os.environ['PYTHONPATH'] - - base_env = os.environ.copy() - base_env.update({ - 'PATH': path, - 'POSIXLY_CORRECT': '1', - 'TOOLDIR': tooldir, - 'srcdir': srcdir, - 'RSYNC': rsync_cmd, - 'RSYNC_PEER': rsync_peer_cmd, - 'TLS_ARGS': tls_args, - 'RUNSHFLAGS': '-e', - 'scratchbase': scratchbase, - 'suitedir': suitedir, - 'TESTRUN_TIMEOUT': str(args.timeout), - 'HOME': scratchbase, - 'PYTHONPATH': pythonpath, - }) - if args.use_tcp: - # Opt-in: daemon tests start a real rsyncd on a claimed loopback port. - # Default (unset) keeps the secure stdio-pipe transport. - base_env['RSYNC_TEST_USE_TCP'] = '1' - if args.race_timeout is not None: - # Only exported when the operator actually passed --race-timeout: its - # mere presence is what tells a race test to override its own default. - base_env['race_timeout'] = str(args.race_timeout) - else: - # A stale value inherited from the environment would silently override - # every test's default; the flag is the only way to set this. - base_env.pop('race_timeout', None) - for k, v in shconfig.items(): - if v: - base_env[k] = v - if setfacl_nodef: - base_env['setfacl_nodef'] = ' '.join(setfacl_nodef) - else: - base_env['setfacl_nodef'] = 'true' - if args.log_level > 8: - base_env['RUNSHFLAGS'] = '-e -x' - - # Collect tests - tests = collect_tests(suitedir, args.tests) - full_run = len(args.tests) == 0 - - # Drop excluded tests entirely (matched by basename against name/glob). - excl = [e.strip() for e in args.exclude.split(',') if e.strip()] - if excl: - before = len(tests) - tests = [t for t in tests - if not any(fnmatch.fnmatch(_testbase(t), pat) for pat in excl)] - if before != len(tests): - print(f"Excluding {before - len(tests)} test(s) matching: " - f"{', '.join(excl)}") - - # Narrow to the daemon-transport tests. The dropped count is always printed: - # a pass that silently ran a third of the suite would read in the report as - # if it had run all of it. - if args.daemon_tests_only: - tests, dropped = select_daemon_tests(tests) - print(f"Daemon-transport tests only: running {len(tests)}, skipping " - f"{len(dropped)} test(s) that cannot observe the transport") - # The expected-skip list describes a full run, so it cannot be enforced - # against a subset -- same rule as naming tests explicitly. - full_run = False - - # An expected-result manifest defines BOTH the run set (its keys) and the - # expected per-test outcome (its values). Used for version-mixing runs. - expect = parse_expect_result(args.expect_result) if args.expect_result else None - if expect is not None: - have = {_testbase(t) for t in tests} - unknown = sorted(k for k in expect if k not in have) - if unknown: - sys.stderr.write( - "runtests.py: --expect-result lists test(s) with no matching " - f"test file (ignored): {', '.join(unknown)}\n" - ) - tests = [t for t in tests if _testbase(t) in expect] - full_run = False - - def _cls(outcome): - """Equivalence class for outcome comparison: fail and xfail both just - mean 'broke', so a manifest 'fail' matches an actual fail OR xfail.""" - return 'broken' if outcome in ('fail', 'xfail') else outcome - - def mismatch(testbase, actual): - """True if actual outcome disagrees with the manifest expectation.""" - return expect is not None and _cls(expect[testbase]) != _cls(actual) - - # Record test order for consistent skipped-list output - test_order = {_testbase(t): i for i, t in enumerate(tests)} - - passed = 0 - failed = 0 - skipped = 0 - xfailed = 0 - skipped_list = [] - outcomes = {} # testbase -> actual outcome string ('pass'/'skip'/'fail'/'xfail') - durations = {} # testbase -> wall-clock seconds (for --timing) - - def process_result(tr): - """Process a TestResult and update counters. Returns True if the test - should count as a failure for --stop-on-fail purposes.""" - nonlocal passed, failed, skipped, xfailed - with _print_lock: - if tr.output: - print(tr.output) - scratchdir = os.path.join(scratchbase, tr.testbase) - oc = outcome_of(tr.result) - outcomes[tr.testbase] = oc - durations[tr.testbase] = tr.duration - if tr.result == Exit.PASS: - passed += 1 - elif tr.result == Exit.SKIP: - skipped_list.append(tr.testbase) - skipped += 1 - elif tr.result == Exit.XFAIL: - # XFAIL: an expected failure (a known, documented residual the test - # asserts against). Reported distinctly but does NOT fail the suite; - # when the underlying issue is fixed the test returns 0 instead. - xfailed += 1 - else: - failed += 1 - if tr.result in (Exit.PASS, Exit.SKIP, Exit.XFAIL) and not args.preserve_scratch \ - and os.path.isdir(scratchdir): - subprocess.run(['rm', '-rf', scratchdir], capture_output=True) - # With a manifest, only a mismatch is a "failure" (an expected fail is - # fine); without one, any non-pass/non-skip/non-xfail result is a failure. - if expect is not None: - return mismatch(tr.testbase, oc) - return tr.result not in (Exit.PASS, Exit.SKIP, Exit.XFAIL) - - run_started = time.monotonic() - - if args.parallel > 1: - # Parallel execution - with concurrent.futures.ThreadPoolExecutor(max_workers=args.parallel) as executor: - futures = {} - for testscript in tests: - testbase = _testbase(testscript) - scratchdir = os.path.join(scratchbase, testbase) - timeout = 600 if ('hardlinks' in testbase or testbase == 'variety') else args.timeout - f = executor.submit( - run_one_test, testscript, testbase, scratchdir, - base_env, timeout, srcdir, tooldir, setfacl_nodef, - args.always_log - ) - futures[f] = testbase - - for f in concurrent.futures.as_completed(futures): - tr = f.result() - is_fail = process_result(tr) - if is_fail and args.stop_on_fail: - # Cancel pending futures - for pending in futures: - pending.cancel() - break - else: - # Sequential execution - for testscript in tests: - testbase = _testbase(testscript) - scratchdir = os.path.join(scratchbase, testbase) - timeout = 600 if ('hardlinks' in testbase or testbase == 'variety') else args.timeout - tr = run_one_test( - testscript, testbase, scratchdir, - base_env, timeout, srcdir, tooldir, setfacl_nodef, - args.always_log - ) - is_fail = process_result(tr) - if is_fail and args.stop_on_fail: - break - - run_wall = time.monotonic() - run_started - - # Check valgrind logs for errors - vg_errors = 0 - if args.valgrind: - for vlog in sorted(glob.glob(os.path.join(scratchbase, 'valgrind-logs', 'valgrind.*.log'))): - try: - with open(vlog) as f: - content = f.read() - for line in content.splitlines(): - if 'ERROR SUMMARY:' in line and 'ERROR SUMMARY: 0 errors' not in line: - vg_errors += 1 - print(f'----- valgrind errors in {os.path.basename(vlog)}:') - print(content) - break - except FileNotFoundError: - pass - - # Summary - print('-' * 60) - print('----- overall results:') - print(f' {passed} passed') - if failed > 0: - print(f' {failed} failed') - if xfailed > 0: - print(f' {xfailed} xfailed (expected)') - if skipped > 0: - print(f' {skipped} skipped') - if vg_errors > 0: - print(f' {vg_errors} valgrind error(s) found (see logs in {os.path.join(scratchbase, "valgrind-logs")})') - - if args.timing: - print_timing_report(durations, outcomes, run_wall, args.parallel) - - if expect is not None: - # Version-mixing mode: the run is judged purely on whether each test's - # actual outcome matched its manifest expectation. An expected 'fail' - # is fine; an UNEXPECTED pass (xpass) or any other divergence is not. - mismatches = [] - for tb in sorted(expect, key=lambda x: test_order.get(x, 1 << 30)): - actual = outcomes.get(tb, 'notrun') - if actual == 'notrun' or mismatch(tb, actual): - mismatches.append((tb, expect[tb], actual)) - if mismatches: - print('----- expected-result mismatches:') - for tb, want, got in mismatches: - tag = ' (xpass)' if _cls(want) == 'broken' and got == 'pass' else '' - print(f' {tb}: expected {want}, got {got}{tag}') - print('-' * 60) - exit_code = len(mismatches) + vg_errors - print(f'overall result is {exit_code}') - sys.exit(exit_code) - - skipped_str = ','.join(sorted(skipped_list, key=lambda x: test_order.get(x, 0))) - if full_run and args.expect_skipped != 'IGNORE': - print('----- skipped results:') - print(f' expected: {args.expect_skipped}') - print(f' got: {skipped_str}') - else: - skipped_str = '' - args.expect_skipped = '' - - print('-' * 60) - - exit_code = failed + vg_errors - if exit_code == 0: - # Compare the skipped set order-insensitively: which tests skipped is - # what matters, not the order runtests happened to collect them in - # (that order is just sorted filenames -- an easy thing to get subtly - # wrong when maintaining the per-platform expected lists). - got = set(s for s in skipped_str.split(',') if s) - want = set(s for s in args.expect_skipped.split(',') if s) - if got != want: - exit_code = 1 - - print(f'overall result is {exit_code}') - sys.exit(exit_code) - - -if __name__ == '__main__': - main() diff --git a/support/rrsh.sh b/support/rrsh.sh deleted file mode 100755 index 83025f2f5..000000000 --- a/support/rrsh.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/bin/sh -# abdiff helper: a "remote shell" that emulates an sshd forced-command of -# `rrsync DIR`. rsync invokes a remote shell as: -# [ssh-opts] -# so when used as -e "sh rrsh.sh " rsync calls us as: -# sh rrsh.sh [opts] lh rsync --server ... -# We hand the server command to rrsync via SSH_ORIGINAL_COMMAND (exactly as -# sshd would) and exec the restricted wrapper, so abdiff can A/B the rrsync -# path itself. Only the pretend hosts "lh"/"localhost" are accepted. -RRSYNC="$1"; DIR="$2"; shift 2 -while [ $# -gt 0 ]; do - case "$1" in - -l) shift 2 ;; - lh|localhost) shift; break ;; - -*) shift ;; - *) break ;; - esac -done -SSH_ORIGINAL_COMMAND="$*" -export SSH_ORIGINAL_COMMAND -exec "$RRSYNC" "$DIR" diff --git a/testsuite/00-hello_test.py b/testsuite/00-hello_test.py deleted file mode 100644 index cdc153798..000000000 --- a/testsuite/00-hello_test.py +++ /dev/null @@ -1,104 +0,0 @@ -#!/usr/bin/env python3 -# Python rewrite of testsuite/00-hello.test. -# -# Foundational smoke test: --version / --info=help / --debug=help all -# work, plus a round-trip transfer of a directory whose name contains -# shell-special characters via the lsh.sh remote-shell stand-in. - -import os - -from rsyncfns import ( - FROMDIR, RSYNC, RSYNC_PEER, SRCDIR, TODIR, - checkit, run_rsync, test_fail, rsync_path_arg, rsh_cmd, -) - - -# Set RSYNC_RSH so rsync picks up lsh.sh for the "lh:" hosts below. -os.environ['RSYNC_RSH'] = rsh_cmd() - -# Basic help dumps must not crash. -if run_rsync('--version', check=False).returncode != 0: - test_fail('--version output failed') -if run_rsync('--info=help', check=False).returncode != 0: - test_fail('--info=help output failed') -if run_rsync('--debug=help', check=False).returncode != 0: - test_fail('--debug=help output failed') - -weird_name = "A weird)name" - -FROMDIR.mkdir(parents=True, exist_ok=True) -weird_dir = FROMDIR / weird_name -weird_dir.mkdir() - - -def append_line(line: str) -> None: - print(line) - with open(weird_dir / 'file', 'a') as f: - f.write(line + '\n') - - -def copy_weird(args: list, src_host: str, dst_host: str) -> None: - checkit( - [*args, f'--rsync-path={rsync_path_arg()}', - f'{src_host}{weird_dir}/', - f'{dst_host}{TODIR / weird_name}'], - FROMDIR, TODIR, - ) - - -append_line('test1') -checkit(['-ai', f'{FROMDIR}/', f'{TODIR}/'], FROMDIR, TODIR) - -append_line('test2') -copy_weird(['-ai'], 'lh:', '') - -append_line('test3') -copy_weird(['-ai'], '', 'lh:') - -append_line('test4') -copy_weird(['-ais'], 'lh:', '') - -append_line('test5') -copy_weird(['-ais'], '', 'lh:') - -# test6: --old-args lets two whitespace-separated names go through as a -# single "one two" remote argument to be re-split by the remote shell. -print('test6') -(FROMDIR / 'one').touch() -(FROMDIR / 'two').touch() - -saved = os.getcwd() -os.chdir(FROMDIR) -try: - run_rsync('-ai', '--old-args', f'--rsync-path={rsync_path_arg()}', - 'lh:one two', f'{TODIR}/') -finally: - os.chdir(saved) - -if not (TODIR / 'one').is_file() or not (TODIR / 'two').is_file(): - test_fail("old-args copy of 'one two' failed") - -# test7: the RSYNC_OLD_ARGS=1 env var should be equivalent to --old-args. -print('test7') -(TODIR / 'one').unlink() -(TODIR / 'two').unlink() - -env = os.environ.copy() -env['RSYNC_OLD_ARGS'] = '1' -import subprocess -from rsyncfns import rsync_argv - -os.chdir(FROMDIR) -try: - subprocess.run( - rsync_argv('-ai', f'--rsync-path={rsync_path_arg()}', - 'lh:one two', f'{TODIR}/'), - env=env, check=True, - ) -finally: - os.chdir(saved) - -# check=True only proves a zero exit; confirm the env-var path actually copied -# both files (as the explicit --old-args case above does). -if not (TODIR / 'one').is_file() or not (TODIR / 'two').is_file(): - test_fail("RSYNC_OLD_ARGS=1 copy of 'one two' failed") diff --git a/testsuite/COVERAGE.md b/testsuite/COVERAGE.md deleted file mode 100644 index 6f09d4527..000000000 --- a/testsuite/COVERAGE.md +++ /dev/null @@ -1,205 +0,0 @@ -# rsync option / daemon-parameter test coverage matrix - -Living checklist for the test-coverage effort that precedes the path-handling -restructure of rsync's path resolution. The restructure rewrites parent-directory -resolution for essentially every option, so the goal here is a regression net -that exercises each option **at directory depth** (≥3 levels) and, where the -option spans trees, **across directory boundaries**, asserting the *specific -property* the option controls — not just `dest == src`. - -How to read the columns: - -* **test(s)** — the `testsuite/*_test.py` that exercise the option. Tests added - by this effort are marked `*new*`. -* **depth** — Y = asserted on entries ≥3 levels deep; `~` = exercised only at/near - the tree root; `n/a` = not a path-resolution option. -* **x-dir** — Y = exercised with the relevant aux tree (temp/backup/dest/partial) - **outside** the main tree; `—` = not a cross-directory option. -* **gap** — what is still missing. - -Status legend: ✓ property asserted · `~` shallow / by an existing ported test · -✗ no coverage. - ---- - -## Command-line options - -### Recursion / structure -| option | test(s) | depth | x-dir | notes / gap | -|---|---|---|---|---| -| -a, --archive | (all) | Y | — | ✓ ubiquitous | -| -r, --recursive | hands, delete-deep*new* | Y | — | ✓ | -| -R, --relative | relative, relative-implied*new* | Y | — | ✓ implied-dir attrs at depth | -| --no-implied-dirs | relative-implied*new* | Y | — | ✓ (proto 30+; proto 29 rejects multi-component path) | -| --inc-recursive / --no-inc-recursive | hardlinks | Y | — | `~` exercised, not isolated | -| -d, --dirs | dirs*new* | Y | — | ✓ no-recurse top layer | -| --old-dirs / --old-d | — | — | — | ✗ | -| -m, --prune-empty-dirs | prune-empty-dirs*new* | Y | — | ✓ incl. filter-emptied chains | - -### Links -| option | test(s) | depth | x-dir | notes / gap | -|---|---|---|---|---| -| -l, --links | links*new*, symlink-ignore | Y | — | ✓ | -| -L, --copy-links | links*new* | Y | — | ✓ deref file+dir | -| -k, --copy-dirlinks | links*new* | Y | — | ✓ follow dir-symlink | -| -K, --keep-dirlinks | symlink-dirlink-basis | Y | — | ✓ #715; skips on no-RESOLVE_BENEATH / --disable-openat2 | -| -H, --hard-links | hardlinks, hardlinks-deep*new* | Y | Y | ✓ cross-directory hardlink | -| --copy-unsafe-links | unsafe-links | `~` | — | `~` | -| --safe-links | safe-links | `~` | — | `~` | -| --munge-links | (daemon-munge*new* covers the daemon param) | — | — | `~` client option not isolated; local mode is a near no-op | - -### Metadata / permissions / ownership -| option | test(s) | depth | x-dir | notes / gap | -|---|---|---|---|---| -| -p, --perms | metadata-depth*new* | Y | — | ✓ exact modes per entry | -| -E, --executability | executability | `~` | — | `~` | -| --chmod | metadata-depth*new*, chmod-option | Y | — | ✓ | -| -A, --acls | acls, acls-depth*new* | Y | — | ✓ | -| -X, --xattrs | xattrs, xattrs-depth*new* | Y | — | ✓ | -| -t, --times | metadata-depth*new* | Y | — | ✓ | -| -U, --atimes | atimes | `~` | — | `~` (same set path as -t, covered deep) | -| --open-noatime | open-noatime | `~` | — | `~` | -| -N, --crtimes | crtimes | `~` | — | `~` (skips without crtimes support) | -| -O, --omit-dir-times | omit-times*new* | Y | — | ✓ | -| -J, --omit-link-times | omit-times*new* | Y | — | ✓ | -| -o, --owner | chown, ownership-depth*new* | Y | — | ✓ uid map root-gated | -| -g, --group | chgrp, ownership-depth*new* | Y | — | ✓ group remap non-root | -| --super / --fake-super | chown, chown-fake | `~` | — | `~` | -| --numeric-ids | — | — | — | ✗ client; daemon `numeric ids` also ✗ | -| --usermap / --groupmap | ownership-depth*new* | Y | — | ✓ groupmap non-root; usermap root-gated | -| --chown | ownership-depth*new* | Y | — | ✓ group half | -| -D / --devices / --specials | devices, devices-fake | `~` | — | `~` root/device-gated | -| --copy-devices / --write-devices | — | — | — | ✗ device-gated | -| -S, --sparse | sparse*new* | Y | — | ✓ hole preserved at depth | - -### Delta / temp / backup / dest (highest restructure risk) -| option | test(s) | depth | x-dir | notes / gap | -|---|---|---|---|---| -| -T, --temp-dir | temp-dir*new*, chmod-temp-dir | Y | Y | ✓ cross-dir rename | -| --partial | partial*new* | Y | — | ✓ partial kept in dest file | -| --partial-dir | partial*new*, symlink-dirlink-basis | Y | Y | ✓ relative (in-tree) + absolute (outside), incl. delta resume from an absolute outside-tree partial | -| --delay-updates | delay-updates, delay-updates-deep*new* | Y | — | ✓ per-dir staging | -| --inplace | inplace*new*, alt-dest | Y | — | ✓ inode preserved | -| --append / --append-verify | append*new* | Y | — | ✓ verify split is proto 30+ | -| -b, --backup / --backup-dir / --suffix | backup, backup-deep*new* | Y | Y | ✓ | -| --compare-dest / --copy-dest / --link-dest | alt-dest, alt-dest-deep*new* | Y | Y | ✓ link=hardlink, copy=copy, compare=skip | -| -y, --fuzzy | fuzzy | `~` | — | `~` | -| -u, --update | update*new* | Y | — | ✓ keeps newer dest, updates older | -| -W, --whole-file | (used widely; --no-whole-file ubiquitous) | n/a | — | `~` | -| --mkpath | mkpath | `~` | — | `~` | -| -x, --one-file-system | — | — | — | ✗ (needs a mount boundary) | -| --preallocate / --fsync | — | — | — | ✗ | -| -B, --block-size | — | — | — | ✗ | -| --max-alloc | max-alloc-zero | — | — | ✓ zero resolves to each peer's supported ceiling; values above the limit are rejected | - -### Filtering -| option | test(s) | depth | x-dir | notes / gap | -|---|---|---|---|---| -| -f, --filter / -F | filter-depth*new*, merge | Y | — | ✓ deep per-dir merge | -| --exclude / --include | filter-depth*new*, exclude, exclude-lsh | Y | — | ✓ | -| --exclude-from / --include-from | files-from-depth*new* | Y | — | ✓ | -| -C, --cvs-exclude | cvs-exclude*new* | Y | — | ✓ incl. deep .cvsignore | -| --files-from | files-from-depth*new* | Y | — | ✓ | -| -0, --from0 | files-from-depth*new* | Y | — | ✓ | -| --max-size / --min-size | size-filter*new* | Y | — | ✓ | -| --existing / --ignore-existing | delete-deep*new* | Y | — | ✓ | -| --ignore-missing-args / --delete-missing-args | — | — | — | ✗ | - -### Deletion -| option | test(s) | depth | x-dir | notes / gap | -|---|---|---|---|---| -| --delete / --del | delete, delete-deep*new* | Y | — | ✓ deep subtree | -| --delete-before/during/delay/after | delete-deep*new* | Y | — | ✓ all four agree | -| --delete-excluded | delete | `~` | — | `~` | -| --max-delete | delete-deep*new* | Y | — | ✓ caps deletions | -| --remove-source-files | delete | `~` | — | `~` | -| --force | update*new* | Y | — | ✓ replaces a non-empty dir with a file | -| --ignore-errors | — | — | — | ✗ (client; daemon `ignore errors` also ✗) | - -### Comparison / checksum / compression -| option | test(s) | depth | x-dir | notes / gap | -|---|---|---|---|---| -| -c, --checksum | compare*new* | Y | — | ✓ catches stealth change | -| -I, --ignore-times | compare*new* | Y | — | ✓ | -| --size-only | compare*new* | Y | — | ✓ | -| -@, --modify-window | compare*new* | Y | — | ✓ | -| --checksum-choice / --checksum-seed | compress-options*new* | Y | — | ✓ every advertised algo | -| -z, --compress | daemon-gzip-{up,down}load, daemon-refuse-compress | `~` | — | `~` | -| --compress-choice / --compress-level / --skip-compress | compress-options*new* | Y | — | ✓ | - -### Output / reporting (path-irrelevant — checked for output shape) -| option | test(s) | notes / gap | -|---|---|---| -| -i, --itemize-changes | output-options*new*, itemize | ✓ | -| -n, --dry-run | output-options*new* | ✓ | -| --stats | output-options*new* | ✓ | -| --out-format | output-options*new* | ✓ | -| --list-only | output-options*new* | ✓ | -| -q, --quiet | output-options*new* | ✓ | -| --progress / -P | output-options*new* | ✓ (--progress) | -| -h, --human-readable / -8, --8-bit-output | output-options*new* | ✓ smoke | -| --version / --help | output-options*new* | ✓ | -| --info / --debug / --stderr / --no-motd / --outbuf | — | ✗ | -| -M, --remote-option / --log-file / --log-file-format | — | ✗ (daemon `log file` covered) | - -### Batch / connection / misc -| option | test(s) | notes / gap | -|---|---|---| -| --write-batch / --only-write-batch / --read-batch | batch-mode | `~` | -| -e, --rsh / --rsync-path | ssh-basic, many | `~` | -| --protocol | check29 / check30 (whole suite) | ✓ | -| --address / --port | daemon tests under --use-tcp | `~` | -| --password-file | daemon-auth*new* | ✓ | -| --early-input / daemon `early exec` | — | ✗ | -| --sockopts / --blocking-io / --timeout / --contimeout | — | ✗ | -| -4/-6, --ipv4/--ipv6 | — | ✗ | -| --stop-after / --stop-at | — | ✗ | -| --bwlimit | partial*new* (used, not asserted) | `~` | -| --copy-as | — | ✗ root-gated | -| --iconv | — | ✗ | -| --nice / --no-nice | niceness | ✓ local, remote shell, local server, daemon and rrsync | -| -s/--secluded-args, --old-args, --trust-sender | (default arg-protection exercised) | `~` | - ---- - -## Daemon (rsyncd.conf) parameters - -| parameter | test(s) | notes / gap | -|---|---|---| -| path | daemon-access*new*, all daemon tests | ✓ incl. deep sub-path | -| read only | daemon-access*new*, daemon | ✓ | -| write only | daemon-access*new* | ✓ | -| list | daemon-access*new*, daemon | ✓ hidden-but-usable | -| use chroot | sender-flist-symlink-leak, daemon-chroot-acl | `~` (no=most tests; yes needs root) | -| munge symlinks | daemon-munge*new* | ✓ /rsyncd-munged/ add+strip | -| exclude / include | daemon-filter*new*, daemon | ✓ exclude | -| filter / exclude from / include from | — | ✗ (exclude covers the mechanism) | -| incoming chmod | daemon-filter*new*, chmod-option | ✓ | -| outgoing chmod | daemon-filter*new* | ✓ | -| auth users / secrets file | daemon-auth*new* | ✓ accept/reject/unauth | -| strict modes | daemon-auth*new* | ✓ rejects world-readable secrets | -| refuse options | daemon-refuse*new*, daemon-refuse-compress | ✓ named/wildcard/allow-list | -| pre-xfer exec / post-xfer exec | daemon-exec*new* | ✓ env + abort | -| early exec | — | ✗ (needs --early-input) | -| hosts allow / hosts deny | daemon (allow), daemon-chroot-acl (deny) | `~` (needs --use-tcp for real peer) | -| reverse lookup / forward lookup | daemon-chroot-acl | `~` reverse only | -| log file / transfer logging / log format | daemon | `~` set, not asserted | -| max verbosity | daemon | `~` | -| comment | daemon, daemon-access*new* | ✓ | -| numeric ids | — | ✗ (hard to observe non-root) | -| fake super | chown-fake (client side) | ✗ as daemon param | -| timeout / max connections / lock file | — | ✗ (need --use-tcp + concurrency) | -| temp dir / open noatime / ignore errors / ignore nonreadable | — | ✗ | -| charset / name converter / dont compress | — | ✗ | -| uid / gid / daemon uid / daemon gid / daemon chroot | build_rsyncd_conf (uid/gid when root), daemon-chroot-acl | `~` root-gated | -| motd file / pid file / port / address / socket options / listen backlog / proxy protocol / syslog facility / syslog tag | — | ✗ (server-startup/connection params) | - ---- - -## Known gaps worth a future pass -* Connection/timeout params (`--timeout`, `--contimeout`, daemon `timeout`, - `max connections`) need a real socket + concurrency (run under `--use-tcp`). -* Root-only behaviours (`-o`/`--usermap` uid remap, real devices, `use chroot - = yes`, daemon uid/gid) skip as non-root; run the suite as root to cover. -* `--ignore-errors`, `-x/--one-file-system`, `--numeric-ids` have no dedicated - test yet (lower restructure risk). diff --git a/testsuite/README.md b/testsuite/README.md index 3536a3660..fdfc861de 100644 --- a/testsuite/README.md +++ b/testsuite/README.md @@ -1,304 +1,65 @@ -# rsync testsuite +# Testing rsync -This directory holds rsync's automated regression tests. Ideally every code -change or bug fix comes with a test that would have caught the problem. +The runner and harness use Python 3.7 or later and its standard library. Some tests also need platform tools for ACLs, extended attributes, SSH or privilege changes. CommonMark is needed to build manual pages rather than to run tests. -The tests are Python scripts named `testsuite/*_test.py`, driven by the -`runtests.py` harness at the top of the tree (the old shell-based `runtests.sh` -is gone). Shared helpers live in `testsuite/rsyncfns.py`. A handful of C helper -programs (`tls`, `getgroups`, `trimslash`, …) are built alongside `rsync` and -used by some tests. Coverage notes are in [COVERAGE.md](COVERAGE.md). - -## Writing tests - -Favour readability — a test is also documentation of the behaviour it pins, so -prefer clarity over cleverness: - -* When a test writes an `rsyncd.conf`, write it as a triple-quoted f-string so - the actual config is readable top-to-bottom, with module parameters indented - with plain spaces. Don't build it from adjacent string literals full of `\n` - (and `\t`) escapes. The daemon's parser accepts space-indented parameters. -* Better still, use the structured helpers in `rsyncfns.py` when a stock config - will do: `write_daemon_conf(modules, globals)` (per-test modules/params) or - `build_rsyncd_conf()` (the four standard modules). They also handle the - root-only `uid`/`gid` lines for you (needed so a `use chroot = no` daemon run - as root can read a root-owned module). -* For config that varies (e.g. those root-only `uid`/`gid` lines), interpolate a - single optional block that expands when needed and is an empty string - otherwise, rather than splicing pieces together: - - ```python - root = get_testuid() == get_rootuid() - ids = f"uid = {get_rootuid()}\ngid = {get_rootgid()}" if root else "" - conf.write_text(f"""\ - pid file = {base}/rsyncd.pid - use chroot = no - {ids} - log file = {base}/rsyncd.log - - [m] - path = {mod} - read only = yes - """) - ``` - -## Running the tests - -### Via make - -Run from the build directory: - -- **`make check`** — build the helper programs and run the whole suite in - parallel (`CHECK_J`, default 8) against the just-built `./rsync`. You do **not** - need `make install` first; indeed you generally should not install before - testing. Use `make check CHECK_J=1` to run serially. -- **`make check29`** / **`make check30`** — the same, forcing protocol version 29 - or 30. -- **`make installcheck`** — run the suite against the *installed* binary (e.g. - `/usr/local/bin/rsync`). Per the GNU standards this does not search `$PATH`. - Handy for testing a distribution build. -- **`make check-progs`** — (re)build just the C helper programs the tests need, - without running anything. -- **`make coverage`** / **`coverage-tcp`** / **`coverage-all`** — generate an HTML - coverage report (needs `./configure --enable-coverage` and `gcovr`); - `coverage-all` merges runs across protocol versions and the tcp transport. - -### Via runtests.py directly - -`make check` just drives `runtests.py`; run it directly for finer control. It -defaults `--rsync-bin` to `./rsync`, so run it from the build directory (or pass -`--rsync-bin` / `--tooldir`): +## Running tests +From a configured build directory: ```sh -./runtests.py # all tests -./runtests.py chmod-temp-dir # a single test by name -./runtests.py 'xattr*' # a glob of test names +make check +make check29 +make check30 ``` -Useful options: - -- `-j N`, `--parallel N` — run up to N tests at once -- `--use-tcp` — run daemon tests against a real `rsyncd` on `127.0.0.1` (the - default runs them over a stdio pipe). **Read the security warning below before - using this on a shared machine.** -- `--protocol VER` — force a protocol version -- `--preserve-scratch` — keep each test's scratch dir afterwards -- `--log-level N`, `--always-log` — more verbose output / show logs for passing tests too -- `--stop-on-fail` — stop after the first failure -- `--timeout SECS` — per-test timeout (default 300) -- `--timing` — after the run, list the tests by wall-clock, slowest first, with - the serial sum and the floor set by the single longest test -- `--race-timeout SECS` — budget a TOCTOU race test may spend trying to win its - race. These are the suite's slowest tests: a race test is a negative oracle, - so it passes by spending its *whole* budget (5–15s each by default). Lowering - this speeds the suite up and weakens the oracle in equal measure. -- `--valgrind`, `--valgrind-opts OPTS` — run rsync under valgrind -- `--rsync-bin PATH`, `--tooldir DIR`, `--srcdir DIR` — locate the binary / build / source dirs -- `--expect-skipped LIST` — see skip enforcement below - -### Security warning: `--use-tcp` - -> **⚠️ Do not use `--use-tcp` on a machine with untrusted local users.** -> -> `--use-tcp` starts a real `rsync` daemon listening on a loopback TCP port -> (`127.0.0.1` / `::1`) and **deliberately configures insecure test scenarios** -> (daemon modules without authentication, unsafe options enabled, etc.). Loopback -> addresses are reachable by *every* local user, so for as long as the tests run, -> any other user on the machine can connect to that daemon and exploit those -> deliberately-insecure modules — potentially reading or writing files with the -> privileges of the user running the tests (which is **root** if you run the suite -> as root). -> -> Only run `--use-tcp` where there are **no possible local users who might try to -> exploit it** — a single-user workstation or a dedicated, isolated CI machine. -> The default stdio-pipe transport carries no such risk: it talks to the daemon -> over a private pipe with nothing listening on the network, so prefer it on any -> shared or multi-user host. - -### Results and exit codes - -Each test prints one result line — `PASS`, `FAIL`, `ERROR`, `SKIP` (with a -reason), or `XFAIL` (an expected failure) — and the run ends with a -`passed / failed / skipped` summary. Per-test exit-code convention: - -| code | meaning | -|------|---------| -| 0 | pass | -| 1 | fail | -| 2 | error | -| 77 | skip | -| 78 | xfail | - -`runtests.py` exits non-zero if any test fails. Some tests need root or another -precondition and otherwise `SKIP` — read the individual test scripts for details. - -**Skip enforcement:** on a full run, set `RSYNC_EXPECT_SKIPPED=a,b,c` (or -`--expect-skipped a,b,c`) and the run fails if the set of skipped tests does not -match. This is how the CI workflows pin each platform's expected skip set. An -`@FILE` entry reads a skip list (one test per line) instead, and several may be -composed: the workflows use -`@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt`. Keeping the -lists one-name-per-line is what stops two branches that each add a skipping test -from conflicting -- see `testsuite/skiplist/README.md`. - -### Scratch dirs and debugging - -Each test runs in `testtmp//`. On failure the scratch directory is left in -place (also `--preserve-scratch`); including its logs in a bug report is helpful. - -### Preconditions - -You need `python3`, `/bin/sh`, and the normal build toolchain. The ACL/xattr -tests need the `acl` and `attr` tools (`getfacl`/`setfacl`, -`getfattr`/`setfattr`) and skip if they are absent. Some tests need root. - -These tests also run in CI via GitHub Actions (see `.github/workflows/`). - -## Fleet testing (fleettest.py) - -`testsuite/fleettest.py` builds the committed HEAD of an rsync checkout on a -fleet of remote machines over ssh and runs the suite under both transports -(stdio-pipe and `--use-tcp`) in parallel, reporting only the *unexpected* -results. It is a fast local pre-flight for the GitHub CI matrix: each target -mirrors a `.github/workflows/*.yml` job — its configure flags, and the -`RSYNC_EXPECT_SKIPPED` list parsed straight from the workflow. - -Because every run includes a `--use-tcp` pass, the fleet stands up the insecure -loopback test daemon on each target — so only point it at machines with **no -untrusted local users** (see the [security warning](#security-warning---use-tcp) -above). - -The fleet — which machines, and how to reach and build on each — is described in -a JSON file. Copy the bundled example (it is git-ignored) and edit it for your -hosts: +The first target runs the suite with the default protocol; the others force protocols 29 and 30. Make runs up to eight tests at once. Set CHECK_J to change that limit. +For an in-tree build, individual tests and filename patterns can be selected after running `make check-progs`: ```sh -cp testsuite/fleettest.json.example testsuite/fleettest.json # then edit -# (or symlink it, or point elsewhere with --fleet PATH) +python3 -S testsuite/runtests.py -j8 chmod-temp-dir 'xattrs*' +python3 -S testsuite/runtests.py --help +python3 -S testsuite/runtests.py --describe-tests ``` -The config is looked up in order: `~/.fleettest.json` first, then -`testsuite/fleettest.json`, unless overridden with `--fleet PATH`. - -Each entry names an ssh host (`null` to run locally), the workflow it mirrors, -and its configure flags, plus optional per-target settings (`make`, `privilege`, -`env_prefix`, …). See the comments in `fleettest.json.example`. - -A target with `"nonroot": true` does an extra pass, after the main (root) run, -that reruns the privilege-sensitive tests as the unprivileged ssh user. Which -tests those are is **not** listed in the fleet config — a test opts in by -setting a module-level `fleet_nonroot = True`, so the set is maintained in the -test files and new privilege-sensitive tests join automatically with no -fleet-config change. +The runner defaults to serial execution. For an out-of-tree build, invoke `$srcdir/testsuite/runtests.py` and pass `--srcdir`, `--tooldir` and `--rsync-bin`. The `installcheck` target uses the installed binary with the build directory's helpers. -A target with `"protocols": [30, 29]` runs one extra stdio-pipe pass per listed -version, each forcing that older wire version with `runtests --protocol=N` — the -fleet analogue of a workflow's `check30`/`check29` steps. Each pass takes the -`RSYNC_EXPECT_SKIPPED` spec from the workflow's own `check30`/`check29` step, so -a lane with extra protocol-gated skips (`check29` adds -`@testsuite/skiplist/proto29.txt`) is enforced correctly. They show up as -`protoNN` columns in the report (and `--timing` breakdown); targets that don't -set `protocols` show `-` there. +Scratch directories default to `testtmp//` and are retained after failures. Root and non-root runs need separate scratch directories. SIGTERM stops queued tests and terminates active test process groups; incomplete runs fail. -Run it from inside a checkout (it builds the current directory's HEAD; use -`--repo PATH` for another tree): +Daemon tests normally use pipes. The --use-tcp option uses loopback listeners, some with unauthenticated writable modules or deliberately unsafe settings. Other local users can reach those listeners. Pipe transport is preferable on shared hosts. The --daemon-tests-only option selects tests which exercise the transport. -```sh -python3 testsuite/fleettest.py # whole fleet, both transports -python3 testsuite/fleettest.py --list # list configured targets -python3 testsuite/fleettest.py --targets NAME[,NAME] -python3 testsuite/fleettest.py --fleet other.json --transport pipe -python3 testsuite/fleettest.py --timing # per-target wall-clock breakdown -python3 testsuite/fleettest.py --keep-on-fail # keep logs + tree where it broke -python3 testsuite/fleettest.py --full-tcp # whole suite in the tcp pass too -``` +## Tests and profiles -`--timing` adds a per-target breakdown after the report — total wall-clock plus -the push / build / pipe / tcp / protoNN / nonroot phases, sorted slowest-first. Targets -run in parallel, so the whole run is gated by the slowest one; the phase columns -show whether that target's hold-up is the push, the build, or a test pass. It -also passes `--timing` down to each target's `runtests.py`, so the captured -output attributes a slow pass to individual tests. +Tests are grouped under tests/ by build, daemon, metadata, path, protocol, rrsync, selftests and transfer. Reusable code lives in harness/ and fixture programs and data in fixtures/. Test names are lowercase, hyphen-separated filenames and must be unique across groups. Selectors and results use the filename without its `.py` ending. -The `tcp` pass runs **only the tests that can reach the daemon transport**, since -it follows a full pipe pass over the very same build. `--use-tcp` is observable -through exactly one path — `RSYNC_TEST_USE_TCP` is read once in `rsyncfns` -(`USE_TCP`) and acted on once, in `start_test_daemon()` — so a test that never -gets there produces an identical result twice. That drops 186 of the 340 tests -and roughly a third of the pass's work; the count skipped is always printed. -Pass `--full-tcp` to sweep the whole suite there anyway. The narrowing applies -only when both transports run: under `--transport tcp` that pass is the only -one, so it runs the whole suite regardless. +Tests declare applicability with `@requires(...)` or a top-level `metadata(...)` call. Transports, protocol versions, peer-version limits and required capabilities determine selection. A min_peer value can set separate primary and secondary release limits. The --describe-tests output lists these declarations. -`--keep-on-fail [DIR]` makes a failure inspectable without repeating the run. -For every target that came back with anything unexpected it writes the full -build and per-transport output to `DIR///` (default -`./fleettest-logs`) and keeps that target's remote run dir, with the scratch -trees its failing tests left behind. Targets that came back clean are swept as -usual. This matters most for the race tests, which may not fail the same way -twice — and because a re-run costs a full configure + build on every machine. +Regression tests should fail against the faulty implementation and check the affected behaviour directly. Fixtures must keep their data inside the test's scratch directory, tolerate parallel execution and stop their child processes. Sleeps cannot establish ordering or prove that a race was exercised. -Each run gets its own randomly-named build dir on every target -(`-`), so two or three runs can share the same fleet without -interfering. The dir is removed when the run ends — on success or failure, and -best-effort on Ctrl-C/kill; pass `--keep` to retain it for inspection. A hard -kill (`SIGKILL`), or a signal arriving mid-push, can leave a stray -`-` behind; sweep leftovers with -`python3 testsuite/fleettest.py --cleanup` (scope it with `--targets`, and only -run it when no other fleet runs are active, since it removes *all* matching run -dirs on the selected targets). +Profiles under profiles/ distinguish tests which do not apply from missing capabilities. Root and nonroot profiles must match the effective UID. Peer profiles identify the archived release and wire protocol. Historical binaries and their build records are under [old_versions](../old_versions/README.md); --rsync-bin2 selects the secondary executable and --archived-role identifies which side runs the archived release. -Each target must be provisioned with the build toolchain its workflow installs -(autoconf, automake, a C compiler, perl, a python3 markdown module such as -cmarkgfm or commonmark unless the flags pass `--disable-md2man`, and the dev -libraries its configure flags enable). A missing piece shows up as `BUILD-FAIL`. +The tools/check.py launcher takes a build directory, binary, profile list and fresh output directory followed by runner options. It saves run.log and receipt.json, then checks the receipt against the executable and profiles. -## Differential regression hunting (abdiff.py) +Result | Meaning +--- | --- +N/A | Excluded by applicability metadata +PASS | Assertions passed +FAIL | An assertion failed +ERROR | The test or harness failed unexpectedly +SKIP | Skipped without a capability +UNSUPPORTED | A required capability was unavailable +PROFILE_ERROR | A result disagreed with the active profile -`testsuite/abdiff.py` is a developer tool — **not** a `*_test.py`, so `runtests.py` -ignores it. It hunts *regressions* by running the **same benign transfer** with -two rsync binaries (`A` = the build under test, `B` = a baseline) and comparing -the OUTCOME. The oracle is: for a benign input, a correctness/behaviour change -between the builds must be **invisible**, so A and B must produce an identical -result. Any divergence is a regression candidate to investigate and, if real, -minimize into a `*_test.py`. +With profiles active, an unsupported result is accepted only when the test declares the capability and the profile permits its absence. A generic skip fails the run. Without profiles, skips and unsupported results are accepted. Receipts record raw outcomes, profile verdicts, exclusions, executable hashes and the effective UID. -It compares exit code, stderr (error markers + normalised text), `--stats` -"Literal data", the destination tree (content + full metadata: mode/uid/gid/ -mtime/size/symlink target/xattrs/ACLs/hardlink grouping), the `--itemize` list, -and — with `--cost` — peak process-group RSS (a resource-regression oracle that -functional comparison misses). A **stability gate** runs each binary several -times and escalates on a candidate diff; nondeterministic scenarios are -quarantined `FLAKY`, never reported as regressions. +## CI -Run it from the build directory (so `./rsync` and `old_versions/` resolve): +[ci.yml](../.github/workflows/ci.yml) defines the platform groups and required checks. Native jobs exercise root and non-root transfers where available. Historical peers run applicable tests in both executable roles over pipe and TCP. Diagnostics cover the minimum Python version, static analysis, sanitisers, coverage and Valgrind. Coverage reports omit daemon work after chroot because its counter files are outside the chroot. -```sh -testsuite/abdiff.py # default: ./rsync vs old_versions/rsync_3.4.1 -testsuite/abdiff.py --sweep all -j5 # broad single pass, 5-way parallel -testsuite/abdiff.py --loop --timelimit 3600 --cost # hunt for an hour, resource oracle on -testsuite/abdiff.py --list --sweep all # list scenarios without running -``` +Linux jobs cover both xattr syscall paths, nested user namespaces, older libc and OpenSSL versions, musl and 32-bit behaviour. We pin one BSD and macOS release unless another release exercises distinct rsync behaviour. Check the workflow files for exact versions. -Each finding is classed `DIFF` (regression candidate), `ALLOW` (an intentional, -documented behaviour change listed in the tool's allowlist), `BETTER` (A succeeds -where B fails), `FLAKY`, or `TIMEOUT`. Findings are printed and appended to a -per-run `abdiff-log_