Skip to content

ci: run native simulator suites on Xcode 27 runner - #179

Closed
rdlabo wants to merge 4 commits into
mainfrom
ci/native-e2e
Closed

rdlabo wants to merge 4 commits into
mainfrom
ci/native-e2e

Conversation

@rdlabo

@rdlabo rdlabo commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add native-e2e.yml workflow: matrix of ui-shell / search / edge suites on the xcode-27 preview image (iOS 27.0, iPhone 17), manual workflow_dispatch + weekly schedule (Mon 03:00 JST).
  • Simulator setup is shared via scripts/ci-simulator.sh in rdlabo-dev/ionic-theme-utils, fetched at the sha pinned in package-lock.json (now ec42938).
  • verify-native-*.sh: honor VERIFY_ARTIFACTS_DIR, and build @rdlabo/ionic-theme-utils dist explicitly when npm skips prepare for the transitive git dependency (observed on CI runners).
  • Screenshots/logs upload always; .xcresult uploads on failure.

Runner availability note

xcode-27 is a GitHub preview label. Jobs are created correctly but wait in queue for preview capacity, so a full green run may not be immediate — the workflow is intended to be exercised manually/weekly until capacity is generally available. A macos-26 probe of the same pipeline reached the actual test phase, confirming the plumbing works.

Test plan

  • scripts/ci-simulator.sh verified locally (sim create/boot/kana on iOS 27 runtime)
  • verify-native-ui-shell.sh end-to-end on a local iOS 27 simulator (9/10 tests; testAllDemoPagesAndRepeatedNavigation fails reproducibly on the toolbar route — real product issue on iOS 27, not CI plumbing)
  • macos-26 probe run executed all pipeline steps through xcodebuild
  • xcode-27 run completes once preview capacity is available

Generated with Devin


Devin Review

rdlabo and others added 4 commits September 16, 2026 16:48
Wire the existing local verification scripts into a scheduled/manual
workflow so the iOS 27 projection checks (page tour, placement, search,
edge gestures) run on real simulators instead of only on maintainer
machines. Three matrix jobs keep wall-clock time low, and DerivedData
is cached per suite. The consumer fixture no longer passes
--ignore-scripts because it skipped the git dependency prepare step
and left @rdlabo/ionic-theme-utils without its dist build.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…deps

- scripts/ci-simulator.sh now lives in rdlabo-dev/ionic-theme-utils so both
  theme repos share one copy; the workflow fetches it at the pinned sha.
- verify scripts build @rdlabo/ionic-theme-utils dist explicitly when npm
  skips `prepare` for the transitive git dependency (seen on CI runners).
- retarget the suite to the xcode-27 image (iOS 27.0, default toolchain).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

Devin Review

Comment on lines +61 to +64
path: ${{ runner.temp }}/native-${{ matrix.suite }}
key: native-${{ matrix.suite }}-${{ steps.meta.outputs.xcode_version }}-${{ hashFiles('**/package-lock.json', 'ios/**', 'demo/ios/**', 'demo/native-*-fixture/**', 'src/**') }}
restore-keys: |
native-${{ matrix.suite }}-${{ steps.meta.outputs.xcode_version }}-

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 キャッシュ復元後のネイティブテストが再実行できない

次回実行では、consumer/ios と tests.xcresult を含む作業ディレクトリ全体が復元されます。初期化せず同じ出力を再生成するため、既存出力と衝突してテスト前またはテスト実行時に失敗します。

Learn more

このキャッシュは Xcode の増分ビルド成果物だけでなく、各検証スクリプトの作業ディレクトリ全体を保存します。次回、同じキーまたは restore key が一致すると、consumer/ios や tests.xcresult も復元されます。検索スイート と UI Shell スイート はそれらを削除せず、同じ場所へ cap add ios と xcodebuild -resultBundlePath を実行します。どちらも新規出力先を前提とするため、復元された成果物と衝突します。

Example: コード変更のない翌週の定期実行では同じ完全キーがヒットします。前週の consumer/ios が復元されるため npx cap add ios は「プラットフォームが既に存在する」として終了し、シミュレーターテストは開始されません。

Recommended fix: キャッシュ対象を demo-build、spm-build、build、test-build などの DerivedData ディレクトリだけに限定してください。または、各スクリプトの開始時に consumer、tests、tests.xcresult、screenshots を削除し、再生成物と増分ビルド成果物を分離してください。

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +7 to +9
workflow_dispatch:
schedule:
- cron: '0 18 * * 0' # Weekly, Monday 03:00 JST

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 定期実行対象は既定ブランチに固定される

schedule は既定ブランチ上のワークフローだけを起動し、そのブランチをチェックアウトします。想定する iOS 27 ブランチを週次検証できる構成か確認してください。

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +45 to +49
- id: meta
run: |
xcodebuild -version
echo "xcode_version=$(xcodebuild -version | head -n 1 | tr ' ' '-')" >> "$GITHUB_OUTPUT"
echo "image_version=${ImageVersion:-unknown}" >> "$GITHUB_OUTPUT"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 ランナーイメージ更新がキャッシュキーに反映されない

取得した image_version は増分ビルドキャッシュキーに使われません。同じ Xcode 表示バージョンでイメージが更新されると、古い DerivedData が復元されます。

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Playwright test results

passed  332 passed

Details

stats  332 tests across 18 suites
duration  4 minutes, 41 seconds
commit  1d0ef36
info  This detailed result covers Ionic 9 only. Ionic 8 runs against the same screenshots in a separate matrix job; check the workflow run for both results. To update the screenshots, comment with /update-screenshots.

@rdlabo rdlabo left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The native simulator CI itself is valuable—it already surfaced the issues tracked in #181. However, this branch is now substantially behind main, conflicts with it, and its two principal suites are failing against the pre-#182/#183 test setup.

I’m going to close this iteration and recreate the workflow from current main after the focused native-shell audit and the valid OS-specific fixes have landed. The replacement should retain the scheduled/manual simulator coverage while avoiding the branch-specific trigger and validating the current test structure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant