External surfaces: home-screen widgets + Live Activities / Dynamic Island#5365
External surfaces: home-screen widgets + Live Activities / Dynamic Island#5365shai-almog wants to merge 27 commits into
Conversation
External surfaces model home-screen widgets (WidgetKit / Android app
widgets) and live activities (ActivityKit / Dynamic Island / ongoing
notifications) as one declarative concept: a serializable layout node
tree with ${key} state placeholders, published as JSON + content-hash
named PNG blobs that platform renderers draw while the app process is
dead. Includes the SurfaceBridge SPI, timeline/live-activity facades,
the shared serializer and cold-start-safe action dispatch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…p floating widgets SurfaceRasterizer renders descriptor JSON to a mutable CN1 Image with Graphics primitives (no theme/component dependence) returning ARGB pixels + action hit-rects + next-tick hints; shared by all desktop ports. The JavaSE bridge persists published surfaces, the simulator gains a Widgets preview window with a mock Dynamic Island, and desktop-packaged apps get frameless always-on-top widget windows with Preferences-persisted pinning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…going-notification live activities The renderer composes pre-baked layout XMLs via RemoteViews.addView (Chronometer countdowns, weighted cells, binder-budgeted bitmaps, night-aware colors). CN1WidgetProvider picks timeline entries and schedules flips with inexact alarms; live activities lower to ongoing silent notifications; taps route through a Theme.NoDisplay trampoline into Surfaces.dispatchAction with cold-start support. The builder scans for surfaces usage, parses surfaces.json, generates per-kind receivers + appwidget-provider XML and injects manifest entries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d, IPhoneBuilder wiring Ships a generic SwiftUI renderer in a generated CN1Widgets extension target (deployment 16.1) that reads published timeline JSON + images from the app group container; ActivityKit live activities render the same descriptors with full Dynamic Island regions. The app-side Swift bridge is reached from ObjC via NSClassFromString behind CN1_USE_WIDGETS, cn1surface:// taps route into Surfaces.dispatchAction, and IPhoneBuilder gains the surfaces.json manifest, app-group plist and entitlement wiring and the xcodeproj ruby extension target. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Floating widgets are frameless layered Win32 windows (premultiplied DIB via UpdateLayeredWindow, pump-thread marshaled through WM_CN1_WIDGET, hit-rect click routing over a string poll queue, per-monitor DPI). Behind windows.msix=true the builder additionally packages a signed MSIX with a C++/WinRT IWidgetProvider (gated by CN1_WIDGETBOARD) that maps descriptors to Adaptive Cards for the Windows 11 Widgets Board; plain-exe distributions keep the floating windows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Undecorated keep-above sticky GTK3 windows with RGBA visuals render rasterized descriptors via cairo; all GTK access marshals through g_idle_add with a slot table, clicks and moves flow back over a string poll queue into Surfaces.dispatchAction. Under Wayland positioning and keep-above degrade to plain floating windows (documented); X11 gets full applet behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sample exercises the full surface family (widget timeline with native countdown, live activity with Dynamic Island regions, action routing incl. cold start, BackgroundFetch republish) and was verified end-to-end in the simulator Widgets preview. Mac Catalyst builds now exclude the widget extension from the Mac slice (platform_filter ios, SUPPORTS_MACCATALYST=NO) since Mac entitlements carry no app group; the extension builder accepts both small/medium and systemSmall/systemMedium family spellings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cloudflare Preview
|
|
Compared 145 screenshots: 145 matched. Native Android coverage
✅ Native Android screenshot tests passed. Native Android coverage
Benchmark ResultsDetailed Performance Metrics
|
|
Compared 168 screenshots: 168 matched. |
|
Compared 142 screenshots: 142 matched. |
|
Compared 142 screenshots: 142 matched. |
|
Compared 141 screenshots: 141 matched. Benchmark ResultsDetailed Performance Metrics
|
|
Compared 141 screenshots: 141 matched. Benchmark ResultsDetailed Performance Metrics
|
|
Compared 143 screenshots: 143 matched. Benchmark Results
Detailed Performance Metrics
|
|
Compared 138 screenshots: 138 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
|
Compared 141 screenshots: 141 matched. Benchmark ResultsDetailed Performance Metrics
|
…cc ICE Four client-side CI breaks: - SurfaceSerializer used Long.toHexString, absent from the CLDC bootclasspath the Ant core build compiles against; format the content-hash by hand (verified with ant compile). - CN1SurfaceStore's unreachable UnsupportedEncodingException fallback called default-encoding String.getBytes(), tripping the forbidden SpotBugs DM_DEFAULT_ENCODING gate; throw instead. - cn1_windows_widgets.cpp used the MSVC STL, which demands Clang 19+ under xwin while the cross-compile CI's clang-cl is older (STL1000); replaced std::deque/string/vector with a C event ring and an owned RECT array, matching how cn1_windows_browser.cpp guards its STL. - The widget bridges' isSystemDark returned a freshly merged boolean from inside a translated try block, which ICEs Alpine 3.20 gcc with SSA corruption on the generated C (reproduced in Docker against the exact compiler: old shape ICEs, new assign-in-try shape compiles clean); restructured both the Linux and Windows twins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
✅ Continuous Quality ReportTest & Coverage
Static Analysis
Generated automatically by the PR CI workflow. |
|
Compared 12 screenshots: 12 matched. |
|
Compared 144 screenshots: 144 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
|
Compared 140 screenshots: 140 matched. |
|
Compared 217 screenshots: 217 matched. |
… fidelity, image GC - areWidgetsSupported() on iOS now gates on the CN1Widgets extension's actual deployment target (new CN1SurfacesMinOS plist key injected by the builder, default 16.1) instead of WidgetKit's iOS 14 floor, so iOS 14-15 no longer accepts publications for a gallery entry that cannot exist. - Android live-activity updates replace the persisted state map wholesale instead of merging, matching every other platform; omitted keys no longer linger. - The desktop rasterizer honors the alpha byte verbatim (fully transparent explicit colors draw nothing), matching the iOS/Android renderers; SurfaceColor.rgb docs now spell out the 0xAARRGGBB contract. - Publishing garbage collects content-hash image blobs the replacement timeline no longer references, on every port (iOS app group, Android files dir, JavaSE/Windows/Linux stores). The serializer's images list now names the COMPLETE reference set including registered-name reuse, which also fixes registered-name references dropping out of the desktop bridges' in-memory image maps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ials) A small vector-op catalog (rects, ellipses, arcs, lines, polygons, anchored text) with rotation groups whose angle can come from the state map, so an analog clock is a static face plus per-minute timeline entries carrying hand angles. Lowered to SwiftUI Canvas on iOS, an in-process anti-aliased Bitmap on Android (RemoteViews cannot draw vectors) and CN1 Graphics shapes with a software affine on desktop. Wire convention: degrees, 0 = 12 o'clock, clockwise; each renderer documents its arc-API conversion. The sample gains an analog-clock widget kind. HTML subset rejected: neither RemoteViews nor widget extensions can host a web engine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…get pull Audited the publish path on every port for background safety (Android publish works from the UI-less BackgroundFetchHandler service context; iOS is app-group file IO plus thread-safe WidgetCenter reload; desktop bridges never block on the EDT) and documented the any-thread contract on Surfaces.publish. New pull direction on Android: when a widget renders with no timeline or an exhausted atEnd timeline, the provider starts the app's declared BackgroundFetch (throttled to once per 15 minutes per kind) so the app republishes fresh content without any UI. spi/package-info and BackgroundFetch document the per-platform story. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five suite tests exercise the framework on the real device VMs: serializer round-trip (all node types, per-size layouts, image blobs), SurfaceRasterizer screenshot (deterministic pinned-clock descriptor, light + dark, bit-identical across runs), timeline logic, action dispatch cold-start ordering on the EDT, and publish/reload no-op safety. The suite app now ships a surfaces.json so the iOS/Android build lowering is exercised by the platform CI legs. Golden screenshots must be seeded from each leg's first run per the screenshots README process. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Covers the concept and dead-process rule, surfaces.json and first publish, the node catalog with the LCD contract, the SurfaceVector analog clock, live activities and Dynamic Island regions, actions and cold start, background refresh, per-platform notes and the build-hint table. Screenshots captured live from the simulator Widgets preview. Passes all guide gates (asciidoctor, Vale, paragraph cap, snippet validation, LanguageTool) at zero findings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Second round pushed ( Review findings
Design feedback
|
|
Developer Guide build artifacts are available for download from this workflow run:
Developer Guide quality checks: |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… Swift sources The schemes ruby re-runs after pods integration; its global deployment-target pass iterated every target, so on the second pass it stomped the already-created CN1Widgets extension down to the app's iOS 14 -- WidgetKit sources then failed to compile (GraphicsContext is 15+). The pass now skips app-extension product types, which own their deployment targets (this also protected the wallet extension). Defense in depth: the renderer sources now compile at ANY deployment target the generated project ends up with -- Canvas/GraphicsContext, Text(timerInterval:), ProgressView(timerInterval:), Gauge and tint are all #available-guarded with the documented degradations as fallbacks (verified with swiftc -typecheck at ios14.0 and ios16.1 targets). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…es tests on JS Goldens seeded from the branch's first CI artifacts and visually verified (light + dark widget tiles, countdown, progress, arc, action outline): Android legs are pixel-identical across JDKs, Linux x64 and arm64 are identical, Windows x64/cross are identical with the arm64 capture inside the 2-pixel tolerance. iOS and Mac goldens follow once those legs build with the deployment-target fix. The JS port fails the serializer round-trip (JSONParser returns mangled tokens, a runtime string/JSON bug rather than a surfaces bug) and NPEs in the rasterizer screenshot setup; both tests are parked on JS per the port.js convention pending a JS runtime fix. All other platform legs gate them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
✅ ByteCodeTranslator Quality ReportTest & Coverage
Benchmark Results
Static Analysis
Generated automatically by the PR CI workflow. |
…natives) The ParparVM iOS runtime has no native implementations for java.io.File's mutating methods; referencing exists/delete/list/mkdir/ renameTo from IOSSurfaceBridge failed the native link with five undefined java_io_File_*Impl symbols (and would have failed at runtime regardless). All bridge file IO now goes through FileSystemStorage, whose iOS implementation tolerates the App Group container's plain absolute paths, with a segment-by-segment mkdirs helper and rename-within-directory for the atomic timeline swap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ActivityKit can be imported on Mac Catalyst but ActivityAttributes and Activity are marked unavailable there, so canImport alone let the app-target glue reach the Catalyst compile and fail; every ActivityKit guard now also excludes targetEnvironment(macCatalyst) (verified with a macabi typecheck of the previously failing file). Seeds the four visually verified iOS-family SurfacesRasterizer goldens (GL, Metal, tvOS, watchOS); the Mac golden follows once its leg builds with this fix. Also swaps the string concat in IOSSurfaceBridge.mkdirs for a StringBuilder per the SpotBugs gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Captured by the first mac-native run after the Catalyst ActivityKit guards landed; visually verified (light + dark tiles, countdown, timer-up, progress bar, arc, action outline). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d JSON numbers
The append(Object) native converted heap objects with the synchronous
toNativeString fallback ('' + value), so boxed numbers rendered as the
literal string [object Object] -- JSONWriter emitted corrupt documents
and every JSONParser round-trip on the JS port mangled (the parked
surfacesJsonRoundTrip / surfacesRasterizerNpe failures, plus the
silently failing timeline-logic test, were all this one defect). The
native now dispatches the real Java toString() through the generator
path; strings/null keep the sync fast path. Both surfaces tests are
un-parked, the JS SurfacesRasterizer golden is seeded from a
CI-calibrated Linux container capture, and a full-suite A/B shows all
135 existing screenshots byte-identical with zero regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… test The exactAlarms build hint (promised in the hint table) now injects SCHEDULE_EXACT_ALARM plus a manifest meta-data flag the provider reads; timeline entry flips use setExactAndAllowWhileIdle when the user has not revoked exact-alarm access, falling back to the inexact window otherwise. A new Android-only cn1ss test exercises the REAL RemoteViews lowering on the emulator legs: it publishes through AndroidSurfaceBridge, renders light + dark via CN1SurfaceRenderer, applies the RemoteViews to live views shown through a PeerComponent and screenshots the result; Chronometer lowering is asserted without a screenshot for determinism. Skips cleanly on every other platform (verified on JavaSE). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A new windows-runner job compiles cn1_windows_widgetboard.cpp with CN1_WIDGETBOARD=1 against the real Windows App SDK (nuget winmds + cppwinrt-generated projections) using clang-cl -- the production toolchain -- plus a second-toolchain compile of the floating-widget TU. Static verification against the winmd and Microsoft Learn found one real defect (missing explicit unknwn.h include enabling classic-COM interop in C++/WinRT) and builder layout bugs: the WinAppSDK nupkg ships winmds only (no prebuilt projections) and puts Bootstrap.lib/dll under lib/win10-<arch> and runtimes/win10-<arch>/native, which resolveWinAppSdkDir/widgetBoardLinkFlags/the MSIX packer now accept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On layer-shell compositors (Sway/Hyprland/Wayfire/KDE) widget windows now anchor as real desktop applets: LAYER_BOTTOM above the wallpaper with left/top margins carrying the persisted position (the pill docks LAYER_TOP centered), keyboard-mode NONE and zero exclusive zone. begin_move_drag does not work on layer surfaces, so a press outside the hit-rects starts a manual margin-updating drag off GTK's implicit grab. The library binds lazily via dlopen (services.c convention); absence or gtk_layer_is_supported()==false leaves the X11/EWMH path byte-for-byte unchanged. Compile-verified at -O2 in an Alpine GTK container. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The suite's surfaces.json now enables liveActivities, so the ActivityKit lowering (CN1LiveActivityWidget.swift, NSSupportsLiveActivities, the Android ongoing-notification manager) is compiled and linked by every platform build instead of only shipping untested. SurfacesPublishTest runs the full start/update/end lifecycle: inert-handle semantics where the platform refuses, the real surface (notification, desktop pill) where supported, with no support value asserted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Running the floating widgets in a real (skinless) desktop app exposed three bugs: widget clicks never focused the app because the port's window field is only set on simulator paths (now falls back to the canvas's top-level ancestor), toFront() could not raise the app over other macOS apps (now also requests foreground activation via java.awt.Desktop when available), and default placement stacked every widget on the same top-right anchor (new windows now cascade below intersecting ones). Verified end to end: rendering, countdown ticks and timeline flips over 70s, drag, position persistence across relaunches, and click-through to the action form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…header isolation The suite settings rewrite in the exactAlarms commit accidentally added an ios.NSMicrophoneUsageDescription, which defines INCLUDE_MICROPHONE_USAGE and compiles requestRecordPermission: -- unavailable on tvOS/watchOS, breaking those legs. The file is restored to its previous content. The new widgetboard compile check did its job and caught two real problems: C++/WinRT at C++17 falls back to <experimental/coroutine>, which the modern MSVC STL rejects under clang (STL1009), and including cn1_globals.h (C11 stdatomic) alongside the winrt STL headers conflicts on atomic fences. The widgetboard TU no longer includes cn1_windows.h (it only needed the logger, now forward-declared with C linkage) and compiles at C++20: the translator's generated CMakeLists now defaults CMAKE_CXX_STANDARD only when unset and WindowsNativeBuilder passes 20 for windows.msix=true builds (both branches; CI step matches). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The wizard now handles the App Group nuance that widgets/Live Activities need. When a project uses external surfaces (surfaces.json present), the one-click auto-setup: finds-or-creates the App Group (group.<package> or the surfaces.json override), enables the APP_GROUPS capability on the main and <package>.CN1Widgets App IDs associated with that group, creates + downloads a distribution profile for the extension, and writes codename1.arg.ios.surfaces.appGroup + codename1.ios.appext.CN1Widgets.provision into the project settings. A manual bundle dialog also gets an App Groups checkbox. The OpenAPI contract gains /app-groups (create/list) and an optional appGroupIds on the capability request; SigningState tracks app groups; MockSigningService + tests cover the full surfaces flow. CN1BuildMojo carries the generated extension's provisioning profile to the cloud build: any codename1.ios.appext.<Name>.provision file is base64-encoded into the ios.appext.<Name>.provisioningData build arg (local Xcode builds use automatic signing and are untouched). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Native fidelity (Android, Material 3)54 pairs compared -- median 95.6%, worst 91.3% ( Distribution --
Geometry vs native (bbox offset / size ratio / center offset / corner radius) -- gated separately from the visual score
Side-by-side comparisons (worst first)
|
Native fidelity (iOS Modern, Metal)68 pairs compared -- median 94.4%, worst 83.5% ( Distribution --
Geometry vs native (bbox offset / size ratio / center offset / corner radius) -- gated separately from the visual score
Side-by-side comparisons (worst first)
|
1 participant
Summary
A unified declarative API for external surfaces -- live information that resides outside the app. One concept, one API family, covering:
CN1Widgetsextension with a generic SwiftUI renderer), Android app widgets (generic RemoteViews renderer, no Compose/Glance dependency), desktop floating widgets (JavaSE/Windows/Linux) and the Windows 11 Widgets Board (MSIX + Adaptive Cards, behindwindows.msix=true).Architecture
Follows the
com.codename1.carpattern (sealed node catalog, fluent builders, SPI bridge, bytecode-scan build gating, zero cost when unused) with one crucial difference: surfaces render while the app process is dead. Descriptors serialize in core to one canonical JSON wire format plus content-hash-named PNG blobs (SurfaceSerializeronJSONWriter); layouts embed${key}placeholders resolved from per-entry state maps, so updates ship data, not layouts. Interactions are string action ids delivered to a registered handler on the EDT with cold-start queueing.SurfaceDynamicTextprovides OS-native countdown/elapsed text (WidgetKitText(date, style:)/ AndroidChronometer) -- timers and ETAs tick with zero app wakeups.Widget kinds are declared at build time in a
surfaces.jsonproject resource (widget galleries are compiled into native apps) and mirrored at runtime viaSurfaces.registerWidgetKind. Refresh is app-driven (publish +BackgroundFetch); the wire format is push-ready for a future server-driven update path.Verification
SurfacesSample(delivery tracker) verified end-to-end in the simulator: Widgets preview rendered the timeline with a ticking countdown, entries flipped on schedule, the mock Dynamic Island tracked live-activity updates, and clicking a widget hit-rect routed throughSurfaces.dispatchActionto the app's handler.arm64-apple-ios16.1-simulator(extension) andx86_64-apple-ios12.0-simulator(app-target glue).Notes for reviewers
<bundleId>.CN1Widgets; automatic signing handles it locally, cloud manual signing usesios.appext.CN1Widgets.provisioningURL. The daemon entitlement gate was widened accordingly.com.codename1.surfacespackage-info (LCD contract table): Android approximations (weights collapse to equal shares, date/relative dynamic text static, circular progress falls back to linear), small iOS widgets honor only the root action, Wayland floating-window degradation.🤖 Generated with Claude Code