From 0e8d49af8993d2f2dc3ea3e5d2858217e0888317 Mon Sep 17 00:00:00 2001 From: Ivan Sekovanikj <31964049+isekovanic@users.noreply.github.com> Date: Wed, 14 Oct 2026 11:19:57 +0200 Subject: [PATCH 1/2] feat: ai-components-js port (#3856) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## ๐ŸŽฏ Goal Deprecate the standalone `@stream-io/chat-react-native-ai` package and ship the AI components from the SDK itself, the same way React did in GetStream/stream-chat-react#3314. This is a direct port of [this part of the AI components repo](https://github.com/GetStream/ai-components-js/tree/main/packages/react-native-sdk), with some minor tweaks and fixes. Apps import them from a separate entry point: - `stream-chat-react-native/ai-components` - `stream-chat-expo/ai-components` Apps that don't import it are unaffected: no extra JS in their bundle (+61 bytes, one empty native-handler slot) and no new required dependencies. ## ๐Ÿ›  Implementation details **Port.** Markdown, syntax highlighting, charts, the AI composer and dictation are ported 1:1. The only deliberate changes are: - `ShimmeringView` is removed; the typing indicator uses the SDK's native shimmer, laid over the text. - If the dictation module isn't compiled in, the mic button hides itself instead of crashing. - `react-syntax-highlighter` moves to v16, for the Prism CVE. - The native code is registered by the two wrappers instead of a separate package. **Isolation.** `validate-entry-isolation` (part of `yarn lint`) fails if the main entry points ever reach the AI code or its dependencies. It also fails if the AI entry reaches the chat SDK, so AI-only apps don't bundle chat. Both directions are confirmed with real Metro bundles. **Dependencies.** The JS-only libraries (zod, the highlighter, simple-markdown, state-store) are regular dependencies. Skia and victory-native are optional peers, needed only by apps that use the AI components. **Native.** - **PerfText:** an Android Fabric text view with a C++ shadow node, built through a custom CMakeLists in each wrapper. - **AIDictation:** a TurboModule for both platforms. - **iOS dictation is opt-in:** enable it with `$StreamChatEnableDictation = true` in the Podfile, or with the new `stream-chat-expo` config plugin. Referencing the Speech framework makes App Store Connect require a speech-recognition purpose string from every app, so it isn't compiled in by default. - **Android:** the library never declares `RECORD_AUDIO`. **Fixes on top of the port**, mostly bugs inherited from upstream: - A final dictation result arriving after recording stops is now applied to the input. - Android no longer cuts users off mid-sentence when the recognizer sends no live results. - The iOS first press no longer races the permission prompts. - Per-method lodash imports (-45.6 KB min) and direct `react-syntax-highlighter` imports (-369 KB min), both in AI bundles only. - The native shimmer now links on RN 0.76 iOS (`StreamShimmerViewCls`), an existing v9 issue. **Samples.** SampleApp gets an "AI Assistant" screen. The Expo ChatGPT sample from `ai-components-js` is ported as `examples/ExpoChatGPTSample` (Expo 57). **Migration.** See `package/src/ai-components/README.md`: the new import path, the Podfile flag and plugin rename, the Babel plugin `zod` needs, and the removed exports. ## ๐Ÿงช Testing ## ๐ŸŽจ UI Changes --------- Co-authored-by: Anton Arnautov --- AGENTS.md | 13 + examples/ExpoChatGPTSample/.gitignore | 40 ++ examples/ExpoChatGPTSample/README.md | 30 + examples/ExpoChatGPTSample/app.json | 32 + examples/ExpoChatGPTSample/app/_layout.tsx | 53 ++ examples/ExpoChatGPTSample/app/index.tsx | 6 + examples/ExpoChatGPTSample/babel.config.js | 12 + .../ExpoChatGPTSample/bottomSheetOptions.tsx | 45 ++ examples/ExpoChatGPTSample/chatConfig.ts | 5 + .../ExpoChatGPTSample/contexts/AppContext.tsx | 42 ++ examples/ExpoChatGPTSample/custom-types.d.ts | 42 ++ examples/ExpoChatGPTSample/http/api.ts | 19 + examples/ExpoChatGPTSample/http/requests.ts | 12 + examples/ExpoChatGPTSample/metro.config.js | 74 +++ examples/ExpoChatGPTSample/package.json | 44 ++ .../ExpoChatGPTSample/screens/ChatContent.tsx | 277 ++++++++ .../ExpoChatGPTSample/screens/MenuDrawer.tsx | 95 +++ examples/ExpoChatGPTSample/tsconfig.json | 12 + examples/SampleApp/fastlane/Fastfile | 2 + examples/SampleApp/ios/Podfile.lock | 4 +- package.json | 10 +- package/ai-components/package.json | 6 + package/bin/validate-entry-isolation.js | 224 +++++++ package/expo-package/ai-components/index.d.ts | 1 + package/expo-package/ai-components/index.js | 21 + .../streamchatexpo/StreamChatExpoPackage.java | 22 +- .../android/src/main/jni/CMakeLists.txt | 63 ++ .../android/src/main/jni/StreamChatExpoSpec.h | 9 + .../src/main/jni/StreamCodegenComponents.h | 7 + .../com/streamchatexpo/AIDictationModule.kt | 49 ++ package/expo-package/app.plugin.js | 1 + package/expo-package/package.json | 15 +- package/expo-package/plugin/index.js | 43 ++ .../plugin/withAndroidManifest.js | 30 + .../plugin/withPodfileProperties.js | 13 + .../expo-package/plugin/withiOSInfoPlist.js | 35 + package/expo-package/react-native.config.js | 8 + .../src/native/NativeAIDictation.ts | 48 ++ .../src/native/PerfTextNativeComponent.ts | 30 + package/expo-package/stream-chat-expo.podspec | 27 + package/jest-setup.tsx | 25 + .../native-package/ai-components/index.d.ts | 1 + package/native-package/ai-components/index.js | 21 + .../StreamChatReactNativePackage.java | 22 +- .../android/src/main/jni/CMakeLists.txt | 63 ++ .../src/main/jni/StreamChatReactNativeSpec.h | 9 + .../src/main/jni/StreamCodegenComponents.h | 7 + .../AIDictationModule.kt | 49 ++ package/native-package/package.json | 13 +- package/native-package/react-native.config.js | 8 + .../src/native/NativeAIDictation.ts | 48 ++ .../src/native/PerfTextNativeComponent.ts | 30 + .../stream-chat-react-native.podspec | 27 + package/package.json | 26 +- package/shared-native/android/PerfTextView.kt | 61 ++ .../android/PerfTextViewManager.kt | 143 ++++ .../dictation/AIDictationController.kt | 514 +++++++++++++++ .../android/jni/PerfTextComponentDescriptor.h | 29 + .../jni/PerfTextMeasurementManager.cpp | 70 ++ .../android/jni/PerfTextMeasurementManager.h | 25 + .../android/jni/PerfTextShadowNode.cpp | 20 + .../android/jni/PerfTextShadowNode.h | 39 ++ package/shared-native/ios/AIDictation.h | 20 + package/shared-native/ios/AIDictation.mm | 114 ++++ .../shared-native/ios/AIDictationModule.swift | 566 ++++++++++++++++ .../ios/StreamShimmerViewComponentView.mm | 9 + .../ai-components/AITypingIndicatorView.tsx | 51 ++ package/src/ai-components/README.md | 279 ++++++++ .../ai-components/StreamingMessageView.tsx | 85 +++ .../__tests__/ComposerView.test.tsx | 113 ++++ .../ai-components/__tests__/Markdown.test.tsx | 94 +++ .../__tests__/StreamingMessageView.test.tsx | 126 ++++ .../__tests__/SyntaxHighlighter.test.tsx | 136 ++++ .../ai-components/__tests__/charts.test.ts | 131 ++++ .../__tests__/dictation.test.tsx | 166 +++++ .../__tests__/generateMarkdownText.test.ts | 54 ++ .../ai-components/__tests__/theme.test.tsx | 41 ++ package/src/ai-components/charts/Chart.tsx | 20 + .../charts/components/PieLegend.tsx | 50 ++ .../charts/components/VictoryChart.tsx | 193 ++++++ .../ai-components/charts/components/index.ts | 2 + package/src/ai-components/charts/index.ts | 13 + .../src/ai-components/charts/mermaid/index.ts | 1 + .../ai-components/charts/mermaid/parser.ts | 33 + package/src/ai-components/charts/types.ts | 25 + .../charts/utils/generate-pie-colors.ts | 82 +++ .../src/ai-components/charts/utils/index.ts | 2 + .../charts/utils/parse-json-chart.ts | 15 + .../ai-components/charts/vega-lite/index.ts | 2 + .../ai-components/charts/vega-lite/parser.ts | 49 ++ .../ai-components/charts/vega-lite/schema.ts | 112 ++++ .../ai-components/components/BottomSheet.tsx | 158 +++++ .../components/MarkdownReactiveScrollView.tsx | 92 +++ package/src/ai-components/components/index.ts | 2 + .../contexts/composer-context.tsx | 73 +++ package/src/ai-components/contexts/index.ts | 2 + .../ai-components/contexts/theme-context.tsx | 68 ++ package/src/ai-components/index.ts | 29 + .../internal/hooks/useStableCallback.ts | 37 ++ .../ai-components/internal/icons/Camera.tsx | 12 + .../internal/icons/CircleStop.tsx | 18 + .../ai-components/internal/icons/Close.tsx | 12 + .../src/ai-components/internal/icons/Mic.tsx | 20 + .../ai-components/internal/icons/Picture.tsx | 16 + .../ai-components/internal/icons/SendUp.tsx | 21 + .../internal/icons/utils/base.tsx | 44 ++ .../markdown/components/Autolink.tsx | 26 + .../markdown/components/BlockQuote.tsx | 19 + .../markdown/components/Bold.tsx | 16 + .../markdown/components/CodeBlock.tsx | 96 +++ .../markdown/components/Emphasis.tsx | 16 + .../markdown/components/Heading.tsx | 30 + .../markdown/components/HorizontalRule.tsx | 9 + .../markdown/components/InlineCode.tsx | 16 + .../markdown/components/LineBreak.tsx | 11 + .../markdown/components/Link.tsx | 28 + .../markdown/components/List.tsx | 65 ++ .../markdown/components/Mailto.tsx | 26 + .../markdown/components/Markdown.tsx | 59 ++ .../markdown/components/NewLine.tsx | 11 + .../markdown/components/Paragraph.tsx | 69 ++ .../markdown/components/Strikethrough.tsx | 16 + .../markdown/components/Table.tsx | 81 +++ .../markdown/components/Text.tsx | 37 ++ .../ai-components/markdown/components/Url.tsx | 26 + .../markdown/components/index.ts | 20 + package/src/ai-components/markdown/index.tsx | 5 + package/src/ai-components/markdown/rules.ts | 216 ++++++ package/src/ai-components/markdown/styles.ts | 236 +++++++ package/src/ai-components/markdown/types.ts | 163 +++++ .../markdown/utils/generate-markdown-text.ts | 117 ++++ .../src/ai-components/markdown/utils/index.ts | 1 + .../message-composer/ActionButton.tsx | 99 +++ .../message-composer/ActionSheet.tsx | 205 ++++++ .../message-composer/AttachmentButton.tsx | 49 ++ .../message-composer/ComposerView.tsx | 187 ++++++ .../message-composer/MediaPreviewList.tsx | 84 +++ .../message-composer/SpeechToTextButton.tsx | 79 +++ .../ai-components/message-composer/index.ts | 5 + .../message-composer/utils/withCloseSheet.ts | 11 + .../native-specs/NativeAIDictation.ts | 48 ++ .../src/ai-components/native/AIDictation.ts | 77 +++ package/src/ai-components/native/PerfText.tsx | 32 + .../services/clipboard-service/expo.ts | 11 + .../services/clipboard-service/index.ts | 4 + .../services/clipboard-service/rncli.ts | 11 + package/src/ai-components/services/index.ts | 2 + .../AbstractMediaPickerService.ts | 62 ++ .../services/media-picker-service/expo.ts | 148 +++++ .../hooks/useMediaPickerState.ts | 14 + .../services/media-picker-service/index.ts | 4 + .../optional-modules.d.ts | 25 + .../services/media-picker-service/rncli.ts | 142 ++++ .../ai-components/store/bottom-sheet/store.ts | 30 + .../store/bottom-sheet/useBottomSheetState.ts | 10 + .../src/ai-components/store/composer/store.ts | 12 + .../store/composer/useComposerHasText.ts | 14 + .../store/composer/useComposerText.ts | 14 + .../ai-components/store/dictation/store.ts | 23 + .../store/dictation/useDictationState.ts | 11 + .../store/dictation/useDictationTranscript.ts | 9 + package/src/ai-components/store/index.ts | 11 + .../syntax-highlighting/SyntaxHighlighter.tsx | 66 ++ .../components/AndroidNativeRenderer.tsx | 29 + .../syntax-highlighting/components/index.ts | 1 + .../syntax-highlighting/index.ts | 11 + .../syntax-highlighting/prism-config.ts | 4 + .../syntax-highlighting/types.ts | 26 + .../syntax-highlighting/utils/converter.ts | 145 +++++ .../utils/flatten-nodes.ts | 150 +++++ .../syntax-highlighting/utils/index.ts | 3 + .../syntax-highlighting/utils/renderer.tsx | 151 +++++ package/src/ai-components/theme.ts | 138 ++++ .../transcription/useDictation.ts | 106 +++ package/src/native.ts | 20 +- yarn.lock | 614 +++++++++++++++++- 176 files changed, 10249 insertions(+), 22 deletions(-) create mode 100644 examples/ExpoChatGPTSample/.gitignore create mode 100644 examples/ExpoChatGPTSample/README.md create mode 100644 examples/ExpoChatGPTSample/app.json create mode 100644 examples/ExpoChatGPTSample/app/_layout.tsx create mode 100644 examples/ExpoChatGPTSample/app/index.tsx create mode 100644 examples/ExpoChatGPTSample/babel.config.js create mode 100644 examples/ExpoChatGPTSample/bottomSheetOptions.tsx create mode 100644 examples/ExpoChatGPTSample/chatConfig.ts create mode 100644 examples/ExpoChatGPTSample/contexts/AppContext.tsx create mode 100644 examples/ExpoChatGPTSample/custom-types.d.ts create mode 100644 examples/ExpoChatGPTSample/http/api.ts create mode 100644 examples/ExpoChatGPTSample/http/requests.ts create mode 100644 examples/ExpoChatGPTSample/metro.config.js create mode 100644 examples/ExpoChatGPTSample/package.json create mode 100644 examples/ExpoChatGPTSample/screens/ChatContent.tsx create mode 100644 examples/ExpoChatGPTSample/screens/MenuDrawer.tsx create mode 100644 examples/ExpoChatGPTSample/tsconfig.json create mode 100644 package/ai-components/package.json create mode 100644 package/bin/validate-entry-isolation.js create mode 100644 package/expo-package/ai-components/index.d.ts create mode 100644 package/expo-package/ai-components/index.js create mode 100644 package/expo-package/android/src/main/jni/CMakeLists.txt create mode 100644 package/expo-package/android/src/main/jni/StreamChatExpoSpec.h create mode 100644 package/expo-package/android/src/main/jni/StreamCodegenComponents.h create mode 100644 package/expo-package/android/src/newarch/com/streamchatexpo/AIDictationModule.kt create mode 100644 package/expo-package/app.plugin.js create mode 100644 package/expo-package/plugin/index.js create mode 100644 package/expo-package/plugin/withAndroidManifest.js create mode 100644 package/expo-package/plugin/withPodfileProperties.js create mode 100644 package/expo-package/plugin/withiOSInfoPlist.js create mode 100644 package/expo-package/src/native/NativeAIDictation.ts create mode 100644 package/expo-package/src/native/PerfTextNativeComponent.ts create mode 100644 package/native-package/ai-components/index.d.ts create mode 100644 package/native-package/ai-components/index.js create mode 100644 package/native-package/android/src/main/jni/CMakeLists.txt create mode 100644 package/native-package/android/src/main/jni/StreamChatReactNativeSpec.h create mode 100644 package/native-package/android/src/main/jni/StreamCodegenComponents.h create mode 100644 package/native-package/android/src/newarch/com/streamchatreactnative/AIDictationModule.kt create mode 100644 package/native-package/src/native/NativeAIDictation.ts create mode 100644 package/native-package/src/native/PerfTextNativeComponent.ts create mode 100644 package/shared-native/android/PerfTextView.kt create mode 100644 package/shared-native/android/PerfTextViewManager.kt create mode 100644 package/shared-native/android/dictation/AIDictationController.kt create mode 100644 package/shared-native/android/jni/PerfTextComponentDescriptor.h create mode 100644 package/shared-native/android/jni/PerfTextMeasurementManager.cpp create mode 100644 package/shared-native/android/jni/PerfTextMeasurementManager.h create mode 100644 package/shared-native/android/jni/PerfTextShadowNode.cpp create mode 100644 package/shared-native/android/jni/PerfTextShadowNode.h create mode 100644 package/shared-native/ios/AIDictation.h create mode 100644 package/shared-native/ios/AIDictation.mm create mode 100644 package/shared-native/ios/AIDictationModule.swift create mode 100644 package/src/ai-components/AITypingIndicatorView.tsx create mode 100644 package/src/ai-components/README.md create mode 100644 package/src/ai-components/StreamingMessageView.tsx create mode 100644 package/src/ai-components/__tests__/ComposerView.test.tsx create mode 100644 package/src/ai-components/__tests__/Markdown.test.tsx create mode 100644 package/src/ai-components/__tests__/StreamingMessageView.test.tsx create mode 100644 package/src/ai-components/__tests__/SyntaxHighlighter.test.tsx create mode 100644 package/src/ai-components/__tests__/charts.test.ts create mode 100644 package/src/ai-components/__tests__/dictation.test.tsx create mode 100644 package/src/ai-components/__tests__/generateMarkdownText.test.ts create mode 100644 package/src/ai-components/__tests__/theme.test.tsx create mode 100644 package/src/ai-components/charts/Chart.tsx create mode 100644 package/src/ai-components/charts/components/PieLegend.tsx create mode 100644 package/src/ai-components/charts/components/VictoryChart.tsx create mode 100644 package/src/ai-components/charts/components/index.ts create mode 100644 package/src/ai-components/charts/index.ts create mode 100644 package/src/ai-components/charts/mermaid/index.ts create mode 100644 package/src/ai-components/charts/mermaid/parser.ts create mode 100644 package/src/ai-components/charts/types.ts create mode 100644 package/src/ai-components/charts/utils/generate-pie-colors.ts create mode 100644 package/src/ai-components/charts/utils/index.ts create mode 100644 package/src/ai-components/charts/utils/parse-json-chart.ts create mode 100644 package/src/ai-components/charts/vega-lite/index.ts create mode 100644 package/src/ai-components/charts/vega-lite/parser.ts create mode 100644 package/src/ai-components/charts/vega-lite/schema.ts create mode 100644 package/src/ai-components/components/BottomSheet.tsx create mode 100644 package/src/ai-components/components/MarkdownReactiveScrollView.tsx create mode 100644 package/src/ai-components/components/index.ts create mode 100644 package/src/ai-components/contexts/composer-context.tsx create mode 100644 package/src/ai-components/contexts/index.ts create mode 100644 package/src/ai-components/contexts/theme-context.tsx create mode 100644 package/src/ai-components/index.ts create mode 100644 package/src/ai-components/internal/hooks/useStableCallback.ts create mode 100644 package/src/ai-components/internal/icons/Camera.tsx create mode 100644 package/src/ai-components/internal/icons/CircleStop.tsx create mode 100644 package/src/ai-components/internal/icons/Close.tsx create mode 100644 package/src/ai-components/internal/icons/Mic.tsx create mode 100644 package/src/ai-components/internal/icons/Picture.tsx create mode 100644 package/src/ai-components/internal/icons/SendUp.tsx create mode 100644 package/src/ai-components/internal/icons/utils/base.tsx create mode 100644 package/src/ai-components/markdown/components/Autolink.tsx create mode 100644 package/src/ai-components/markdown/components/BlockQuote.tsx create mode 100644 package/src/ai-components/markdown/components/Bold.tsx create mode 100644 package/src/ai-components/markdown/components/CodeBlock.tsx create mode 100644 package/src/ai-components/markdown/components/Emphasis.tsx create mode 100644 package/src/ai-components/markdown/components/Heading.tsx create mode 100644 package/src/ai-components/markdown/components/HorizontalRule.tsx create mode 100644 package/src/ai-components/markdown/components/InlineCode.tsx create mode 100644 package/src/ai-components/markdown/components/LineBreak.tsx create mode 100644 package/src/ai-components/markdown/components/Link.tsx create mode 100644 package/src/ai-components/markdown/components/List.tsx create mode 100644 package/src/ai-components/markdown/components/Mailto.tsx create mode 100644 package/src/ai-components/markdown/components/Markdown.tsx create mode 100644 package/src/ai-components/markdown/components/NewLine.tsx create mode 100644 package/src/ai-components/markdown/components/Paragraph.tsx create mode 100644 package/src/ai-components/markdown/components/Strikethrough.tsx create mode 100644 package/src/ai-components/markdown/components/Table.tsx create mode 100644 package/src/ai-components/markdown/components/Text.tsx create mode 100644 package/src/ai-components/markdown/components/Url.tsx create mode 100644 package/src/ai-components/markdown/components/index.ts create mode 100644 package/src/ai-components/markdown/index.tsx create mode 100644 package/src/ai-components/markdown/rules.ts create mode 100644 package/src/ai-components/markdown/styles.ts create mode 100644 package/src/ai-components/markdown/types.ts create mode 100644 package/src/ai-components/markdown/utils/generate-markdown-text.ts create mode 100644 package/src/ai-components/markdown/utils/index.ts create mode 100644 package/src/ai-components/message-composer/ActionButton.tsx create mode 100644 package/src/ai-components/message-composer/ActionSheet.tsx create mode 100644 package/src/ai-components/message-composer/AttachmentButton.tsx create mode 100644 package/src/ai-components/message-composer/ComposerView.tsx create mode 100644 package/src/ai-components/message-composer/MediaPreviewList.tsx create mode 100644 package/src/ai-components/message-composer/SpeechToTextButton.tsx create mode 100644 package/src/ai-components/message-composer/index.ts create mode 100644 package/src/ai-components/message-composer/utils/withCloseSheet.ts create mode 100644 package/src/ai-components/native-specs/NativeAIDictation.ts create mode 100644 package/src/ai-components/native/AIDictation.ts create mode 100644 package/src/ai-components/native/PerfText.tsx create mode 100644 package/src/ai-components/services/clipboard-service/expo.ts create mode 100644 package/src/ai-components/services/clipboard-service/index.ts create mode 100644 package/src/ai-components/services/clipboard-service/rncli.ts create mode 100644 package/src/ai-components/services/index.ts create mode 100644 package/src/ai-components/services/media-picker-service/AbstractMediaPickerService.ts create mode 100644 package/src/ai-components/services/media-picker-service/expo.ts create mode 100644 package/src/ai-components/services/media-picker-service/hooks/useMediaPickerState.ts create mode 100644 package/src/ai-components/services/media-picker-service/index.ts create mode 100644 package/src/ai-components/services/media-picker-service/optional-modules.d.ts create mode 100644 package/src/ai-components/services/media-picker-service/rncli.ts create mode 100644 package/src/ai-components/store/bottom-sheet/store.ts create mode 100644 package/src/ai-components/store/bottom-sheet/useBottomSheetState.ts create mode 100644 package/src/ai-components/store/composer/store.ts create mode 100644 package/src/ai-components/store/composer/useComposerHasText.ts create mode 100644 package/src/ai-components/store/composer/useComposerText.ts create mode 100644 package/src/ai-components/store/dictation/store.ts create mode 100644 package/src/ai-components/store/dictation/useDictationState.ts create mode 100644 package/src/ai-components/store/dictation/useDictationTranscript.ts create mode 100644 package/src/ai-components/store/index.ts create mode 100644 package/src/ai-components/syntax-highlighting/SyntaxHighlighter.tsx create mode 100644 package/src/ai-components/syntax-highlighting/components/AndroidNativeRenderer.tsx create mode 100644 package/src/ai-components/syntax-highlighting/components/index.ts create mode 100644 package/src/ai-components/syntax-highlighting/index.ts create mode 100644 package/src/ai-components/syntax-highlighting/prism-config.ts create mode 100644 package/src/ai-components/syntax-highlighting/types.ts create mode 100644 package/src/ai-components/syntax-highlighting/utils/converter.ts create mode 100644 package/src/ai-components/syntax-highlighting/utils/flatten-nodes.ts create mode 100644 package/src/ai-components/syntax-highlighting/utils/index.ts create mode 100644 package/src/ai-components/syntax-highlighting/utils/renderer.tsx create mode 100644 package/src/ai-components/theme.ts create mode 100644 package/src/ai-components/transcription/useDictation.ts diff --git a/AGENTS.md b/AGENTS.md index f49f37c2ff..2b9e6679c3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -62,6 +62,7 @@ Respect repo-specific rules. Do not suppress lint rules broadly; justify and sco - `icons/` โ€” SVG icon components - `mock-builders/` โ€” test fixtures and fakes (also aliased as `mock-builders` in Jest) - `native.ts` โ€” native-capability interfaces + `registerNativeHandlers()` +- `ai-components/` โ€” the AI components (markdown, syntax highlighting, charts, AI composer, dictation), published as the separate entry point `stream-chat-react-native/ai-components` (`stream-chat-expo/ai-components`). See [AI components entry point](#ai-components-entry-point). Use the closest folder's patterns and conventions when editing. @@ -97,6 +98,7 @@ cd package && yarn test:typecheck # SDK src + tests + mock-b # Lint / format yarn lint # prettier --list-different + eslint --max-warnings 0 + validate-translations yarn lint-fix # ALWAYS run this before committing +yarn workspace stream-chat-react-native-core validate-entry-isolation # main and ai-components entry points stay apart (part of yarn lint) yarn eslint # eslint a single path # Translations @@ -185,6 +187,17 @@ Both wrappers are thin. They: Native code shared by both wrappers lives in `package/shared-native/{ios,android}` and is copied into each wrapper by `shared-native:sync`. **Edit `shared-native/`, never the synced copies.** +### AI components entry point + +`package/src/ai-components/` (ported 1:1 from the deprecated `@stream-io/chat-react-native-ai`) is reachable only through the `ai-components` subpath, implemented as physical folders (`package/ai-components/package.json`, `package/{native,expo}-package/ai-components/`) rather than an `exports` map, which would break deep imports. Metro does not tree-shake, so `bin/validate-entry-isolation.js` (run by `yarn lint`) enforces both directions: + +- the main entry points must never reach `src/ai-components` or its dependencies (`react-syntax-highlighter`, `@khanacademy/simple-markdown`, `@stream-io/state-store`, `zod`, `victory-native`, Skia); +- the AI entry points may only reach `src/ai-components` plus an explicit allowlist (`src/native.ts`, the native shimmer). The wrappers' `ai-components/index.js` registers the native pieces the AI code needs itself and must never import the wrappers' `src/index.js`. + +Do not add AI slots to `DEFAULT_COMPONENTS`; core `require`s the defaults, which would break isolation. Skia and `victory-native` are optional peers (native libraries must be owned by the app); the JS-only AI dependencies are regular dependencies. + +Native pieces live in `package/shared-native/`: the Android-only `PerfText` Fabric view (Kotlin plus a C++ shadow node in `shared-native/android/jni`, built through each wrapper's `android/src/main/jni/CMakeLists.txt`, registered with `cmakeListsPath`/`componentDescriptors` in `react-native.config.js`) and the `AIDictation` TurboModule. **The iOS dictation sources are excluded from the pod unless the app opts in** (`$StreamChatEnableDictation = true` in the Podfile, or the `stream-chat-expo` config plugin in `package/expo-package/plugin`), because referencing the Speech framework makes App Store Connect demand a speech recognition usage description from every app. Never declare `RECORD_AUDIO` in a library manifest. The dictation codegen spec is duplicated into each wrapper's `src/native/` (codegen only reads a package's own `jsSrcsDir`); the isolation check keeps the copies identical. + ## Critical architectural patterns - **Memoization:** components use `React.memo()` with custom `areEqual` comparators (not HOCs). Comparators check cheap props before deep message comparison โ€” keep that ordering when extending one. diff --git a/examples/ExpoChatGPTSample/.gitignore b/examples/ExpoChatGPTSample/.gitignore new file mode 100644 index 0000000000..5f1ef56382 --- /dev/null +++ b/examples/ExpoChatGPTSample/.gitignore @@ -0,0 +1,40 @@ +# Learn more https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files + +# dependencies +node_modules/ + +# Expo +.expo/ +ios/ +android/ +dist/ +web-build/ +expo-env.d.ts + +# Native +*.orig.* +*.jks +*.p8 +*.p12 +*.key +*.mobileprovision + +# Metro +.metro-health-check* + +# debug +npm-debug.* +yarn-debug.* +yarn-error.* + +# macOS +.DS_Store +*.pem + +# local env files +.env*.local + +# typescript +*.tsbuildinfo + +app-example diff --git a/examples/ExpoChatGPTSample/README.md b/examples/ExpoChatGPTSample/README.md new file mode 100644 index 0000000000..7f68eeacef --- /dev/null +++ b/examples/ExpoChatGPTSample/README.md @@ -0,0 +1,30 @@ +# ExpoChatGPTSample + +A ChatGPT-style assistant built with [`stream-chat-expo`](../../package/expo-package) and its AI components +(`stream-chat-expo/ai-components`): streamed markdown answers with syntax highlighting, tables and charts, an "AI is +thinking" indicator, and a composer with attachments, a custom actions sheet and speech-to-text. + +Ported from the `ExpoChatGPTSample` of the deprecated `@stream-io/chat-react-native-ai` repository +([`GetStream/ai-components-js`](https://github.com/GetStream/ai-components-js)). + +## How it works + +- The drawer lists the user's conversations; `+` starts a new one. +- A new conversation is created on its first message. Its title is summarized from that message by the sample backend. +- Before sending, the app asks the sample backend to add an AI agent to the channel (`http/requests.ts`). The agent + replies with streamed `ai_generated` messages, rendered by `StreamingMessageView`. + +The Stream app, user and backend in `chatConfig.ts` and `http/requests.ts` are Stream's public demo setup. To use your +own, replace them and point `http/requests.ts` at your own AI agent server. + +## Running + +From the repository root: + +```bash +yarn install +yarn workspace expochatgptsample ios # or: android +``` + +`ios` and `android` sync the SDK's shared native code, run `expo prebuild` and build the app. The native `ios` and +`android` folders are generated and not committed. diff --git a/examples/ExpoChatGPTSample/app.json b/examples/ExpoChatGPTSample/app.json new file mode 100644 index 0000000000..7310f41c67 --- /dev/null +++ b/examples/ExpoChatGPTSample/app.json @@ -0,0 +1,32 @@ +{ + "expo": { + "name": "ExpoChatGPTSample", + "slug": "ExpoChatGPTSample", + "scheme": "expochatgptsample", + "ios": { + "bundleIdentifier": "io.getstream.expochatgptsample", + "appleTeamId": "EHV7XZLAHA" + }, + "android": { + "permissions": ["android.permission.RECORD_AUDIO"], + "package": "io.getstream.expochatgptsample" + }, + "plugins": [ + "expo-router", + [ + "expo-image-picker", + { + "photosPermission": "The app accesses your photos to let them share with others.", + "cameraPermission": "The app accesses your camera to let you take photos and share with others." + } + ], + [ + "stream-chat-expo", + { + "dictationMicrophoneUsageDescription": "$(PRODUCT_NAME) would like to access your microphone to capture your voice.", + "dictationSpeechRecognitionUsageDescription": "$(PRODUCT_NAME) would like to access speech recognition to transcribe your voice." + } + ] + ] + } +} diff --git a/examples/ExpoChatGPTSample/app/_layout.tsx b/examples/ExpoChatGPTSample/app/_layout.tsx new file mode 100644 index 0000000000..aa358a0e9c --- /dev/null +++ b/examples/ExpoChatGPTSample/app/_layout.tsx @@ -0,0 +1,53 @@ +import React from 'react'; + +import { StyleSheet } from 'react-native'; + +import { GestureHandlerRootView } from 'react-native-gesture-handler'; +import { SafeAreaProvider } from 'react-native-safe-area-context'; + +import { Drawer } from 'expo-router/drawer'; + +import type { LocalMessage } from 'stream-chat'; +import { Chat, OverlayProvider, useCreateChatClient } from 'stream-chat-expo'; + +import { StreamTheme } from 'stream-chat-expo/ai-components'; + +import { chatApiKey, chatUserId, chatUserName, chatUserToken } from '../chatConfig'; +import { AppProvider } from '../contexts/AppContext'; + +import { MenuDrawer } from '../screens/MenuDrawer'; + +const isMessageAIGenerated = (message: LocalMessage) => !!message.ai_generated; + +export default function Layout() { + const chatClient = useCreateChatClient({ + apiKey: chatApiKey, + tokenOrProvider: chatUserToken, + userData: { id: chatUserId, name: chatUserName }, + }); + + if (!chatClient) { + return null; + } + return ( + + + + + + + + + + + + + + + + ); +} + +const styles = StyleSheet.create({ + container: { flex: 1 }, +}); diff --git a/examples/ExpoChatGPTSample/app/index.tsx b/examples/ExpoChatGPTSample/app/index.tsx new file mode 100644 index 0000000000..062b54b1a5 --- /dev/null +++ b/examples/ExpoChatGPTSample/app/index.tsx @@ -0,0 +1,6 @@ +import { ChatContent } from '../screens/ChatContent'; + +const Page = () => { + return ; +}; +export default Page; diff --git a/examples/ExpoChatGPTSample/babel.config.js b/examples/ExpoChatGPTSample/babel.config.js new file mode 100644 index 0000000000..5b84251884 --- /dev/null +++ b/examples/ExpoChatGPTSample/babel.config.js @@ -0,0 +1,12 @@ +module.exports = function (api) { + api.cache(true); + return { + presets: ['babel-preset-expo'], + plugins: [ + // Needed by `stream-chat-expo/ai-components`: its `zod` dependency ships `export * as`. + '@babel/plugin-transform-export-namespace-from', + // react-native-worklets/plugin has to be the last one + 'react-native-worklets/plugin', + ], + }; +}; diff --git a/examples/ExpoChatGPTSample/bottomSheetOptions.tsx b/examples/ExpoChatGPTSample/bottomSheetOptions.tsx new file mode 100644 index 0000000000..b83226ad5d --- /dev/null +++ b/examples/ExpoChatGPTSample/bottomSheetOptions.tsx @@ -0,0 +1,45 @@ +import React from 'react'; +import { Alert } from 'react-native'; + +import { Copy, Edit, FilePickerIcon, Flag, IconProps, Picture } from 'stream-chat-expo'; + +// The sheet renders each icon without props, and some SDK icons (e.g. `Flag`, `Copy`) only draw +// with an explicit stroke, so every icon gets the same colour here. +const withStroke = (Icon: React.ComponentType) => { + const StrokedIcon = () => ; + StrokedIcon.displayName = `Stroked(${Icon.displayName ?? Icon.name})`; + return StrokedIcon; +}; + +export const bottomSheetOptions = [ + { + title: 'Create Image', + subtitle: 'Visualize anything', + action: () => Alert.alert('Pressed on Create Image !'), + Icon: withStroke(Picture), + }, + { + title: 'Thinking', + subtitle: 'Think longer for better answers', + action: () => Alert.alert('Pressed on Thinking !'), + Icon: withStroke(Flag), + }, + { + title: 'Deep research', + subtitle: 'Get a detailed report', + action: () => Alert.alert('Pressed on Deep research !'), + Icon: withStroke(FilePickerIcon), + }, + { + title: 'Web search', + subtitle: 'Find real-time news and info', + action: () => Alert.alert('Pressed on Web search !'), + Icon: withStroke(Copy), + }, + { + title: 'Study and learn', + subtitle: 'Learn a new concept', + action: () => Alert.alert('Pressed on Study and learn !'), + Icon: withStroke(Edit), + }, +]; diff --git a/examples/ExpoChatGPTSample/chatConfig.ts b/examples/ExpoChatGPTSample/chatConfig.ts new file mode 100644 index 0000000000..e385f5f427 --- /dev/null +++ b/examples/ExpoChatGPTSample/chatConfig.ts @@ -0,0 +1,5 @@ +export const chatApiKey = 'zcgvnykxsfm8'; +export const chatUserId = 'rn-ai-test-user-3'; +export const chatUserToken = + 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2lkIjoicm4tYWktdGVzdC11c2VyLTMifQ.kOg3zLkzHc-b_5IyckQr-RixbebN-GDs7N3c570oA5A'; +export const chatUserName = 'Bob'; diff --git a/examples/ExpoChatGPTSample/contexts/AppContext.tsx b/examples/ExpoChatGPTSample/contexts/AppContext.tsx new file mode 100644 index 0000000000..5f0f685291 --- /dev/null +++ b/examples/ExpoChatGPTSample/contexts/AppContext.tsx @@ -0,0 +1,42 @@ +import type { PropsWithChildren } from 'react'; +import React, { useMemo, useState } from 'react'; + +import type { Channel, StreamChat } from 'stream-chat'; + +import { chatUserId } from '../chatConfig'; + +export type AppContextValue = { + channel: Channel | undefined; + setChannel: (channel: Channel) => void; +}; + +export const AppContext = React.createContext({ + setChannel: () => {}, + channel: undefined, +}); + +const ALPHABET = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz-'; + +export const nanoid = (size: number = 21): string => { + let id = ''; + for (let i = 0; i < size; i++) { + const r = Math.floor(Math.random() * ALPHABET.length); + id += ALPHABET[r]; + } + return id; +}; + +export const createChannel = (client: StreamChat) => + client.channel('messaging', nanoid(), { + members: [chatUserId], + }); + +export const AppProvider = ({ client, children }: PropsWithChildren<{ client: StreamChat }>) => { + const [channel, setChannel] = useState(() => createChannel(client)); + + const contextValue = useMemo(() => ({ channel, setChannel }), [channel]); + + return {children}; +}; + +export const useAppContext = () => React.useContext(AppContext); diff --git a/examples/ExpoChatGPTSample/custom-types.d.ts b/examples/ExpoChatGPTSample/custom-types.d.ts new file mode 100644 index 0000000000..a54ef2f03b --- /dev/null +++ b/examples/ExpoChatGPTSample/custom-types.d.ts @@ -0,0 +1,42 @@ +import { + DefaultAttachmentData, + DefaultChannelData, + DefaultCommandData, + DefaultEventData, + DefaultMemberData, + DefaultMessageData, + DefaultPollData, + DefaultPollOptionData, + DefaultReactionData, + DefaultThreadData, + DefaultUserData, +} from 'stream-chat-expo'; + +declare module 'stream-chat' { + // Module augmentation: the empty interfaces are how the default data types are applied. + /* eslint-disable @typescript-eslint/no-empty-object-type */ + + interface CustomAttachmentData extends DefaultAttachmentData {} + + interface CustomChannelData extends DefaultChannelData {} + + interface CustomCommandData extends DefaultCommandData {} + + interface CustomEventData extends DefaultEventData {} + + interface CustomMemberData extends DefaultMemberData {} + + interface CustomUserData extends DefaultUserData {} + + interface CustomMessageData extends DefaultMessageData { + ai_generated?: boolean; + } + + interface CustomPollOptionData extends DefaultPollOptionData {} + + interface CustomPollData extends DefaultPollData {} + + interface CustomReactionData extends DefaultReactionData {} + + interface CustomThreadData extends DefaultThreadData {} +} diff --git a/examples/ExpoChatGPTSample/http/api.ts b/examples/ExpoChatGPTSample/http/api.ts new file mode 100644 index 0000000000..4ff8fba92f --- /dev/null +++ b/examples/ExpoChatGPTSample/http/api.ts @@ -0,0 +1,19 @@ +export const post = async (url: string, data: unknown = {}) => { + try { + const response = await fetch(url, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + }, + body: JSON.stringify(data), + }); + + if (!response.ok) { + throw new Error(`An HTTP Error has occurred. Status: ${response.status}`); + } + + return await response.json(); + } catch (error) { + console.error('Error:', error); + } +}; diff --git a/examples/ExpoChatGPTSample/http/requests.ts b/examples/ExpoChatGPTSample/http/requests.ts new file mode 100644 index 0000000000..e8c20536b0 --- /dev/null +++ b/examples/ExpoChatGPTSample/http/requests.ts @@ -0,0 +1,12 @@ +import { post } from './api'; + +export const startAI = (channelId: string) => + post('https://ai-sdk-server-0f347d455e2e.herokuapp.com/start-ai-agent', { + channel_id: channelId, + }); +export const stopAI = (channelId: string) => + post('https://ai-sdk-server-0f347d455e2e.herokuapp.com/stop-ai-agent', { + channel_id: channelId, + }); +export const summarize = (text: string) => + post('https://ai-sdk-server-0f347d455e2e.herokuapp.com/summarize', { text }); diff --git a/examples/ExpoChatGPTSample/metro.config.js b/examples/ExpoChatGPTSample/metro.config.js new file mode 100644 index 0000000000..5fa02acffa --- /dev/null +++ b/examples/ExpoChatGPTSample/metro.config.js @@ -0,0 +1,74 @@ +const { getDefaultConfig } = require('@expo/metro-config'); +const { exclusionList, resolveUniqueModule } = require('@rnx-kit/metro-config'); + +/** + * Metro configuration + * https://facebook.github.io/metro/docs/configuration + * + * @type {import('metro-config').MetroConfig} + */ +const config = getDefaultConfig(__dirname); + +const PATH = require('path'); +const packageDirPath = PATH.resolve(__dirname, '../../package'); +const expoPackageDirPath = PATH.resolve(__dirname, '../../package/expo-package'); + +const symlinked = { + 'stream-chat-expo': expoPackageDirPath, + 'stream-chat-react-native-core': packageDirPath, +}; + +// find what all modules need to be unique for the app (mainly react and react-native) +// note: we filter the symlinked modules as they are already unique +// and as they dont follow the workspace pattern the auto-generated path to the module is incorrect +const dependencyPackageNames = Object.keys(require('./package.json').dependencies); + +const uniqueModules = dependencyPackageNames.map((packageName) => { + if (symlinked[packageName]) { + const modulePath = symlinked[packageName]; + const escapedPackageName = PATH.normalize(packageName).replace(/\\/g, '\\\\'); + + // exclude the symlinked package from being resolved from node_modules + // example: .*\/node_modules\/stream-chat-react-native-core\/.* + // the above would avoid native-package to resolve core from its own node_modules + const exclusionRE = new RegExp( + `.*${PATH.sep}node_modules\\${PATH.sep}${escapedPackageName}\\${PATH.sep}.*`, + ); + + return { + packageName, // name of the package + modulePath, + blockPattern: exclusionRE, // paths that match this pattern will be blocked from being resolved + }; + } + const [modulePath, blockPattern] = resolveUniqueModule(packageName, __dirname); + // resolveUniqueModule emits a regex ending in `/` (package-dir boundary). metro-config's + // exclusionList anchors every alternation branch with `$`, which would require file paths to + // literally end with `/node_modules//`. Append `.*` so the anchor lands at end-of-file + // instead of end-of-directory, otherwise the blocklist silently matches nothing. + const fixedBlockPattern = new RegExp(blockPattern.source + '.*'); + return { + packageName, // name of the package + modulePath, // actual path to the module in the project's node modules + blockPattern: fixedBlockPattern, // paths that match this pattern will be blocked from being resolved + }; +}); + +// block the other paths for unique modules from being resolved +const blockList = uniqueModules.map(({ blockPattern }) => blockPattern); + +// provide the path for the unique modules +const extraNodeModules = uniqueModules.reduce((acc, item) => { + acc[item.packageName] = item.modulePath; + return acc; +}, {}); + +config.resolver.blockList = exclusionList(blockList); +config.resolver.extraNodeModules = extraNodeModules; + +config.resolver.nodeModulesPaths = [PATH.resolve(__dirname, 'node_modules')]; + +// add the package dir for metro to access the package folder +config.watchFolders = [packageDirPath]; + +module.exports = config; diff --git a/examples/ExpoChatGPTSample/package.json b/examples/ExpoChatGPTSample/package.json new file mode 100644 index 0000000000..ed33a9c288 --- /dev/null +++ b/examples/ExpoChatGPTSample/package.json @@ -0,0 +1,44 @@ +{ + "name": "expochatgptsample", + "version": "1.0.1", + "main": "expo-router/entry", + "scripts": { + "sync-native": "bash ../../package/scripts/reconcile-shared-native.sh expo-package && bash ../../package/scripts/sync-shared-native.sh expo-package", + "start": "yarn sync-native && expo start --dev-client", + "android": "yarn sync-native && expo run:android", + "ios": "yarn sync-native && expo run:ios", + "typecheck": "tsc --noEmit" + }, + "dependencies": { + "@react-native-community/netinfo": "12.0.1", + "@shopify/react-native-skia": "^2.13.0", + "expo": "^57.0.7", + "expo-clipboard": "~57.0.1", + "expo-constants": "~57.0.6", + "expo-image-manipulator": "~57.0.5", + "expo-image-picker": "~57.0.5", + "expo-linking": "~57.0.3", + "expo-router": "~57.0.7", + "react": "19.2.3", + "react-native": "0.86.0", + "react-native-gesture-handler": "~2.32.0", + "react-native-reanimated": "4.5.2", + "react-native-safe-area-context": "~5.7.0", + "react-native-screens": "4.26.2", + "react-native-svg": "15.15.4", + "react-native-teleport": "^1.1.12", + "react-native-worklets": "0.11.1", + "stream-chat": "^9.54.0", + "stream-chat-expo": "workspace:^", + "stream-chat-react-native-core": "workspace:^", + "victory-native": "^42.0.1" + }, + "devDependencies": { + "@babel/plugin-transform-export-namespace-from": "^7.27.1", + "@rnx-kit/metro-config": "^2.1.2", + "@stream-io/typescript-config": "workspace:^", + "@types/react": "~19.2.10", + "typescript": "6.0.3" + }, + "private": true +} diff --git a/examples/ExpoChatGPTSample/screens/ChatContent.tsx b/examples/ExpoChatGPTSample/screens/ChatContent.tsx new file mode 100644 index 0000000000..924d8fb02b --- /dev/null +++ b/examples/ExpoChatGPTSample/screens/ChatContent.tsx @@ -0,0 +1,277 @@ +import { useCallback, useMemo } from 'react'; +import { Platform, StatusBar, StyleSheet, Text, View } from 'react-native'; +import Animated, { FadeIn, FadeOut } from 'react-native-reanimated'; +import { useSafeAreaInsets } from 'react-native-safe-area-context'; + +import { useHeaderHeight } from 'expo-router/react-navigation'; + +import type { LocalMessage } from 'stream-chat'; +import type { MessageProps } from 'stream-chat-expo'; +import { + AIStates, + Channel, + isLocalUrl, + mergeThemes, + Message, + MessageList, + ThemeProvider, + useAIState, + useChannelContext, + useMessageComposer, + useMessageContext, + useMessageInputContext, + useStableCallback, + useTheme, + WithComponents, +} from 'stream-chat-expo'; + +import { + AITypingIndicatorView, + ComposerView, + type ComposerViewProps, + StreamingMessageView, +} from 'stream-chat-expo/ai-components'; + +import { bottomSheetOptions } from '../bottomSheetOptions'; +import { useAppContext } from '../contexts/AppContext'; +import { startAI, summarize } from '../http/requests'; + +const CustomMessage = (props: MessageProps) => { + const { theme } = useTheme(); + const { message } = props; + const isFromBot = message.ai_generated; + const hasPendingAttachments = useMemo( + () => + (message.attachments ?? []).some( + (attachment) => + (attachment.image_url && isLocalUrl(attachment.image_url)) || + (attachment.asset_url && isLocalUrl(attachment.asset_url)), + ), + [message.attachments], + ); + + const modifiedTheme = useMemo( + () => + mergeThemes({ + theme, + style: { + messageItemView: isFromBot + ? { + content: { + containerInner: { + backgroundColor: 'transparent', + borderRadius: 0, + borderColor: 'transparent', + }, + textContainer: { maxWidth: '100%' }, + }, + } + : { + container: { + opacity: hasPendingAttachments ? 0.5 : 1, + }, + }, + }, + }), + [theme, isFromBot, hasPendingAttachments], + ); + + return ( + + + + ); +}; + +const CustomStreamingMessageView = () => { + const { message } = useMessageContext(); + return ( + + + + ); +}; + +const CustomComposerView = () => { + const messageComposer = useMessageComposer(); + const { sendMessage } = useMessageInputContext(); + const { channel } = useChannelContext(); + + const { aiState } = useAIState(channel); + + const stopGenerating = useCallback(() => channel?.stopAIResponse(), [channel]); + + const isGenerating = [AIStates.Thinking, AIStates.Generating].includes(aiState); + + const safeAreaInsets = useSafeAreaInsets(); + const insets = useMemo( + () => ({ + ...safeAreaInsets, + bottom: + safeAreaInsets.bottom + + (Platform.OS === 'android' ? (StatusBar.currentHeight ?? 0) * 2 : 0), + }), + [safeAreaInsets], + ); + + const serializeToMessage = useStableCallback( + async ({ text, attachments }: Parameters[0]) => { + messageComposer.textComposer.setText(text); + if (attachments && attachments.length > 0) { + const localAttachments = await Promise.all( + attachments.map((asset) => + messageComposer.attachmentManager.fileToLocalUploadAttachment({ + ...asset, + // The picker can leave these out (e.g. some iOS photos have no file name). + name: asset.name ?? asset.uri.split('/').pop() ?? 'image', + size: asset.size ?? 0, + type: asset.type ?? 'image/jpeg', + }), + ), + ); + messageComposer.attachmentManager.upsertAttachments(localAttachments); + } + + await sendMessage(); + }, + ); + + return ( + + ); +}; + +const EmptyStateIndicator = () => ( + + What can I help with ? + +); + +const CustomAITypingIndicatorView = () => { + const { channel } = useChannelContext(); + const { aiState } = useAIState(channel); + + const allowedStates = { + [AIStates.Thinking]: 'Thinking about the question...', + [AIStates.Generating]: 'Generating a response...', + [AIStates.ExternalSources]: 'Checking external sources...', + }; + + const indicatorText = allowedStates[aiState as keyof typeof allowedStates]; + + if (!indicatorText) { + return null; + } + + return ( + + + + ); +}; + +const RenderNull = () => null; + +const additionalFlatListProps = { + maintainVisibleContentPosition: { + minIndexForVisible: 0, + autoscrollToTopThreshold: 0, + }, + ListHeaderComponent: null, +}; + +export const ChatContent = () => { + const { channel } = useAppContext(); + const { bottom } = useSafeAreaInsets(); + // The channel sits below the drawer header, so the keyboard offset has to account for it + // (without it, Android keeps the composer behind the keyboard). + const headerHeight = useHeaderHeight(); + + const preSendMessageRequest = useStableCallback( + async ({ localMessage }: { localMessage: LocalMessage }) => { + if (!channel) { + return; + } + + if (!channel.initialized) { + await channel.watch({ + created_by_id: localMessage.user_id, + }); + summarize(localMessage.text ?? '').then((response) => { + // `post` resolves to undefined when the request fails; the channel then keeps its id. + const summary = (response as { summary?: string } | undefined)?.summary; + if (summary) { + channel.update({ name: summary }); + } + }); + } + + if ( + !Object.keys(channel.state.watchers).some((watcher) => watcher.startsWith('ai-bot')) && + channel.id + ) { + await startAI(channel.id); + } + }, + ); + + if (!channel) { + return null; + } + + return ( + + + + + + + + + + ); +}; + +const styles = StyleSheet.create({ + wrapper: { flex: 1, backgroundColor: '#fcfcfc' }, + emptyContainer: { + flex: 1, + width: '100%', + backgroundColor: 'transparent', + justifyContent: 'center', + alignItems: 'center', + }, + emptyContainerText: { fontSize: 24, fontWeight: 'bold' }, + streamingMessageViewWrapper: { width: '100%', paddingHorizontal: 16 }, + aiTypingIndicatorWrapper: { + paddingHorizontal: 24, + paddingVertical: 12, + }, +}); diff --git a/examples/ExpoChatGPTSample/screens/MenuDrawer.tsx b/examples/ExpoChatGPTSample/screens/MenuDrawer.tsx new file mode 100644 index 0000000000..10fe99de57 --- /dev/null +++ b/examples/ExpoChatGPTSample/screens/MenuDrawer.tsx @@ -0,0 +1,95 @@ +import React from 'react'; +import { Pressable, StyleSheet, Text, View } from 'react-native'; + +import { SafeAreaView } from 'react-native-safe-area-context'; + +import type { DrawerContentComponentProps } from 'expo-router/drawer'; +import type { ChannelSort } from 'stream-chat'; + +import type { Channel as ChannelClass } from 'stream-chat'; +import { + ChannelList, + ChannelPreviewProps, + Plus, + useChannelsContext, + useChatContext, + useStableCallback, + WithComponents, +} from 'stream-chat-expo'; + +import { chatUserId } from '../chatConfig'; + +import { createChannel, useAppContext } from '../contexts/AppContext'; + +const filters = { + members: { + $in: [chatUserId], + }, +}; + +const sort: ChannelSort = { last_updated: -1 }; + +const ChannelPreview = (props: ChannelPreviewProps) => { + const channel = props.channel; + const { onSelect } = useChannelsContext(); + const onPress = useStableCallback(() => { + onSelect?.(channel); + }); + return ( + ({ + paddingVertical: 8, + paddingHorizontal: 12, + opacity: pressed ? 0.6 : 1, + })} + onPress={onPress} + > + + {channel.data?.name ?? channel.cid} + + + ); +}; + +export const MenuDrawer = ({ navigation }: DrawerContentComponentProps) => { + const { client } = useChatContext(); + const { setChannel } = useAppContext(); + + const onSelect = useStableCallback((channel: ChannelClass) => { + setChannel(channel); + navigation.closeDrawer(); + }); + + const onCreateNewChat = useStableCallback(() => { + setChannel(createChannel(client)); + navigation.closeDrawer(); + }); + + return ( + + + Conversations + + + + + + + + + ); +}; + +const styles = StyleSheet.create({ + wrapper: { flex: 1, backgroundColor: '#FFFFFF' }, + container: { + flexDirection: 'row', + justifyContent: 'space-between', + marginHorizontal: 12, + paddingVertical: 16, + borderBottomWidth: 1, + borderBottomColor: 'grey', + }, + title: { fontSize: 15, fontWeight: 'bold' }, + previewText: { fontSize: 15, fontWeight: 'bold' }, +}); diff --git a/examples/ExpoChatGPTSample/tsconfig.json b/examples/ExpoChatGPTSample/tsconfig.json new file mode 100644 index 0000000000..570678c294 --- /dev/null +++ b/examples/ExpoChatGPTSample/tsconfig.json @@ -0,0 +1,12 @@ +{ + "extends": ["expo/tsconfig.base", "@stream-io/typescript-config/base.json"], + "compilerOptions": { + "paths": { + // Same as ExpoMessaging: this workspace keeps its own React Native (Expo 57 -> RN 0.86) while + // the SDK workspace is on RN 0.87, so the whole program is pinned to this workspace's copy. + "react-native": ["./node_modules/react-native"], + "react-native/*": ["./node_modules/react-native/*"] + } + }, + "include": ["**/*.ts", "**/*.tsx"] +} diff --git a/examples/SampleApp/fastlane/Fastfile b/examples/SampleApp/fastlane/Fastfile index bc15b10477..f7993958b3 100644 --- a/examples/SampleApp/fastlane/Fastfile +++ b/examples/SampleApp/fastlane/Fastfile @@ -203,8 +203,10 @@ lane :frameworks_sizes do yarn_all sh('yarn build') sh('yarn minify-bundle') + sh('yarn minify-ai-components-bundle') js_bundle_size = file_size(path: 'package/lib/module/bundle.min.js') + UI.message("ai_components_bundle_size: #{file_size(path: 'package/lib/ai-components.min.js').round} KB") { js_bundle_size: js_bundle_size } end end diff --git a/examples/SampleApp/ios/Podfile.lock b/examples/SampleApp/ios/Podfile.lock index 4de2187ad1..0be1f3f2d0 100644 --- a/examples/SampleApp/ios/Podfile.lock +++ b/examples/SampleApp/ios/Podfile.lock @@ -2640,7 +2640,7 @@ PODS: - SDWebImage/Core (~> 5.17) - SocketRocket (0.7.1): - ReactNativeDependencies - - stream-chat-react-native (9.9.2): + - stream-chat-react-native (9.9.4): - hermes-engine - RCTRequired - RCTTypeSafety @@ -3223,7 +3223,7 @@ SPEC CHECKSUMS: SDWebImageSVGCoder: 15a300a97ec1c8ac958f009c02220ac0402e936c SDWebImageWebPCoder: 0e06e365080397465cc73a7a9b472d8a3bd0f377 SocketRocket: 37aec555668fb852ec12e3c0de59a86ca58f0871 - stream-chat-react-native: bb3a2c7f8949f4ebb735d83f549f43ae37c3ae7b + stream-chat-react-native: 999be0a94189d35c11938d7e0a0f91af73f93fb5 Teleport: c42718cf1f14dd53b250c0fa29b9235a42f861ec Yoga: d1c536142c5ff8ec8cd856ab2a7c227a1d875c8e diff --git a/package.json b/package.json index 79f28d9e77..f2af3840b3 100644 --- a/package.json +++ b/package.json @@ -7,7 +7,8 @@ "package/native-package", "package/expo-package", "examples/SampleApp", - "examples/ExpoMessaging" + "examples/ExpoMessaging", + "examples/ExpoChatGPTSample" ], "resolutions": { "@types/react": "^19.2.0" @@ -66,15 +67,16 @@ "release-next": "yarn workspaces foreach -A --topological-dev --include stream-chat-react-native-core --include sampleapp run release-next", "extract-changelog": "rm -rf NEXT_RELEASE_CHANGELOG.md && yarn workspaces foreach -A --topological-dev --include stream-chat-react-native-core --include sampleapp run extract-changelog", "eslint": "eslint .", - "lint": "prettier --list-different . && eslint . --max-warnings 0 && yarn workspace stream-chat-react-native-core validate-translations", + "lint": "prettier --list-different . && eslint . --max-warnings 0 && yarn workspace stream-chat-react-native-core validate-translations && yarn workspace stream-chat-react-native-core validate-entry-isolation", "lint-fix": "prettier --write . && eslint . --fix --max-warnings 0", "prettier": "prettier --list-different .", "prettier-fix": "prettier --write .", - "typecheck": "yarn workspaces foreach -A --parallel --include 'stream-chat-react-native-core' --include 'stream-chat-react-native' --include 'stream-chat-expo' --include 'sampleapp' --include 'ExpoMessaging' run typecheck", + "typecheck": "yarn workspaces foreach -A --parallel --include 'stream-chat-react-native-core' --include 'stream-chat-react-native' --include 'stream-chat-expo' --include 'sampleapp' --include 'ExpoMessaging' --include 'expochatgptsample' run typecheck", "build": "yarn workspace stream-chat-react-native-core build", "test:coverage": "yarn workspace stream-chat-react-native-core test:coverage", "test:unit": "yarn workspace stream-chat-react-native-core test:unit", - "minify-bundle": "uglifyjs package/lib/module/**/*.js -o package/lib/module/bundle.min.js -c -m" + "minify-bundle": "uglifyjs package/lib/module/*.js package/lib/module/!(ai-components)/**/*.js -o package/lib/module/bundle.min.js -c -m", + "minify-ai-components-bundle": "uglifyjs package/lib/module/ai-components/**/*.js -o package/lib/ai-components.min.js -c -m" }, "packageManager": "yarn@4.15.0" } diff --git a/package/ai-components/package.json b/package/ai-components/package.json new file mode 100644 index 0000000000..59e65e8706 --- /dev/null +++ b/package/ai-components/package.json @@ -0,0 +1,6 @@ +{ + "main": "../lib/commonjs/ai-components/index.js", + "module": "../lib/module/ai-components/index.js", + "react-native": "../src/ai-components/index.ts", + "types": "../lib/typescript/ai-components/index.d.ts" +} diff --git a/package/bin/validate-entry-isolation.js b/package/bin/validate-entry-isolation.js new file mode 100644 index 0000000000..f5e4e4f527 --- /dev/null +++ b/package/bin/validate-entry-isolation.js @@ -0,0 +1,224 @@ +/* global __dirname, process, require */ +/** + * Keeps the `ai-components` entry point isolated from the main SDK entry point, in both directions. + * + * Metro does not tree-shake: everything reachable from an imported module ends up in the app's + * bundle. So: + * + * 1. The main entry points (core `src/index.ts` and the wrappers' `src/index.js`) must never reach + * the AI components or their dependencies. Apps that don't use AI must not bundle any of it. + * 2. The AI entry points (core `src/ai-components/index.ts` and the wrappers' + * `ai-components/index.js`) may only reach the AI sources plus an explicit allowlist of other + * files. Apps that only use the AI components must not bundle the chat SDK. + * 3. The codegen spec copies the wrappers need (codegen only reads a package's own `jsSrcsDir`) stay + * byte-identical to the core source of truth. + * + * Type-only imports and exports are ignored, since they are erased at build time. + */ +const fs = require('fs'); +const path = require('path'); + +const ts = require('typescript'); + +const PACKAGE_DIR = path.resolve(__dirname, '..'); +const CORE_SRC = path.join(PACKAGE_DIR, 'src'); +const AI_SRC = path.join(CORE_SRC, 'ai-components'); +const WRAPPERS = ['native-package', 'expo-package'].map((name) => path.join(PACKAGE_DIR, name)); + +// Third-party packages only the AI components may use. (`lodash` is not one of them: the main +// entry point imports `lodash/*` modules too.) +const AI_ONLY_PACKAGES = [ + '@khanacademy/simple-markdown', + '@shopify/react-native-skia', + '@stream-io/state-store', + 'react-syntax-highlighter', + 'victory-native', + 'zod', +]; + +// Files outside `src/ai-components` that the AI entry points are allowed to reach. +const AI_ALLOWED_FILES = [ + path.join(CORE_SRC, 'native.ts'), + path.join(CORE_SRC, 'components/UIComponents/NativeShimmerView.tsx'), + ...WRAPPERS.flatMap((wrapper) => [ + path.join(wrapper, 'ai-components/index.js'), + path.join(wrapper, 'src/optionalDependencies/NativeShimmerView.ts'), + path.join(wrapper, 'src/native/StreamShimmerViewNativeComponent.ts'), + path.join(wrapper, 'src/native/PerfTextNativeComponent.ts'), + ]), +]; + +// Third-party packages the AI entry points must never reach. +const AI_FORBIDDEN_PACKAGES = ['stream-chat', 'stream-chat-react-native-core']; + +const EXTENSIONS = ['.ts', '.tsx', '.js', '.jsx']; + +const resolveFile = (candidate) => { + if (fs.existsSync(candidate) && fs.statSync(candidate).isFile()) return candidate; + for (const ext of EXTENSIONS) { + if (fs.existsSync(candidate + ext)) return candidate + ext; + } + for (const ext of EXTENSIONS) { + const index = path.join(candidate, 'index' + ext); + if (fs.existsSync(index)) return index; + } + return undefined; +}; + +const packageNameOf = (specifier) => { + const parts = specifier.split('/'); + return specifier.startsWith('@') ? parts.slice(0, 2).join('/') : parts[0]; +}; + +/** Resolves a specifier to `{ file }` for first-party code or `{ pkg }` for a dependency. */ +const resolveSpecifier = (specifier, fromFile) => { + if (specifier.startsWith('.')) { + const file = resolveFile(path.resolve(path.dirname(fromFile), specifier)); + if (!file) throw new Error(`Cannot resolve "${specifier}" from ${fromFile}`); + return { file }; + } + if (specifier === 'stream-chat-react-native-core/ai-components') { + return { file: path.join(AI_SRC, 'index.ts') }; + } + return { pkg: packageNameOf(specifier), specifier }; +}; + +const isTypeOnlyImport = (node) => { + const clause = node.importClause; + if (!clause) return false; // side-effect import + if (clause.isTypeOnly) return true; + if (clause.name) return false; // default import + const bindings = clause.namedBindings; + if (!bindings || ts.isNamespaceImport(bindings)) return false; + return bindings.elements.length > 0 && bindings.elements.every((element) => element.isTypeOnly); +}; + +/** Returns the runtime module specifiers of a file. */ +const runtimeSpecifiers = (file) => { + const source = ts.createSourceFile( + file, + fs.readFileSync(file, 'utf8'), + ts.ScriptTarget.Latest, + true, + ); + const specifiers = []; + const visit = (node) => { + if (ts.isImportDeclaration(node)) { + if (!isTypeOnlyImport(node)) specifiers.push(node.moduleSpecifier.text); + } else if (ts.isExportDeclaration(node)) { + if (node.moduleSpecifier && !node.isTypeOnly) { + const onlyTypes = + node.exportClause && + ts.isNamedExports(node.exportClause) && + node.exportClause.elements.length > 0 && + node.exportClause.elements.every((element) => element.isTypeOnly); + if (!onlyTypes) specifiers.push(node.moduleSpecifier.text); + } + } else if ( + ts.isCallExpression(node) && + node.arguments.length === 1 && + ts.isStringLiteralLike(node.arguments[0]) && + ((ts.isIdentifier(node.expression) && node.expression.text === 'require') || + node.expression.kind === ts.SyntaxKind.ImportKeyword) + ) { + specifiers.push(node.arguments[0].text); + } + ts.forEachChild(node, visit); + }; + visit(source); + return specifiers; +}; + +/** Walks the runtime import graph, returning reached files and packages with an example path. */ +const walk = (entry) => { + const files = new Map([[entry, [entry]]]); + const packages = new Map(); + const queue = [entry]; + while (queue.length) { + const file = queue.shift(); + for (const specifier of runtimeSpecifiers(file)) { + const resolved = resolveSpecifier(specifier, file); + if (resolved.file) { + if (!files.has(resolved.file)) { + files.set(resolved.file, [...files.get(file), resolved.file]); + queue.push(resolved.file); + } + } else if (!packages.has(resolved.specifier)) { + packages.set(resolved.specifier, [...files.get(file), resolved.specifier]); + } + } + } + return { files, packages }; +}; + +const relative = (p) => (path.isAbsolute(p) ? path.relative(PACKAGE_DIR, p) : p); +const errors = []; +const report = (message, chain) => + errors.push(`${message}\n via: ${chain.map(relative).join('\n -> ')}`); + +// 1. Main entry points must not reach the AI components. +for (const entry of [ + path.join(CORE_SRC, 'index.ts'), + ...WRAPPERS.map((w) => path.join(w, 'src/index.js')), +]) { + const { files, packages } = walk(entry); + for (const [file, chain] of files) { + if ( + file.startsWith(AI_SRC + path.sep) || + file.includes(`${path.sep}ai-components${path.sep}`) + ) { + report(`${relative(entry)} reaches the AI components (${relative(file)})`, chain); + } + } + for (const [specifier, chain] of packages) { + if (AI_ONLY_PACKAGES.includes(packageNameOf(specifier))) { + report(`${relative(entry)} reaches the AI-only dependency "${specifier}"`, chain); + } + } +} + +// 2. AI entry points must not reach the rest of the SDK. +for (const entry of [ + path.join(AI_SRC, 'index.ts'), + ...WRAPPERS.map((w) => path.join(w, 'ai-components/index.js')), +]) { + const { files, packages } = walk(entry); + for (const [file, chain] of files) { + if (!file.startsWith(AI_SRC + path.sep) && !AI_ALLOWED_FILES.includes(file)) { + report( + `${relative(entry)} reaches ${relative(file)}, which is not in the AI allowlist`, + chain, + ); + } + } + for (const [specifier, chain] of packages) { + if (AI_FORBIDDEN_PACKAGES.includes(specifier)) { + report(`${relative(entry)} reaches "${specifier}"`, chain); + } + } +} + +// 3. Codegen spec copies must not drift. +const copies = [ + [ + path.join(AI_SRC, 'native-specs/NativeAIDictation.ts'), + ...WRAPPERS.map((w) => path.join(w, 'src/native/NativeAIDictation.ts')), + ], + WRAPPERS.map((w) => path.join(w, 'src/native/PerfTextNativeComponent.ts')), +]; +for (const [source, ...targets] of copies) { + const expected = fs.readFileSync(source, 'utf8'); + for (const target of targets) { + if (!fs.existsSync(target) || fs.readFileSync(target, 'utf8') !== expected) { + errors.push(`${relative(target)} must be identical to ${relative(source)}`); + } + } +} + +if (errors.length) { + console.error( + `Entry point isolation check failed:\n\n${errors.map((e) => ` - ${e}`).join('\n\n')}`, + ); + process.exit(1); +} +console.log('Entry point isolation check passed.'); diff --git a/package/expo-package/ai-components/index.d.ts b/package/expo-package/ai-components/index.d.ts new file mode 100644 index 0000000000..6bdce055e3 --- /dev/null +++ b/package/expo-package/ai-components/index.d.ts @@ -0,0 +1 @@ +export * from 'stream-chat-react-native-core/ai-components'; diff --git a/package/expo-package/ai-components/index.js b/package/expo-package/ai-components/index.js new file mode 100644 index 0000000000..f53d6b982a --- /dev/null +++ b/package/expo-package/ai-components/index.js @@ -0,0 +1,21 @@ +import { Platform } from 'react-native'; + +import { registerNativeHandlers } from 'stream-chat-react-native-core/ai-components'; + +import { NativeShimmerView } from '../src/optionalDependencies/NativeShimmerView'; + +// This entry point deliberately does not import `../src/index.js`: that file re-exports the whole +// core SDK, which Metro would then bundle into apps that only use the AI components. Instead, only +// the native pieces the AI components need are registered here. Registering them again when the +// main entry point is imported as well is harmless. + +// `PerfText` is an Android-only Fabric component. +registerNativeHandlers({ + NativeShimmerView, + PerfText: + Platform.OS === 'android' + ? require('../src/native/PerfTextNativeComponent').default + : undefined, +}); + +export * from 'stream-chat-react-native-core/ai-components'; diff --git a/package/expo-package/android/src/main/java/com/streamchatexpo/StreamChatExpoPackage.java b/package/expo-package/android/src/main/java/com/streamchatexpo/StreamChatExpoPackage.java index 8f0d071417..4442d860eb 100644 --- a/package/expo-package/android/src/main/java/com/streamchatexpo/StreamChatExpoPackage.java +++ b/package/expo-package/android/src/main/java/com/streamchatexpo/StreamChatExpoPackage.java @@ -7,8 +7,9 @@ import com.facebook.react.module.model.ReactModuleInfo; import com.facebook.react.module.model.ReactModuleInfoProvider; import com.facebook.react.uimanager.ViewManager; +import com.streamchatreactnative.PerfTextViewManager; import com.streamchatreactnative.StreamShimmerViewManager; -import java.util.Collections; +import java.util.Arrays; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -16,6 +17,7 @@ public class StreamChatExpoPackage extends TurboReactPackage { private static final String STREAM_MULTIPART_UPLOADER_MODULE = "StreamMultipartUploader"; private static final String STREAM_VIDEO_THUMBNAIL_MODULE = "StreamVideoThumbnail"; + private static final String AI_DICTATION_MODULE = "AIDictation"; @Nullable @Override @@ -28,6 +30,10 @@ public NativeModule getModule(String name, ReactApplicationContext reactContext) return createNewArchModule("com.streamchatexpo.StreamVideoThumbnailModule", reactContext); } + if (name.equals(AI_DICTATION_MODULE)) { + return createNewArchModule("com.streamchatexpo.AIDictationModule", reactContext); + } + return null; } @@ -57,13 +63,25 @@ public ReactModuleInfoProvider getReactModuleInfoProvider() { false, // isCxxModule true // isTurboModule )); + + moduleInfos.put( + AI_DICTATION_MODULE, + new ReactModuleInfo( + AI_DICTATION_MODULE, + AI_DICTATION_MODULE, + false, // canOverrideExistingModule + false, // needsEagerInit + false, // hasConstants + false, // isCxxModule + true // isTurboModule + )); return moduleInfos; }; } @Override public List createViewManagers(ReactApplicationContext reactContext) { - return Collections.singletonList(new StreamShimmerViewManager()); + return Arrays.asList(new StreamShimmerViewManager(), new PerfTextViewManager()); } @Nullable diff --git a/package/expo-package/android/src/main/jni/CMakeLists.txt b/package/expo-package/android/src/main/jni/CMakeLists.txt new file mode 100644 index 0000000000..0da704177a --- /dev/null +++ b/package/expo-package/android/src/main/jni/CMakeLists.txt @@ -0,0 +1,63 @@ +cmake_minimum_required(VERSION 3.13) +set(CMAKE_VERBOSE_MAKEFILE on) + +# Custom CMakeLists for this package's codegen library, registered through `cmakeListsPath` in +# react-native.config.js. It builds everything the codegen-generated CMakeLists would, plus the +# hand-written C++ of the Android-only PerfText component used by `ai-components`. + +set(LIB_LITERAL StreamChatExpoSpec) +set(LIB_TARGET_NAME react_codegen_${LIB_LITERAL}) + +# REALPATH: the package is often reached through a symlink (workspaces, `yarn link`), and CMake +# collapses `..` lexically, which would otherwise resolve relative paths against the symlink. +get_filename_component(LIB_ANDROID_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../.. REALPATH) +set(LIB_GENERATED_JNI_DIR ${LIB_ANDROID_DIR}/build/generated/source/codegen/jni) +set(LIB_GENERATED_COMPONENTS_DIR ${LIB_GENERATED_JNI_DIR}/react/renderer/components/${LIB_LITERAL}) + +# Shared C++ sources: read straight from the monorepo during development, otherwise from the copy +# synced into this package (scripts/sync-shared-native.sh, run before publishing). +get_filename_component(MONOREPO_SHARED_CPP_DIR ${LIB_ANDROID_DIR}/../../shared-native/android/jni REALPATH) +if(EXISTS ${MONOREPO_SHARED_CPP_DIR}) + set(SHARED_CPP_DIR ${MONOREPO_SHARED_CPP_DIR}) +else() + set(SHARED_CPP_DIR ${LIB_ANDROID_DIR}/src/main/java/com/streamchatreactnative/shared/jni) +endif() + +file(GLOB LIB_SRCS CONFIGURE_DEPENDS + ${LIB_GENERATED_JNI_DIR}/*.cpp + ${LIB_GENERATED_COMPONENTS_DIR}/*.cpp + ${SHARED_CPP_DIR}/*.cpp +) + +add_library(${LIB_TARGET_NAME} OBJECT ${LIB_SRCS}) + +# This directory must come first: see StreamChatExpoSpec.h next to this file. +target_include_directories( + ${LIB_TARGET_NAME} + PUBLIC + ${CMAKE_CURRENT_SOURCE_DIR} + ${SHARED_CPP_DIR} + ${LIB_GENERATED_JNI_DIR} + ${LIB_GENERATED_COMPONENTS_DIR} +) + +target_link_libraries( + ${LIB_TARGET_NAME} + fbjni + jsi + reactnative +) + +if(ReactAndroid_VERSION_MINOR GREATER_EQUAL 80) + target_compile_reactnative_options(${LIB_TARGET_NAME} PRIVATE) +else() + target_compile_options( + ${LIB_TARGET_NAME} + PRIVATE + -DLOG_TAG=\"ReactNative\" + -fexceptions + -frtti + -std=c++20 + -Wall + ) +endif() diff --git a/package/expo-package/android/src/main/jni/StreamChatExpoSpec.h b/package/expo-package/android/src/main/jni/StreamChatExpoSpec.h new file mode 100644 index 0000000000..aa2e9d4132 --- /dev/null +++ b/package/expo-package/android/src/main/jni/StreamChatExpoSpec.h @@ -0,0 +1,9 @@ +#pragma once + +// The app's autolinking includes `` to register this package's TurboModules and +// component descriptors. This directory comes first on the include path (see CMakeLists.txt), so +// that include lands here: forward to the codegen-generated header, and add the hand-written +// PerfText component descriptor, which codegen does not generate (`interfaceOnly` component). +#include_next + +#include "PerfTextComponentDescriptor.h" diff --git a/package/expo-package/android/src/main/jni/StreamCodegenComponents.h b/package/expo-package/android/src/main/jni/StreamCodegenComponents.h new file mode 100644 index 0000000000..3f94f7073f --- /dev/null +++ b/package/expo-package/android/src/main/jni/StreamCodegenComponents.h @@ -0,0 +1,7 @@ +#pragma once + +// The codegen-generated component headers of this package, for the shared PerfText C++ +// (shared-native/android/jni), which is compiled into both SDK packages and so cannot name the +// package's codegen library itself. +#include +#include diff --git a/package/expo-package/android/src/newarch/com/streamchatexpo/AIDictationModule.kt b/package/expo-package/android/src/newarch/com/streamchatexpo/AIDictationModule.kt new file mode 100644 index 0000000000..b603246c5d --- /dev/null +++ b/package/expo-package/android/src/newarch/com/streamchatexpo/AIDictationModule.kt @@ -0,0 +1,49 @@ +package com.streamchatexpo + +import com.facebook.react.bridge.ReactApplicationContext +import com.facebook.react.bridge.ReadableMap +import com.facebook.react.modules.core.DeviceEventManagerModule +import com.streamchatreactnative.shared.dictation.AIDictationController + +/** + * The `AIDictation` TurboModule used by `ai-components` speech-to-text. The implementation is + * shared with the other SDK package; see [AIDictationController]. + */ +class AIDictationModule( + reactContext: ReactApplicationContext, +) : NativeAIDictationSpec(reactContext) { + private val controller = + AIDictationController(reactContext) { eventName, payload -> + reactApplicationContext + .getJSModule(DeviceEventManagerModule.RCTDeviceEventEmitter::class.java) + .emit(eventName, payload) + } + + override fun getName(): String = NAME + + override fun initialize() { + super.initialize() + controller.initialize() + } + + override fun invalidate() { + super.invalidate() + controller.invalidate() + } + + override fun start(options: ReadableMap?) = controller.start(options) + + override fun stop() = controller.stop() + + override fun cancel() = controller.cancel() + + override fun isRecording(): Boolean = controller.isRecording() + + override fun addListener(eventName: String) = Unit + + override fun removeListeners(count: Double) = Unit + + companion object { + const val NAME = AIDictationController.NAME + } +} diff --git a/package/expo-package/app.plugin.js b/package/expo-package/app.plugin.js new file mode 100644 index 0000000000..7590f2e3ce --- /dev/null +++ b/package/expo-package/app.plugin.js @@ -0,0 +1 @@ +module.exports = require('./plugin'); diff --git a/package/expo-package/package.json b/package/expo-package/package.json index 73420997bc..b93b59053e 100644 --- a/package/expo-package/package.json +++ b/package/expo-package/package.json @@ -20,7 +20,10 @@ "ios", "*.podspec", "react-native.config.js", - "package.json" + "package.json", + "ai-components", + "app.plugin.js", + "plugin" ], "license": "SEE LICENSE IN LICENSE", "main": "src/index.js", @@ -30,6 +33,7 @@ "stream-chat-react-native-core": "workspace:^" }, "peerDependencies": { + "@shopify/react-native-skia": ">=2.0.0", "expo": ">=52.0.0", "expo-audio": "*", "expo-av": "*", @@ -42,9 +46,13 @@ "expo-media-library": "*", "expo-sharing": "*", "expo-video": "*", - "react-native": ">=0.76.0" + "react-native": ">=0.76.0", + "victory-native": ">=41.0.0" }, "peerDependenciesMeta": { + "@shopify/react-native-skia": { + "optional": true + }, "expo-audio": { "optional": true }, @@ -77,6 +85,9 @@ }, "expo-video": { "optional": true + }, + "victory-native": { + "optional": true } }, "devDependencies": { diff --git a/package/expo-package/plugin/index.js b/package/expo-package/plugin/index.js new file mode 100644 index 0000000000..0fc7ed9036 --- /dev/null +++ b/package/expo-package/plugin/index.js @@ -0,0 +1,43 @@ +const { createRunOncePlugin, withPlugins } = require('expo/config-plugins'); + +const pkg = require('../package.json'); + +const withAndroidManifestPermissions = require('./withAndroidManifest'); +const withiOSInfoPlist = require('./withiOSInfoPlist'); +const withPodfileProperties = require('./withPodfileProperties'); + +/** + * Config plugin of `stream-chat-expo`. It sets up the speech-to-text dictation used by + * `stream-chat-expo/ai-components`: + * + * - iOS: compiles the dictation module in (it is left out by default, see the podspec) and adds the + * microphone and speech recognition usage descriptions. + * - Android: adds the `RECORD_AUDIO` permission. + * + * Props: + * - `enableDictation` (default `true`): set to `false` to leave dictation out. + * - `dictationMicrophoneUsageDescription`, `dictationSpeechRecognitionUsageDescription`: the iOS + * usage descriptions; defaults are used when the app does not define its own. + * + * @type {import('expo/config-plugins').ConfigPlugin<{ + * enableDictation?: boolean; + * dictationMicrophoneUsageDescription?: string; + * dictationSpeechRecognitionUsageDescription?: string; + * } | void>} + */ +const withStreamChatExpo = (config, props = {}) => { + const options = props ?? {}; + if (options.enableDictation === false) { + return withPodfileProperties(config, options); + } + + return withPlugins(config, [ + // ios + [withiOSInfoPlist, options], + [withPodfileProperties, options], + // android + [withAndroidManifestPermissions, options], + ]); +}; + +module.exports = createRunOncePlugin(withStreamChatExpo, pkg.name, pkg.version); diff --git a/package/expo-package/plugin/withAndroidManifest.js b/package/expo-package/plugin/withAndroidManifest.js new file mode 100644 index 0000000000..f6381eb22a --- /dev/null +++ b/package/expo-package/plugin/withAndroidManifest.js @@ -0,0 +1,30 @@ +const { withAndroidManifest } = require('expo/config-plugins'); + +/** Adds the `RECORD_AUDIO` permission dictation needs. */ +const withStreamAndroidManifest = (configuration) => + withAndroidManifest(configuration, (config) => { + const androidManifest = config.modResults; + const manifest = androidManifest.manifest; + + if (!Array.isArray(manifest['uses-permission'])) { + manifest['uses-permission'] = []; + } + + const permissions = manifest['uses-permission']; + + const hasRecordAudio = permissions.some( + (item) => item.$?.['android:name'] === 'android.permission.RECORD_AUDIO', + ); + + if (!hasRecordAudio) { + permissions.push({ + $: { + 'android:name': 'android.permission.RECORD_AUDIO', + }, + }); + } + + return config; + }); + +module.exports = withStreamAndroidManifest; diff --git a/package/expo-package/plugin/withPodfileProperties.js b/package/expo-package/plugin/withPodfileProperties.js new file mode 100644 index 0000000000..f76ef7bbb8 --- /dev/null +++ b/package/expo-package/plugin/withPodfileProperties.js @@ -0,0 +1,13 @@ +const { withPodfileProperties } = require('expo/config-plugins'); + +/** + * Tells the stream-chat-expo podspec whether to compile the dictation module in, through the + * `stream-chat.enableDictation` key of ios/Podfile.properties.json. + */ +const withStreamPodfileProperties = (configuration, props = {}) => + withPodfileProperties(configuration, (config) => { + config.modResults['stream-chat.enableDictation'] = String(props.enableDictation !== false); + return config; + }); + +module.exports = withStreamPodfileProperties; diff --git a/package/expo-package/plugin/withiOSInfoPlist.js b/package/expo-package/plugin/withiOSInfoPlist.js new file mode 100644 index 0000000000..355d4d3f99 --- /dev/null +++ b/package/expo-package/plugin/withiOSInfoPlist.js @@ -0,0 +1,35 @@ +const { withInfoPlist } = require('expo/config-plugins'); + +const DEFAULT_MICROPHONE_USAGE_DESCRIPTION = + '$(PRODUCT_NAME) uses the microphone to transcribe your voice into messages.'; +const DEFAULT_SPEECH_RECOGNITION_USAGE_DESCRIPTION = + '$(PRODUCT_NAME) uses speech recognition to transcribe your voice into messages.'; + +/** + * Adds the usage descriptions dictation needs. Explicit props win; otherwise descriptions the app + * already defines are kept, and defaults are only added when there are none, since App Store Connect + * rejects binaries that reference the Speech framework without one. + */ +const withiOSInfoPlist = (configuration, props = {}) => + withInfoPlist(configuration, (config) => { + const infoPlist = config.modResults; + + const { dictationMicrophoneUsageDescription, dictationSpeechRecognitionUsageDescription } = + props; + + if (typeof dictationMicrophoneUsageDescription === 'string') { + infoPlist.NSMicrophoneUsageDescription = dictationMicrophoneUsageDescription; + } else if (!infoPlist.NSMicrophoneUsageDescription) { + infoPlist.NSMicrophoneUsageDescription = DEFAULT_MICROPHONE_USAGE_DESCRIPTION; + } + + if (typeof dictationSpeechRecognitionUsageDescription === 'string') { + infoPlist.NSSpeechRecognitionUsageDescription = dictationSpeechRecognitionUsageDescription; + } else if (!infoPlist.NSSpeechRecognitionUsageDescription) { + infoPlist.NSSpeechRecognitionUsageDescription = DEFAULT_SPEECH_RECOGNITION_USAGE_DESCRIPTION; + } + + return config; + }); + +module.exports = withiOSInfoPlist; diff --git a/package/expo-package/react-native.config.js b/package/expo-package/react-native.config.js index 5159c2e46f..869c72c44f 100644 --- a/package/expo-package/react-native.config.js +++ b/package/expo-package/react-native.config.js @@ -4,6 +4,14 @@ module.exports = { android: { packageImportPath: 'import com.streamchatexpo.StreamChatExpoPackage;', packageInstance: 'new StreamChatExpoPackage()', + // A custom CMakeLists, needed to compile the hand-written PerfText shadow node next to the + // codegen output. Listing the descriptors replaces the ones autolinking would detect, so the + // codegen-generated StreamShimmerView descriptor has to be listed as well. + cmakeListsPath: 'src/main/jni/CMakeLists.txt', + componentDescriptors: [ + 'StreamShimmerViewComponentDescriptor', + 'PerfTextComponentDescriptor', + ], }, }, }, diff --git a/package/expo-package/src/native/NativeAIDictation.ts b/package/expo-package/src/native/NativeAIDictation.ts new file mode 100644 index 0000000000..b36c97e284 --- /dev/null +++ b/package/expo-package/src/native/NativeAIDictation.ts @@ -0,0 +1,48 @@ +import type { TurboModule } from 'react-native'; +import { TurboModuleRegistry } from 'react-native'; + +/** + * Codegen spec of the native dictation module. + * + * This file is duplicated (byte for byte) in `package/native-package/src/native/NativeAIDictation.ts` + * and `package/expo-package/src/native/NativeAIDictation.ts`, which are the copies codegen reads. + * `yarn validate-entry-isolation` fails if they drift apart. + * + * Events are delivered through `addListener`/`removeListeners` and a `NativeEventEmitter` (see + * `../native/AIDictation.ts`) rather than codegen `EventEmitter` properties, since the latter are + * not understood by the codegen of every React Native version the SDK supports (>= 0.76). + */ + +export type DictationStartOptions = { + language?: string; + interimResults?: boolean; + silenceTimeoutMs?: number; +}; + +export type DictationResult = { + text: string; + isFinal: boolean; +}; + +export type DictationStateValue = 'idle' | 'starting' | 'listening' | 'stopping'; +export type DictationState = { state: DictationStateValue }; + +export type DictationError = { + code: string; + message: string; +}; + +export interface Spec extends TurboModule { + start(options?: DictationStartOptions): void; + stop(): void; + cancel(): void; + isRecording(): boolean; + + addListener(eventName: string): void; + removeListeners(count: number): void; +} + +// `get` rather than `getEnforcing`: the module is only compiled in when dictation is enabled +// (always on Android, opt-in on iOS), and importing the AI entry point +// must not throw when it is absent. +export default TurboModuleRegistry.get('AIDictation'); diff --git a/package/expo-package/src/native/PerfTextNativeComponent.ts b/package/expo-package/src/native/PerfTextNativeComponent.ts new file mode 100644 index 0000000000..3c9ee1ce59 --- /dev/null +++ b/package/expo-package/src/native/PerfTextNativeComponent.ts @@ -0,0 +1,30 @@ +import type { ViewProps } from 'react-native'; +import type { Float, Int32 } from 'react-native/Libraries/Types/CodegenTypes'; +import codegenNativeComponent from 'react-native/Libraries/Utilities/codegenNativeComponent'; + +export type ColorRange = { + start: Int32; + end: Int32; + color: Int32; // processed to Android int on JS side +}; + +export interface NativeProps extends ViewProps { + text: string; + colorRanges?: ReadonlyArray; + fontSize?: Float; + fontFamily?: string; + lineHeight?: Float; +} + +// Applying interfaceOnly is extremely important, as we are going to +// override some of the codegen-generated code within our C++ bindings +// and without it it won't compile as there will be naming collisions. +// This essentially allows the compiler to pick the interface that's +// defined first (which we make sure is ours within CMakeLists.txt) and +// stick with that, ignoring anything else. +export default codegenNativeComponent('PerfText', { + // PerfText only exists on Android. Without this, RN 0.76 iOS codegen registers the component + // and the app fails to link on the missing `PerfTextCls` symbol. + excludedPlatforms: ['iOS'], + interfaceOnly: true, +}); diff --git a/package/expo-package/stream-chat-expo.podspec b/package/expo-package/stream-chat-expo.podspec index 0a9f977668..ead4ddac26 100644 --- a/package/expo-package/stream-chat-expo.podspec +++ b/package/expo-package/stream-chat-expo.podspec @@ -21,5 +21,32 @@ Pod::Spec.new do |s| s.source_files = "ios/**/*.{h,m,mm,swift}" s.private_header_files = "ios/**/*.h" + # Dictation (`ai-components` speech-to-text) references the Speech framework, and App Store Connect + # requires a speech recognition usage description from every app whose binary does. It is therefore + # only compiled into apps that opt in, through any of: + # - `$StreamChatEnableDictation = true` in the Podfile (before `use_native_modules!`) + # - `"stream-chat.enableDictation": "true"` in Podfile.properties.json (set by the Expo plugin) + # - the `STREAM_CHAT_ENABLE_DICTATION=1` environment variable during `pod install` + dictation_enabled = + begin + if defined?($StreamChatEnableDictation) && !$StreamChatEnableDictation.nil? + $StreamChatEnableDictation.to_s == "true" + elsif ENV["STREAM_CHAT_ENABLE_DICTATION"] + ["1", "true"].include?(ENV["STREAM_CHAT_ENABLE_DICTATION"]) + else + properties_path = File.join(Pod::Config.instance.installation_root.to_s, "Podfile.properties.json") + File.exist?(properties_path) && + JSON.parse(File.read(properties_path))["stream-chat.enableDictation"].to_s == "true" + end + rescue StandardError + false + end + + if dictation_enabled + s.frameworks = "Speech" + else + s.exclude_files = ["ios/shared/AIDictation.h", "ios/shared/AIDictation.mm", "ios/shared/AIDictationModule.swift"] + end + install_modules_dependencies(s) end diff --git a/package/jest-setup.tsx b/package/jest-setup.tsx index 2b30bb48f0..e6e7031cc1 100644 --- a/package/jest-setup.tsx +++ b/package/jest-setup.tsx @@ -141,6 +141,31 @@ jest.mock('react-native-teleport', () => { jest.mock('react-native-safe-area-context', () => mockSafeAreaContext); +// Skia and victory-native are optional peers used by the `ai-components` charts. Skia's JS needs its +// native module at import time, so both are replaced with inert stand-ins that render plain views. +jest.mock('@shopify/react-native-skia', () => ({ + LinearGradient: () => null, + matchFont: () => ({}), + vec: (x: number, y: number) => ({ x, y }), +})); + +jest.mock('victory-native', () => { + const React = require('react'); + const rn = require('react-native'); + const Primitive = () => null; + const Chart = ({ children, testID }: { children?: unknown; testID?: string }) => + React.createElement(rn.View, { testID }, typeof children === 'function' ? null : children); + return { + Area: Primitive, + Bar: Primitive, + CartesianChart: (props: object) => Chart({ ...props, testID: 'victory-cartesian-chart' }), + Line: Primitive, + Pie: { Chart: Primitive }, + PolarChart: (props: object) => Chart({ ...props, testID: 'victory-polar-chart' }), + Scatter: Primitive, + }; +}); + jest.mock('./src/components/Message/utils/measureInWindow', () => ({ measureInWindow: jest.fn(async () => ({ x: 10, diff --git a/package/native-package/ai-components/index.d.ts b/package/native-package/ai-components/index.d.ts new file mode 100644 index 0000000000..6bdce055e3 --- /dev/null +++ b/package/native-package/ai-components/index.d.ts @@ -0,0 +1 @@ +export * from 'stream-chat-react-native-core/ai-components'; diff --git a/package/native-package/ai-components/index.js b/package/native-package/ai-components/index.js new file mode 100644 index 0000000000..f53d6b982a --- /dev/null +++ b/package/native-package/ai-components/index.js @@ -0,0 +1,21 @@ +import { Platform } from 'react-native'; + +import { registerNativeHandlers } from 'stream-chat-react-native-core/ai-components'; + +import { NativeShimmerView } from '../src/optionalDependencies/NativeShimmerView'; + +// This entry point deliberately does not import `../src/index.js`: that file re-exports the whole +// core SDK, which Metro would then bundle into apps that only use the AI components. Instead, only +// the native pieces the AI components need are registered here. Registering them again when the +// main entry point is imported as well is harmless. + +// `PerfText` is an Android-only Fabric component. +registerNativeHandlers({ + NativeShimmerView, + PerfText: + Platform.OS === 'android' + ? require('../src/native/PerfTextNativeComponent').default + : undefined, +}); + +export * from 'stream-chat-react-native-core/ai-components'; diff --git a/package/native-package/android/src/main/java/com/streamchatreactnative/StreamChatReactNativePackage.java b/package/native-package/android/src/main/java/com/streamchatreactnative/StreamChatReactNativePackage.java index fc3b5e060e..0a07a2c2b4 100644 --- a/package/native-package/android/src/main/java/com/streamchatreactnative/StreamChatReactNativePackage.java +++ b/package/native-package/android/src/main/java/com/streamchatreactnative/StreamChatReactNativePackage.java @@ -9,13 +9,14 @@ import com.facebook.react.uimanager.ViewManager; import java.util.HashMap; -import java.util.Collections; +import java.util.Arrays; import java.util.List; import java.util.Map; public class StreamChatReactNativePackage extends TurboReactPackage { private static final String STREAM_MULTIPART_UPLOADER_MODULE = "StreamMultipartUploader"; private static final String STREAM_VIDEO_THUMBNAIL_MODULE = "StreamVideoThumbnail"; + private static final String AI_DICTATION_MODULE = "AIDictation"; @Nullable @Override @@ -32,6 +33,11 @@ public NativeModule getModule(String name, ReactApplicationContext reactContext) "com.streamchatreactnative.StreamVideoThumbnailModule", reactContext ); + } else if (name.equals(AI_DICTATION_MODULE)) { + return createNewArchModule( + "com.streamchatreactnative.AIDictationModule", + reactContext + ); } else { return null; } @@ -74,13 +80,25 @@ public ReactModuleInfoProvider getReactModuleInfoProvider() { false, // isCxxModule true // isTurboModule )); + + moduleInfos.put( + AI_DICTATION_MODULE, + new ReactModuleInfo( + AI_DICTATION_MODULE, + AI_DICTATION_MODULE, + false, // canOverrideExistingModule + false, // needsEagerInit + false, // hasConstants + false, // isCxxModule + true // isTurboModule + )); return moduleInfos; }; } @Override public List createViewManagers(ReactApplicationContext reactContext) { - return Collections.singletonList(new StreamShimmerViewManager()); + return Arrays.asList(new StreamShimmerViewManager(), new PerfTextViewManager()); } @Nullable diff --git a/package/native-package/android/src/main/jni/CMakeLists.txt b/package/native-package/android/src/main/jni/CMakeLists.txt new file mode 100644 index 0000000000..b6edc3eb05 --- /dev/null +++ b/package/native-package/android/src/main/jni/CMakeLists.txt @@ -0,0 +1,63 @@ +cmake_minimum_required(VERSION 3.13) +set(CMAKE_VERBOSE_MAKEFILE on) + +# Custom CMakeLists for this package's codegen library, registered through `cmakeListsPath` in +# react-native.config.js. It builds everything the codegen-generated CMakeLists would, plus the +# hand-written C++ of the Android-only PerfText component used by `ai-components`. + +set(LIB_LITERAL StreamChatReactNativeSpec) +set(LIB_TARGET_NAME react_codegen_${LIB_LITERAL}) + +# REALPATH: the package is often reached through a symlink (workspaces, `yarn link`), and CMake +# collapses `..` lexically, which would otherwise resolve relative paths against the symlink. +get_filename_component(LIB_ANDROID_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../.. REALPATH) +set(LIB_GENERATED_JNI_DIR ${LIB_ANDROID_DIR}/build/generated/source/codegen/jni) +set(LIB_GENERATED_COMPONENTS_DIR ${LIB_GENERATED_JNI_DIR}/react/renderer/components/${LIB_LITERAL}) + +# Shared C++ sources: read straight from the monorepo during development, otherwise from the copy +# synced into this package (scripts/sync-shared-native.sh, run before publishing). +get_filename_component(MONOREPO_SHARED_CPP_DIR ${LIB_ANDROID_DIR}/../../shared-native/android/jni REALPATH) +if(EXISTS ${MONOREPO_SHARED_CPP_DIR}) + set(SHARED_CPP_DIR ${MONOREPO_SHARED_CPP_DIR}) +else() + set(SHARED_CPP_DIR ${LIB_ANDROID_DIR}/src/main/java/com/streamchatreactnative/shared/jni) +endif() + +file(GLOB LIB_SRCS CONFIGURE_DEPENDS + ${LIB_GENERATED_JNI_DIR}/*.cpp + ${LIB_GENERATED_COMPONENTS_DIR}/*.cpp + ${SHARED_CPP_DIR}/*.cpp +) + +add_library(${LIB_TARGET_NAME} OBJECT ${LIB_SRCS}) + +# This directory must come first: see StreamChatReactNativeSpec.h next to this file. +target_include_directories( + ${LIB_TARGET_NAME} + PUBLIC + ${CMAKE_CURRENT_SOURCE_DIR} + ${SHARED_CPP_DIR} + ${LIB_GENERATED_JNI_DIR} + ${LIB_GENERATED_COMPONENTS_DIR} +) + +target_link_libraries( + ${LIB_TARGET_NAME} + fbjni + jsi + reactnative +) + +if(ReactAndroid_VERSION_MINOR GREATER_EQUAL 80) + target_compile_reactnative_options(${LIB_TARGET_NAME} PRIVATE) +else() + target_compile_options( + ${LIB_TARGET_NAME} + PRIVATE + -DLOG_TAG=\"ReactNative\" + -fexceptions + -frtti + -std=c++20 + -Wall + ) +endif() diff --git a/package/native-package/android/src/main/jni/StreamChatReactNativeSpec.h b/package/native-package/android/src/main/jni/StreamChatReactNativeSpec.h new file mode 100644 index 0000000000..1f2c244c68 --- /dev/null +++ b/package/native-package/android/src/main/jni/StreamChatReactNativeSpec.h @@ -0,0 +1,9 @@ +#pragma once + +// The app's autolinking includes `` to register this package's TurboModules and +// component descriptors. This directory comes first on the include path (see CMakeLists.txt), so +// that include lands here: forward to the codegen-generated header, and add the hand-written +// PerfText component descriptor, which codegen does not generate (`interfaceOnly` component). +#include_next + +#include "PerfTextComponentDescriptor.h" diff --git a/package/native-package/android/src/main/jni/StreamCodegenComponents.h b/package/native-package/android/src/main/jni/StreamCodegenComponents.h new file mode 100644 index 0000000000..8295288aa5 --- /dev/null +++ b/package/native-package/android/src/main/jni/StreamCodegenComponents.h @@ -0,0 +1,7 @@ +#pragma once + +// The codegen-generated component headers of this package, for the shared PerfText C++ +// (shared-native/android/jni), which is compiled into both SDK packages and so cannot name the +// package's codegen library itself. +#include +#include diff --git a/package/native-package/android/src/newarch/com/streamchatreactnative/AIDictationModule.kt b/package/native-package/android/src/newarch/com/streamchatreactnative/AIDictationModule.kt new file mode 100644 index 0000000000..25f0e64430 --- /dev/null +++ b/package/native-package/android/src/newarch/com/streamchatreactnative/AIDictationModule.kt @@ -0,0 +1,49 @@ +package com.streamchatreactnative + +import com.facebook.react.bridge.ReactApplicationContext +import com.facebook.react.bridge.ReadableMap +import com.facebook.react.modules.core.DeviceEventManagerModule +import com.streamchatreactnative.shared.dictation.AIDictationController + +/** + * The `AIDictation` TurboModule used by `ai-components` speech-to-text. The implementation is + * shared with the other SDK package; see [AIDictationController]. + */ +class AIDictationModule( + reactContext: ReactApplicationContext, +) : NativeAIDictationSpec(reactContext) { + private val controller = + AIDictationController(reactContext) { eventName, payload -> + reactApplicationContext + .getJSModule(DeviceEventManagerModule.RCTDeviceEventEmitter::class.java) + .emit(eventName, payload) + } + + override fun getName(): String = NAME + + override fun initialize() { + super.initialize() + controller.initialize() + } + + override fun invalidate() { + super.invalidate() + controller.invalidate() + } + + override fun start(options: ReadableMap?) = controller.start(options) + + override fun stop() = controller.stop() + + override fun cancel() = controller.cancel() + + override fun isRecording(): Boolean = controller.isRecording() + + override fun addListener(eventName: String) = Unit + + override fun removeListeners(count: Double) = Unit + + companion object { + const val NAME = AIDictationController.NAME + } +} diff --git a/package/native-package/package.json b/package/native-package/package.json index 608e47a985..2cade0adf6 100644 --- a/package/native-package/package.json +++ b/package/native-package/package.json @@ -22,7 +22,8 @@ "ios", "*.podspec", "react-native.config.js", - "package.json" + "package.json", + "ai-components" ], "license": "SEE LICENSE IN LICENSE", "main": "src/index.js", @@ -36,6 +37,7 @@ "@react-native-camera-roll/camera-roll": ">=7.9.0", "@react-native-clipboard/clipboard": ">=1.16.0", "@react-native-documents/picker": ">=10.1.1", + "@shopify/react-native-skia": ">=2.0.0", "react-native": ">=0.76.0", "react-native-audio-recorder-player": ">=3.6.13", "react-native-blob-util": ">=0.22.0", @@ -43,7 +45,8 @@ "react-native-image-picker": ">=7.1.2", "react-native-nitro-sound": ">=0.2.9", "react-native-share": ">=11.0.0", - "react-native-video": ">=6.18.0" + "react-native-video": ">=6.18.0", + "victory-native": ">=41.0.0" }, "peerDependenciesMeta": { "@react-native-camera-roll/camera-roll": { @@ -55,6 +58,9 @@ "@react-native-documents/picker": { "optional": true }, + "@shopify/react-native-skia": { + "optional": true + }, "react-native-audio-recorder-player": { "optional": true }, @@ -75,6 +81,9 @@ }, "react-native-video": { "optional": true + }, + "victory-native": { + "optional": true } }, "scripts": { diff --git a/package/native-package/react-native.config.js b/package/native-package/react-native.config.js index 401a529c65..765869cc32 100644 --- a/package/native-package/react-native.config.js +++ b/package/native-package/react-native.config.js @@ -4,6 +4,14 @@ module.exports = { android: { packageImportPath: 'import com.streamchatreactnative.StreamChatReactNativePackage;', packageInstance: 'new StreamChatReactNativePackage()', + // A custom CMakeLists, needed to compile the hand-written PerfText shadow node next to the + // codegen output. Listing the descriptors replaces the ones autolinking would detect, so the + // codegen-generated StreamShimmerView descriptor has to be listed as well. + cmakeListsPath: 'src/main/jni/CMakeLists.txt', + componentDescriptors: [ + 'StreamShimmerViewComponentDescriptor', + 'PerfTextComponentDescriptor', + ], }, }, }, diff --git a/package/native-package/src/native/NativeAIDictation.ts b/package/native-package/src/native/NativeAIDictation.ts new file mode 100644 index 0000000000..b36c97e284 --- /dev/null +++ b/package/native-package/src/native/NativeAIDictation.ts @@ -0,0 +1,48 @@ +import type { TurboModule } from 'react-native'; +import { TurboModuleRegistry } from 'react-native'; + +/** + * Codegen spec of the native dictation module. + * + * This file is duplicated (byte for byte) in `package/native-package/src/native/NativeAIDictation.ts` + * and `package/expo-package/src/native/NativeAIDictation.ts`, which are the copies codegen reads. + * `yarn validate-entry-isolation` fails if they drift apart. + * + * Events are delivered through `addListener`/`removeListeners` and a `NativeEventEmitter` (see + * `../native/AIDictation.ts`) rather than codegen `EventEmitter` properties, since the latter are + * not understood by the codegen of every React Native version the SDK supports (>= 0.76). + */ + +export type DictationStartOptions = { + language?: string; + interimResults?: boolean; + silenceTimeoutMs?: number; +}; + +export type DictationResult = { + text: string; + isFinal: boolean; +}; + +export type DictationStateValue = 'idle' | 'starting' | 'listening' | 'stopping'; +export type DictationState = { state: DictationStateValue }; + +export type DictationError = { + code: string; + message: string; +}; + +export interface Spec extends TurboModule { + start(options?: DictationStartOptions): void; + stop(): void; + cancel(): void; + isRecording(): boolean; + + addListener(eventName: string): void; + removeListeners(count: number): void; +} + +// `get` rather than `getEnforcing`: the module is only compiled in when dictation is enabled +// (always on Android, opt-in on iOS), and importing the AI entry point +// must not throw when it is absent. +export default TurboModuleRegistry.get('AIDictation'); diff --git a/package/native-package/src/native/PerfTextNativeComponent.ts b/package/native-package/src/native/PerfTextNativeComponent.ts new file mode 100644 index 0000000000..3c9ee1ce59 --- /dev/null +++ b/package/native-package/src/native/PerfTextNativeComponent.ts @@ -0,0 +1,30 @@ +import type { ViewProps } from 'react-native'; +import type { Float, Int32 } from 'react-native/Libraries/Types/CodegenTypes'; +import codegenNativeComponent from 'react-native/Libraries/Utilities/codegenNativeComponent'; + +export type ColorRange = { + start: Int32; + end: Int32; + color: Int32; // processed to Android int on JS side +}; + +export interface NativeProps extends ViewProps { + text: string; + colorRanges?: ReadonlyArray; + fontSize?: Float; + fontFamily?: string; + lineHeight?: Float; +} + +// Applying interfaceOnly is extremely important, as we are going to +// override some of the codegen-generated code within our C++ bindings +// and without it it won't compile as there will be naming collisions. +// This essentially allows the compiler to pick the interface that's +// defined first (which we make sure is ours within CMakeLists.txt) and +// stick with that, ignoring anything else. +export default codegenNativeComponent('PerfText', { + // PerfText only exists on Android. Without this, RN 0.76 iOS codegen registers the component + // and the app fails to link on the missing `PerfTextCls` symbol. + excludedPlatforms: ['iOS'], + interfaceOnly: true, +}); diff --git a/package/native-package/stream-chat-react-native.podspec b/package/native-package/stream-chat-react-native.podspec index d1c4a58a24..aef09c54f1 100644 --- a/package/native-package/stream-chat-react-native.podspec +++ b/package/native-package/stream-chat-react-native.podspec @@ -21,5 +21,32 @@ Pod::Spec.new do |s| s.source_files = "ios/**/*.{h,m,mm,swift}" s.private_header_files = "ios/**/*.h" + # Dictation (`ai-components` speech-to-text) references the Speech framework, and App Store Connect + # requires a speech recognition usage description from every app whose binary does. It is therefore + # only compiled into apps that opt in, through any of: + # - `$StreamChatEnableDictation = true` in the Podfile (before `use_native_modules!`) + # - `"stream-chat.enableDictation": "true"` in Podfile.properties.json (set by the Expo plugin) + # - the `STREAM_CHAT_ENABLE_DICTATION=1` environment variable during `pod install` + dictation_enabled = + begin + if defined?($StreamChatEnableDictation) && !$StreamChatEnableDictation.nil? + $StreamChatEnableDictation.to_s == "true" + elsif ENV["STREAM_CHAT_ENABLE_DICTATION"] + ["1", "true"].include?(ENV["STREAM_CHAT_ENABLE_DICTATION"]) + else + properties_path = File.join(Pod::Config.instance.installation_root.to_s, "Podfile.properties.json") + File.exist?(properties_path) && + JSON.parse(File.read(properties_path))["stream-chat.enableDictation"].to_s == "true" + end + rescue StandardError + false + end + + if dictation_enabled + s.frameworks = "Speech" + else + s.exclude_files = ["ios/shared/AIDictation.h", "ios/shared/AIDictation.mm", "ios/shared/AIDictationModule.swift"] + end + install_modules_dependencies(s) end diff --git a/package/package.json b/package/package.json index ae0d4f189c..b71db5fb89 100644 --- a/package/package.json +++ b/package/package.json @@ -20,7 +20,8 @@ "src", "lib", "package.json", - "README.md" + "README.md", + "ai-components" ], "scripts": { "shared-native:sync": "bash ./scripts/sync-shared-native.sh all", @@ -35,6 +36,7 @@ "test:unit": "TZ=UTC jest", "typecheck": "tsc --noEmit -p tsconfig.test.json", "validate-translations": "node bin/validate-translations.js", + "validate-entry-isolation": "node bin/validate-entry-isolation.js", "get-version": "echo $npm_package_version", "version": "bash ./bin/before-tag.sh", "bootstrap": "yarn install", @@ -67,32 +69,39 @@ }, "dependencies": { "@gorhom/bottom-sheet": "^5.2.14", + "@khanacademy/simple-markdown": "^2.1.0", + "@stream-io/state-store": "^1.1.6", "@ungap/structured-clone": "^1.3.0", "dayjs": "1.11.13", "emoji-regex": "^10.4.0", "i18next": "^25.2.1", "intl-pluralrules": "^2.0.1", "linkifyjs": "^4.3.2", + "lodash": "4.18.1", "lodash-es": "4.18.1", "mime-types": "^2.1.35", "path": "0.12.7", "react-native-markdown-package": "1.8.2", "react-native-url-polyfill": "^2.0.0", + "react-syntax-highlighter": "^16.1.0", "stream-chat": "^9.54.0", - "use-sync-external-store": "^1.5.0" + "use-sync-external-store": "^1.5.0", + "zod": "^4.1.12" }, "peerDependencies": { "@emoji-mart/data": ">=1.1.0", "@op-engineering/op-sqlite": ">=14.0.0", "@react-native-community/netinfo": ">=11.3.1", "@shopify/flash-list": ">=2.1.0", + "@shopify/react-native-skia": ">=2.0.0", "emoji-mart": ">=5.4.0", "react-native": ">=0.76.0", "react-native-gesture-handler": ">=2.18.0", "react-native-reanimated": ">=3.16.0", "react-native-safe-area-context": ">=5.4.1", "react-native-svg": ">=15.8.0", - "react-native-teleport": ">=0.5.4" + "react-native-teleport": ">=0.5.4", + "victory-native": ">=41.0.0" }, "peerDependenciesMeta": { "@emoji-mart/data": { @@ -104,8 +113,14 @@ "@shopify/flash-list": { "optional": true }, + "@shopify/react-native-skia": { + "optional": true + }, "emoji-mart": { "optional": true + }, + "victory-native": { + "optional": true } }, "devDependencies": { @@ -116,6 +131,7 @@ "@react-native/babel-preset": "0.87.0", "@react-native/jest-preset": "0.87.0", "@shopify/flash-list": "^2.3.3", + "@shopify/react-native-skia": "^2.13.0", "@stream-io/typescript-config": "workspace:^", "@testing-library/jest-native": "^5.4.3", "@testing-library/react-native": "13.2.0", @@ -127,6 +143,7 @@ "@types/lodash": "4.17.16", "@types/mime-types": "2.1.4", "@types/react": "^19.2.0", + "@types/react-syntax-highlighter": "^15.5.13", "@types/react-test-renderer": "19.1.0", "@types/ungap__structured-clone": "^1.2.0", "@types/use-sync-external-store": "^1.5.0", @@ -150,7 +167,8 @@ "react-test-renderer": "19.2.3", "rimraf": "^6.0.1", "typescript": "6.0.3", - "uuid": "^11.1.0" + "uuid": "^11.1.0", + "victory-native": "^42.0.1" }, "resolutions": { "@types/react": "^19.2.0" diff --git a/package/shared-native/android/PerfTextView.kt b/package/shared-native/android/PerfTextView.kt new file mode 100644 index 0000000000..02f6103c6c --- /dev/null +++ b/package/shared-native/android/PerfTextView.kt @@ -0,0 +1,61 @@ +package com.streamchatreactnative + +import android.content.Context +import android.text.SpannableString +import android.text.Spanned +import android.text.style.ForegroundColorSpan +import android.graphics.Typeface +import androidx.appcompat.widget.AppCompatTextView +import com.facebook.react.common.assets.ReactFontManager + +class PerfTextView(context: Context) : AppCompatTextView(context) { + + data class ColorRange(val start: Int, val end: Int, val color: Int) + + private var colorRanges: List = emptyList() + + fun setTextValue(value: String) { + applyTextAndColors(value, colorRanges) + } + + fun setColorRanges(ranges: List) { + colorRanges = ranges + applyTextAndColors(text?.toString().orEmpty(), colorRanges) + } + + fun setFontFamilyCompat(fontFamily: String?) { + val style = typeface?.style ?: Typeface.NORMAL + val fontManager = ReactFontManager.getInstance() + + val tf: Typeface = fontManager.getTypeface( + fontFamily ?: "monospace", + style, + context.assets, + ) + + typeface = tf + } + + private fun applyTextAndColors(value: String, ranges: List) { + if (value.isEmpty()) { + setText(value) + return + } + + val spannable = SpannableString(value) + for (r in ranges) { + val start = r.start.coerceAtLeast(0).coerceAtMost(value.length) + val end = r.end.coerceAtLeast(start).coerceAtMost(value.length) + if (start == end) continue + spannable.setSpan( + ForegroundColorSpan(r.color), + start, + end, + Spanned.SPAN_EXCLUSIVE_EXCLUSIVE, + ) + } + text = spannable + } +} + + diff --git a/package/shared-native/android/PerfTextViewManager.kt b/package/shared-native/android/PerfTextViewManager.kt new file mode 100644 index 0000000000..76a03d5df8 --- /dev/null +++ b/package/shared-native/android/PerfTextViewManager.kt @@ -0,0 +1,143 @@ +package com.streamchatreactnative + +import android.content.Context +import android.view.View +import com.facebook.react.bridge.ColorPropConverter +import com.facebook.react.bridge.Dynamic +import com.facebook.react.bridge.ReadableArray +import com.facebook.react.bridge.ReadableMap +import com.facebook.react.bridge.ReadableType +import com.facebook.react.module.annotations.ReactModule +import com.facebook.react.uimanager.PixelUtil +import com.facebook.react.uimanager.SimpleViewManager +import com.facebook.react.uimanager.ThemedReactContext +import com.facebook.react.uimanager.annotations.ReactProp +import com.facebook.yoga.YogaMeasureMode +import com.facebook.yoga.YogaMeasureOutput +import kotlin.math.ceil + +@ReactModule(name = "PerfText") +class PerfTextViewManager : SimpleViewManager() { + + override fun getName() = "PerfText" + override fun createViewInstance(rc: ThemedReactContext) = PerfTextView(rc) + + @ReactProp(name = "text") + fun setText(v: PerfTextView, text: String?) { + v.setTextValue(text ?: "") + } + + @ReactProp(name = "colorRanges") + fun setColorRanges(v: PerfTextView, arr: ReadableArray?) { + val out = mutableListOf() + if (arr != null) { + for (i in 0 until arr.size()) { + val m = arr.getMap(i) + val start = m!!.getInt("start") + val end = m.getInt("end") + val dyn: Dynamic = m.getDynamic("color") + val color = when (dyn.type) { + ReadableType.Number -> dyn.asInt() + ReadableType.String -> ColorPropConverter.getColor(dyn.asString(), v.context) + else -> 0xff000000.toInt() + } ?: 0xff000000.toInt() + out.add(PerfTextView.ColorRange(start, end, color)) + } + } + v.setColorRanges(out) + } + + @ReactProp(name = "fontSize") + fun setFontSize(v: PerfTextView, sizeSp: Double) { + if (sizeSp > 0) v.textSize = sizeSp.toFloat() + } + + @ReactProp(name = "lineHeight") + fun setLineHeight(v: PerfTextView, height: Double) { + val targetPx = PixelUtil.toPixelFromDIP(height.toFloat()) + if (height > 0) { + v.setLineSpacing(0f, targetPx / v.textSize) + } else { + // default to no multiplier rather than default line height + v.setLineSpacing(0.0f, 1.0f) + }; + } + + @ReactProp(name = "fontFamily") + fun setFontFamily(v: PerfTextView, family: String?) { + v.setFontFamilyCompat(family) + } + + // A fabric measurement hook. + // Will only be called by FabricUIManager.measure(), via the C++ measurement manager. + override fun measure( + context: Context, + localData: ReadableMap?, + props: ReadableMap?, + state: ReadableMap?, + width: Float, + widthMode: YogaMeasureMode?, + height: Float, + heightMode: YogaMeasureMode?, + attachmentsPositions: FloatArray? + ): Long { + val view = PerfTextView(context) + + // Apply props that influence size (text, fontSize, lineHeight) + props?.let { p -> + if (p.hasKey("fontSize") && !p.isNull("fontSize")) { + val fs = p.getDouble("fontSize") + if (fs > 0.0) { + view.textSize = fs.toFloat() + } + } + + if (p.hasKey("fontFamily") && !p.isNull("fontFamily")) { + view.setFontFamilyCompat(p.getString("fontFamily")) + } + + if (p.hasKey("lineHeight") && !p.isNull("lineHeight")) { + val lh = p.getDouble("lineHeight") + if (lh > 0.0) { + val targetPx = PixelUtil.toPixelFromDIP(lh.toFloat()) + view.setLineSpacing(0.0f, targetPx / view.textSize) + } else { + view.setLineSpacing(0.0f, 1.0f) + } + } + + if (p.hasKey("text") && !p.isNull("text")) { + view.setTextValue(p.getString("text") ?: "") + } + // colorRanges donโ€™t affect size so we can ignore them for measurement + } + val widthPx = if (!width.isNaN()) PixelUtil.toPixelFromDIP(width) else 0f + val heightPx = if (!height.isNaN()) PixelUtil.toPixelFromDIP(height) else 0f + + val widthSpec = when (widthMode) { + YogaMeasureMode.EXACTLY -> + View.MeasureSpec.makeMeasureSpec(widthPx.toInt(), View.MeasureSpec.EXACTLY) + YogaMeasureMode.AT_MOST -> + View.MeasureSpec.makeMeasureSpec(widthPx.toInt(), View.MeasureSpec.AT_MOST) + else -> + View.MeasureSpec.makeMeasureSpec(0, View.MeasureSpec.UNSPECIFIED) + } + + val heightSpec = when (heightMode) { + YogaMeasureMode.EXACTLY -> + View.MeasureSpec.makeMeasureSpec(heightPx.toInt(), View.MeasureSpec.EXACTLY) + YogaMeasureMode.AT_MOST -> + View.MeasureSpec.makeMeasureSpec(heightPx.toInt(), View.MeasureSpec.AT_MOST) + else -> + View.MeasureSpec.makeMeasureSpec(0, View.MeasureSpec.UNSPECIFIED) + } + + view.measure(widthSpec, heightSpec) + + val measuredWidthDp = ceil(PixelUtil.toDIPFromPixel(view.measuredWidth.toFloat())) + val measuredHeightDp = ceil(PixelUtil.toDIPFromPixel(view.measuredHeight.toFloat())) + + return YogaMeasureOutput.make(measuredWidthDp * 1.6f, measuredHeightDp * 1.0f) + } +} + diff --git a/package/shared-native/android/dictation/AIDictationController.kt b/package/shared-native/android/dictation/AIDictationController.kt new file mode 100644 index 0000000000..a06fae8cf9 --- /dev/null +++ b/package/shared-native/android/dictation/AIDictationController.kt @@ -0,0 +1,514 @@ +package com.streamchatreactnative.shared.dictation + +import android.Manifest +import android.content.Intent +import android.content.pm.PackageManager +import android.os.Bundle +import android.os.Handler +import android.os.Looper +import android.speech.RecognitionListener +import android.speech.RecognizerIntent +import android.speech.SpeechRecognizer +import androidx.core.content.ContextCompat +import com.facebook.react.bridge.Arguments +import com.facebook.react.bridge.LifecycleEventListener +import com.facebook.react.bridge.ReactApplicationContext +import com.facebook.react.bridge.ReadableMap +import com.facebook.react.bridge.ReadableType +import com.facebook.react.bridge.WritableMap +import java.util.Locale +import java.util.concurrent.atomic.AtomicBoolean + +/** + * Speech-to-text for the `ai-components` dictation button, backed by [SpeechRecognizer]. + * + * Shared by the `AIDictation` TurboModule of both SDK packages (each package's codegen generates its + * own `NativeAIDictationSpec`), which forward their calls here. Events are delivered through + * [emitEvent] with the names in [EVENT_ON_STATE], [EVENT_ON_RESULT] and [EVENT_ON_ERROR]. + */ +class AIDictationController( + private val reactContext: ReactApplicationContext, + private val emitEvent: (eventName: String, payload: WritableMap) -> Unit, +) : LifecycleEventListener { + + // --------------------------------------------------------------------------- + // Module constants + // --------------------------------------------------------------------------- + + companion object { + const val NAME = "AIDictation" + + const val EVENT_ON_STATE = "AIDictationOnState" + const val EVENT_ON_RESULT = "AIDictationOnResult" + const val EVENT_ON_ERROR = "AIDictationOnError" + + private const val STATE_IDLE = "idle" + private const val STATE_STARTING = "starting" + private const val STATE_LISTENING = "listening" + private const val STATE_STOPPING = "stopping" + + private const val ERROR_NO_PERMISSION = "NO_PERMISSION" + private const val ERROR_NOT_AVAILABLE = "NOT_AVAILABLE" + private const val ERROR_INTERNAL = "INTERNAL_ERROR" + private const val ERROR_ANDROID = "ANDROID_ERROR" + + private const val DEFAULT_SILENCE_TIMEOUT_MS = 2500L + } + + private val mainHandler = Handler(Looper.getMainLooper()) + private var speechRecognizer: SpeechRecognizer? = null + + private val isListening = AtomicBoolean(false) + private var interimResultsEnabled: Boolean = true + private var currentLocale: Locale = Locale.getDefault() + private var silenceTimeoutMs: Long = DEFAULT_SILENCE_TIMEOUT_MS + private var lastSpeechTimeMs: Long = 0L + + private var isSpeechInProgress: Boolean = false + + private var silenceRunnable: Runnable? = null + + /** Called from the owning module's `initialize()`. */ + fun initialize() { + reactContext.addLifecycleEventListener(this) + } + + /** Called from the owning module's `invalidate()`. */ + fun invalidate() { + reactContext.removeLifecycleEventListener(this) + cleanupRecognizer() + } + + // --------------------------------------------------------------------------- + // AIDictation API implementation + // --------------------------------------------------------------------------- + + /** + * Starts a new dictation session with the given options. + * + * - No-ops if a session is already active + * - Verifies the RECORD_AUDIO permission and speech recognition availability before starting + * - On success, configures and starts [SpeechRecognizer] as well as the silence monitor + * + * @param options Optional configuration map: + * - `interimResults` (boolean, default `true`) โ€“ whether to emit partial/interim results + * - `language` (string, BCP-47 tag) โ€“ language/locale for recognition; falls back to + * [Locale.getDefault] if omitted or blank + * - `silenceTimeoutMs` (number) โ€“ max allowed silence in milliseconds before + * auto-stopping; falls back to [DEFAULT_SILENCE_TIMEOUT_MS] if missing or invalid + */ + fun start(options: ReadableMap?) { + if (isListening.get()) return + + emitState(STATE_STARTING) + + interimResultsEnabled = options?.getBooleanSafe("interimResults") ?: true + + val languageTag = options?.getStringSafe("language") + currentLocale = if (languageTag.isNullOrBlank()) { + Locale.getDefault() + } else { + Locale.forLanguageTag(languageTag) + } + + val silenceOptMs = options?.getDoubleSafe("silenceTimeoutMs") + silenceTimeoutMs = if (silenceOptMs != null && silenceOptMs > 0) { + silenceOptMs.toLong() + } else { + DEFAULT_SILENCE_TIMEOUT_MS + } + + if (!hasAudioPermission()) { + emitError( + ERROR_NO_PERMISSION, + "RECORD_AUDIO permission not granted. Request it in JS before calling start().", + ) + emitState(STATE_IDLE) + return + } + + if (!SpeechRecognizer.isRecognitionAvailable(reactContext)) { + emitError( + ERROR_NOT_AVAILABLE, + "Speech recognition is not available on this device.", + ) + emitState(STATE_IDLE) + return + } + + mainHandler.post { + try { + setupRecognizerIfNeeded() + + val intent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply { + putExtra( + RecognizerIntent.EXTRA_LANGUAGE_MODEL, + RecognizerIntent.LANGUAGE_MODEL_FREE_FORM, + ) + putExtra(RecognizerIntent.EXTRA_LANGUAGE, currentLocale.toLanguageTag()) + putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true) + putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 3) + putExtra(RecognizerIntent.EXTRA_PROMPT, "Listening...") + } + + lastSpeechTimeMs = System.currentTimeMillis() + isSpeechInProgress = false + isListening.set(true) + emitState(STATE_LISTENING) + + speechRecognizer?.startListening(intent) + startSilenceMonitor() + } catch (t: Throwable) { + isListening.set(false) + emitState(STATE_IDLE) + emitError( + ERROR_INTERNAL, + "Failed to start recognition: ${t.message ?: t.javaClass.simpleName}", + ) + } + } + } + + /** + * Requests a graceful stop of the current dictation session. + * + * - No-ops if we are not currently listening + * - Emits [STATE_STOPPING] so JS can show a "stopping" state + * - Stops the silence monitor + * - On the main thread, calls [SpeechRecognizer.stopListening] to let + * Android finish the current utterance and deliver final results + * + * The recognizer will transition to [STATE_IDLE] via [onResults] or [onError]. + */ + fun stop() { + if (!isListening.get()) return + + emitState(STATE_STOPPING) + stopSilenceMonitor() + + mainHandler.post { + try { + speechRecognizer?.stopListening() + } catch (_: Throwable) { + // ignore + } + } + } + + /** + * Immediately cancels the current dictation session. + * + * - No-ops if we are not currently listening + * - Stops the silence monitor + * - On the main thread, calls [SpeechRecognizer.cancel] to abort recognition + * - Regardless of the cancel outcome, marks `isListening` false and emits + * [STATE_IDLE] so JS knows dictation has fully stopped + */ + fun cancel() { + if (!isListening.get()) return + + stopSilenceMonitor() + + mainHandler.post { + try { + speechRecognizer?.cancel() + } catch (_: Throwable) { + // ignore + } finally { + isListening.set(false) + emitState(STATE_IDLE) + } + } + } + + /** + * Synchronously reports whether a dictation session is currently active. + * + * Exposed as a blocking synchronous method to JS so callers can quickly check + * recording state without going through an async event. + */ + fun isRecording(): Boolean { + return isListening.get() + } + + // --------------------------------------------------------------------------- + // React Native host lifecycle callbacks + // --------------------------------------------------------------------------- + + + /** + * Called when the React Native host Activity is resumed (app comes to foreground). + * + * We currently donโ€™t need to resume or re-attach anything for dictation here, + * so this is intentionally a no-op. + */ + override fun onHostResume() = Unit + + /** + * Called when the React Native host Activity is paused (app goes to background + * or another Activity is shown on top). + * + * If we are actively listening for speech when this happens, we cancel the + * current dictation session. This avoids: + * + * - Keeping the microphone active while the app is in the background + * - Letting the platform speech recognizer run in an invalid/paused state + * + * `cancel()` is used instead of `stop()` so the partial utterance is treated as + * aborted and not as a successfully completed transcription. + */ + override fun onHostPause() { + if (isListening.get()) { + cancel() + } + } + + /** + * Called when the React Native host Activity is being destroyed. + * + * This is the last chance to clean up any native resources held by the + * speech recognizer (listeners, microphone, underlying platform objects). + * Technically, we typically want to do this within React Native code itself, + * (i.e hook cleanup) however that might not always be enough in certain + * scenarios and we keep this as a safety measure to make sure any memory leaks + * don't occur. + */ + override fun onHostDestroy() { + cleanupRecognizer() + } + + // --------------------------------------------------------------------------- + // Internal + // --------------------------------------------------------------------------- + + /** + * Creates (if needed) and configures the Android [SpeechRecognizer] instance. + * + * The recognizer is bound to the current [reactContext] and wired to forward + * Android recognition callbacks into the JavaScript side via: + * + * - [emitResult] for partial and final transcription text + * - [emitAndroidError] for platform error codes + * - [emitState] to update the high-level dictation state (i.e IDLE vs LISTENING) + * + * Callback behavior: + * + * - [RecognitionListener.onError]: + * - Cleans up all relevant timers, resets the state to [STATE_IDLE] and emits a JS error event + * + * - [RecognitionListener.onResults]: + * - Considered to be the end of the listening session (happening natively, so not stopped or canceled) + * and extracts the best final transcript from [SpeechRecognizer.RESULTS_RECOGNITION] + * + * - [RecognitionListener.onPartialResults]: + * - Similar to [RecognitionListener.onResults], but only runs when interim results are enabled + * ([interimResultsEnabled] == true) + * - Will try to extract the best segmented transcript + * + * - [RecognitionListener.onBeginningOfSpeech] / [RecognitionListener.onEndOfSpeech]: + * - Mark speech as in progress, which keeps the silence monitor from stopping the session + * + * - All other callbacks ([onReadyForSpeech], [onRmsChanged], [onBufferReceived], [onEvent]) + * are currently treated as no-ops + * + * This method should be called before starting a new dictation session to ensure + * that a configured [SpeechRecognizer] instance is available. + */ + + private fun setupRecognizerIfNeeded() { + if (speechRecognizer != null) return + + val ctx = reactContext.currentActivity ?: reactContext + + speechRecognizer = SpeechRecognizer.createSpeechRecognizer(reactContext).apply { + setRecognitionListener(object : RecognitionListener { + override fun onReadyForSpeech(params: Bundle?) = Unit + override fun onBeginningOfSpeech() { + isSpeechInProgress = true + lastSpeechTimeMs = System.currentTimeMillis() + } + + override fun onRmsChanged(rmsdB: Float) = Unit + override fun onBufferReceived(buffer: ByteArray?) = Unit + + override fun onEndOfSpeech() { + isSpeechInProgress = false + lastSpeechTimeMs = System.currentTimeMillis() + } + + override fun onError(error: Int) { + stopSilenceMonitor() + isListening.set(false) + emitAndroidError(error) + emitState(STATE_IDLE) + } + + override fun onResults(results: Bundle?) { + stopSilenceMonitor() + isListening.set(false) + + val text = results + ?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION) + ?.firstOrNull() + ?: "" + + if (text.isNotBlank()) { + emitResult(text, true) + } + emitState(STATE_IDLE) + } + + override fun onPartialResults(partialResults: Bundle?) { + val text = partialResults + ?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION) + ?.firstOrNull() + ?: "" + + if (text.isNotBlank()) { + lastSpeechTimeMs = System.currentTimeMillis() + if (interimResultsEnabled) { + emitResult(text, false) + } + } + } + + override fun onEvent(eventType: Int, params: Bundle?) = Unit + }) + } + } + + /** + * Fully tears down the current [SpeechRecognizer] instance and resets dictation state. + * + * This should be called when we are done with speech recognition for the current + * lifecycle of the host (e.g. `onHostDestroy`) or when we want to guarantee that + * no recognizer resources are left hanging around. + */ + private fun cleanupRecognizer() { + stopSilenceMonitor() + mainHandler.post { + try { + speechRecognizer?.setRecognitionListener(null) + speechRecognizer?.destroy() + } catch (_: Throwable) { + // ignore + } finally { + speechRecognizer = null + isListening.set(false) + emitState(STATE_IDLE) + } + } + } + + /** + * Returns `true` if the app has the RECORD_AUDIO permission. + */ + private fun hasAudioPermission(): Boolean { + val granted = ContextCompat.checkSelfPermission( + reactContext, + Manifest.permission.RECORD_AUDIO, + ) + return granted == PackageManager.PERMISSION_GRANTED + } + + /** + * Starts (or restarts) a rudimentary silence monitor. + * + * Periodically checks the time since the last speech activity and if the + * elapsed time exceeds [silenceTimeoutMs] while still listening, calls [stop]. + * Never stops while the recognizer reports speech in progress. + */ + private fun startSilenceMonitor() { + stopSilenceMonitor() + + val runnable = object : Runnable { + override fun run() { + if (!isListening.get()) return + + val elapsed = System.currentTimeMillis() - lastSpeechTimeMs + if (!isSpeechInProgress && elapsed >= silenceTimeoutMs) { + stop() + return + } + + mainHandler.postDelayed(this, 300L) + } + } + + silenceRunnable = runnable + mainHandler.postDelayed(runnable, 300L) + } + + /** + * Stops the silence monitor by removing any pending callbacks. + */ + private fun stopSilenceMonitor() { + silenceRunnable?.let { mainHandler.removeCallbacks(it) } + silenceRunnable = null + } + + /** + * Emits a dictation result event to JS. + * + * @param text Transcribed text + * @param isFinal Whether this result is final (`true`) or interim (`false`) + */ + private fun emitResult(text: String, isFinal: Boolean) { + val map = Arguments.createMap().apply { + putString("text", text) + putBoolean("isFinal", isFinal) + } + emitEvent(EVENT_ON_RESULT, map) + } + + /** + * Emits a state change event to JS (e.g. IDLE, LISTENING etc). + */ + private fun emitState(state: String) { + val map = Arguments.createMap().apply { + putString("state", state) + } + emitEvent(EVENT_ON_STATE, map) + } + + /** + * Emits a generic error event to JS with a code and message. + */ + private fun emitError(code: String, message: String) { + val map = Arguments.createMap().apply { + putString("code", code) + putString("message", message) + } + emitEvent(EVENT_ON_ERROR, map) + } + + /** + * Maps an Android SpeechRecognizer error code to a human-readable message + * and emits it as a JS error event. + */ + private fun emitAndroidError(errorCode: Int) { + val msg = when (errorCode) { + SpeechRecognizer.ERROR_AUDIO -> "Audio recording error" + SpeechRecognizer.ERROR_CLIENT -> "Client side error" + SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS -> "Insufficient permissions" + SpeechRecognizer.ERROR_NETWORK -> "Network error" + SpeechRecognizer.ERROR_NETWORK_TIMEOUT -> "Network timeout" + SpeechRecognizer.ERROR_NO_MATCH -> "No recognition result matched" + SpeechRecognizer.ERROR_RECOGNIZER_BUSY -> "RecognitionService busy" + SpeechRecognizer.ERROR_SERVER -> "Server error" + SpeechRecognizer.ERROR_SPEECH_TIMEOUT -> "No speech input" + else -> "Unknown speech recognizer error ($errorCode)" + } + + emitError(ERROR_ANDROID, msg) + } + + private fun ReadableMap.getBooleanSafe(key: String): Boolean? = + if (hasKey(key) && getType(key) == ReadableType.Boolean) getBoolean(key) else null + + private fun ReadableMap.getStringSafe(key: String): String? = + if (hasKey(key) && getType(key) == ReadableType.String) getString(key) else null + + private fun ReadableMap.getDoubleSafe(key: String): Double? = + if (hasKey(key) && getType(key) == ReadableType.Number) getDouble(key) else null +} diff --git a/package/shared-native/android/jni/PerfTextComponentDescriptor.h b/package/shared-native/android/jni/PerfTextComponentDescriptor.h new file mode 100644 index 0000000000..7e26055eed --- /dev/null +++ b/package/shared-native/android/jni/PerfTextComponentDescriptor.h @@ -0,0 +1,29 @@ +#pragma once + +#include "PerfTextShadowNode.h" +#include "PerfTextMeasurementManager.h" + +#include + +namespace facebook::react { + +class PerfTextComponentDescriptor final + : public ConcreteComponentDescriptor { + public: + using ConcreteComponentDescriptor::ConcreteComponentDescriptor; + PerfTextComponentDescriptor( + const ComponentDescriptorParameters ¶meters) + : ConcreteComponentDescriptor(parameters), + measurementManager_(std::make_shared(contextContainer_)) {} + + void adopt(ShadowNode &shadowNode) const override { + ConcreteComponentDescriptor::adopt(shadowNode); + auto &node = static_cast(shadowNode); + node.setMeasurementManager(measurementManager_); + } + + private: + const std::shared_ptr measurementManager_; +}; + +} // namespace facebook::react diff --git a/package/shared-native/android/jni/PerfTextMeasurementManager.cpp b/package/shared-native/android/jni/PerfTextMeasurementManager.cpp new file mode 100644 index 0000000000..d0a5fb5324 --- /dev/null +++ b/package/shared-native/android/jni/PerfTextMeasurementManager.cpp @@ -0,0 +1,70 @@ +#include "PerfTextMeasurementManager.h" + +#include +#include +#include +#include + +using namespace facebook::jni; + +namespace facebook::react { + +Size PerfTextMeasurementManager::measure( + SurfaceId surfaceId, + LayoutConstraints layoutConstraints, + const PerfTextProps &props) const { + + // This comes from ContextContainer set up by RN + const auto &fabricUIManager = + contextContainer_->at>("FabricUIManager"); + + static const auto measureMethod = + facebook::jni::findClassStatic("com/facebook/react/fabric/FabricUIManager") + ->getMethod("measure"); + + auto minimumSize = layoutConstraints.minimumSize; + auto maximumSize = layoutConstraints.maximumSize; + + local_ref componentName = make_jstring("PerfText"); + + // Serialize relevant props. For now we'll handpick the ones we want + // to do this for and may extend later in the future. + folly::dynamic serializedProps = folly::dynamic::object(); + serializedProps["text"] = props.text; + serializedProps["fontSize"] = props.fontSize; + serializedProps["lineHeight"] = props.lineHeight; + + auto propsRNM = ReadableNativeMap::newObjectCxxArgs(serializedProps); + local_ref propsRM = + make_local(reinterpret_cast(propsRNM.get())); + // ----------------------------------------------- + + const auto measureResult = measureMethod( + fabricUIManager, + static_cast(surfaceId), + componentName.get(), + nullptr, // localData + propsRM.get(), // props + nullptr, // state + minimumSize.width, + maximumSize.width, + minimumSize.height, + maximumSize.height + ); + + // Use RN's yogaMeassureToSize(int64_t) from conversions.h + return yogaMeassureToSize(static_cast(measureResult)); +} + +} // namespace facebook::react + + diff --git a/package/shared-native/android/jni/PerfTextMeasurementManager.h b/package/shared-native/android/jni/PerfTextMeasurementManager.h new file mode 100644 index 0000000000..1e6011a251 --- /dev/null +++ b/package/shared-native/android/jni/PerfTextMeasurementManager.h @@ -0,0 +1,25 @@ +#pragma once + +#include "StreamCodegenComponents.h" +#include +#include + +namespace facebook::react { + +class PerfTextMeasurementManager { + public: + PerfTextMeasurementManager( + const std::shared_ptr &contextContainer) + : contextContainer_(contextContainer) {} + + // Props type name comes from codegen, hence why we don't override Props.h + Size measure( + SurfaceId surfaceId, + LayoutConstraints layoutConstraints, + const PerfTextProps &props) const; + + private: + const std::shared_ptr contextContainer_; +}; + +} // namespace facebook::react diff --git a/package/shared-native/android/jni/PerfTextShadowNode.cpp b/package/shared-native/android/jni/PerfTextShadowNode.cpp new file mode 100644 index 0000000000..f76fc236fb --- /dev/null +++ b/package/shared-native/android/jni/PerfTextShadowNode.cpp @@ -0,0 +1,20 @@ +#include "PerfTextShadowNode.h" + +namespace facebook::react { + +extern const char PerfTextComponentName[] = "PerfText"; + +void PerfTextShadowNode::setMeasurementManager( + const std::shared_ptr &mm) { + ensureUnsealed(); + measurementManager_ = mm; +} + +Size PerfTextShadowNode::measureContent( + const LayoutContext &layoutContext, + const LayoutConstraints &layoutConstraints) const { + // Ask measurement manager, drilling props so it can size based on text, fontSize, etc. + return measurementManager_->measure(getSurfaceId(), layoutConstraints, getConcreteProps()); +} + +} // namespace facebook::react diff --git a/package/shared-native/android/jni/PerfTextShadowNode.h b/package/shared-native/android/jni/PerfTextShadowNode.h new file mode 100644 index 0000000000..3dfeea5532 --- /dev/null +++ b/package/shared-native/android/jni/PerfTextShadowNode.h @@ -0,0 +1,39 @@ +#pragma once + +#include "PerfTextMeasurementManager.h" + +// Provided by the including SDK package (android/src/main/jni), so that the codegen-generated +// headers of that package are used. +#include "StreamCodegenComponents.h" +#include + +namespace facebook::react { + +extern const char PerfTextComponentName[]; + +class PerfTextShadowNode final : public ConcreteViewShadowNode< + PerfTextComponentName, + PerfTextProps, + PerfTextEventEmitter> { + public: + using ConcreteViewShadowNode::ConcreteViewShadowNode; + + static ShadowNodeTraits BaseTraits() { + auto traits = ConcreteViewShadowNode::BaseTraits(); + traits.set(ShadowNodeTraits::Trait::LeafYogaNode); + traits.set(ShadowNodeTraits::Trait::MeasurableYogaNode); + return traits; + } + + void setMeasurementManager( + const std::shared_ptr &mm); + + Size measureContent( + const LayoutContext &layoutContext, + const LayoutConstraints &layoutConstraints) const override; + + private: + std::shared_ptr measurementManager_; +}; + +} // namespace facebook::react diff --git a/package/shared-native/ios/AIDictation.h b/package/shared-native/ios/AIDictation.h new file mode 100644 index 0000000000..c33bc553b3 --- /dev/null +++ b/package/shared-native/ios/AIDictation.h @@ -0,0 +1,20 @@ +#ifdef RCT_NEW_ARCH_ENABLED + +#import + +#if __has_include("StreamChatReactNativeSpec.h") +#import "StreamChatReactNativeSpec.h" +#elif __has_include("StreamChatExpoSpec.h") +#import "StreamChatExpoSpec.h" +#else +#error "Unable to find generated codegen spec header for AIDictation." +#endif + +// Speech-to-text for the `ai-components` dictation button. The implementation lives in +// AIDictationModule.swift. These files are only compiled when dictation is enabled +// (`$StreamChatEnableDictation = true` in the Podfile, or the stream-chat-expo config plugin), since +// referencing the Speech framework requires apps to declare a speech recognition usage description. +@interface AIDictation : RCTEventEmitter +@end + +#endif diff --git a/package/shared-native/ios/AIDictation.mm b/package/shared-native/ios/AIDictation.mm new file mode 100644 index 0000000000..5c657e9668 --- /dev/null +++ b/package/shared-native/ios/AIDictation.mm @@ -0,0 +1,114 @@ +#import "AIDictation.h" + +#ifdef RCT_NEW_ARCH_ENABLED + +#if __has_include() +#import +#elif __has_include() +#import +#elif __has_include("stream_chat_react_native-Swift.h") +#import "stream_chat_react_native-Swift.h" +#elif __has_include("stream_chat_expo-Swift.h") +#import "stream_chat_expo-Swift.h" +#else +#error "Unable to import generated Swift header for AIDictation." +#endif + +static NSString *const AIDictationOnStateEventName = @"AIDictationOnState"; +static NSString *const AIDictationOnResultEventName = @"AIDictationOnResult"; +static NSString *const AIDictationOnErrorEventName = @"AIDictationOnError"; + +@interface AIDictation () +@end + +@implementation AIDictation { + AIDictationModule *moduleInstance; +} + +// Registered here rather than through `codegenConfig.ios.modulesProvider`: the provider list is +// resolved at startup and reports a missing class as an error, while this module is only compiled +// in when dictation is enabled. +RCT_EXPORT_MODULE(AIDictation) + ++ (BOOL)requiresMainQueueSetup +{ + return NO; +} + +- (instancetype)init +{ + if (self = [super init]) { + moduleInstance = [AIDictationModule new]; + moduleInstance.eventSink = self; + } + return self; +} + +- (NSArray *)supportedEvents +{ + return @[ AIDictationOnStateEventName, AIDictationOnResultEventName, AIDictationOnErrorEventName ]; +} + +- (std::shared_ptr)getTurboModule: + (const facebook::react::ObjCTurboModule::InitParams &)params +{ + return std::make_shared(params); +} + +- (void)start:(JS::NativeAIDictation::DictationStartOptions &)options +{ + NSMutableDictionary *dict = [NSMutableDictionary dictionary]; + + auto interimOpt = options.interimResults(); + if (interimOpt.has_value()) { + dict[@"interimResults"] = @(*interimOpt); + } + + auto langOpt = options.language(); + if (langOpt && langOpt != (id)kCFNull) { + dict[@"language"] = langOpt; + } + + auto silenceOpt = options.silenceTimeoutMs(); + if (silenceOpt.has_value()) { + dict[@"silenceTimeoutMs"] = @(*silenceOpt); + } + + [moduleInstance startWithOptions:dict eventEmitter:self]; +} + +- (void)stop +{ + [moduleInstance stop]; +} + +- (void)cancel +{ + [moduleInstance cancel]; +} + +- (NSNumber *)isRecording +{ + return @([moduleInstance isRecording]); +} + +#pragma mark - AIDictationEventSink + +- (void)emitOnState:(NSDictionary *)value +{ + [self sendEventWithName:AIDictationOnStateEventName body:value]; +} + +- (void)emitOnResult:(NSDictionary *)value +{ + [self sendEventWithName:AIDictationOnResultEventName body:value]; +} + +- (void)emitOnError:(NSDictionary *)value +{ + [self sendEventWithName:AIDictationOnErrorEventName body:value]; +} + +@end + +#endif diff --git a/package/shared-native/ios/AIDictationModule.swift b/package/shared-native/ios/AIDictationModule.swift new file mode 100644 index 0000000000..b753af2c93 --- /dev/null +++ b/package/shared-native/ios/AIDictationModule.swift @@ -0,0 +1,566 @@ +import Foundation +import Speech +import AVFoundation +import React + +/** + * A sink for dictation events emitted from the native module. + * + * Implemented by the Obj-C++ TurboModule (`AIDictation.mm`), which forwards these + * calls into React Nativeโ€™s event system through `RCTEventEmitter`. + * + * All payloads are plain `NSDictionary` objects. + */ +@objc public protocol AIDictationEventSink { + func emitOnState(_ value: NSDictionary) + func emitOnResult(_ value: NSDictionary) + func emitOnError(_ value: NSDictionary) +} + +@objc(AIDictationModule) +public class AIDictationModule: NSObject { + + // ------------------------------------------------------------------------- + // Constants + // ------------------------------------------------------------------------- + + /** + * The logical module name exposed to JS. + * + * This must stay in sync with the TurboModule spec name so that + * `NativeAIDictation` (JS) binds to this native implementation. + */ + static let NAME = "NativeAIDictation" + + // High-level dictation state labels. These are emitted to JS via + // the `state` field as part of a `type: "state"` event payload. + private let STATE_IDLE = "idle" + private let STATE_STARTING = "starting" + private let STATE_LISTENING = "listening" + private let STATE_STOPPING = "stopping" + + // Error codes used in `type: "error"` events. + private let ERROR_NO_PERMISSION = "NO_PERMISSION" + private let ERROR_NOT_AVAILABLE = "NOT_AVAILABLE" + private let ERROR_INTERNAL = "INTERNAL_ERROR" + + /// Default maximum allowed silence (in milliseconds) before auto-stopping. + private let DEFAULT_SILENCE_TIMEOUT_MS: Double = 2500.0 + + // ------------------------------------------------------------------------- + // Internal state + // ------------------------------------------------------------------------- + + // AVFoundation/SpeechKit primitives driving the recognition session. + private let audioEngine = AVAudioEngine() + private var speechRecognizer: SFSpeechRecognizer? + private var recognitionRequest: SFSpeechAudioBufferRecognitionRequest? + private var recognitionTask: SFSpeechRecognitionTask? + + // Runtime configuration and state of the current dictation session. + private var isListening: Bool = false + private var interimResultsEnabled: Bool = true + private var currentLocale: Locale = .current + private var silenceTimeoutMs: Double = 2500.0 + private var lastSpeechTimeMs: Double = 0.0 + + // Timer used for the simple silence monitor. + private var silenceTimer: Timer? + + // Set while the permission prompts of the first `start()` are on screen. `stop()` and + // `cancel()` clear it, which abandons that start. + private var isAwaitingPermission: Bool = false + + // ------------------------------------------------------------------------- + // Event emitter plumbing + // ------------------------------------------------------------------------- + + /** + * Event sink provided by the Obj-C TurboModule wrapper. + * + * The Obj-C class sets this to itself and forwards + * calls into the codegen-generated `emitOn` + * helpers, which in turn send events to JS. + */ + @objc public weak var eventSink: AIDictationEventSink? + + /** + * Convenience entry point wired from Obj-C if needed. + * + * This signature exists so Objective-C can call + * `[_impl startWithOptions:options eventEmitter:self]` without having + * to construct Swift-specific types. Internally it just forwards to + * the primary `start(_:)` implementation. + */ + @objc(startWithOptions:eventEmitter:) + public func startWithOptions(_ options: NSDictionary?, eventEmitter: NSObject) { + start(options) + } + + // ------------------------------------------------------------------------- + // NativeAIDictationSpec API + // ------------------------------------------------------------------------- + + /** + * Starts a new dictation session with the given options. + * + * - No-ops if a session is already active + * - Validates microphone + speech recognition permissions + * - On success, configures and starts `AVAudioEngine` + `SFSpeechRecognizer` + * and arms a basic silence monitor + * + * Expected `options` keys (all optional): + * + * - `interimResults` (`Bool`, default `true`): + * Whether to emit partial / interim results while the user is speaking. + * + * - `language` (`String`, BCP-47 tag): + * Locale identifier (e.g. `"en-US"`). Falls back to `Locale.current` + * if omitted or blank. + * + * - `silenceTimeoutMs` (`Double`): + * Maximum allowed silence, in milliseconds, before auto-stopping. + * Falls back to `DEFAULT_SILENCE_TIMEOUT_MS` if missing or invalid. + */ + @objc + public func start(_ options: NSDictionary?) { + if isListening || isAwaitingPermission { + return + } + + emitState(STATE_STARTING) + + // Runtime configuration derived from options. + interimResultsEnabled = (options?["interimResults"] as? Bool) ?? true + + if let languageTag = options?["language"] as? String, + !languageTag.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + currentLocale = Locale(identifier: languageTag) + } else { + currentLocale = .current + } + + if let silenceOpt = options?["silenceTimeoutMs"] as? Double, silenceOpt > 0 { + silenceTimeoutMs = silenceOpt + } else { + silenceTimeoutMs = DEFAULT_SILENCE_TIMEOUT_MS + } + + // Permissions: on first use, wait for the user to answer the prompts before starting. + // Starting right away would race the prompts and fail. + if !hasAudioPermission() { + isAwaitingPermission = true + requestPermissions { [weak self] granted in + guard let self = self, self.isAwaitingPermission else { return } + self.isAwaitingPermission = false + + if !granted { + self.emitError( + self.ERROR_NO_PERMISSION, + message: "Microphone or speech recognition permission not granted." + ) + self.emitState(self.STATE_IDLE) + return + } + self.startSession() + } + return + } + + startSession() + } + + /** + * Checks availability and starts the audio engine and recognition task. Called once the + * permissions are granted. + */ + private func startSession() { + // Availability + guard let recognizer = SFSpeechRecognizer(locale: currentLocale), recognizer.isAvailable else { + emitError( + ERROR_NOT_AVAILABLE, + message: "Speech recognition is not available for this locale or on this device." + ) + emitState(STATE_IDLE) + return + } + + speechRecognizer = recognizer + + // Configure and start the audio engine + recognition task on the main thread. + DispatchQueue.main.async { + do { + try self.setupAudioSession() + try self.startRecognition() + } catch { + self.isListening = false + self.emitState(self.STATE_IDLE) + self.emitError( + self.ERROR_INTERNAL, + message: "Failed to start recognition: \(error.localizedDescription)" + ) + } + } + } + + /** + * Requests a graceful stop of the current dictation session. + * + * - No-ops if we are not currently listening + * - Calls `endAudio()` on the recognition request, allowing + * `SFSpeechRecognizer` to deliver final results before terminating + * + * Final state transitions are handled via the recognition task callback + * (`finishRecognitionSession()` or the error branch). + */ + @objc + public func stop() { + if abandonPendingStart() { + return + } + if !isListening { + return + } + + emitState(STATE_STOPPING) + stopSilenceMonitor() + + DispatchQueue.main.async { + self.recognitionRequest?.endAudio() + } + } + + /** + * Immediately cancels the current dictation session. + * + * - No-ops if we are not currently listening + * - Stops the audio engine and cancels the recognition task + * - Resets all internal state and emits `STATE_IDLE` + * + * Use this when the session should be aborted (e.g. user cancelled) rather + * than gracefully completed. + */ + @objc + public func cancel() { + if abandonPendingStart() { + return + } + if !isListening { + return + } + + stopSilenceMonitor() + + DispatchQueue.main.async { + self.stopAudioEngine() + self.recognitionTask?.cancel() + self.recognitionTask = nil + self.recognitionRequest = nil + self.isListening = false + self.emitState(self.STATE_IDLE) + } + } + + /** + * Synchronously reports whether a dictation session is currently active. + * + * This is exposed to JS as a sync method so callers can quickly check + * recording state without waiting on an event. + */ + @objc + public func isRecording() -> Bool { + return isListening + } + + // ------------------------------------------------------------------------- + // Internal setup / teardown + // ------------------------------------------------------------------------- + + /** + * Configures the shared `AVAudioSession` for recording. + * + * - Uses `.record` category and `.measurement` mode for best recognition + * quality. + * - Ducks other audio and notifies other sessions on deactivation. + * + * Throws if the session cannot be configured or activated. + */ + private func setupAudioSession() throws { + let session = AVAudioSession.sharedInstance() + try session.setCategory(.record, mode: .measurement, options: [.duckOthers]) + try session.setActive(true, options: .notifyOthersOnDeactivation) + } + + /** + * Starts a fresh `SFSpeechRecognitionTask` and connects it to `AVAudioEngine`. + * + * Responsibilities: + * - Cleans up any previous recognition task + * - Creates a new `SFSpeechAudioBufferRecognitionRequest` + * - Installs an input node tap to feed audio buffers into the request + * + * The recognition task callback: + * - Emits interim and final transcripts via `emitResult(...)` + * - Advances the silence timer on non-empty results + * - On final results, calls `finishRecognitionSession()` + * - On error, emits an error event, resets state, and tears down the + * engine / task + */ + private func startRecognition() throws { + // Clean up any previous task + recognitionTask?.cancel() + recognitionTask = nil + + let request = SFSpeechAudioBufferRecognitionRequest() + // Always requested: partial results also drive the silence monitor. They are only + // forwarded to JS when `interimResults` is enabled. + request.shouldReportPartialResults = true + recognitionRequest = request + + let inputNode = audioEngine.inputNode + let recordingFormat = inputNode.outputFormat(forBus: 0) + + inputNode.removeTap(onBus: 0) + inputNode.installTap(onBus: 0, bufferSize: 1024, format: recordingFormat) { + [weak self] (buffer, _) in + self?.recognitionRequest?.append(buffer) + } + + audioEngine.prepare() + try audioEngine.start() + + lastSpeechTimeMs = currentTimeMs() + isListening = true + emitState(STATE_LISTENING) + + startSilenceMonitor() + + recognitionTask = speechRecognizer?.recognitionTask(with: request) { [weak self] result, error in + guard let self = self else { return } + + if let result = result { + let text = result.bestTranscription.formattedString + if !text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + self.lastSpeechTimeMs = self.currentTimeMs() + if result.isFinal || self.interimResultsEnabled { + self.emitResult(text: text, isFinal: result.isFinal) + } + } + + if result.isFinal { + self.finishRecognitionSession() + } + } + + if let error = error { + self.stopSilenceMonitor() + self.isListening = false + self.emitError( + self.ERROR_INTERNAL, + message: "Speech recognizer error: \(error.localizedDescription)" + ) + self.emitState(self.STATE_IDLE) + self.stopAudioEngine() + self.recognitionTask = nil + self.recognitionRequest = nil + } + } + } + + /** + * Handles the normal end-of-session flow after a final recognition result. + * + * - Stops the silence monitor + * - Marks `isListening = false` + * - Emits `STATE_IDLE` + * - Stops the audio engine and clears recognition objects + */ + private func finishRecognitionSession() { + stopSilenceMonitor() + isListening = false + emitState(STATE_IDLE) + stopAudioEngine() + recognitionTask = nil + recognitionRequest = nil + } + + /** + * Abandons a `start()` that is still waiting for the permission prompts, so the session does + * not begin once they are answered. Returns `true` if there was one. + */ + private func abandonPendingStart() -> Bool { + guard isAwaitingPermission else { return false } + isAwaitingPermission = false + emitState(STATE_IDLE) + return true + } + + /** + * Fully tears down the recognizer and audio engine and resets state. + * + * Intended for hard cleanup paths (e.g. deinit, React instance teardown). + * Ensures we: + * - stop the silence monitor + * - stop and detach taps from the audio engine + * - cancel and nil out the recognition task / request + * - release the `SFSpeechRecognizer` instance + * - emit `STATE_IDLE` + */ + private func cleanupRecognizer() { + stopSilenceMonitor() + DispatchQueue.main.async { + self.stopAudioEngine() + self.recognitionTask?.cancel() + self.recognitionTask = nil + self.recognitionRequest = nil + self.speechRecognizer = nil + self.isListening = false + self.emitState(self.STATE_IDLE) + } + } + + /** + * Stops the audio engine and removes any installed tap on the input node. + */ + private func stopAudioEngine() { + if audioEngine.isRunning { + audioEngine.stop() + audioEngine.inputNode.removeTap(onBus: 0) + } + } + + // ------------------------------------------------------------------------- + // Permissions / silence monitor + // ------------------------------------------------------------------------- + + /** + * Returns `true` if both microphone and speech recognition permissions + * are already granted. + * + * This is a simple synchronous check used before starting recognition. + */ + private func hasAudioPermission() -> Bool { + let micAllowed = AVAudioSession.sharedInstance().recordPermission == .granted + let speechAllowed = SFSpeechRecognizer.authorizationStatus() == .authorized + return micAllowed && speechAllowed + } + + /** + * Requests microphone + speech recognition permissions. + * + * The completion handler is invoked on the main thread with `true` + * only if both: + * - `AVAudioSession` record permission is granted + * - `SFSpeechRecognizer` authorization status is `.authorized` + */ + private func requestPermissions(completion: @escaping (Bool) -> Void) { + let audioSession = AVAudioSession.sharedInstance() + + audioSession.requestRecordPermission { micGranted in + // Without the microphone there is nothing to transcribe, so the speech recognition + // prompt is not shown. + guard micGranted else { + DispatchQueue.main.async { completion(false) } + return + } + SFSpeechRecognizer.requestAuthorization { status in + let speechGranted = (status == .authorized) + DispatchQueue.main.async { + completion(speechGranted) + } + } + } + } + + /** + * Starts (or restarts) a rudimentary silence monitor. + * + * Periodically checks the time since the last speech activity and if the + * elapsed time exceeds `silenceTimeoutMs` while still listening, calls `stop()`. + * + * The timer runs on the main run loop in `.common` mode so that it continues + * to fire during typical UI interactions. + */ + private func startSilenceMonitor() { + stopSilenceMonitor() + + guard silenceTimeoutMs > 0 else { return } + + silenceTimer = Timer.scheduledTimer(withTimeInterval: 0.3, repeats: true) { [weak self] _ in + guard let self = self, self.isListening else { return } + + let elapsed = self.currentTimeMs() - self.lastSpeechTimeMs + if elapsed >= self.silenceTimeoutMs { + self.stop() + } + } + if let timer = silenceTimer { + RunLoop.main.add(timer, forMode: .common) + } + } + + /** + * Stops the silence monitor by invalidating any pending timer. + */ + private func stopSilenceMonitor() { + silenceTimer?.invalidate() + silenceTimer = nil + } + + /** + * Returns the current wall-clock time in milliseconds. + * + * Used to track the last time non-empty speech was observed so that the + * silence monitor can auto-stop idle sessions. + */ + private func currentTimeMs() -> Double { + return Date().timeIntervalSince1970 * 1000.0 + } + + // ------------------------------------------------------------------------- + // Event helpers + // ------------------------------------------------------------------------- + + /** + * Emits a dictation result event to JS. + * + * Payload fields: + * - `text` (`String`): recognized transcript + * - `isFinal` (`Bool`): `true` for final results, `false` for interim + */ + private func emitResult(text: String, isFinal: Bool) { + let payload: NSDictionary = [ + "text": text, + "isFinal": isFinal + ] + eventSink?.emitOnResult(payload) + } + + /** + * Emits a state change event to JS. + * + * Payload fields: + * - `state` (`String`): one of `"idle"`, `"starting"`, `"listening"`, `"stopping"` + */ + private func emitState(_ state: String) { + let payload: NSDictionary = [ + "state": state + ] + eventSink?.emitOnState(payload) + } + + /** + * Emits a generic error event to JS with a code and message. + * + * Payload fields: + * - `code` (`String`): internal error code (e.g. `NO_PERMISSION`, `NOT_AVAILABLE`) + * - `message` (`String`): human-readable description for logging / debugging + */ + private func emitError(_ code: String, message: String) { + let payload: NSDictionary = [ + "code": code, + "message": message + ] + eventSink?.emitOnError(payload) + } +} diff --git a/package/shared-native/ios/StreamShimmerViewComponentView.mm b/package/shared-native/ios/StreamShimmerViewComponentView.mm index 9cad9af460..a1dfa8b952 100644 --- a/package/shared-native/ios/StreamShimmerViewComponentView.mm +++ b/package/shared-native/ios/StreamShimmerViewComponentView.mm @@ -105,4 +105,13 @@ - (void)dealloc @end +// React Native 0.76 resolves third-party Fabric components through a `Cls()` function +// (RCTThirdPartyFabricComponentsProvider); 0.77+ uses `codegenConfig.ios.componentProvider` instead. +// The provider declares it with C linkage, so it must be defined as such to link. +// This can be removed once the SDK's minimum version moves beyond 0.76. +extern "C" Class StreamShimmerViewCls(void) +{ + return StreamShimmerViewComponentView.class; +} + #endif diff --git a/package/src/ai-components/AITypingIndicatorView.tsx b/package/src/ai-components/AITypingIndicatorView.tsx new file mode 100644 index 0000000000..ac230bf6f8 --- /dev/null +++ b/package/src/ai-components/AITypingIndicatorView.tsx @@ -0,0 +1,51 @@ +import { useMemo } from 'react'; +import { StyleSheet, Text, View } from 'react-native'; + +import { useTheme } from './contexts'; + +import { NativeShimmerView } from '../components/UIComponents/NativeShimmerView'; + +export const AITypingIndicatorView = ({ text }: { text: string }) => { + const { + theme: { + colors: { shimmer }, + }, + } = useTheme(); + const styles = useStyles(); + + // The native shimmer paints behind its children, so it is laid over the text instead: the + // highlight has to sweep across the glyphs, which a transparent background would otherwise hide. + return ( + + {text} + + + ); +}; + +const useStyles = () => { + const { theme } = useTheme(); + + return useMemo( + () => + StyleSheet.create({ + // The native highlight band is soft and wide, so it is kept fairly strong over the text. + shimmer: { opacity: 0.8 }, + text: { + fontSize: 16, + color: theme.colors.grey_dark, + }, + wrapper: { + alignSelf: 'flex-start', + backgroundColor: 'transparent', + }, + }), + [theme], + ); +}; diff --git a/package/src/ai-components/README.md b/package/src/ai-components/README.md new file mode 100644 index 0000000000..d68538c1dd --- /dev/null +++ b/package/src/ai-components/README.md @@ -0,0 +1,279 @@ +# AI components for Stream Chat React Native + +`stream-chat-react-native/ai-components` (and `stream-chat-expo/ai-components`) contains Stream Chat's UI components for AI-first applications. Paired with the [Chat API](https://getstream.io/chat/), they render responses from LLM providers such as ChatGPT, Gemini, Anthropic or any custom backend: Markdown, code blocks with syntax highlighting, tables, charts, thinking indicators and more. + +- `StreamingMessageView` - renders text, markdown and code in real time, using a ChatGPT-like typewriter animation +- `ComposerView` - a prompt composer with attachments and speech input +- `SpeechToTextButton` - records voice input and streams the recognized transcript into your UI +- `AITypingIndicatorView` - displays the state of the LLM (thinking, checking external sources, etc.) + +The components live in their own entry point. Apps that never import `stream-chat-react-native/ai-components` do not bundle any of it, and apps that only use the AI components do not bundle the rest of the chat SDK. + +## Migrating from `@stream-io/chat-react-native-ai` + +These components used to ship as the standalone `@stream-io/chat-react-native-ai` package, which is now deprecated. Their API is unchanged; to migrate: + +1. Uninstall `@stream-io/chat-react-native-ai` and import from `stream-chat-react-native/ai-components` (or `stream-chat-expo/ai-components`) instead: + + ```diff + - import { ComposerView, StreamingMessageView } from '@stream-io/chat-react-native-ai'; + + import { ComposerView, StreamingMessageView } from 'stream-chat-react-native/ai-components'; + ``` + +2. Dictation is now opt-in on iOS (see [Dictation](#dictation-speech-to-text)): add `$StreamChatEnableDictation = true` to your `Podfile`, or, with Expo, replace the `@stream-io/chat-react-native-ai` config plugin with `stream-chat-expo`. +3. Keep the `export-namespace-from` Babel plugin (see [Installation](#๏ธ-installation)); it is needed by the `zod` dependency. +4. `ShimmeringView` and `toHexFromRNColor` were removed. `AITypingIndicatorView` now uses the SDK's native shimmer, so `@shopify/react-native-skia` is only needed for charts. +5. `SpeechToTextButton` and `isDictationAvailable` are now exported. + +## ๐Ÿ› ๏ธ Installation + +The AI components ship with `stream-chat-react-native` and `stream-chat-expo`; their JavaScript dependencies are installed with them. Charts additionally need the following peer dependencies, which only apps using the AI components have to install: + +```bash +yarn add victory-native @shopify/react-native-skia +``` + +`react-native-reanimated`, `react-native-gesture-handler` and `react-native-svg` are already required by the chat SDK. + +The `zod` dependency ships `export * as` syntax, which the React Native Babel preset does not transform, so add the `export-namespace-from` plugin to your `babel.config.js`: + +```bash +yarn add -D @babel/plugin-transform-export-namespace-from +``` + +```js +module.exports = { + presets: ['module:@react-native/babel-preset'], + plugins: [ + // ... rest of your @babel plugins + '@babel/plugin-transform-export-namespace-from', + // react-native-worklets/plugin has to be the last one + 'react-native-worklets/plugin', + ], +}; +``` + +Apps that do not import the AI components do not bundle `zod` and do not need the plugin. + +### Dictation (speech-to-text) + +The speech-to-text button uses the platform speech recognizers. + +#### `iOS` + +Because App Store Connect requires a speech recognition usage description from every app whose binary references the Speech framework, the dictation module is only compiled into apps that opt in. Add this to your `Podfile`, before `use_native_modules!`, and run `pod install`: + +```ruby +$StreamChatEnableDictation = true +``` + +(`STREAM_CHAT_ENABLE_DICTATION=1 pod install` works as well.) Then add the usage descriptions to your `Info.plist`: + +```txt +NSMicrophoneUsageDescription +$(PRODUCT_NAME) would like to access your microphone to capture your voice. +NSSpeechRecognitionUsageDescription +$(PRODUCT_NAME) would like to access speech recognition to transcribe your voice. +``` + +The text you pick for the capabilities is up to you. Without the opt-in the speech-to-text button is hidden, and a development-only warning explains how to enable it. + +#### `Android` + +Add the permission to `android/app/src/main/AndroidManifest.xml`: + +```xml + +``` + +#### `Expo` + +Add the `stream-chat-expo` config plugin to your `app.json`/`app.config.[js|ts]`. It enables the dictation module and adds the permissions (with default descriptions unless you provide your own): + +```json +"plugins": [ + // ... rest of your plugins + [ + "stream-chat-expo", + { + "dictationMicrophoneUsageDescription": "$(PRODUCT_NAME) would like to access your microphone to capture your voice.", + "dictationSpeechRecognitionUsageDescription": "$(PRODUCT_NAME) would like to access speech recognition to transcribe your voice." + } + ] +], +``` + +Pass `"enableDictation": false` to leave dictation out. + +## Optional features + +### Media Picker + +The media picker of the `ComposerView` picks existing images and takes photos with the camera. It works with whichever of these libraries you have installed: + +- `react-native-image-picker` (React Native CLI). For the camera, add `NSCameraUsageDescription` to your `Info.plist` and the `android.permission.CAMERA` permission to your `AndroidManifest.xml`. +- `expo-image-picker` (Expo). See the [Expo documentation](https://docs.expo.dev/versions/latest/sdk/imagepicker/) for its permissions. + +### Clipboard + +The copy button of code blocks works with whichever of these libraries you have installed: + +- `@react-native-clipboard/clipboard` (React Native CLI) +- `expo-clipboard` (Expo) + +### Syntax highlighting on Android + +On Android with the New Architecture, highlighted code is rendered by a native `PerfText` view: a single text view with colour spans instead of one `Text` per token, which keeps long code blocks cheap to render and scroll. Elsewhere, code is rendered with nested `Text`. + +## โš™๏ธ Usage + +All of the components listed below are designed to work seamlessly with our existing React Native [Chat SDK](https://getstream.io/chat/react-native-chat/tutorial/). Our [developer guide](https://getstream.io/chat/solutions/ai-integration/) explains how to get started building AI integrations with Stream. + +In order to use them, they have to be wrapped within our `StreamTheme` provider that will make sure theming is applied to them. It is recommended that you put the provider somewhere high up in the component tree so that it encapsulates all components used downstream. + +### `StreamingMessageView` + +The `StreamingMessageView` is a component that can render markdown content efficiently. It has code syntax highlighting, supporting all the major languages. It can render most of the standard markdown content, such as tables, inline code, headings, lists etc. + +Under the hood, it implements a letter-by-letter typewriter animation with a character queue, similar to ChatGPT. + +| Name | Type | Required | Description | +| ------------------------ | -------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `text` | `string` | yes | The text we want to pass as markdown. | +| `paragraphNumberOfLines` | `number` | no | The `numberOfLines` applied to markdown `Paragraph` and `Text` components. Particularly useful if we want to display the same message, but in a "cut" fashion (for example when replying to someone). | +| `rules` | [`MarkdownRules`](./markdown/types.ts) | no | An object of `MarkdownRules` that is then going to be deeply merged with our default rules, based on the [`SimpleMarkdown` parsing engine](https://github.com/Khan/perseus/tree/main/packages/simple-markdown#adding-a-simple-extension). Can be used to add custom rules or change existing rules. You can disable a rule by passing `{ [ruleName]: { match: () => null }}`. | +| `onLink` | `(url: string) => void` | no | A function that is going to be invoked whenever a link is pressed within markdown parsed text. | +| `letterInterval` | `number` | no | A number signifying the interval at which the typewriter animation is going to render characters. Defaults to `0` | +| `renderingLetterCount` | `number` | no | A number signifying the number of letters that are going to be rendered per tick of the interval during the typewriter animation. Defaults to `2` | + +#### Example + +Provided below is an example of how to use the component. + +````tsx +const markdownText = ``` +# Heading + +some text + +## Another heading +```; + +; +```` + +### `AITypingIndicatorView` + +The `AITypingIndicatorView` is used to represent different states of the LLM, such as `Thinking`, `Checking External Sources` and so on, depending on the states you've defined on your backend. The only thing that needs to be passed to the component is the `text` property, which will then be displayed with a shimmering animation. + +| Name | Type | Required | Description | +| ------ | -------- | -------- | ---------------------------------------------------- | +| `text` | `string` | yes | The text we want to be displayed inside of the view. | + +#### Example + +```tsx + +``` + +### `ComposerView` + +The `ComposerView` gives users a modern text entry surface with attachment previews, actionable bottom sheet, speech-to-text button and an integrated send button. + +| Name | Type | Required | Description | +| -------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `onSendMessage` | `(opts: { text: string; attachments?: MediaPickerState['assets']; custom?: Record; }) => Promise` | yes | A callback that will be invoked whenever the send button is pressed. The `text`, `attachments` and any `custom` data we've added to the state will be passed to it. | +| `bottomSheetOptions` | [`BottomSheetOption[]`](./message-composer/ComposerView.tsx) | no | An array of [`BottomSheetOption`](./message-composer/ComposerView.tsx) objects that will render the extra options in the bottom sheet. | +| `bottomSheetInsets` | `{ top: number; bottom: number; left: number; right: number }` | no | An object containing extra insets we can pass to the `ComposerView` in order to make sure the bottom sheet can extend properly beyond them. | +| `isGenerating` | `boolean` | no | A boolean signifying whether the LLM is currently generating a response or not. It will be used to render the stop-generating button in the composer instead of the send button whenever this happens. | +| `stopGenerating` | `() => Promise` | no | A callback that is going to be invoked if the stop-generating button is clicked. | +| `mediaPickerService` | `AbstractMediaPickerService` | no | An instance of the `MediaPickerService` we may decide to inject from the outside for more fine-grained control over attachment state. You can create an instance as `const customInstance = MediaPickerService()` and it will automatically detect which library you're using. | +| `state` | `StateStore` | no | A state store of the `ComposerState` we may decide to inject from the outside for more fine-grained control over the composer state. You can create an instance as `const customComposerState = createNewComposerStore()`. | + +#### Example + +```tsx +import { useSafeAreaInsets } from 'react-native-safe-area-context'; + +const options = [ + { + title: 'Create Image', + subtitle: 'Visualize anything', + action: () => Alert.alert('Pressed on Create Image !'), + Icon: DownloadArrow, + }, + { + title: 'Thinking', + subtitle: 'Think longer for better answers', + action: () => Alert.alert('Pressed on Thinking !'), + Icon: Flag, + }, +]; + +const insets = useSafeAreaInsets(); + +; +``` + +### `SpeechToTextButton` + +The `SpeechToTextButton` turns voice input into text using native implementations of the iOS and Android speech frameworks, respectively. When tapped it asks for microphone access, records audio and forwards the recognized transcript to the `ComposerState` directly. + +It uses the `useDictation` hook under the hood, which can also be used without the button as well for voice transcription purposes outside of the button. + +It takes a single property named `options` that has the following keys: + +| Name | Type | Required | Description | +| ------------------ | --------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `language` | `string` | no | The language we want to transcribe from. It will default to `en-US`. | +| `interimResults` | `boolean` | no | A boolean signifying whether we want to receive the intermediate results from the transcription or just the final result when the transcription is deemed done. Defaults to `true` | +| `silenceTimeoutMs` | `number` | no | A number signifying the number of milliseconds of silence until transcription is deemed finished. Defaults to `2500`. | + +```tsx +const options = { + language: 'de-DE', // set the language to german + interimResults: false, // disable intermediate results and only use the final results + silenceTimeoutMs: 3500 // set the silence timeout to 3.5 seconds +} + +``` + +The `SpeechToTextButton` is already integrated within the `ComposerView`, however feel free to use it elsewhere as well. It renders nothing when dictation is not part of the build (see [Dictation](#dictation-speech-to-text)); `isDictationAvailable()` tells you whether it is. + +### Theming + +Each one of the components in the SDK is fully theme-compatible. The `StreamTheme` provider takes care of this for you. + +In order to modify the theme, you may refer to our full fledged theme object [as seen here](./theme.ts). + +#### Example + +In the example below, we introduce a dark color scheme through the theming system. + +```tsx +const customTheme = { + colors: colorScheme === 'dark' + ? { + accent_blue: '#4C9DFF', + accent_red: '#FF636E', + black: '#FFFFFF', + code_block: '#1E1E22', + grey: '#A1A1AA', + grey_neutral: '#C5C5C8', + grey_dark: '#71717A', + grey_gainsboro: '#3F3F46', + grey_whisper: '#27272F', + overlay: '#000000CC', + transparent: 'transparent', + white: '#050509', + white_smoke: '#121214', + shimmer: '#FFFFFF', + } : {} +} + +{children} +``` diff --git a/package/src/ai-components/StreamingMessageView.tsx b/package/src/ai-components/StreamingMessageView.tsx new file mode 100644 index 0000000000..eb23f1ee2b --- /dev/null +++ b/package/src/ai-components/StreamingMessageView.tsx @@ -0,0 +1,85 @@ +import React, { useEffect, useMemo, useRef, useState } from 'react'; + +import { Linking } from 'react-native'; + +import { useStableCallback } from './internal/hooks/useStableCallback'; +import { generateMarkdownText } from './markdown'; +import { Markdown } from './markdown'; +import type { MarkdownRules } from './markdown'; + +export type UseStreamingMessageProps = { + text: string; + letterInterval?: number; + renderingLetterCount?: number; +}; + +export const useStreamingMessage = ({ + letterInterval = 0, + renderingLetterCount = 2, + text, +}: UseStreamingMessageProps): { streamedMessageText: string } => { + const [streamedMessageText, setStreamedMessageText] = useState(text); + const textCursor = useRef(text.length); + + useEffect(() => { + const textLength = text.length; + const interval = setInterval(() => { + if (!text || textCursor.current >= textLength) { + clearInterval(interval); + } + const newCursorValue = textCursor.current + renderingLetterCount; + const newText = text.substring(0, newCursorValue); + textCursor.current += newText.length - textCursor.current; + const codeBlockCounts = (newText.match(/```/g) || []).length; + const shouldOptimisticallyCloseCodeBlock = codeBlockCounts > 0 && codeBlockCounts % 2 > 0; + setStreamedMessageText(shouldOptimisticallyCloseCodeBlock ? newText + '```' : newText); + }, letterInterval); + + return () => { + clearInterval(interval); + }; + }, [letterInterval, renderingLetterCount, text]); + + return { streamedMessageText }; +}; + +export const StreamingMessageView = ({ + text, + paragraphNumberOfLines, + rules, + onLink: onLinkParam, + letterInterval, + renderingLetterCount, +}: { + text: string; + paragraphNumberOfLines?: number; + rules?: MarkdownRules; + onLink?: (url: string) => void; + letterInterval?: number; + renderingLetterCount?: number; +}) => { + const markdownText = useMemo(() => generateMarkdownText(text), [text]); + + const { streamedMessageText } = useStreamingMessage({ + text: markdownText ?? '', + letterInterval, + renderingLetterCount, + }); + + const onLink = useStableCallback((url: string): void | Promise => + onLinkParam + ? onLinkParam(url) + : Linking.canOpenURL(url).then((canOpenUrl) => { + if (canOpenUrl) { + return Linking.openURL(url); + } + return undefined; + }), + ); + + return ( + + {streamedMessageText} + + ); +}; diff --git a/package/src/ai-components/__tests__/ComposerView.test.tsx b/package/src/ai-components/__tests__/ComposerView.test.tsx new file mode 100644 index 0000000000..ba6b8c5318 --- /dev/null +++ b/package/src/ai-components/__tests__/ComposerView.test.tsx @@ -0,0 +1,113 @@ +import React from 'react'; +import { Image, Text } from 'react-native'; + +import { act, fireEvent, render, screen, waitFor } from '@testing-library/react-native'; + +import { + ComposerView, + type ComposerViewProps, + sheetStore, + sheetStoreApi, + StreamTheme, +} from '../index'; +import { + AbstractMediaPickerService, + type MediaAsset, +} from '../services/media-picker-service/AbstractMediaPickerService'; + +const flushPromises = () => new Promise((resolve) => setTimeout(resolve, 0)); + +class FakeMediaPickerService extends AbstractMediaPickerService { + pickMedia = jest.fn(() => { + const assets: MediaAsset[] = [{ name: 'a.png', uri: 'file:///a.png' }]; + this.appendAssets(assets); + return Promise.resolve({ assets, cancelled: false }); + }); + + takeMedia = jest.fn(() => Promise.resolve({ cancelled: true })); +} + +const renderComposer = (props: Partial = {}) => { + const onSendMessage = jest.fn(() => Promise.resolve()); + const utils = render( + + + , + ); + return { ...utils, onSendMessage }; +}; + +describe('ai-components ComposerView', () => { + afterEach(() => { + act(() => { + sheetStoreApi.closeSheet(); + }); + }); + + it('sends the typed text and clears the input', async () => { + const { onSendMessage } = renderComposer(); + + fireEvent.changeText(screen.getByPlaceholderText('Ask anything'), 'Hello there'); + await act(async () => { + fireEvent.press(screen.getByTestId('send-up')); + await flushPromises(); + }); + + expect(onSendMessage).toHaveBeenCalledWith(expect.objectContaining({ text: 'Hello there' })); + expect(screen.getByPlaceholderText('Ask anything').props.value).toBe(''); + }); + + it('shows the stop button while generating', () => { + const stopGenerating = jest.fn(() => Promise.resolve()); + renderComposer({ isGenerating: true, stopGenerating }); + + fireEvent.changeText(screen.getByPlaceholderText('Ask anything'), 'Hello'); + expect(screen.queryByTestId('send-up')).toBeNull(); + }); + + it('opens the attachment sheet, picks media and sends it along', async () => { + const mediaPickerService = new FakeMediaPickerService(); + const action = jest.fn(); + const { onSendMessage } = renderComposer({ + bottomSheetOptions: [{ action, subtitle: 'Do it', title: 'Custom option' }], + mediaPickerService, + }); + + fireEvent.press(screen.getByText('+')); + expect(sheetStore.getLatestValue().open).toBe(true); + expect(screen.getByText('Camera')).toBeTruthy(); + expect(screen.getByText('Custom option')).toBeTruthy(); + + await act(async () => { + fireEvent.press(screen.getByText('Photos')); + await flushPromises(); + }); + expect(mediaPickerService.pickMedia).toHaveBeenCalled(); + expect(sheetStore.getLatestValue().open).toBe(false); + await waitFor(() => expect(screen.UNSAFE_getAllByType(Image)).toHaveLength(1)); + + act(() => { + sheetStoreApi.openSheet(); + }); + await act(async () => { + fireEvent.press(screen.getByText('Custom option')); + await flushPromises(); + }); + expect(action).toHaveBeenCalled(); + + fireEvent.changeText(screen.getByPlaceholderText('Ask anything'), 'With an image'); + await act(async () => { + fireEvent.press(screen.getByTestId('send-up')); + await flushPromises(); + }); + expect(onSendMessage).toHaveBeenCalledWith({ + attachments: [{ name: 'a.png', uri: 'file:///a.png' }], + custom: undefined, + text: 'With an image', + }); + expect(mediaPickerService.state.getLatestValue().assets).toEqual([]); + expect( + screen.UNSAFE_queryAllByType(Text).some((t) => t.props.children === 'With an image'), + ).toBe(false); + }); +}); diff --git a/package/src/ai-components/__tests__/Markdown.test.tsx b/package/src/ai-components/__tests__/Markdown.test.tsx new file mode 100644 index 0000000000..c691af40a7 --- /dev/null +++ b/package/src/ai-components/__tests__/Markdown.test.tsx @@ -0,0 +1,94 @@ +import React from 'react'; + +import { fireEvent, render, screen } from '@testing-library/react-native'; + +import { Markdown, mergeThemes, StreamTheme } from '../index'; + +const mockSetString = jest.fn(); +jest.mock( + '@react-native-clipboard/clipboard', + () => ({ __esModule: true, default: { setString: (text: string) => mockSetString(text) } }), + { virtual: true }, +); + +const renderMarkdown = (text: string, props: Partial> = {}) => + render( + + {text} + , + ); + +describe('ai-components Markdown', () => { + beforeEach(() => { + mockSetString.mockClear(); + }); + + it('renders headings, emphasis, lists and links', () => { + const onLink = jest.fn(); + renderMarkdown( + '# Title\n\nSome **bold** and *italic* and ~~gone~~ text.\n\n- one\n- two\n\n1. first\n2. second\n\n[docs](https://getstream.io)', + { onLink }, + ); + + expect(screen.getByText('Title')).toBeTruthy(); + expect(screen.getByText('bold')).toBeTruthy(); + expect(screen.getByText('italic')).toBeTruthy(); + expect(screen.getByText('one')).toBeTruthy(); + expect(screen.getByText('second')).toBeTruthy(); + + fireEvent.press(screen.getByText('docs')); + expect(onLink).toHaveBeenCalledWith('https://getstream.io'); + }); + + it('renders tables, block quotes, inline code and horizontal rules', () => { + renderMarkdown('| a | b |\n|---|---|\n| 1 | 2 |\n\n> quoted\n\nuse `npm i`\n\n---\n\nafter'); + + expect(screen.getByText('a')).toBeTruthy(); + expect(screen.getByText('2')).toBeTruthy(); + expect(screen.getByText('quoted')).toBeTruthy(); + expect(screen.getByText('npm i')).toBeTruthy(); + expect(screen.getByText('after')).toBeTruthy(); + }); + + it('renders a code block with its language and copies its content', () => { + renderMarkdown('```js\nconst a = 1;\n```'); + + expect(screen.getByText('js')).toBeTruthy(); + expect(screen.getByText(/const/)).toBeTruthy(); + + fireEvent.press(screen.getByText('\u29C9')); + expect(mockSetString).toHaveBeenCalledWith('const a = 1;'); + }); + + it('renders a mermaid pie chart instead of a code block', () => { + renderMarkdown('```mermaid\npie\n"Dogs" : 3\n"Cats" : 2\n```'); + + expect(screen.getByTestId('victory-polar-chart')).toBeTruthy(); + expect(screen.getByText('Dogs')).toBeTruthy(); + expect(screen.queryByText('mermaid')).toBeNull(); + }); + + it('renders a vega-lite JSON chart, and falls back to code for other JSON', () => { + const chart = JSON.stringify({ + data: { values: [{ a: 'A', b: 1 }] }, + encoding: { x: { field: 'a' }, y: { field: 'b' } }, + mark: 'bar', + }); + renderMarkdown('```json\n' + chart + '\n```'); + expect(screen.getByTestId('victory-cartesian-chart')).toBeTruthy(); + + renderMarkdown('```json\n{ "hello": "world" }\n```'); + expect(screen.getByText('json')).toBeTruthy(); + }); + + it('applies theme overrides', () => { + const theme = mergeThemes({ style: { markdown: { text: { color: 'rgb(1, 2, 3)' } } } }); + render( + + {'# Themed'} + , + ); + + expect(screen.getByText('Themed')).toHaveStyle({ color: 'rgb(1, 2, 3)' }); + }); +}); diff --git a/package/src/ai-components/__tests__/StreamingMessageView.test.tsx b/package/src/ai-components/__tests__/StreamingMessageView.test.tsx new file mode 100644 index 0000000000..1c1824f99a --- /dev/null +++ b/package/src/ai-components/__tests__/StreamingMessageView.test.tsx @@ -0,0 +1,126 @@ +import React from 'react'; +import { Linking, StyleSheet, View } from 'react-native'; + +import { act, fireEvent, render, renderHook, screen } from '@testing-library/react-native'; + +import { registerNativeHandlers } from '../../native'; +import { + AITypingIndicatorView, + StreamingMessageView, + StreamTheme, + useStreamingMessage, +} from '../index'; + +describe('ai-components streaming', () => { + beforeEach(() => { + jest.useFakeTimers(); + }); + + afterEach(() => { + jest.useRealTimers(); + }); + + describe('useStreamingMessage', () => { + it('renders the initial text immediately and animates text appended later', () => { + const { result, rerender } = renderHook( + ({ text }: { text: string }) => + useStreamingMessage({ letterInterval: 10, renderingLetterCount: 2, text }), + { initialProps: { text: 'Hello' } }, + ); + expect(result.current.streamedMessageText).toBe('Hello'); + + rerender({ text: 'Hello world' }); + act(() => { + jest.advanceTimersByTime(10); + }); + expect(result.current.streamedMessageText).toBe('Hello w'); + + act(() => { + jest.advanceTimersByTime(100); + }); + expect(result.current.streamedMessageText).toBe('Hello world'); + }); + + it('optimistically closes a code fence while it is being streamed', () => { + const { result, rerender } = renderHook( + ({ text }: { text: string }) => + useStreamingMessage({ letterInterval: 10, renderingLetterCount: 4, text }), + { initialProps: { text: '' } }, + ); + + rerender({ text: '```js\ncode' }); + act(() => { + jest.advanceTimersByTime(10); + }); + expect(result.current.streamedMessageText).toBe('```j```'); + }); + }); + + describe('StreamingMessageView', () => { + it('renders markdown and handles links with the provided callback', () => { + const onLink = jest.fn(); + render( + + + , + ); + + expect(screen.getByText('Hi')).toBeTruthy(); + fireEvent.press(screen.getByText('getstream.io')); + expect(onLink).toHaveBeenCalledWith('http://getstream.io'); + }); + + it('opens links with Linking by default', async () => { + const canOpenURL = jest.spyOn(Linking, 'canOpenURL').mockResolvedValue(true); + const openURL = jest.spyOn(Linking, 'openURL').mockResolvedValue(undefined); + render( + + + , + ); + + await act(async () => { + fireEvent.press(screen.getByText('getstream.io')); + // Let the `canOpenURL` promise resolve. + await Promise.resolve(); + }); + + expect(canOpenURL).toHaveBeenCalledWith('http://getstream.io'); + expect(openURL).toHaveBeenCalledWith('http://getstream.io'); + }); + }); + + describe('AITypingIndicatorView', () => { + it('lays the SDK native shimmer over the text', () => { + const MockShimmer = jest.fn((props) => ); + registerNativeHandlers({ NativeShimmerView: MockShimmer }); + + render( + + + , + ); + + expect(screen.getByText('Thinking...')).toBeTruthy(); + const shimmer = screen.getByTestId('native-shimmer'); + // The native shimmer paints behind its children, so it must be a sibling drawn after the + // text, covering it, rather than its parent. + expect(shimmer.props.children).toBeUndefined(); + expect(MockShimmer.mock.calls[0]?.[0]).toMatchObject({ + baseColor: 'transparent', + duration: 1200, + gradientColor: '#FFFFFF', + pointerEvents: 'none', + }); + expect(StyleSheet.flatten(shimmer.props.style)).toMatchObject({ + bottom: 0, + left: 0, + position: 'absolute', + right: 0, + top: 0, + }); + + registerNativeHandlers({ NativeShimmerView: null as unknown as undefined }); + }); + }); +}); diff --git a/package/src/ai-components/__tests__/SyntaxHighlighter.test.tsx b/package/src/ai-components/__tests__/SyntaxHighlighter.test.tsx new file mode 100644 index 0000000000..5e58209473 --- /dev/null +++ b/package/src/ai-components/__tests__/SyntaxHighlighter.test.tsx @@ -0,0 +1,136 @@ +import React from 'react'; +import { Platform, StyleSheet, Text, View } from 'react-native'; + +import { render, screen } from '@testing-library/react-native'; + +import { registerNativeHandlers } from '../../native'; +import { + cssToRNTextStyle, + flattenRowsToTextAndColorRanges, + generateNewStylesheet, + StreamTheme, + SyntaxHighlighter, +} from '../index'; + +const setPlatform = (os: typeof Platform.OS) => + Object.defineProperty(Platform, 'OS', { configurable: true, get: () => os }); + +describe('ai-components syntax highlighting', () => { + const originalOS = Platform.OS; + + afterEach(() => { + setPlatform(originalOS); + registerNativeHandlers({ PerfText: null as unknown as undefined }); + }); + + it('converts web CSS into React Native text styles', () => { + expect( + cssToRNTextStyle({ + color: '#fff', + font: 'italic bold 14px/20px "Fira Code", monospace', + letterSpacing: '0.5em', + overflow: 'auto', + }), + ).toEqual({ + color: '#fff', + fontFamily: 'Fira Code', + fontSize: 14, + fontStyle: 'italic', + fontWeight: 'bold', + letterSpacing: 8, + lineHeight: 20, + overflow: 'scroll', + }); + expect( + cssToRNTextStyle({ fontSize: '1.5em', lineHeight: 'nope' }, { baseFontSize: 10 }), + ).toEqual({ + fontSize: 15, + }); + }); + + it('flattens highlighted rows into one text with merged colour ranges', () => { + const { defaultColor, transformedStyle } = generateNewStylesheet({ + highlighter: 'prism', + stylesheet: { + 'pre[class*="language-"]': { color: '#111111' }, + keyword: { color: '#ff0000' }, + }, + }); + expect(defaultColor).toBe('#111111'); + + const rows = [ + [ + { + children: [{ type: 'text', value: 'const' }], + properties: { className: ['token', 'keyword'] }, + tagName: 'span', + type: 'element', + }, + { type: 'text', value: ' a = 1;' }, + ], + [{ type: 'text', value: 'next' }], + ] as unknown as rendererNode[]; + + const { ranges, text } = flattenRowsToTextAndColorRanges(rows, { + defaultColor: defaultColor as string, + // The renderer passes the converted stylesheet through react-syntax-highlighter untyped. + stylesheet: transformedStyle as unknown as Parameters< + typeof flattenRowsToTextAndColorRanges + >[1]['stylesheet'], + }); + + expect(text).toBe('const a = 1;\nnext'); + expect(ranges).toHaveLength(3); + expect(ranges[0]).toMatchObject({ end: 5, start: 0 }); + expect(ranges[1]).toMatchObject({ end: 12, start: 5 }); + expect(ranges[2]).toMatchObject({ end: 17, start: 13 }); + }); + + it('renders nested Text on iOS', () => { + setPlatform('ios'); + render( + + + {'const a = 1;'} + + , + ); + + expect(screen.getByText('const')).toBeTruthy(); + }); + + it('highlights with highlight.js by default', () => { + setPlatform('ios'); + render( + + {'const a = 1;'} + , + ); + + // Tokenized: the keyword is its own text node, inside a span styled by the highlight.js default + // theme (`hljs-keyword` is bold). + expect(screen.getByText('const')).toBeTruthy(); + expect( + screen + .UNSAFE_getAllByType(Text) + .some((node) => StyleSheet.flatten(node.props.style)?.fontWeight === 'bold'), + ).toBe(true); + }); + + it('renders a single PerfText on Android', () => { + setPlatform('android'); + const MockPerfText = jest.fn((props) => ); + registerNativeHandlers({ PerfText: MockPerfText }); + + render( + + + {'const a = 1;'} + + , + ); + + expect(screen.getByTestId('perf-text')).toBeTruthy(); + expect(MockPerfText.mock.calls[0]?.[0]).toMatchObject({ text: 'const a = 1;' }); + }); +}); diff --git a/package/src/ai-components/__tests__/charts.test.ts b/package/src/ai-components/__tests__/charts.test.ts new file mode 100644 index 0000000000..f5a38095e5 --- /dev/null +++ b/package/src/ai-components/__tests__/charts.test.ts @@ -0,0 +1,131 @@ +import { colorFromLabel, hashLabel, parseJsonChart, parseMermaid, xorshift32 } from '../charts'; + +describe('ai-components chart parsers', () => { + describe('parseMermaid', () => { + it('parses a pie chart with a title', () => { + const spec = parseMermaid('pie\ntitle Pets\n"Dogs" : 386\n"Cats" : 85.5\nRats : 15'); + + expect(spec.type).toBe('pie'); + expect(spec.title).toBe('Pets'); + expect(spec.data.map(({ dimension, value }) => ({ dimension, value }))).toEqual([ + { dimension: 'Dogs', value: 386 }, + { dimension: 'Cats', value: 85.5 }, + { dimension: 'Rats', value: 15 }, + ]); + spec.data.forEach(({ color }) => expect(color).toMatch(/^#[0-9a-f]{6}$/i)); + }); + + it('throws on unsupported diagrams', () => { + expect(() => parseMermaid('graph TD\nA-->B')).toThrow('Unknown mermaid diagram type'); + }); + }); + + describe('parseJsonChart', () => { + it('parses a vega-lite bar chart', () => { + const spec = parseJsonChart( + JSON.stringify({ + data: { + values: [ + { a: 'A', b: 28 }, + { a: 'B', b: 55 }, + { a: 'C', b: 'not a number' }, + ], + }, + encoding: { + x: { field: 'a', type: 'nominal' }, + y: { field: 'b', type: 'quantitative' }, + }, + mark: 'bar', + }), + ); + + expect(spec.type).toBe('bar'); + expect(spec.data).toEqual([ + { color: undefined, dimension: 'A', value: 28 }, + { color: undefined, dimension: 'B', value: 55 }, + ]); + }); + + it('parses a vega-lite arc chart as a pie chart', () => { + const spec = parseJsonChart( + JSON.stringify({ + data: { + values: [ + { category: 'x', value: 4 }, + { category: 'y', value: 6 }, + ], + }, + encoding: { + color: { field: 'category', type: 'nominal' }, + theta: { field: 'value', type: 'quantitative' }, + }, + mark: { type: 'arc' }, + }), + ); + + expect(spec.type).toBe('pie'); + expect(spec.data.map(({ dimension, value }) => ({ dimension, value }))).toEqual([ + { dimension: 'x', value: 4 }, + { dimension: 'y', value: 6 }, + ]); + }); + + it('takes the mark from the first layer', () => { + const spec = parseJsonChart( + JSON.stringify({ + data: { values: [{ t: '2024-01-01', v: 1 }] }, + encoding: { x: { field: 't', type: 'temporal' }, y: { field: 'v' } }, + layer: [{ mark: { type: 'line' } }], + }), + ); + + expect(spec.type).toBe('line'); + expect(spec.isTemporalDim).toBe(true); + expect(spec.data[0]?.dimension).toBeInstanceOf(Date); + }); + + it('rejects JSON that is not a supported chart', () => { + expect(() => parseJsonChart(JSON.stringify({ hello: 'world' }))).toThrow( + 'Unknown type of JSON formatted chart.', + ); + expect(() => + parseJsonChart( + JSON.stringify({ + data: { values: [] }, + encoding: { x: { field: 'a' }, y: { field: 'b' } }, + mark: 'geoshape', + }), + ), + ).toThrow(); + expect(() => parseJsonChart('{ not json')).toThrow(); + }); + }); + + describe('colours', () => { + it('hashes labels case and whitespace insensitively', () => { + expect(hashLabel(' Dogs ')).toBe(hashLabel('dogs')); + expect(hashLabel('dogs')).not.toBe(hashLabel('cats')); + }); + + it('produces a deterministic pseudo random sequence in [0, 1]', () => { + const a = xorshift32(42); + const b = xorshift32(42); + for (let i = 0; i < 5; i++) { + const value = a(); + expect(value).toBe(b()); + expect(value).toBeGreaterThanOrEqual(0); + expect(value).toBeLessThanOrEqual(1); + } + }); + + it('gives stable, distinct colours per label', () => { + const firstRun = ['a', 'b', 'c'].map((label) => colorFromLabel(label, [])); + const secondRun = ['a', 'b', 'c'].map((label) => colorFromLabel(label, [])); + expect(firstRun).toEqual(secondRun); + + const usedHues: number[] = []; + const distinct = ['a', 'b', 'c', 'd'].map((label) => colorFromLabel(label, usedHues)); + expect(new Set(distinct).size).toBe(4); + }); + }); +}); diff --git a/package/src/ai-components/__tests__/dictation.test.tsx b/package/src/ai-components/__tests__/dictation.test.tsx new file mode 100644 index 0000000000..83219584f9 --- /dev/null +++ b/package/src/ai-components/__tests__/dictation.test.tsx @@ -0,0 +1,166 @@ +import React from 'react'; +import { DeviceEventEmitter } from 'react-native'; + +import { act, fireEvent, render, screen } from '@testing-library/react-native'; + +import * as aiComponents from '../index'; +import { AIDictation } from '../native/AIDictation'; + +const flushPromises = () => new Promise((resolve) => setTimeout(resolve, 0)); + +const createNativeModule = () => ({ + addListener: jest.fn(), + cancel: jest.fn(), + isRecording: jest.fn(() => false), + removeListeners: jest.fn(), + start: jest.fn(), + stop: jest.fn(), +}); + +// The AI components read the native module through this getter, so every test can decide whether +// the module is "compiled in". +let mockNativeModule: ReturnType | null = null; +jest.mock('../native-specs/NativeAIDictation', () => ({ + __esModule: true, + get default() { + return mockNativeModule; + }, +})); + +const renderComposer = () => + render( + + Promise.resolve())} /> + , + ); + +describe('ai-components dictation', () => { + afterEach(() => { + mockNativeModule = null; + act(() => { + aiComponents.dictationStoreApi.reset(); + }); + }); + + describe('when the native module is not compiled in', () => { + it('reports dictation as unavailable, hides the mic button and warns once in development', () => { + const warn = jest.spyOn(console, 'warn').mockImplementation(() => {}); + + expect(aiComponents.isDictationAvailable()).toBe(false); + renderComposer(); + renderComposer(); + + expect(screen.UNSAFE_queryAllByProps({ fill: '#7A7A7A' })).toHaveLength(0); + expect(warn).toHaveBeenCalledTimes(1); + expect(warn).toHaveBeenCalledWith(expect.stringContaining('$StreamChatEnableDictation')); + warn.mockRestore(); + }); + + it('turns the low-level API into no-ops', () => { + expect(() => AIDictation.start()).not.toThrow(); + expect(AIDictation.isRecording()).toBe(false); + const unsubscribe = AIDictation.addResultListener(jest.fn()); + expect(() => unsubscribe()).not.toThrow(); + }); + }); + + describe('when the native module is compiled in', () => { + it('starts dictation and streams the transcript into the input', async () => { + const nativeModule = createNativeModule(); + mockNativeModule = nativeModule; + expect(aiComponents.isDictationAvailable()).toBe(true); + + renderComposer(); + + // The mic is the only icon rendered by an empty composer. + const mic = screen.UNSAFE_getByProps({ fill: '#7A7A7A' }); + await act(async () => { + fireEvent.press(mic); + await flushPromises(); + }); + expect(nativeModule.start).toHaveBeenCalledWith({ + interimResults: true, + language: 'en-US', + silenceTimeoutMs: 2500, + }); + + act(() => { + DeviceEventEmitter.emit('AIDictationOnState', { state: 'listening' }); + }); + // While recording the mic is highlighted. + expect(screen.UNSAFE_getByProps({ fill: '#005FFF' })).toBeTruthy(); + + act(() => { + DeviceEventEmitter.emit('AIDictationOnResult', { isFinal: false, text: 'hello wor' }); + }); + expect(screen.getByPlaceholderText('Ask anything').props.value).toBe('hello wor'); + + // Pressing the mic while listening stops dictation. + fireEvent.press(screen.UNSAFE_getByProps({ fill: '#005FFF' })); + expect(nativeModule.stop).toHaveBeenCalled(); + + act(() => { + DeviceEventEmitter.emit('AIDictationOnState', { state: 'stopping' }); + DeviceEventEmitter.emit('AIDictationOnResult', { isFinal: true, text: 'hello world' }); + }); + expect(aiComponents.dictationStore.getLatestValue()).toMatchObject({ + isRecording: false, + state: 'idle', + transcript: 'hello world', + }); + // Once dictation finishes with text in the input, the send button replaces the mic. + expect(screen.getByTestId('send-up')).toBeTruthy(); + expect(nativeModule.cancel).toHaveBeenCalled(); + }); + + it('applies a final result that arrives after the recognizer stopped listening', async () => { + mockNativeModule = createNativeModule(); + renderComposer(); + + await act(async () => { + fireEvent.press(screen.UNSAFE_getByProps({ fill: '#7A7A7A' })); + await flushPromises(); + }); + // Some recognizers (e.g. the Android emulator) send no interim results, only a final one + // after `stopping`. + act(() => { + DeviceEventEmitter.emit('AIDictationOnState', { state: 'listening' }); + DeviceEventEmitter.emit('AIDictationOnState', { state: 'stopping' }); + DeviceEventEmitter.emit('AIDictationOnResult', { isFinal: true, text: 'test test' }); + }); + + expect(screen.getByPlaceholderText('Ask anything').props.value).toBe('test test'); + }); + + it('does not restore the transcript of an earlier dictation when the input mounts', () => { + mockNativeModule = createNativeModule(); + act(() => { + aiComponents.dictationStore.partialNext({ transcript: 'an earlier dictation' }); + }); + + renderComposer(); + + expect(screen.getByPlaceholderText('Ask anything').props.value).toBe(''); + }); + + it('stores native errors', async () => { + mockNativeModule = createNativeModule(); + renderComposer(); + + await act(async () => { + fireEvent.press(screen.UNSAFE_getByProps({ fill: '#7A7A7A' })); + await flushPromises(); + }); + act(() => { + DeviceEventEmitter.emit('AIDictationOnState', { state: 'listening' }); + DeviceEventEmitter.emit('AIDictationOnError', { code: 'X', message: 'boom' }); + }); + + expect(aiComponents.dictationStore.getLatestValue()).toMatchObject({ + error: { code: 'X', message: 'boom' }, + isRecording: false, + state: 'idle', + }); + }); + }); +}); diff --git a/package/src/ai-components/__tests__/generateMarkdownText.test.ts b/package/src/ai-components/__tests__/generateMarkdownText.test.ts new file mode 100644 index 0000000000..f45e5b4cac --- /dev/null +++ b/package/src/ai-components/__tests__/generateMarkdownText.test.ts @@ -0,0 +1,54 @@ +import { escapeRegExp, generateMarkdownText, parseLinksFromText } from '../markdown'; + +describe('ai-components generateMarkdownText', () => { + it('returns null for empty input', () => { + expect(generateMarkdownText(undefined)).toBeNull(); + expect(generateMarkdownText('')).toBeNull(); + }); + + it('converts a raw link into a markdown link', () => { + expect(generateMarkdownText('Visit getstream.io today')).toBe( + 'Visit [getstream.io](http://getstream.io) today', + ); + }); + + // Regression: every link used to be replaced on the original text, so only the last one survived. + it('converts every raw link in the text', () => { + expect(generateMarkdownText('See getstream.io and reactnative.dev')).toBe( + 'See [getstream.io](http://getstream.io) and [reactnative.dev](http://reactnative.dev)', + ); + }); + + it('keeps emails as mailto links', () => { + expect(generateMarkdownText('Mail hello@getstream.io')).toBe( + 'Mail [hello@getstream.io](mailto:hello@getstream.io)', + ); + }); + + it('closes an unterminated code fence', () => { + const result = generateMarkdownText('Code:\n```js\nconst a = 1;'); + expect((result?.match(/```/g) ?? []).length % 2).toBe(0); + }); + + it('collapses indentation after newlines but keeps nested list indentation', () => { + expect(generateMarkdownText('Line one\n indented line')).toBe('Line one\nindented line'); + expect(generateMarkdownText('- parent\n - child')).toBe('- parent\n - child'); + }); + + it('preserves code blocks verbatim', () => { + const result = generateMarkdownText('```py\ndef f():\n return 1\n```'); + expect(result).toContain('def f():\n return 1'); + }); + + it('parses links and emails, ignoring markdown links', () => { + expect(parseLinksFromText('a getstream.io b [x](https://example.com) c me@x.io')).toEqual([ + { raw: 'getstream.io', url: 'http://getstream.io' }, + { raw: 'me@x.io', url: 'mailto:me@x.io' }, + ]); + expect(parseLinksFromText(undefined)).toEqual([]); + }); + + it('escapes regular expression characters', () => { + expect(escapeRegExp('a.b*c')).toBe('a\\.b\\*c'); + }); +}); diff --git a/package/src/ai-components/__tests__/theme.test.tsx b/package/src/ai-components/__tests__/theme.test.tsx new file mode 100644 index 0000000000..8903050385 --- /dev/null +++ b/package/src/ai-components/__tests__/theme.test.tsx @@ -0,0 +1,41 @@ +import React from 'react'; + +import { renderHook } from '@testing-library/react-native'; + +import { defaultTheme, mergeThemes, StreamTheme, useTheme } from '../index'; + +describe('ai-components theme', () => { + it('throws when used outside of StreamTheme', () => { + jest.spyOn(console, 'error').mockImplementation(() => {}); + expect(() => renderHook(() => useTheme())).toThrow( + 'The useTheme hook was called outside the StreamTheme Provider.', + ); + jest.restoreAllMocks(); + }); + + it('deep merges style overrides without mutating the default theme', () => { + const merged = mergeThemes({ + style: { colors: { accent_blue: 'red' }, composer: { inputPillHeight: 60 } }, + }); + + expect(merged.colors.accent_blue).toBe('red'); + expect(merged.colors.black).toBe(defaultTheme.colors.black); + expect(merged.composer.inputPillHeight).toBe(60); + expect(defaultTheme.colors.accent_blue).toBe('#005FFF'); + }); + + it('provides the merged theme, or a pre-merged one', () => { + const { result } = renderHook(() => useTheme(), { + wrapper: ({ children }) => ( + {children} + ), + }); + expect(result.current.theme.colors.grey).toBe('blue'); + + const mergedStyle = mergeThemes({ style: { colors: { grey: 'green' } } }); + const { result: preMerged } = renderHook(() => useTheme(), { + wrapper: ({ children }) => {children}, + }); + expect(preMerged.current.theme).toBe(mergedStyle); + }); +}); diff --git a/package/src/ai-components/charts/Chart.tsx b/package/src/ai-components/charts/Chart.tsx new file mode 100644 index 0000000000..a047d4c7b0 --- /dev/null +++ b/package/src/ai-components/charts/Chart.tsx @@ -0,0 +1,20 @@ +import React from 'react'; + +import { VictoryChart } from './components'; +import type { ChartSpec } from './types'; + +export type ChartFromBlockProps = { + spec: ChartSpec; + height?: number; + width?: number; +}; + +const Chart = ({ spec, height = 260, width }: ChartFromBlockProps) => { + if (!spec) { + return null; + } + + return ; +}; + +export default Chart; diff --git a/package/src/ai-components/charts/components/PieLegend.tsx b/package/src/ai-components/charts/components/PieLegend.tsx new file mode 100644 index 0000000000..dcf8dd0cc6 --- /dev/null +++ b/package/src/ai-components/charts/components/PieLegend.tsx @@ -0,0 +1,50 @@ +import React from 'react'; +import { StyleSheet, Text, View } from 'react-native'; + +import type { Datum } from '../types'; + +export const PieLegend = ({ items, maxRows = 3 }: { items: Datum[]; maxRows?: number }) => { + return ( + + {items.slice(0, maxRows * 4).map((item, i) => ( + + + + {item.dimension} + + + ))} + + ); +}; + +const styles = StyleSheet.create({ + container: { + marginTop: 8, + flexDirection: 'row', + justifyContent: 'center', + flexWrap: 'wrap', + gap: 10, + }, + item: { + flexDirection: 'row', + alignItems: 'center', + gap: 6, + maxWidth: '48%', // lets two items sit per row, tweak as needed + }, + swatch: { + borderRadius: 2, + width: 12, + height: 12, + }, + label: { + fontSize: 13, + }, +}); diff --git a/package/src/ai-components/charts/components/VictoryChart.tsx b/package/src/ai-components/charts/components/VictoryChart.tsx new file mode 100644 index 0000000000..fa728c7449 --- /dev/null +++ b/package/src/ai-components/charts/components/VictoryChart.tsx @@ -0,0 +1,193 @@ +import React from 'react'; +import { Platform, View } from 'react-native'; + +import { LinearGradient, matchFont, vec } from '@shopify/react-native-skia'; +import { + Area, + Bar, + CartesianChart, + Line, + Pie, + PolarChart, + Scatter, + // useChartTransformState, +} from 'victory-native'; + +import { PieLegend } from './PieLegend'; + +import type { ChartSpec } from '../types'; + +const font = matchFont({ + fontFamily: Platform.select({ + ios: 'Helvetica', + android: 'sans-serif', + default: 'serif', + }), + fontSize: 12, + fontWeight: 'normal', + fontStyle: 'normal', +}); + +export type VictoryChartProps = { + spec: ChartSpec; + width?: number; + height?: number; +}; + +const DEFAULT_HEIGHT = 260; +const DEFAULT_WIDTH = 225; + +export const VictoryChart = (props: VictoryChartProps) => { + const { spec, height = DEFAULT_HEIGHT, width = DEFAULT_WIDTH } = props; + + // const { state: transformState } = useChartTransformState({ + // scaleX: 1.5, + // scaleY: 1.0, + // }); + + if (spec.type === 'pie') { + if (!spec?.data.length) return null; + return ( + <> + + + + + + + + ); + } + + // Group by color into series + const seriesNames = Array.from(new Set(spec.data.map((d) => d.color ?? 'series'))); + const yKeys = seriesNames.map((_name, idx) => `y${idx}` as const); + + let minY = 0; + + // Normalize data into one row per x, with columns y0,y1,... for each series + // eslint-disable-next-line @typescript-eslint/no-explicit-any -- victory-native rows are keyed by dynamic series names + const rowsMap = new Map>(); + for (const s of seriesNames) { + const keyIndex = seriesNames.indexOf(s); + const yKey = yKeys[keyIndex]; + if (yKey) { + for (const pt of spec.data.filter((d) => (d.color ?? 'series') === s)) { + const xKey = spec.isTemporalDim ? new Date(pt.dimension).getTime() : pt.dimension; + const row = rowsMap.get(xKey) ?? { x: pt.dimension }; + row[yKey] = pt.value; + minY = Math.min(minY, pt.value); + rowsMap.set(xKey, row); + } + } + } + + const table = Array.from(rowsMap.values()); + + const visibleBars = table.length; + + const estimatedBarWidth = (width - 30 - visibleBars * 2) / visibleBars; + + const maxTickCharCount = spec.isTemporalDim + ? 3 + : Math.max( + ...table.map((entry) => + typeof entry.x === 'string' ? entry.x.length : String(entry.x).length, + ), + ); + + const hasLongLabels = + (visibleBars > 4 ? maxTickCharCount > 1 : maxTickCharCount > 5) && !spec.isNumericDim; + + const lowestValue = minY < 0 ? minY - 20 : 0; + + return ( + + { + const parsedLabel = (typeof label === 'number' ? String(label) : label) ?? ''; + if (hasLongLabels && parsedLabel.length > 4) { + return `${parsedLabel.slice(0, 3)}...`; + } + return parsedLabel; + }, + labelRotate: hasLongLabels ? 90 : 0, + tickCount: visibleBars, + labelOffset: hasLongLabels ? 0 : 2, + }} + yAxis={[{ font }]} + padding={{ bottom: hasLongLabels ? 2 * maxTickCharCount + 12 : 0 }} + // transformState={transformState} + // transformConfig={{ + // pinch: { enabled: false }, + // pan: { enabled: true, dimensions: 'x', activateAfterLongPress: 10 }, + // }} + > + {({ points, chartBounds }) => ( + <> + {/* One primitive per series */} + {yKeys.map((yk) => { + // eslint-disable-next-line @typescript-eslint/no-explicit-any -- victory-native types points per static key, ours are dynamic + const pts = (points as Record)[yk]!; + switch (spec.type) { + case 'bar': + return ( + + + + ); + case 'area': + return ( + + + + ); + case 'point': + return ; + case 'line': + default: + return ( + + ); + } + })} + + )} + + + ); +}; diff --git a/package/src/ai-components/charts/components/index.ts b/package/src/ai-components/charts/components/index.ts new file mode 100644 index 0000000000..64b181088f --- /dev/null +++ b/package/src/ai-components/charts/components/index.ts @@ -0,0 +1,2 @@ +export * from './PieLegend'; +export * from './VictoryChart'; diff --git a/package/src/ai-components/charts/index.ts b/package/src/ai-components/charts/index.ts new file mode 100644 index 0000000000..2eb7939f6d --- /dev/null +++ b/package/src/ai-components/charts/index.ts @@ -0,0 +1,13 @@ +// components +export * from './Chart'; +export * from './components'; + +// schemas and parsers +export * from './vega-lite'; +export * from './mermaid'; + +// utils +export * from './utils'; + +// types +export * from './types'; diff --git a/package/src/ai-components/charts/mermaid/index.ts b/package/src/ai-components/charts/mermaid/index.ts new file mode 100644 index 0000000000..75aa8b9397 --- /dev/null +++ b/package/src/ai-components/charts/mermaid/index.ts @@ -0,0 +1 @@ +export * from './parser'; diff --git a/package/src/ai-components/charts/mermaid/parser.ts b/package/src/ai-components/charts/mermaid/parser.ts new file mode 100644 index 0000000000..f36893e898 --- /dev/null +++ b/package/src/ai-components/charts/mermaid/parser.ts @@ -0,0 +1,33 @@ +import type { ChartSpec, Datum } from '../types'; +import { colorFromLabel } from '../utils'; + +export const parseMermaidPie = (lines: string[]): ChartSpec => { + let title: string | undefined; + const data: Datum[] = []; + const usedHues: number[] = []; + for (const ln of lines.slice(1)) { + const t = ln.match(/^title\s+(.+)$/i); + // TODO: Check if this is correct ? We only have one group. + if (t && t[1]) { + title = t[1]?.trim(); + continue; + } + const d = ln.match(/^(?:"([^"]+)"|([^"]+))\s*:\s*([+-]?\d+(?:\.\d+)?)/); + if (d) { + const label = (d[1] ?? d[2] ?? '').trim(); + data.push({ + dimension: label, + value: Number(d[3]), + color: colorFromLabel(label, usedHues), + }); + } + } + return { type: 'pie', title, data }; +}; + +export const parseMermaid = (code: string) => { + const lines = code.split(/\r?\n/).map((l) => l.trim()); + if (/^pie\b/i.test(lines[0] || '')) return parseMermaidPie(lines); + + throw new Error('Unknown mermaid diagram type'); +}; diff --git a/package/src/ai-components/charts/types.ts b/package/src/ai-components/charts/types.ts new file mode 100644 index 0000000000..889599a389 --- /dev/null +++ b/package/src/ai-components/charts/types.ts @@ -0,0 +1,25 @@ +export type Dim = string; + +export type Datum = { + /** The categorical/temporal dimension (label, category, or time). */ + dimension: Dim; + /** The numeric measure/value. */ + value: number; + /** Color for this datum; the renderer will fall back to series color. */ + color: string; + /** Optional series key for multi-series charts (e.g., region). */ + series?: string; +}; + +export type ChartType = 'bar' | 'line' | 'area' | 'point' | 'pie'; + +export type ChartSpec = { + type: ChartType; + data: Datum[]; + seriesColors?: Record; + title?: string; + orientation?: 'vertical' | 'horizontal'; + isTemporalDim?: boolean; + isNumericDim?: boolean; + dimDomain?: Dim[]; +}; diff --git a/package/src/ai-components/charts/utils/generate-pie-colors.ts b/package/src/ai-components/charts/utils/generate-pie-colors.ts new file mode 100644 index 0000000000..8eccf723a5 --- /dev/null +++ b/package/src/ai-components/charts/utils/generate-pie-colors.ts @@ -0,0 +1,82 @@ +/* eslint-disable no-bitwise -- FNV-1a hashing and xorshift PRNG are bitwise by definition */ +// hashing and rng +export const hashLabel = (raw: string) => { + const s = raw.trim().toLowerCase().normalize('NFKC'); + let h = 2166136261 >>> 0; // FNV-1a + for (let i = 0; i < s.length; i++) { + h ^= s.charCodeAt(i); + h = Math.imul(h, 16777619) >>> 0; + } + return h >>> 0; +}; + +export const xorshift32 = (seed: number) => { + let x = seed || 123456789; + return () => { + x ^= x << 13; + x ^= x >>> 17; + x ^= x << 5; + return (x >>> 0) / 0xffffffff; + }; +}; + +// color utils +export const hslToHex = (h: number, s = 68, l = 52) => { + const S = s / 100, + L = l / 100; + const C = (1 - Math.abs(2 * L - 1)) * S; + const X = C * (1 - Math.abs(((h / 60) % 2) - 1)); + const m = L - C / 2; + const [r1, g1, b1] = + h < 60 + ? [C, X, 0] + : h < 120 + ? [X, C, 0] + : h < 180 + ? [0, C, X] + : h < 240 + ? [0, X, C] + : h < 300 + ? [X, 0, C] + : [C, 0, X]; + const r = Math.round((r1 + m) * 255) + .toString(16) + .padStart(2, '0'); + const g = Math.round((g1 + m) * 255) + .toString(16) + .padStart(2, '0'); + const b = Math.round((b1 + m) * 255) + .toString(16) + .padStart(2, '0'); + return `#${r}${g}${b}`; +}; + +// ensure hues not closer than MIN_SEP degrees within the same chart +export const colorFromLabel = ( + label: string, + usedHues: number[], + opts?: { minSep?: number; sat?: number; light?: number }, +) => { + const { minSep = 25, sat = 68, light = 52 } = opts || {}; + const seed = hashLabel(label); + const rand = xorshift32(seed); + + // base hue from RNG (looks random, but stable) + let hue = Math.floor(rand() * 360); + + // nudge away from already used hues in THIS chart + let attempts = 0; + const tooClose = (h: number) => + usedHues.some((u) => { + const d = Math.abs(h - u); + return Math.min(d, 360 - d) < minSep; + }); + + while (tooClose(hue) && attempts < 48) { + hue = (hue + 137.508) % 360; // golden-angle hop + attempts++; + } + + usedHues.push(hue); + return hslToHex(hue, sat, light); +}; diff --git a/package/src/ai-components/charts/utils/index.ts b/package/src/ai-components/charts/utils/index.ts new file mode 100644 index 0000000000..47bd877534 --- /dev/null +++ b/package/src/ai-components/charts/utils/index.ts @@ -0,0 +1,2 @@ +export * from './generate-pie-colors'; +export * from './parse-json-chart'; diff --git a/package/src/ai-components/charts/utils/parse-json-chart.ts b/package/src/ai-components/charts/utils/parse-json-chart.ts new file mode 100644 index 0000000000..8c28bba2ec --- /dev/null +++ b/package/src/ai-components/charts/utils/parse-json-chart.ts @@ -0,0 +1,15 @@ +import type { ChartSpec } from '../types'; +import { parseVegalite, VegaLiteSchema } from '../vega-lite'; + +export const parseJsonChart = (code: string): ChartSpec => { + const json = JSON.parse(code); + + try { + const parsedVegaLite = VegaLiteSchema.parse(json); + return parseVegalite(parsedVegaLite); + } catch (_error) { + /* do nothing */ + } + + throw new Error('Unknown type of JSON formatted chart.'); +}; diff --git a/package/src/ai-components/charts/vega-lite/index.ts b/package/src/ai-components/charts/vega-lite/index.ts new file mode 100644 index 0000000000..2747256d7a --- /dev/null +++ b/package/src/ai-components/charts/vega-lite/index.ts @@ -0,0 +1,2 @@ +export * from './parser'; +export * from './schema'; diff --git a/package/src/ai-components/charts/vega-lite/parser.ts b/package/src/ai-components/charts/vega-lite/parser.ts new file mode 100644 index 0000000000..b86989d50e --- /dev/null +++ b/package/src/ai-components/charts/vega-lite/parser.ts @@ -0,0 +1,49 @@ +import type { VegaLite } from './schema'; + +import type { ChartSpec, Datum } from '../types'; +import { colorFromLabel } from '../utils'; + +export const parseVegalite = (spec: VegaLite): ChartSpec => { + if (spec.type === 'pie') { + const tField = spec.encoding?.theta?.field; + const cField = spec.encoding?.color?.field; + const usedHues: number[] = []; + const data = (spec.data?.values ?? []) + .map((d) => { + const y = Number(d[tField]); + const x = cField ? String(d[cField]) : ''; + return Number.isFinite(y) + ? { + dimension: x.trim(), + value: y, + color: colorFromLabel(x, usedHues), + } + : null; + }) + .filter(Boolean) as Datum[]; + return { type: 'pie', data }; + } + + const { x, y, color } = spec.encoding; + const xField = x?.field, + yField = y?.field; + const xIsTime = (x?.type ?? '').toLowerCase() === 'temporal'; + const xIsNumeric = (x?.type ?? '').toLowerCase() === 'quantitative'; + + const data = (spec.data?.values ?? []) + .map((d) => { + const xv = xIsTime && typeof d[xField] === 'string' ? new Date(d[xField]) : d[xField]; + const yv = Number(d[yField]); + if (!Number.isFinite(yv)) return null; + const c = color?.field ? String(d[color.field]) : undefined; + return { dimension: xv, value: yv, color: c }; + }) + .filter(Boolean) as Datum[]; + + return { + type: spec.type === 'xy' && spec.markName !== 'arc' ? spec.markName : 'pie', + data, + isTemporalDim: xIsTime, + isNumericDim: xIsNumeric, + }; +}; diff --git a/package/src/ai-components/charts/vega-lite/schema.ts b/package/src/ai-components/charts/vega-lite/schema.ts new file mode 100644 index 0000000000..2f86626e7b --- /dev/null +++ b/package/src/ai-components/charts/vega-lite/schema.ts @@ -0,0 +1,112 @@ +import { z } from 'zod'; + +/* primitives */ +export const VLType = z.enum(['quantitative', 'ordinal', 'nominal', 'temporal']); + +const FieldRef = z.object({ + field: z.string(), + type: VLType.optional(), +}); + +const ColorXY = z.object({ + field: z.string().optional(), + type: VLType.optional(), +}); +const ColorPie = z.object({ field: z.string(), type: VLType.optional() }); + +/* we support only these currently */ +const MarkName = z.enum([ + 'bar', + 'line', + 'area', + 'point', + 'arc', + // 'rect', + // 'rule', + // 'text', + // 'tick', + // 'circle', + // 'square', + // 'trail', + // 'geoshape', + // 'image', +]); + +const MarkDef = z.looseObject({ + type: z.union([MarkName, z.string()]).optional(), +}); + +/* encodings */ +const XyEncoding = z.object({ + x: FieldRef, + y: FieldRef, + color: ColorXY.optional(), +}); + +const PieEncoding = z.object({ + theta: FieldRef, + color: ColorPie.optional(), +}); + +/* base spec */ +const BaseVL = z.object({ + archetype: z.literal('vegalite'), + $schema: z.string().optional(), + data: z.object({ + values: z.array(z.record(z.string(), z.unknown())), + }), + mark: z.union([MarkName, MarkDef, z.string()]).optional(), + layer: z.array(z.looseObject({ mark: MarkDef })).optional(), + marks: z.array(MarkDef).optional(), + markName: MarkName, +}); + +/* branches (with explicit kind) */ +const VL_XY = BaseVL.extend({ + type: z.literal('xy'), + encoding: XyEncoding, +}).strip(); + +const VL_PIE = BaseVL.extend({ + type: z.literal('pie'), + encoding: PieEncoding, +}).strip(); + +const normalizeMarkName = (m: unknown): string | undefined => { + if (typeof m === 'string') return m.toLowerCase(); + if (m && typeof m === 'object' && typeof (m as { type?: unknown }).type === 'string') { + return String((m as { type?: unknown }).type).toLowerCase(); + } + return undefined; +}; + +// eslint-disable-next-line @typescript-eslint/no-explicit-any -- untyped JSON input, validated by the zod schema below +const deriveMarkName = (o: any): string | undefined => { + // spec.mark (string or MarkDef) + const direct = normalizeMarkName(o?.mark); + if (direct) return direct; + + // first layer's mark.type + const firstLayerMark = + Array.isArray(o?.layer) && o.layer[0]?.mark ? normalizeMarkName(o.layer[0].mark) : undefined; + if (firstLayerMark) return firstLayerMark; + + // first marks[] item .type + return Array.isArray(o?.marks) && o.marks.length > 0 ? normalizeMarkName(o.marks[0]) : undefined; +}; + +export const VegaLiteCore = z.discriminatedUnion('type', [VL_XY, VL_PIE]); + +/* final schema: preprocess to inject needed metadata, then discriminated union */ +export const VegaLiteSchema = z.preprocess((raw) => { + // eslint-disable-next-line @typescript-eslint/no-explicit-any -- untyped JSON input, validated by the zod schema below + const o: any = raw ?? {}; + const markName = deriveMarkName(o); + const hasTheta = !!o?.encoding?.theta; + + const type = markName === 'arc' || hasTheta ? 'pie' : 'xy'; + + return { ...o, type, archetype: 'vegalite', markName }; +}, VegaLiteCore); + +export type VegaLite = z.infer; diff --git a/package/src/ai-components/components/BottomSheet.tsx b/package/src/ai-components/components/BottomSheet.tsx new file mode 100644 index 0000000000..4268db2e95 --- /dev/null +++ b/package/src/ai-components/components/BottomSheet.tsx @@ -0,0 +1,158 @@ +import React, { type PropsWithChildren, useEffect, useMemo, useRef, useState } from 'react'; +import { Dimensions, Modal, Platform, Pressable, StyleSheet, View } from 'react-native'; +import { + Gesture, + GestureDetector, + GestureHandlerRootView, + type GestureType, +} from 'react-native-gesture-handler'; +import Animated, { + runOnJS, + useAnimatedStyle, + useSharedValue, + withTiming, +} from 'react-native-reanimated'; + +import { sheetStoreApi, useBottomSheetState } from '../store'; + +type Props = PropsWithChildren; + +const { height: SCREEN_H } = Dimensions.get('window'); +const SPRING = { duration: 200 }; +const MAX_BOTTOM_SHEET_HEIGHT = Math.round(SCREEN_H * 0.4); +const BACKDROP_OPACITY = 0.4; + +export function BottomSheet({ children }: Props) { + const { open, height } = useBottomSheetState(); + const [bottomSheetModalOpen, setBottomSheetModalOpen] = useState(open); + const maxTranslateY = SCREEN_H; + const translateY = useSharedValue(maxTranslateY); + + const bottomSheetHeight = useMemo(() => Math.min(height, MAX_BOTTOM_SHEET_HEIGHT), [height]); + const bottomSheetY = SCREEN_H - bottomSheetHeight; + + useEffect(() => { + if (open) { + setBottomSheetModalOpen(true); + } else { + translateY.value = withTiming(maxTranslateY, SPRING, (finished) => { + if (finished) { + runOnJS(setBottomSheetModalOpen)(false); + } + }); + } + }, [setBottomSheetModalOpen, maxTranslateY, open, translateY]); + + useEffect(() => { + if (bottomSheetModalOpen && open) { + translateY.value = withTiming(bottomSheetY, SPRING); + } + }, [open, bottomSheetModalOpen, bottomSheetY, translateY]); + + const dragStartY = useRef(0); + const panRef = useRef(undefined); + const scrollRef = useRef(undefined); + + const nativeScroll = useMemo(() => Gesture.Native().withRef(scrollRef), []); + + const pan = Gesture.Pan() + .withRef(panRef) + .requireExternalGestureToFail(scrollRef) + .onBegin(() => { + dragStartY.current = 0; + }) + .onUpdate((e) => { + if (dragStartY.current === 0) dragStartY.current = translateY.value; + const next = dragStartY.current + e.translationY; + const minY = bottomSheetY; + const maxY = maxTranslateY; + const overTop = next < minY; + const overBottom = next > maxY; + translateY.value = overTop + ? minY - (minY - next) * 0.2 + : overBottom + ? maxY + (next - maxY) * 0.2 + : next; + }) + .onEnd((e) => { + // very basic momentum bias + const projected = e.translationY + e.velocityY * 0.15; + const nearest = projected >= bottomSheetHeight * 0.5 ? maxTranslateY : bottomSheetY; + translateY.value = withTiming(nearest, SPRING, (finished) => { + if (finished && nearest === maxTranslateY) runOnJS(sheetStoreApi.closeSheet)(); + }); + }); + + const backdropStyle = useAnimatedStyle(() => { + const minY = bottomSheetY; + const t = 1 - Math.min(1, Math.max(0, (translateY.value - minY) / (maxTranslateY - minY))); + return { + opacity: t * BACKDROP_OPACITY, + pointerEvents: t > 0.02 ? ('auto' as const) : ('none' as const), + }; + }); + + const sheetStyle = useAnimatedStyle(() => ({ + transform: [{ translateY: translateY.value }], + })); + + const sheetContentStyle = useMemo(() => ({ height: bottomSheetHeight }), [bottomSheetHeight]); + + return ( + + + + + + + + + + + + + + + + {open ? ( + {children} + ) : null} + + + + + + + + ); +} + +const styles = StyleSheet.create({ + gestureHandlerRootView: { flex: 1 }, + backdrop: { backgroundColor: '#000' }, + sheet: { + position: 'absolute', + overflow: 'hidden', + left: 0, + right: 0, + top: 0, + height: SCREEN_H, + transform: [{ translateY: SCREEN_H }], + borderTopLeftRadius: 20, + borderTopRightRadius: 20, + backgroundColor: 'white', + ...Platform.select({ + ios: { + shadowColor: '#000', + shadowOpacity: 0.2, + shadowOffset: { width: 0, height: -4 }, + shadowRadius: 12, + }, + android: { elevation: 18 }, + }), + }, + handleContainer: { alignItems: 'center', paddingVertical: 8 }, + handle: { width: 40, height: 4, borderRadius: 2, backgroundColor: '#444' }, + content: { flexGrow: 0 }, + contentContainer: { flex: 1, minHeight: 0 }, +}); diff --git a/package/src/ai-components/components/MarkdownReactiveScrollView.tsx b/package/src/ai-components/components/MarkdownReactiveScrollView.tsx new file mode 100644 index 0000000000..1e8c722995 --- /dev/null +++ b/package/src/ai-components/components/MarkdownReactiveScrollView.tsx @@ -0,0 +1,92 @@ +import { type PropsWithChildren, useMemo } from 'react'; +import { type LayoutChangeEvent, StyleSheet, View } from 'react-native'; +import { Gesture, GestureDetector } from 'react-native-gesture-handler'; +import Animated, { clamp, scrollTo, useAnimatedRef, useSharedValue } from 'react-native-reanimated'; + +import { useStableCallback } from '../internal/hooks/useStableCallback'; + +export const MarkdownReactiveScrollView = ({ children }: PropsWithChildren) => { + const scrollViewRef = useAnimatedRef(); + const contentWidth = useSharedValue(0); + const visibleContentWidth = useSharedValue(0); + const offsetBeforeScroll = useSharedValue(0); + const touchStart = useSharedValue<{ x: number; y: number } | null>(null); + + const panGesture = useMemo( + () => + Gesture.Pan() + .onBegin((event) => { + touchStart.value = { x: event.x, y: event.y }; + }) + .onTouchesMove((event, state) => { + if (!touchStart.value || !event.changedTouches.length) { + state.fail(); + return; + } + + const xDiff = Math.abs(event.changedTouches[0]!.x - touchStart.value.x); + const yDiff = Math.abs(event.changedTouches[0]!.y - touchStart.value.y); + const isHorizontalPanning = xDiff > yDiff; + + if (isHorizontalPanning) { + state.activate(); + } else { + state.fail(); + } + }) + .onUpdate((event) => { + const { translationX } = event; + + scrollTo(scrollViewRef, offsetBeforeScroll.value - translationX, 0, false); + }) + .onEnd((event) => { + const { translationX } = event; + + const velocityEffect = event.velocityX * 0.3; + + const finalPosition = clamp( + offsetBeforeScroll.value - translationX - velocityEffect, + 0, + contentWidth.value - visibleContentWidth.value, + ); + + offsetBeforeScroll.value = finalPosition; + + scrollTo(scrollViewRef, finalPosition, 0, true); + }), + [contentWidth, offsetBeforeScroll, scrollViewRef, touchStart, visibleContentWidth], + ); + + const onContentSizeChange = useStableCallback((width: number) => { + contentWidth.value = width; + }); + + const onLayout = useStableCallback((event: LayoutChangeEvent) => { + visibleContentWidth.value = event.nativeEvent.layout.width; + }); + + return ( + + + + {children} + + + + ); +}; + +const styles = StyleSheet.create({ + wrapper: { + width: '100%', + }, + contentContainer: { flexGrow: 1 }, +}); diff --git a/package/src/ai-components/components/index.ts b/package/src/ai-components/components/index.ts new file mode 100644 index 0000000000..a96097f4ff --- /dev/null +++ b/package/src/ai-components/components/index.ts @@ -0,0 +1,2 @@ +export * from './MarkdownReactiveScrollView'; +export * from '../native/PerfText'; diff --git a/package/src/ai-components/contexts/composer-context.tsx b/package/src/ai-components/contexts/composer-context.tsx new file mode 100644 index 0000000000..b825409744 --- /dev/null +++ b/package/src/ai-components/contexts/composer-context.tsx @@ -0,0 +1,73 @@ +import React, { type PropsWithChildren, useMemo, useState } from 'react'; + +import type { StateStore } from '@stream-io/state-store'; + +import { useStableCallback } from '../internal/hooks/useStableCallback'; +import type { ComposerViewProps } from '../message-composer'; +import { MediaPickerService } from '../services'; +import type { AbstractMediaPickerService } from '../services/media-picker-service/AbstractMediaPickerService'; +import { type ComposerState, createNewComposerStore } from '../store'; + +export type ComposerContext = { + state: StateStore; + setText: (text: string) => void | Promise; + sendMessage: () => void | Promise; + mediaPickerService?: AbstractMediaPickerService; +}; + +export type ComposerContextProps = PropsWithChildren< + Pick +>; + +export const ComposerContext = React.createContext(undefined); + +export const ComposerProvider = ({ + onSendMessage, + mediaPickerService: mediaPickerServiceOverride, + state: stateOverride, + children, +}: ComposerContextProps) => { + const [mediaPickerService] = useState( + () => mediaPickerServiceOverride ?? (MediaPickerService ? new MediaPickerService() : undefined), + ); + const [state] = useState(() => stateOverride ?? createNewComposerStore()); + + const setText = useStableCallback((text: string) => state.partialNext({ text })); + + const clearState = useStableCallback(() => { + setText(''); + mediaPickerService?.clearAssets(); + }); + + const sendMessage = useStableCallback(async () => { + const { text, custom } = state.getLatestValue(); + + const data = { + text, + attachments: mediaPickerService?.state.getLatestValue().assets, + custom, + }; + + clearState(); + + await onSendMessage(data); + }); + + const contextValue = useMemo( + () => ({ mediaPickerService, state, setText, sendMessage, clearState }), + [state, mediaPickerService, setText, sendMessage, clearState], + ); + + return {children}; +}; + +export const useMessageComposerContext = () => { + const value = React.useContext(ComposerContext); + if (value === undefined) { + throw new Error( + 'The useMessageComposerContext hook was called outside of the ComposerContext provider.', + ); + } + + return value; +}; diff --git a/package/src/ai-components/contexts/index.ts b/package/src/ai-components/contexts/index.ts new file mode 100644 index 0000000000..d84d8bb745 --- /dev/null +++ b/package/src/ai-components/contexts/index.ts @@ -0,0 +1,2 @@ +export * from './composer-context'; +export * from './theme-context'; diff --git a/package/src/ai-components/contexts/theme-context.tsx b/package/src/ai-components/contexts/theme-context.tsx new file mode 100644 index 0000000000..b1f68bf1ae --- /dev/null +++ b/package/src/ai-components/contexts/theme-context.tsx @@ -0,0 +1,68 @@ +import React, { createContext, type PropsWithChildren, useContext, useMemo } from 'react'; + +import merge from 'lodash/merge'; + +import { defaultTheme, type Theme } from '../theme'; + +export type DeepPartial = T extends (...args: never[]) => unknown + ? T + : T extends object + ? { [K in keyof T]?: DeepPartial } + : T; + +export type StreamThemeInputValue = { + mergedStyle?: Theme; + style?: DeepPartial; +}; + +export type MergedThemesParams = { + style?: DeepPartial; + theme?: Theme; +}; + +export type ThemeContextValue = { + theme: Theme; +}; + +export const mergeThemes = (params: MergedThemesParams) => { + const { style, theme } = params; + // TODO: Use structured-clone here or something similar. + const finalTheme = ( + !theme || Object.keys(theme).length === 0 + ? JSON.parse(JSON.stringify(defaultTheme)) + : JSON.parse(JSON.stringify(theme)) + ) as Theme; + + if (style) { + merge(finalTheme, style); + } + + return finalTheme; +}; + +export const ThemeContext = createContext(undefined); + +export const StreamTheme = ( + props: PropsWithChildren>, +) => { + const { children, mergedStyle, style, theme } = props; + + const modifiedTheme = useMemo(() => { + if (mergedStyle) { + return mergedStyle; + } + + return mergeThemes({ style, theme }); + }, [mergedStyle, style, theme]); + + return {children}; +}; + +export const useTheme = () => { + const theme = useContext(ThemeContext); + + if (theme === undefined) { + throw new Error('The useTheme hook was called outside the StreamTheme Provider.'); + } + return { theme }; +}; diff --git a/package/src/ai-components/index.ts b/package/src/ai-components/index.ts new file mode 100644 index 0000000000..c0eec23884 --- /dev/null +++ b/package/src/ai-components/index.ts @@ -0,0 +1,29 @@ +// primitives +export * from './markdown'; +export * from './syntax-highlighting'; +export * from './charts'; +export * from './message-composer'; +export * from './StreamingMessageView'; +export * from './AITypingIndicatorView'; + +// components +export * from './components'; + +// contexts +export * from './contexts'; + +// services +export * from './services'; +export * from './transcription/useDictation'; +export { isDictationAvailable } from './native/AIDictation'; + +// state-stores +export * from './store'; + +// theme +export * from './theme'; + +// Native handler registration, used by the `stream-chat-react-native/ai-components` and +// `stream-chat-expo/ai-components` entry points to register only the native pieces this entry +// point needs (without importing, and therefore bundling, the main SDK entry point). +export { registerNativeHandlers } from '../native'; diff --git a/package/src/ai-components/internal/hooks/useStableCallback.ts b/package/src/ai-components/internal/hooks/useStableCallback.ts new file mode 100644 index 0000000000..8ef86ffb05 --- /dev/null +++ b/package/src/ai-components/internal/hooks/useStableCallback.ts @@ -0,0 +1,37 @@ +import { useCallback, useRef } from 'react'; + +export type StableCallback = (...args: A) => R; + +/** + * A utility hook implementing a stable callback. It takes in an unstable method that + * is supposed to be invoked somewhere deeper in the DOM tree without making it + * change its reference every time the parent component rerenders. It will also return + * the value of the callback if it does return one. + * A common use-case would be having a function whose invocation depends on state + * somewhere high up in the DOM tree and wanting to use the same function deeper + * down, for example in a leaf node and simply using useCallback results in + * cascading dependency hell. If we wrap it in useStableCallback, we would be able + * to: + * - Use the same function as a dependency of another hook (since it is stable) + * - Still invoke it and get the latest state + * + * **Caveats:** + * - Never wrap a function that is supposed to return a React.ReactElement in + * useStableCallback, since React will not know that the DOM needs to be updated + * whenever the callback value changes (for example, renderItem from FlatList must + * never be wrapped in this hook) + * - Always prefer using a standard useCallback/stable function wherever possible + * (the purpose of useStableCallback is to bridge the gap between top level contexts + * and cascading rereders in downstream components - **not** as an escape hatch) + * @param callback - the callback we want to stabilize + */ +export const useStableCallback = ( + callback: StableCallback, +): StableCallback => { + const ref = useRef(callback); + ref.current = callback; + + return useCallback>((...args) => { + return ref.current(...args); + }, []); +}; diff --git a/package/src/ai-components/internal/icons/Camera.tsx b/package/src/ai-components/internal/icons/Camera.tsx new file mode 100644 index 0000000000..fa8cf5c3e4 --- /dev/null +++ b/package/src/ai-components/internal/icons/Camera.tsx @@ -0,0 +1,12 @@ +import React from 'react'; + +import { type IconProps, RootPath, RootSvg } from './utils/base'; + +export const Camera = (props: IconProps) => ( + + + +); diff --git a/package/src/ai-components/internal/icons/CircleStop.tsx b/package/src/ai-components/internal/icons/CircleStop.tsx new file mode 100644 index 0000000000..6d919e5eb4 --- /dev/null +++ b/package/src/ai-components/internal/icons/CircleStop.tsx @@ -0,0 +1,18 @@ +import React from 'react'; + +import { Path, Svg } from 'react-native-svg'; + +import type { IconProps } from './utils/base'; + +type Props = IconProps & { + size: number; +}; + +export const CircleStop = ({ size, ...rest }: Props) => ( + + + +); diff --git a/package/src/ai-components/internal/icons/Close.tsx b/package/src/ai-components/internal/icons/Close.tsx new file mode 100644 index 0000000000..3f8f4bbecb --- /dev/null +++ b/package/src/ai-components/internal/icons/Close.tsx @@ -0,0 +1,12 @@ +import React from 'react'; + +import { type IconProps, RootPath, RootSvg } from './utils/base'; + +export const Close = (props: IconProps) => ( + + + +); diff --git a/package/src/ai-components/internal/icons/Mic.tsx b/package/src/ai-components/internal/icons/Mic.tsx new file mode 100644 index 0000000000..de7a79a7ae --- /dev/null +++ b/package/src/ai-components/internal/icons/Mic.tsx @@ -0,0 +1,20 @@ +import React from 'react'; + +import { G, Path, Svg } from 'react-native-svg'; + +import type { IconProps } from './utils/base'; + +type Props = IconProps & { + size: number; +}; + +export const Mic = ({ size, ...rest }: Props) => ( + + + + + +); diff --git a/package/src/ai-components/internal/icons/Picture.tsx b/package/src/ai-components/internal/icons/Picture.tsx new file mode 100644 index 0000000000..043a3256a3 --- /dev/null +++ b/package/src/ai-components/internal/icons/Picture.tsx @@ -0,0 +1,16 @@ +import React from 'react'; + +import { type IconProps, RootPath, RootSvg } from './utils/base'; + +export const Picture = (props: IconProps) => ( + + + + +); diff --git a/package/src/ai-components/internal/icons/SendUp.tsx b/package/src/ai-components/internal/icons/SendUp.tsx new file mode 100644 index 0000000000..5bd09033b9 --- /dev/null +++ b/package/src/ai-components/internal/icons/SendUp.tsx @@ -0,0 +1,21 @@ +import React from 'react'; + +import Svg, { Path } from 'react-native-svg'; + +import type { IconProps } from './utils/base'; + +type Props = IconProps & { + size: number; +}; + +export const SendUp = ({ size, ...rest }: Props) => ( + + {/**/} + + +); diff --git a/package/src/ai-components/internal/icons/utils/base.tsx b/package/src/ai-components/internal/icons/utils/base.tsx new file mode 100644 index 0000000000..d7e66580db --- /dev/null +++ b/package/src/ai-components/internal/icons/utils/base.tsx @@ -0,0 +1,44 @@ +import React from 'react'; +import Svg, { Path, type PathProps, type SvgProps } from 'react-native-svg'; + +export type IconProps = Partial & + Omit & { + height?: number; + width?: number; + }; + +export const RootSvg = (props: IconProps) => { + const { children, height = 24, viewBox = '0 0 24 24', width = 24 } = props; + return ( + + {children} + + ); +}; + +export type RootPathProps = Pick & { + pathFill?: SvgProps['fill']; + pathOpacity?: PathProps['opacity']; +}; + +export const RootPath = (props: RootPathProps) => { + const { d, pathFill = 'black', pathOpacity } = props; + return ( + + ); +}; diff --git a/package/src/ai-components/markdown/components/Autolink.tsx b/package/src/ai-components/markdown/components/Autolink.tsx new file mode 100644 index 0000000000..4a77514654 --- /dev/null +++ b/package/src/ai-components/markdown/components/Autolink.tsx @@ -0,0 +1,26 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const Autolink = ({ children, styles, onPress }: MarkdownComponentProps) => ( + + {children} + +); + +export const renderAutolink: RuleRenderFunction = ({ node, output, state, styles, onLink }) => { + state.withinLink = true; + const onLinkPress = () => onLink && onLink(node.target); + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/BlockQuote.tsx b/package/src/ai-components/markdown/components/BlockQuote.tsx new file mode 100644 index 0000000000..c04a75f1bc --- /dev/null +++ b/package/src/ai-components/markdown/components/BlockQuote.tsx @@ -0,0 +1,19 @@ +import { Text, View } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const BlockQuote = ({ children, styles }: MarkdownComponentProps) => ( + + + {children} + +); + +export const renderBlockQuote: RuleRenderFunction = ({ node, output, state, styles }) => { + state.withinQuote = true; + return ( +
+ {output(node.content, state)} +
+ ); +}; diff --git a/package/src/ai-components/markdown/components/Bold.tsx b/package/src/ai-components/markdown/components/Bold.tsx new file mode 100644 index 0000000000..d258a8bcca --- /dev/null +++ b/package/src/ai-components/markdown/components/Bold.tsx @@ -0,0 +1,16 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const Bold = ({ children, styles }: MarkdownComponentProps) => ( + {children} +); + +export const renderBold: RuleRenderFunction = ({ node, output, state, styles }) => { + state.withinText = true; + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/CodeBlock.tsx b/package/src/ai-components/markdown/components/CodeBlock.tsx new file mode 100644 index 0000000000..3e1d627a71 --- /dev/null +++ b/package/src/ai-components/markdown/components/CodeBlock.tsx @@ -0,0 +1,96 @@ +import React, { type PropsWithChildren, useCallback, useMemo } from 'react'; +import { Pressable, type PressableProps, Text, View } from 'react-native'; + +import { parseJsonChart } from '../../charts'; +import { parseMermaid } from '../../charts'; +import Chart from '../../charts/Chart'; +import { MarkdownReactiveScrollView } from '../../components'; +import { useStableCallback } from '../../internal/hooks/useStableCallback'; +import { setClipboardString } from '../../services'; +import { SyntaxHighlighter } from '../../syntax-highlighting'; +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const CodeBlockCopyButton = ({ onPress }: { onPress?: PressableProps['onPress'] }) => ( + ({ opacity: pressed ? 0.5 : 1 })} onPress={onPress}> + {'\u29C9'} + +); + +const areEqual = (prev: MarkdownComponentProps, next: MarkdownComponentProps) => + prev.node.content === next.node.content; + +export const CodeBlock = React.memo(({ styles, node }: MarkdownComponentProps) => { + const text = useMemo(() => node.content?.trim(), [node.content]); + + const copyText = useStableCallback(() => { + if (setClipboardString) { + setClipboardString(text); + } + }); + + const CodeTag = useCallback( + ({ children }: PropsWithChildren) => ( + + {children} + + ), + [styles], + ); + + const CodeBlockHeader = useCallback( + () => ( + + {node.lang} + + + ), + [styles, node.lang, copyText], + ); + + const CodeBlockWrapper = useCallback( + ({ children }: PropsWithChildren) => ( + + + {children} + + ), + [CodeBlockHeader, styles.codeBlockWrapper], + ); + + if (node.lang === 'mermaid') { + try { + const parsed = parseMermaid(text); + if (parsed) { + return ; + } + } catch (_e) { + /* do nothing */ + } + } + + if (node.lang === 'json') { + try { + const parsed = parseJsonChart(text); + if (parsed) { + return ; + } + } catch (_e) { + /* do nothing */ + } + } + + return ( + + {text} + + ); +}, areEqual); + +export const renderCodeBlock: RuleRenderFunction = ({ node, output, state, styles }) => ( + +); diff --git a/package/src/ai-components/markdown/components/Emphasis.tsx b/package/src/ai-components/markdown/components/Emphasis.tsx new file mode 100644 index 0000000000..f5f883bae9 --- /dev/null +++ b/package/src/ai-components/markdown/components/Emphasis.tsx @@ -0,0 +1,16 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const Emphasis = ({ children, styles }: MarkdownComponentProps) => ( + {children} +); + +export const renderEmphasis: RuleRenderFunction = ({ node, output, state, styles }) => { + state.withinText = true; + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/Heading.tsx b/package/src/ai-components/markdown/components/Heading.tsx new file mode 100644 index 0000000000..b85a35883d --- /dev/null +++ b/package/src/ai-components/markdown/components/Heading.tsx @@ -0,0 +1,30 @@ +import { useMemo } from 'react'; +import { Text } from 'react-native'; + +import type { HeadingLevel, MarkdownComponentProps, RuleRenderFunction } from '../types'; + +const DEFAULT_HEADING_LEVEL = undefined; + +const isValidHeadingLevel = (n: unknown): n is HeadingLevel => + (Number.isInteger(n) && (n as number) >= 1 && (n as number) <= 6) || n === undefined; + +export const Heading = ({ children, node, styles }: MarkdownComponentProps) => { + const headingLevel = isValidHeadingLevel(node.level) ? node.level : DEFAULT_HEADING_LEVEL; + + const headingStyle = useMemo( + () => styles[`heading${headingLevel ?? ''}`], + [headingLevel, styles], + ); + + return {children}; +}; + +export const renderHeading: RuleRenderFunction = ({ node, output, state, styles }) => { + state.withinText = true; + state.withinHeading = true; + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/HorizontalRule.tsx b/package/src/ai-components/markdown/components/HorizontalRule.tsx new file mode 100644 index 0000000000..7090c1fc87 --- /dev/null +++ b/package/src/ai-components/markdown/components/HorizontalRule.tsx @@ -0,0 +1,9 @@ +import { View } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const HorizontalRule = ({ styles }: MarkdownComponentProps) => ; + +export const renderHorizontalRule: RuleRenderFunction = ({ node, output, state, styles }) => ( + +); diff --git a/package/src/ai-components/markdown/components/InlineCode.tsx b/package/src/ai-components/markdown/components/InlineCode.tsx new file mode 100644 index 0000000000..8663c8c3d0 --- /dev/null +++ b/package/src/ai-components/markdown/components/InlineCode.tsx @@ -0,0 +1,16 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const InlineCode = ({ children, styles }: MarkdownComponentProps) => ( + {children} +); + +export const renderInlineCode: RuleRenderFunction = ({ node, output, state, styles }) => { + state.withinText = true; + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/LineBreak.tsx b/package/src/ai-components/markdown/components/LineBreak.tsx new file mode 100644 index 0000000000..24c89ed392 --- /dev/null +++ b/package/src/ai-components/markdown/components/LineBreak.tsx @@ -0,0 +1,11 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const LineBreak = ({ styles }: MarkdownComponentProps) => ( + {'\n\n'} +); + +export const renderLineBreak: RuleRenderFunction = ({ node, output, state, styles }) => ( + +); diff --git a/package/src/ai-components/markdown/components/Link.tsx b/package/src/ai-components/markdown/components/Link.tsx new file mode 100644 index 0000000000..fc7b17883f --- /dev/null +++ b/package/src/ai-components/markdown/components/Link.tsx @@ -0,0 +1,28 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const Link = ({ children, styles, onPress }: MarkdownComponentProps) => ( + + {children} + +); + +export const renderLink: RuleRenderFunction = ({ node, output, state, styles, onLink }) => { + const url = state.parentLink ?? node.target; + state.withinLink = true; + state.parentLink = url; + const onLinkPress = () => onLink && onLink(url); + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/List.tsx b/package/src/ai-components/markdown/components/List.tsx new file mode 100644 index 0000000000..ae2644914d --- /dev/null +++ b/package/src/ai-components/markdown/components/List.tsx @@ -0,0 +1,65 @@ +import type { PropsWithChildren } from 'react'; +import { Text, type TextProps, View, type ViewProps } from 'react-native'; + +import type { SingleASTNode } from '@khanacademy/simple-markdown'; + +import type { BulletProps, MarkdownComponentProps, RuleRenderFunction } from '../types'; + +const parentTypes = ['text', 'paragraph', 'strong']; + +export const List = ({ node, output, state, styles }: MarkdownComponentProps) => { + let isSublist = state.withinList; + + return ( + + {node.items.map((item: SingleASTNode, index: number) => { + const indexAfterStart = node.start + index; + + if (item === null) { + return ( + + + + ); + } + + isSublist = item.length > 1 && item[1].type === 'list'; + const isSublistWithinText = parentTypes.includes((item[0] ?? {}).type) && isSublist; + const style = isSublistWithinText ? { marginBottom: 0 } : {}; + + return ( + + + + {output(item, { ...state, withinList: true })} + + + ); + })} + + ); +}; + +const Bullet = ({ index, style }: BulletProps) => ( + + {index ? `${index}. ` : '\u2022 '} + +); + +const ListRow = ({ children, style }: PropsWithChildren) => ( + {children} +); + +const ListItem = ({ children, style }: PropsWithChildren) => ( + {children} +); + +export const renderList: RuleRenderFunction = ({ node, output, state, styles }) => ( + +); diff --git a/package/src/ai-components/markdown/components/Mailto.tsx b/package/src/ai-components/markdown/components/Mailto.tsx new file mode 100644 index 0000000000..74dad85d1f --- /dev/null +++ b/package/src/ai-components/markdown/components/Mailto.tsx @@ -0,0 +1,26 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const Mailto = ({ children, styles, onPress }: MarkdownComponentProps) => ( + + {children} + +); + +export const renderMailto: RuleRenderFunction = ({ node, output, state, styles, onLink }) => { + state.withinLink = true; + const onLinkPress = () => onLink && onLink(node.target); + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/Markdown.tsx b/package/src/ai-components/markdown/components/Markdown.tsx new file mode 100644 index 0000000000..f3d2c6a76f --- /dev/null +++ b/package/src/ai-components/markdown/components/Markdown.tsx @@ -0,0 +1,59 @@ +import React, { type PropsWithChildren, useMemo } from 'react'; + +import { View } from 'react-native'; + +import SimpleMarkdown, { + type OutputRules, + type ParserRules, + type ReactOutputRule, +} from '@khanacademy/simple-markdown'; +import isArray from 'lodash/isArray'; +import isEqual from 'lodash/isEqual'; +import merge from 'lodash/merge'; + +import { useTheme } from '../../contexts'; +import { getLocalRules } from '../rules'; + +import { getLocalStyles } from '../styles'; +import type { MarkdownProps } from '../types'; + +const UnmemoizedMarkdown = (props: PropsWithChildren) => { + const { theme } = useTheme(); + + const { onLink, rules: rulesProp, paragraphNumberOfLines, children } = props; + + const mergedStyles = useMemo(() => merge({}, getLocalStyles({ theme }), theme.markdown), [theme]); + + const localRules = useMemo( + () => + merge( + {}, + SimpleMarkdown.defaultRules, + getLocalRules(mergedStyles, { onLink, paragraphNumberOfLines }), + rulesProp, + ) as unknown as ParserRules, + [mergedStyles, onLink, rulesProp, paragraphNumberOfLines], + ); + + const parser = useMemo(() => SimpleMarkdown.parserFor(localRules), [localRules]); + const renderer = useMemo( + () => SimpleMarkdown.outputFor(localRules as unknown as OutputRules, 'react'), + [localRules], + ); + + const childText = useMemo(() => (isArray(children) ? children.join('') : children), [children]); + + const toRender = useMemo(() => { + const blockSource = `${childText ?? ''}\n\n`; + return parser(blockSource, { inline: false }); + }, [childText, parser]); + + const tree = useMemo(() => renderer(toRender), [renderer, toRender]); + + return {tree}; +}; + +const areEqual = (prevProps: PropsWithChildren, nextProps: PropsWithChildren) => + isEqual(prevProps.children, nextProps.children); + +export const Markdown = React.memo(UnmemoizedMarkdown, areEqual); diff --git a/package/src/ai-components/markdown/components/NewLine.tsx b/package/src/ai-components/markdown/components/NewLine.tsx new file mode 100644 index 0000000000..9b3bbb0352 --- /dev/null +++ b/package/src/ai-components/markdown/components/NewLine.tsx @@ -0,0 +1,11 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const NewLine = ({ styles }: MarkdownComponentProps) => ( + {'\n'} +); + +export const renderNewLine: RuleRenderFunction = ({ node, output, state, styles }) => ( + +); diff --git a/package/src/ai-components/markdown/components/Paragraph.tsx b/package/src/ai-components/markdown/components/Paragraph.tsx new file mode 100644 index 0000000000..92bd796354 --- /dev/null +++ b/package/src/ai-components/markdown/components/Paragraph.tsx @@ -0,0 +1,69 @@ +import { useMemo } from 'react'; +import { Text, type TextStyle, View } from 'react-native'; + +import size from 'lodash/size'; +import some from 'lodash/some'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const Paragraph = ({ + children, + styles, + node, + state, + paragraphNumberOfLines, +}: MarkdownComponentProps) => { + const paragraphStyle = useMemo(() => { + let style: TextStyle | (TextStyle | undefined)[] | undefined = + size(node.content) < 3 && some(node.content, { type: 'strong' }) + ? styles.paragraphCenter + : styles.paragraph; + + if (state.withinList) { + style = [style, styles.noMargin]; + } + + return style; + }, [node.content, state.withinList, styles.paragraph, styles.paragraphCenter, styles.noMargin]); + + return ( + + {children} + + ); +}; + +export const ParagraphWithImage = ({ children, styles }: MarkdownComponentProps) => ( + {children} +); + +export const renderParagraph: RuleRenderFunction = ({ + node, + output, + state, + styles, + paragraphNumberOfLines, +}) => { + if (some(node.content, { type: 'image' })) { + state.withinParagraphWithImage = true; + + return ( + + {output(node.content, state)} + + ); + } + + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/Strikethrough.tsx b/package/src/ai-components/markdown/components/Strikethrough.tsx new file mode 100644 index 0000000000..2aa451b963 --- /dev/null +++ b/package/src/ai-components/markdown/components/Strikethrough.tsx @@ -0,0 +1,16 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const Strikethrough = ({ children, styles }: MarkdownComponentProps) => ( + {children} +); + +export const renderStrikethrough: RuleRenderFunction = ({ node, output, state, styles }) => { + state.withinText = true; + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/Table.tsx b/package/src/ai-components/markdown/components/Table.tsx new file mode 100644 index 0000000000..def5c82a62 --- /dev/null +++ b/package/src/ai-components/markdown/components/Table.tsx @@ -0,0 +1,81 @@ +import { useCallback, useMemo } from 'react'; +import { Text, View } from 'react-native'; + +import type { SingleASTNode } from '@khanacademy/simple-markdown'; + +import { MarkdownReactiveScrollView } from '../../components'; +import type { + MarkdownComponentProps, + MarkdownOutputProps, + MarkdownTableRowProps, + RuleRenderFunction, +} from '../types'; + +export const Table = ({ node, output, styles, state }: MarkdownComponentProps) => ( + + + +); + +const transpose = (matrix: SingleASTNode[][]) => + // TS gets confused because it considers the matrix to be potentially ragged, + // while the markdown parser can never output a ragged matrix of AST nodes + // when parsing a table. Hence, we use the forced type coercion. + (matrix[0] ?? []).map((_, colIndex) => matrix.map((row) => row[colIndex]!)); + +const MarkdownTable = ({ node, output, state, styles }: MarkdownOutputProps) => { + const content = useMemo(() => { + const nodeContent = [node?.header, ...(node?.cells ?? [])]; + return transpose(nodeContent); + }, [node?.cells, node?.header]); + const columns = content?.map((column, idx) => ( + + )); + + return ( + + {columns} + + ); +}; + +const MarkdownTableColumn = ({ items, output, state, styles }: MarkdownTableRowProps) => { + const [headerCellContent, ...columnCellContents] = items; + + const ColumnCell = useCallback( + ({ content }: { content: SingleASTNode }) => + content ? ( + + {output(content, state)} + + ) : null, + [output, state, styles], + ); + + return ( + + {headerCellContent ? ( + + {output(headerCellContent, state)} + + ) : null} + {columnCellContents && + columnCellContents.map((content, idx) => ( + + ))} + + ); +}; + +export const renderTable: RuleRenderFunction = ({ node, output, state, styles }) => ( + + {node.content?.trim()} +
+); diff --git a/package/src/ai-components/markdown/components/Text.tsx b/package/src/ai-components/markdown/components/Text.tsx new file mode 100644 index 0000000000..b6287e02fa --- /dev/null +++ b/package/src/ai-components/markdown/components/Text.tsx @@ -0,0 +1,37 @@ +import { useMemo } from 'react'; +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const TextUI = ({ children, styles, state }: MarkdownComponentProps) => { + const textStyle = useMemo(() => { + const style = [styles.text, state.style]; + + if (state.withinQuote) { + style.push(styles.blockQuoteText); + } + + if (state.withinLink) { + style.push(styles.autolink); + } + + return style; + }, [ + state.style, + state.withinLink, + state.withinQuote, + styles.text, + styles.autolink, + styles.blockQuoteText, + ]); + + return {children}; +}; + +export const renderText: RuleRenderFunction = ({ node, output, state, styles }) => { + return ( + + {node.content} + + ); +}; diff --git a/package/src/ai-components/markdown/components/Url.tsx b/package/src/ai-components/markdown/components/Url.tsx new file mode 100644 index 0000000000..feb9669b22 --- /dev/null +++ b/package/src/ai-components/markdown/components/Url.tsx @@ -0,0 +1,26 @@ +import { Text } from 'react-native'; + +import type { MarkdownComponentProps, RuleRenderFunction } from '../types'; + +export const Url = ({ children, styles, onPress }: MarkdownComponentProps) => ( + + {children} + +); + +export const renderUrl: RuleRenderFunction = ({ node, output, state, styles, onLink }) => { + state.withinLink = true; + const onLinkPress = () => onLink && onLink(node.target); + return ( + + {output(node.content, state)} + + ); +}; diff --git a/package/src/ai-components/markdown/components/index.ts b/package/src/ai-components/markdown/components/index.ts new file mode 100644 index 0000000000..7553ce4a17 --- /dev/null +++ b/package/src/ai-components/markdown/components/index.ts @@ -0,0 +1,20 @@ +export * from './Autolink'; +export * from './BlockQuote'; +export * from './Bold'; +export * from './CodeBlock'; +export * from './Emphasis'; +export * from './Heading'; +export * from './HorizontalRule'; +export * from './InlineCode'; +export * from './LineBreak'; +export * from './Link'; +export * from './List'; +export * from './Mailto'; +export * from './NewLine'; +export * from './Paragraph'; +export * from './Strikethrough'; +export * from './Table'; +export * from './Text'; +export * from './Url'; + +export * from './Markdown'; diff --git a/package/src/ai-components/markdown/index.tsx b/package/src/ai-components/markdown/index.tsx new file mode 100644 index 0000000000..1ca12ecdf1 --- /dev/null +++ b/package/src/ai-components/markdown/index.tsx @@ -0,0 +1,5 @@ +export * from './rules'; +export * from './styles'; +export * from './components'; +export * from './utils'; +export * from './types'; diff --git a/package/src/ai-components/markdown/rules.ts b/package/src/ai-components/markdown/rules.ts new file mode 100644 index 0000000000..6e39f7c85b --- /dev/null +++ b/package/src/ai-components/markdown/rules.ts @@ -0,0 +1,216 @@ +import type React from 'react'; + +import type { + MatchFunction, + Output, + OutputRules, + ParseFunction, + Parser, + ReactOutputRule, + SingleASTNode, +} from '@khanacademy/simple-markdown'; +import SimpleMarkdown from '@khanacademy/simple-markdown'; + +import { + renderAutolink, + renderBlockQuote, + renderBold, + renderCodeBlock, + renderEmphasis, + renderHeading, + renderHorizontalRule, + renderInlineCode, + renderLineBreak, + renderLink, + renderList, + renderMailto, + renderNewLine, + renderParagraph, + renderStrikethrough, + renderTable, + renderText, + renderUrl, +} from './components'; +import type { + MarkdownOptions, + MarkdownState, + MarkdownStyle, + RuleRenderFunction, + RuleRenderFunctionEnrichedProps, +} from './types'; + +const LINK_INSIDE = '(?:\\[(?:\\\\.|[^\\\\\\[\\]])*\\]|\\\\.|[^\\[\\]\\\\])*'; +/** + * Href + optional title (escape-aware). Group 1 === href (angle or bare). + * The title is in groups 2, 3 or 4. + */ +const LINK_HREF_AND_TITLE = + '\\s*]*?)>?\\s*' + // 1: href + '(?:' + + '\\\\"((?:\\\\.|[^"\\\\])*)\\\\"' + // 2: "title" + "|\\\\'((?:\\\\.|[^'\\\\])*)\\\\'" + // 3: 'title' + '|\\(((?:\\\\.|[^)\\\\])*)\\)' + // 4: (title) + ')?\\s*'; + +const LINK_REGEX = new RegExp('^\\[(' + LINK_INSIDE + ')\\]\\(' + LINK_HREF_AND_TITLE + '\\)'); + +export const getLocalRules = ( + styles: MarkdownStyle, + opts: MarkdownOptions = {}, +): OutputRules => { + const { onLink, paragraphNumberOfLines } = opts; + const openLinkHandler = (target: string) => { + if (onLink) { + // user-supplied handler may be async; we keep the behavior + Promise.resolve(onLink(target)).catch((error: unknown) => { + const msg = + error && typeof error === 'object' && 'toString' in error + ? String(error) + : 'Unknown error'; + + console.log('There has been a problem with this action. ' + msg); + throw error; + }); + } + }; + + const parseInline = function ( + parse: Parser, + content: string, + state: MarkdownState, + ): SingleASTNode[] { + const isCurrentlyInline = state.inline || false; + state.inline = true; + const result = parse(content, state); + state.inline = isCurrentlyInline; + return result; + }; + + const parseCaptureInline: ParseFunction = (capture, parse, state) => { + return { + content: parseInline(parse, capture[2] as string, state as MarkdownState), + }; + }; + + const parseFence: ParseFunction = (capture) => { + // capture[2] = language string (may be undefined/empty) + // capture[3] = code content + let lang = (capture[2] || '').trim() || undefined; + let content = capture[3] || ''; + + // If we've found no lang on the fence line (the opening fence) but the first code line + // is a lone token, treat it as lang and strip it from content. Helps with some specific + // use-cases such as "```sql...". + if (!lang) { + const nl = content.indexOf('\n'); + const firstLine = (nl === -1 ? content : content.slice(0, nl)).trim(); + if (/^[A-Za-z0-9_+.-]+$/.test(firstLine)) { + lang = firstLine; + content = nl === -1 ? '' : content.slice(nl + 1); + } + } + + // Mirror your trimming behavior + content = content.replace(/\n+$/, ''); + + return { type: 'codeBlock', lang, content }; + }; + + const enrichedRenderFunction = + (render: RuleRenderFunction, options: Partial = {}) => + (node: SingleASTNode, output: Output, { ...state }: MarkdownState) => + render({ node, output, state, styles, ...options }); + + return { + autolink: { + react: enrichedRenderFunction(renderAutolink, { + onLink: openLinkHandler, + }), + }, + blockQuote: { + react: enrichedRenderFunction(renderBlockQuote), + }, + br: { + react: enrichedRenderFunction(renderLineBreak), + }, + codeBlock: { + react: enrichedRenderFunction(renderCodeBlock), + }, + fence: { + match: SimpleMarkdown.blockRegex( + // 1: fence (``` or ~~~) + // 2: info string (lang etc.) - SAME CAPTURE INDEX AS YOURS + // 3: code content + /^ {0,3}(`{3,}|~{3,})[ \t]*(\S+)?[ \t]*\r?\n([\s\S]*?)\r?\n?(?: {0,3})\1[ \t]*(?:\r?\n+|$)/, + ), + parse: parseFence, + }, + del: { + react: enrichedRenderFunction(renderStrikethrough), + }, + em: { + react: enrichedRenderFunction(renderEmphasis), + }, + heading: { + match: SimpleMarkdown.blockRegex(/^ *(#{1,6}) *([^\n]+?) *#* *(?:\n *)+/), + react: enrichedRenderFunction(renderHeading), + }, + hr: { + react: enrichedRenderFunction(renderHorizontalRule), + }, + image: { + // We intentionally disable parsing images; keep the shape + match: () => null, + }, + inlineCode: { + parse: parseCaptureInline, + react: enrichedRenderFunction(renderInlineCode), + }, + link: { + match: SimpleMarkdown.inlineRegex(LINK_REGEX) as MatchFunction, + react: enrichedRenderFunction(renderLink, { + onLink: openLinkHandler, + }), + }, + list: { + react: enrichedRenderFunction(renderList), + }, + mailto: { + react: enrichedRenderFunction(renderMailto, { + onLink: openLinkHandler, + }), + }, + newline: { + react: enrichedRenderFunction(renderNewLine), + }, + paragraph: { + react: enrichedRenderFunction(renderParagraph, { + paragraphNumberOfLines, + }), + }, + strong: { + react: enrichedRenderFunction(renderBold), + }, + sublist: { + react: enrichedRenderFunction(renderList), + }, + table: { + react: enrichedRenderFunction(renderTable), + }, + text: { + react: enrichedRenderFunction(renderText), + }, + u: { + // no support for underlines yet + match: () => null, + }, + url: { + react: enrichedRenderFunction(renderUrl, { + onLink: openLinkHandler, + }), + }, + // no support for reflinks + reflink: { match: () => null }, + refimage: { match: () => null }, + } as unknown as OutputRules; +}; diff --git a/package/src/ai-components/markdown/styles.ts b/package/src/ai-components/markdown/styles.ts new file mode 100644 index 0000000000..d3517f36f4 --- /dev/null +++ b/package/src/ai-components/markdown/styles.ts @@ -0,0 +1,236 @@ +import { Dimensions, Platform, StyleSheet } from 'react-native'; + +import type { Theme } from '../theme'; + +export const getLocalStyles = ({ theme }: { theme: Theme }) => { + const colors = theme.colors; + + return StyleSheet.create({ + autolink: { + color: colors.accent_blue, + }, + bgImage: { + bottom: 0, + flex: 1, + left: 0, + position: 'absolute', + right: 0, + top: 0, + }, + bgImageView: { + flex: 1, + overflow: 'hidden', + }, + blockQuoteSection: { + flexDirection: 'row', + padding: 8, + }, + blockQuoteSectionBar: { + backgroundColor: colors.grey_gainsboro, + height: null, + marginRight: 8, + width: 2, + }, + blockQuoteText: { + color: colors.grey_dark, + }, + codeBlock: { + backgroundColor: colors.code_block, + fontFamily: Platform.OS === 'ios' ? 'Courier' : 'Monospace', + fontWeight: '500', + color: colors.black, + paddingLeft: 8, + lineHeight: 14, + }, + codeBlockWrapper: { + backgroundColor: colors.code_block, + borderRadius: 8, + marginVertical: 8, + padding: 12, + }, + codeBlockContainer: { + flexDirection: 'row', + }, + codeBlockLineNumberGutter: { + flexDirection: 'column', + color: colors.black, + }, + codeBlockLineNumberCell: { + fontFamily: Platform.OS === 'ios' ? 'Courier' : 'Monospace', + color: colors.grey, + paddingVertical: 1, // should be (codeBlock.lineHeight - this.fontSize) / 2 + fontSize: 12, + }, + codeBlockHeaderContainer: { + flexDirection: 'row', + justifyContent: 'space-between', + marginBottom: 8, + }, + codeBlockHeaderTitle: { + fontSize: 12, + color: colors.grey, + fontFamily: Platform.OS === 'ios' ? 'Courier' : 'Monospace', + }, + del: { + textDecorationLine: 'line-through', + textDecorationStyle: 'solid', + }, + em: { + fontStyle: 'italic', + }, + heading: { + fontWeight: '200', + }, + heading1: { + fontSize: 32, + }, + heading2: { + fontSize: 24, + }, + heading3: { + fontSize: 18, + }, + heading4: { + fontSize: 16, + }, + heading5: { + fontSize: 13, + }, + heading6: { + fontSize: 11, + }, + hr: { + backgroundColor: colors.grey_gainsboro, + height: 1, + }, + image: { + alignSelf: 'center', + height: 200, + resizeMode: 'contain', + width: Dimensions.get('window').width - 30, + }, + imageBox: { + flex: 1, + resizeMode: 'cover', + }, + inlineCode: { + backgroundColor: colors.white_smoke, + borderColor: colors.grey_gainsboro, + borderRadius: 3, + borderWidth: 1, + color: colors.accent_red, + fontFamily: Platform.OS === 'ios' ? 'Courier' : 'Monospace', + fontWeight: '500', + fontSize: 13, + padding: 3, + paddingHorizontal: 5, + }, + list: { + marginBottom: 8, + marginTop: 8, + }, + listItem: { + flexDirection: 'row', + }, + listItemBullet: { + fontSize: 20, + lineHeight: 20, + }, + listItemNumber: { + fontWeight: '500', + }, + listItemText: { + flex: 0, + }, + listRow: { + flexDirection: 'row', + }, + mentions: { + color: colors.accent_blue, + fontWeight: '700', + }, + noMargin: { + marginBottom: 0, + marginTop: 0, + }, + paragraph: { + alignItems: 'flex-start', + flexDirection: 'row', + flexWrap: 'wrap', + justifyContent: 'flex-start', + marginBottom: 8, + marginTop: 8, + }, + paragraphCenter: { + alignItems: 'flex-start', + flexDirection: 'row', + flexWrap: 'wrap', + justifyContent: 'center', + marginBottom: 8, + marginTop: 8, + textAlign: 'center', + }, + paragraphWithImage: { + alignItems: 'flex-start', + flex: 1, + justifyContent: 'flex-start', + marginBottom: 8, + marginTop: 8, + }, + strong: { + fontWeight: 'bold', + }, + sublist: { + paddingLeft: 10, + width: Dimensions.get('window').width - 60, + }, + table: { + borderColor: colors.grey_dark, + borderRadius: 3, + borderWidth: 1, + flex: 1, + flexDirection: 'row', + marginVertical: 8, + }, + tableHeader: { + backgroundColor: colors.grey, + flexDirection: 'row', + justifyContent: 'space-around', + }, + tableHeaderCell: { + color: colors.white, + fontWeight: '500', + padding: 5, + }, + tableRow: { + alignItems: 'center', + borderColor: colors.grey_dark, + flexDirection: 'row', + justifyContent: 'space-around', + }, + tableRowCell: { + borderColor: colors.grey_dark, + flex: 1, + padding: 5, + }, + tableRowLast: { + borderColor: 'transparent', + }, + text: { + color: colors.black, + }, + textRow: { + flexDirection: 'row', + }, + u: { + borderBottomWidth: 1, + borderColor: colors.grey_dark, + }, + url: { + color: colors.accent_blue, + }, + view: { + alignSelf: 'stretch', + }, + }); +}; diff --git a/package/src/ai-components/markdown/types.ts b/package/src/ai-components/markdown/types.ts new file mode 100644 index 0000000000..276fa43fc8 --- /dev/null +++ b/package/src/ai-components/markdown/types.ts @@ -0,0 +1,163 @@ +import type { JSX } from 'react'; +import type React from 'react'; +import { type PropsWithChildren } from 'react'; +import type { + GestureResponderEvent, + ImageStyle, + TextProps, + TextStyle, + ViewStyle, +} from 'react-native'; + +import type SimpleMarkdown from '@khanacademy/simple-markdown'; +import type { Output } from '@khanacademy/simple-markdown'; +import type { + ParserRule, + ReactOutputRule, + SingleASTNode, + State, +} from '@khanacademy/simple-markdown'; + +import type { DeepPartial } from '../contexts'; + +export type LinkInfo = { + raw: string; + url: string; +}; + +export type ReactOutput = Output; + +export type DefaultRules = typeof SimpleMarkdown.defaultRules & { + [key: string]: ParserRule & ReactOutputRule; +}; + +export type MarkdownRules = DeepPartial; + +export type MarkdownProps = { + onLink?: (url: string) => void | Promise; + rules?: MarkdownRules; + styles?: MarkdownStyle; + paragraphNumberOfLines?: number; +}; + +export type MarkdownOptions = Partial>; + +export type MarkdownStyleProp = TextStyle | ViewStyle; + +export type MarkdownState = State & { + withinText?: boolean; + withinQuote?: boolean; + withinHeading?: boolean; + withinLink?: boolean; + withinList?: boolean; + withinParagraphWithImage?: boolean; + parentLink?: string; + style: MarkdownStyleProp; +}; + +export type NodeWithContent = SingleASTNode & { content: SingleASTNode[] }; +export type NodeWithStringContent = SingleASTNode & { content: string }; +export type HeadingNode = SingleASTNode & { + level: number; + content: SingleASTNode[]; +}; +export type ListNode = SingleASTNode & { + ordered: boolean; + items: SingleASTNode[] | SingleASTNode[][]; +}; +export type TableNode = SingleASTNode & { + header: SingleASTNode[]; + cells: SingleASTNode[][]; +}; +export type TargetNode = SingleASTNode & { target: string }; + +// Allow dynamic heading style access like styles["heading1"] +export type HeadingLevel = 1 | 2 | 3 | 4 | 5 | 6 | undefined; + +export type MarkdownStyle = Partial<{ + autolink: TextStyle; + blockQuoteBar: ViewStyle; + blockQuoteSection: ViewStyle; + blockQuoteSectionBar: ViewStyle; + blockQuoteText: TextStyle | ViewStyle; + br: TextStyle; + codeBlock: TextStyle; + codeBlockLineNumberGutter: ViewStyle; + codeBlockContainer: ViewStyle; + codeBlockWrapper: ViewStyle; + codeBlockLineNumberCell: TextStyle; + codeBlockHeaderContainer: ViewStyle; + codeBlockHeaderTitle: TextStyle; + del: TextStyle; + em: TextStyle; + heading: TextStyle; + heading1: TextStyle; + heading2: TextStyle; + heading3: TextStyle; + heading4: TextStyle; + heading5: TextStyle; + heading6: TextStyle; + hr: ViewStyle; + image: ImageStyle; + inlineCode: TextStyle; + list: ViewStyle; + listItem: ViewStyle; + listItemBullet: TextStyle; + listItemNumber: TextStyle; + listItemText: TextStyle; + listRow: ViewStyle; + mailTo: TextStyle; + mentions: TextStyle; + newline: TextStyle; + noMargin: TextStyle; + paragraph: TextStyle; + paragraphCenter: TextStyle; + paragraphWithImage: ViewStyle; + strong: TextStyle; + sublist: ViewStyle; + table: ViewStyle; + tableHeader: ViewStyle; + tableHeaderCell: TextStyle; + tableRow: ViewStyle; + tableRowCell: TextStyle; + tableRowLast: ViewStyle; + text: TextStyle; + u: TextStyle; + url: TextStyle; + view: ViewStyle; +}>; + +export interface MarkdownOutputProps { + node: SingleASTNode; + output: ReactOutput; + state: State; + styles: Partial; +} + +export type MarkdownComponentProps = PropsWithChildren; + +export interface MarkdownTableRowProps extends MarkdownOutputProps { + items: SingleASTNode[]; +} + +export interface BulletProps extends TextProps { + index?: number; +} + +export type RuleOutputProps = { + node: SingleASTNode; + output: ReactOutput; + state: MarkdownState; +}; + +export type RuleRenderFunctionEnrichedProps = { + styles: MarkdownStyle; + onPress?: (event: GestureResponderEvent) => void; + onLongPress?: (event: GestureResponderEvent) => void; + onLink?: (url: string) => void; + paragraphNumberOfLines?: number; +}; + +export type RuleRenderFunctionProps = RuleOutputProps & RuleRenderFunctionEnrichedProps; + +export type RuleRenderFunction = (props: RuleRenderFunctionProps) => JSX.Element; diff --git a/package/src/ai-components/markdown/utils/generate-markdown-text.ts b/package/src/ai-components/markdown/utils/generate-markdown-text.ts new file mode 100644 index 0000000000..28944b57b4 --- /dev/null +++ b/package/src/ai-components/markdown/utils/generate-markdown-text.ts @@ -0,0 +1,117 @@ +import { find } from 'linkifyjs'; +import truncate from 'lodash/truncate'; + +import type { LinkInfo } from '../index'; + +export function escapeRegExp(text: string) { + return text.replace(/[-[\]{}()*+?.,/\\^$|#]/g, '\\$&'); +} + +/** + * This is done to remove all markdown formatted links. + * eg: [google.com](https://www.google.com), [Google](https://www.google.com), [https://www.google.com](https://www.google.com) + * */ +const removeMarkdownLinksFromText = (input: string) => input.replace(/\[.*\]\(.*\)/g, ''); + +/** + * This is done to avoid parsing usernames with dot as well as an email address in it. + */ +const removeUserNamesWithEmailFromText = (input: string) => + input.replace(/@(\w+(\.\w+)?)(@\w+\.\w+)/g, ''); + +export const parseLinksFromText = (input?: string): LinkInfo[] => { + if (!input) { + return []; + } + const strippedInput = [removeMarkdownLinksFromText, removeUserNamesWithEmailFromText].reduce( + (acc, fn) => fn(acc), + input, + ); + + const links = find(strippedInput, 'url'); + const emails = find(strippedInput, 'email'); + + const result: LinkInfo[] = [...links, ...emails].map(({ href, value }) => { + let hrefWithProtocol = href; + // Matching these: https://reactnative.dev/docs/0.73/linking?syntax=ios#built-in-url-schemes + const pattern = new RegExp(/^(mailto:|tel:|sms:|\S+:\/\/)/); + if (!pattern.test(hrefWithProtocol)) { + hrefWithProtocol = 'http://' + hrefWithProtocol; + } + + return { + raw: value, + url: hrefWithProtocol, + }; + }); + + return result; +}; + +export const generateMarkdownText = (text?: string) => { + if (!text) { + return null; + } + + // Trim the extra spaces from the text. + let resultText = text; + + // List of all the links present in the text. + const linkInfos = parseLinksFromText(resultText); + + for (const linkInfo of linkInfos) { + const displayLink = truncate(linkInfo.raw, { + length: 200, + omission: '...', + }); + // Convert raw links/emails in the text to respective markdown syntax. + // Eg: Hi @getstream.io -> Hi @[getstream.io](getstream.io). + const normalRegEx = new RegExp(escapeRegExp(linkInfo.raw), 'g'); + const markdown = `[${displayLink}](${linkInfo.url})`; + resultText = resultText.replace(normalRegEx, markdown); + + // After previous step, in some cases, the mentioned user after `@` might have a link/email so we convert it back to normal raw text. + // Eg: Hi, @[test.user@gmail.com](mailto:test.user@gmail.com) to @test.user@gmail.com. + const mentionsRegex = new RegExp( + `@\\[${escapeRegExp(displayLink)}\\]\\(${escapeRegExp(linkInfo.url)}\\)`, + 'g', + ); + resultText = resultText.replace(mentionsRegex, `@${displayLink}`); + } + + // Remove whitespaces that come directly after newlines except in code blocks where we deem this allowed. + // A line starts a (possibly quoted) list item if, after indentation, it has + // - unordered bullet: -, *, + + // - or ordered marker: 1. 2. 10. + // and at least one space after the marker. + const LIST_START_RE = /^[ \t]*(?:> ?)*?(?:[*+-]|\d+\.)(?: |\t)(?:\[(?: |x|X)\] )?/; + + resultText = resultText.replace( + // 1) fenced code blocks (``` ```) + // 2) inline code (`โ€ฆ`) + // 3) newline + 2 + spaces/tabs + /(```[\s\S]*?```|`[^`]*?`)|\n([ \t]{2,})/g, + (match, code, spaces, offset, full) => { + // Preserve code spans / fences exactly as they are + if (code) return code; + + // Peek what follows this newline + the indent on the same line + const after = full.slice(offset + match.length); + + // If that line begins with a list item (any nesting depth, optional > quotes), + // keep the indentation; otherwise collapse to a single newline. + return LIST_START_RE.test(after) ? `\n${spaces}` : '\n'; + }, + ); + + // Always replace \n``` with \n\n``` to force the markdown state machine to treat it as a separate block. Otherwise, code blocks inside of list + // items for example were broken. We clean up the code block closing state within the rendering itself. + resultText = resultText.replace(/\n```/g, '\n\n```\n'); + + const codeBlockCounts = (resultText.match(/```/g) || []).length; + if (codeBlockCounts % 2 > 0) { + resultText += '```'; + } + + return resultText; +}; diff --git a/package/src/ai-components/markdown/utils/index.ts b/package/src/ai-components/markdown/utils/index.ts new file mode 100644 index 0000000000..dce593138b --- /dev/null +++ b/package/src/ai-components/markdown/utils/index.ts @@ -0,0 +1 @@ +export * from './generate-markdown-text'; diff --git a/package/src/ai-components/message-composer/ActionButton.tsx b/package/src/ai-components/message-composer/ActionButton.tsx new file mode 100644 index 0000000000..dce7d571a2 --- /dev/null +++ b/package/src/ai-components/message-composer/ActionButton.tsx @@ -0,0 +1,99 @@ +import React, { useMemo } from 'react'; +import { Pressable, StyleSheet, View } from 'react-native'; +import Animated, { ZoomIn, ZoomOut } from 'react-native-reanimated'; + +import { useStateStore } from '@stream-io/state-store/react-bindings'; + +import type { ComposerViewProps } from './ComposerView'; + +import { SpeechToTextButton } from './SpeechToTextButton'; + +import { useMessageComposerContext, useTheme } from '../contexts'; +import { CircleStop } from '../internal/icons/CircleStop'; +import { SendUp } from '../internal/icons/SendUp'; +import { dictationStore, type DictationStoreState, useComposerHasText } from '../store'; + +const selector = ({ isRecording }: DictationStoreState) => ({ + isRecording, +}); + +export const ActionButton = ({ + isGenerating, + stopGenerating, +}: Pick) => { + const { hasText } = useComposerHasText(); + const { isRecording } = useStateStore(dictationStore, selector); + const { sendMessage } = useMessageComposerContext(); + const { + theme: { + colors: { black, white }, + composer: { iconButton, stopGeneratingIcon, sendIcon }, + }, + } = useTheme(); + const styles = useStyles(); + + if (isGenerating) { + return ( + + + + + + + + ); + } + + return hasText && !isRecording ? ( + + + + + + + + ) : ( + + ); +}; + +const useStyles = () => { + const { theme } = useTheme(); + + return useMemo( + () => + StyleSheet.create({ + micIcon: { + width: 32, + height: 32, + justifyContent: 'center', + alignItems: 'center', + }, + sendIcon: { + width: 32, + height: 32, + backgroundColor: theme.colors.black, + borderRadius: 16, + }, + stopGeneratingIcon: { + width: 32, + height: 32, + backgroundColor: theme.colors.transparent, + borderRadius: 16, + }, + iconButton: { + justifyContent: 'center', + alignItems: 'center', + }, + }), + [theme], + ); +}; diff --git a/package/src/ai-components/message-composer/ActionSheet.tsx b/package/src/ai-components/message-composer/ActionSheet.tsx new file mode 100644 index 0000000000..5b4f999342 --- /dev/null +++ b/package/src/ai-components/message-composer/ActionSheet.tsx @@ -0,0 +1,205 @@ +import React, { useCallback, useMemo } from 'react'; +import { Pressable, ScrollView, StyleSheet, Text, View } from 'react-native'; + +import type { BottomSheetOption, ComposerViewProps } from './ComposerView'; + +import { withCloseSheet } from './utils/withCloseSheet'; + +import { useMessageComposerContext, useTheme } from '../contexts'; +import { Camera } from '../internal/icons/Camera'; +import { Picture } from '../internal/icons/Picture'; +import { sheetStoreApi } from '../store'; + +type BottomSheetContentProps = Pick; + +export const BottomSheetContent = ({ + bottomSheetOptions, + bottomSheetInsets, +}: BottomSheetContentProps) => { + const { + theme: { + sheet: { contentContainer, quickActionsCard, divider, listSection }, + }, + } = useTheme(); + const styles = useStyles(); + + const { mediaPickerService } = useMessageComposerContext(); + + const onContentSizeChange = useCallback( + (_: unknown, height: number) => sheetStoreApi.setHeight(height), + [], + ); + + const insetStyle = useMemo( + () => ({ paddingBottom: 16 + (bottomSheetInsets?.bottom ?? 0) }), + [bottomSheetInsets], + ); + + return ( + + + mediaPickerService?.takeMedia({}))} + /> + mediaPickerService?.pickMedia({}))} + /> + + + {bottomSheetOptions && bottomSheetOptions.length > 0 ? ( + <> + + + {bottomSheetOptions.map((option, index) => ( + + ))} + + + ) : null} + + ); +}; + +type QuickActionButtonProps = { + label: string; + Icon: React.ComponentType; + onPress?: () => void; +}; + +export const QuickActionButton = ({ label, Icon, onPress }: QuickActionButtonProps) => { + const { + theme: { + sheet: { quickAction, quickActionLabel }, + }, + } = useTheme(); + const styles = useStyles(); + + return ( + [styles.quickAction, quickAction, pressed && styles.pressedView]} + > + + {label} + + ); +}; + +export type ListItemProps = { + option: BottomSheetOption; +}; + +export const ListItem = ({ option }: ListItemProps) => { + const { + theme: { + sheet: { listItem, listIcon, listTextContainer, listTitle, listSubtitle }, + }, + } = useTheme(); + const styles = useStyles(); + + const { title, subtitle, action, Icon } = option; + + return ( + [styles.listItem, listItem, pressed && styles.pressedView]} + > + {Icon ? ( + + + + ) : null} + + + {title} + {subtitle} + + + ); +}; + +const useStyles = () => { + const { theme } = useTheme(); + + return useMemo(() => { + const { colors } = theme; + + return StyleSheet.create({ + container: { + flex: 1, + }, + contentContainer: { + paddingHorizontal: 16, + paddingTop: 12, + }, + quickActionsCard: { + flexDirection: 'row', + justifyContent: 'center', + gap: 12, + borderRadius: 18, + marginBottom: 16, + }, + quickAction: { + flex: 1, + alignItems: 'center', + padding: 16, + borderRadius: 14, + marginHorizontal: 4, + backgroundColor: colors.white_smoke, + }, + pressedView: { + opacity: 0.7, + }, + quickActionLabel: { + fontSize: 13, + fontWeight: '500', + color: colors.black, + marginTop: 4, + }, + listSection: { + marginTop: 4, + borderRadius: 18, + backgroundColor: colors.white, + paddingHorizontal: 12, + paddingVertical: 4, + }, + listItem: { + flexDirection: 'row', + alignItems: 'flex-start', + paddingVertical: 10, + }, + listIcon: { + width: 24, + height: 24, + marginRight: 12, + marginTop: 2, + }, + listTextContainer: { + flex: 1, + }, + listTitle: { + fontSize: 15, + fontWeight: '500', + color: colors.black, + }, + listSubtitle: { + marginTop: 2, + fontSize: 13, + color: colors.grey_dark, + }, + divider: { + height: StyleSheet.hairlineWidth, + backgroundColor: colors.grey_whisper, + marginTop: 10, + }, + }); + }, [theme]); +}; diff --git a/package/src/ai-components/message-composer/AttachmentButton.tsx b/package/src/ai-components/message-composer/AttachmentButton.tsx new file mode 100644 index 0000000000..36a9407856 --- /dev/null +++ b/package/src/ai-components/message-composer/AttachmentButton.tsx @@ -0,0 +1,49 @@ +import React, { useMemo } from 'react'; +import { Pressable, StyleSheet, Text } from 'react-native'; + +import { useTheme } from '../contexts'; +import { sheetStoreApi } from '../store'; + +export const AttachmentButton = () => { + const { + theme: { + composer: { roundButton, attachIcon }, + }, + } = useTheme(); + const styles = useStyles(); + + return ( + + + + + ); +}; + +const useStyles = () => { + const { theme } = useTheme(); + + return useMemo(() => { + const { colors, composer } = theme; + + const pillHeight = composer.inputPillHeight; + + return StyleSheet.create({ + attachIcon: { + fontSize: 32, + textAlign: 'center', + alignSelf: 'center', + lineHeight: 32, + color: colors.grey, + }, + roundButton: { + width: pillHeight, + height: pillHeight, + borderRadius: pillHeight / 2, + backgroundColor: colors.white_smoke, + justifyContent: 'center', + alignItems: 'center', + marginRight: 8, + }, + }); + }, [theme]); +}; diff --git a/package/src/ai-components/message-composer/ComposerView.tsx b/package/src/ai-components/message-composer/ComposerView.tsx new file mode 100644 index 0000000000..c2ae35564a --- /dev/null +++ b/package/src/ai-components/message-composer/ComposerView.tsx @@ -0,0 +1,187 @@ +import React, { useEffect, useMemo, useRef } from 'react'; +import { Platform, StyleSheet, TextInput, View } from 'react-native'; + +import Animated, { LinearTransition } from 'react-native-reanimated'; + +import type { StateStore } from '@stream-io/state-store'; + +import { ActionButton } from './ActionButton'; +import { BottomSheetContent } from './ActionSheet'; + +import { AttachmentButton } from './AttachmentButton'; + +import { MediaPreviewList } from './MediaPreviewList'; + +import { BottomSheet } from '../components/BottomSheet'; + +import { ComposerProvider, useMessageComposerContext, useTheme } from '../contexts'; +import type { + AbstractMediaPickerService, + MediaPickerState, +} from '../services/media-picker-service/AbstractMediaPickerService'; +import { useComposerText } from '../store'; +import type { ComposerState } from '../store'; +import { useDictationState } from '../store/dictation/useDictationState'; +import { useDictationTranscript } from '../store/dictation/useDictationTranscript'; + +export type BottomSheetOption = { + title: string; + action: () => void | Promise; + subtitle?: string; + Icon?: React.ComponentType; +}; + +export type BottomSheetInsets = { + top: number; + bottom: number; + left: number; + right: number; +}; + +export type ComposerViewProps = { + onSendMessage: (opts: { + text: string; + attachments?: MediaPickerState['assets']; + custom?: Record; + }) => Promise; + bottomSheetOptions?: BottomSheetOption[]; + isGenerating?: boolean; + stopGenerating?: () => Promise; + bottomSheetInsets?: BottomSheetInsets; + mediaPickerService?: AbstractMediaPickerService; + state?: StateStore; +}; + +export const ComposerViewUI = ({ + bottomSheetOptions = [], + bottomSheetInsets, + isGenerating, + stopGenerating, +}: ComposerViewProps) => { + const { + theme: { + composer: { container, containerRow, inputPillContainer, inputPill }, + }, + } = useTheme(); + const styles = useStyles(); + + return ( + + + + + + + + + + + + + + + + + ); +}; + +export const AIMessageInput = () => { + const { + theme: { + colors: { grey_neutral }, + composer: { textInput }, + }, + } = useTheme(); + const styles = useStyles(); + + const { error, isRecording } = useDictationState(); + const { text } = useComposerText(); + const { transcript } = useDictationTranscript(); + const { setText } = useMessageComposerContext(); + const previousTranscript = useRef(transcript); + + useEffect(() => { + const transcriptChanged = transcript !== previousTranscript.current; + previousTranscript.current = transcript; + if (!error && (isRecording || transcriptChanged)) { + setText(transcript); + } + }, [isRecording, transcript, error, setText]); + + return ( + + ); +}; + +export const ComposerView = (props: ComposerViewProps) => ( + + + +); + +const useStyles = () => { + const { theme } = useTheme(); + + return useMemo(() => { + const { colors, composer } = theme; + + const pillHeight = composer.inputPillHeight; + + return StyleSheet.create({ + container: { + paddingHorizontal: 12, + paddingVertical: 8, + }, + row: { + flexDirection: 'row', + alignItems: 'flex-end', + }, + inputPill: { + alignItems: 'center', + flexDirection: 'row', + maxHeight: pillHeight * 3, + }, + inputPillContainer: { + flex: 1, + minHeight: pillHeight, + alignItems: 'center', + justifyContent: 'center', + borderRadius: pillHeight / 2, + backgroundColor: colors.white_smoke, + paddingHorizontal: 14, + paddingVertical: Platform.OS === 'ios' ? 10 : 6, + shadowColor: colors.black, + shadowOpacity: 0.08, + shadowOffset: { width: 0, height: 1 }, + shadowRadius: 4, + elevation: 2, + }, + textInput: { + flex: 1, + fontSize: 16, + color: colors.black, + paddingVertical: 0, + paddingHorizontal: 0, + marginRight: 24, + }, + }); + }, [theme]); +}; diff --git a/package/src/ai-components/message-composer/MediaPreviewList.tsx b/package/src/ai-components/message-composer/MediaPreviewList.tsx new file mode 100644 index 0000000000..c69e859606 --- /dev/null +++ b/package/src/ai-components/message-composer/MediaPreviewList.tsx @@ -0,0 +1,84 @@ +import React, { useMemo } from 'react'; +import { Image, Pressable, ScrollView, StyleSheet } from 'react-native'; +import Animated, { FadeIn, FadeOut, LinearTransition } from 'react-native-reanimated'; + +import { useMessageComposerContext, useTheme } from '../contexts'; +import { Close } from '../internal/icons/Close'; +import { useMediaPickerState } from '../services/media-picker-service/hooks/useMediaPickerState'; + +export const MediaPreviewList = () => { + const { mediaPickerService } = useMessageComposerContext(); + const { attachments } = useMediaPickerState({ service: mediaPickerService }) ?? {}; + const { + theme: { + colors: { white }, + composer: { + mediaPreviewStyle, + mediaPreviewContentContainerStyle, + mediaPreviewImage, + mediaPreviewRemoveButton, + }, + }, + } = useTheme(); + const styles = useStyles(); + + if (!attachments?.length) { + return null; + } + + return ( + + {(attachments ?? []).map((attachment, index) => ( + + + mediaPickerService?.removeAsset(index)} + > + + + + ))} + + ); +}; + +const useStyles = () => { + const { theme } = useTheme(); + + return useMemo( + () => + StyleSheet.create({ + mediaPreviewStyle: { + width: '100%', + }, + mediaPreviewContentContainerStyle: { flexGrow: 1, paddingBottom: 8 }, + mediaPreviewImage: { borderRadius: 12, marginRight: 8 }, + mediaPreviewRemoveButton: { + position: 'absolute', + top: 8, + right: 12, + backgroundColor: theme.colors.overlay, + borderRadius: 24, + }, + }), + [theme], + ); +}; diff --git a/package/src/ai-components/message-composer/SpeechToTextButton.tsx b/package/src/ai-components/message-composer/SpeechToTextButton.tsx new file mode 100644 index 0000000000..99336f7e35 --- /dev/null +++ b/package/src/ai-components/message-composer/SpeechToTextButton.tsx @@ -0,0 +1,79 @@ +import React, { useEffect } from 'react'; +import { Pressable, StyleSheet, View } from 'react-native'; +import Animated, { ZoomIn, ZoomOut } from 'react-native-reanimated'; + +import { useStateStore } from '@stream-io/state-store/react-bindings'; + +import { useTheme } from '../contexts'; +import { Mic } from '../internal/icons/Mic'; + +import { isDictationAvailable, warnIfDictationUnavailable } from '../native/AIDictation'; +import type { DictationStartOptions } from '../native-specs/NativeAIDictation'; +import { dictationStore, type DictationStoreState } from '../store'; +import { useDictation } from '../transcription/useDictation'; + +const selector = ({ isRecording }: DictationStoreState) => ({ + isRecording, +}); + +type SpeechToTextButtonProps = { + options?: DictationStartOptions; +}; + +/** + * Renders nothing when the native dictation module is not part of the build (see + * `isDictationAvailable`), so that pressing the button can never reach a missing module. + */ +export const SpeechToTextButton = (props: SpeechToTextButtonProps) => { + if (!isDictationAvailable()) { + warnIfDictationUnavailable(); + return null; + } + + return ; +}; + +const AvailableSpeechToTextButton = ({ options }: SpeechToTextButtonProps) => { + const { + theme: { + colors: { grey, accent_blue }, + composer: { iconButton, micIcon }, + }, + } = useTheme(); + + const { cancel, toggle } = useDictation(options); + const { isRecording } = useStateStore(dictationStore, selector); + + useEffect(() => { + return () => { + cancel(); + }; + }, [cancel]); + + return ( + + + + + + + + ); +}; + +const styles = StyleSheet.create({ + micIcon: { + width: 32, + height: 32, + justifyContent: 'center', + alignItems: 'center', + }, + iconButton: { + justifyContent: 'center', + alignItems: 'center', + }, +}); diff --git a/package/src/ai-components/message-composer/index.ts b/package/src/ai-components/message-composer/index.ts new file mode 100644 index 0000000000..415e82ed71 --- /dev/null +++ b/package/src/ai-components/message-composer/index.ts @@ -0,0 +1,5 @@ +export * from './ComposerView'; +export * from './ActionButton'; +export * from './AttachmentButton'; +export * from './MediaPreviewList'; +export * from './SpeechToTextButton'; diff --git a/package/src/ai-components/message-composer/utils/withCloseSheet.ts b/package/src/ai-components/message-composer/utils/withCloseSheet.ts new file mode 100644 index 0000000000..14d9118815 --- /dev/null +++ b/package/src/ai-components/message-composer/utils/withCloseSheet.ts @@ -0,0 +1,11 @@ +import { sheetStoreApi } from '../../store'; + +export const withCloseSheet = ( + callback: (...args: T) => void | Promise | unknown | Promise | undefined, +) => { + return async (...args: T) => { + const result = await callback(...args); + sheetStoreApi.closeSheet(); + return result; + }; +}; diff --git a/package/src/ai-components/native-specs/NativeAIDictation.ts b/package/src/ai-components/native-specs/NativeAIDictation.ts new file mode 100644 index 0000000000..b36c97e284 --- /dev/null +++ b/package/src/ai-components/native-specs/NativeAIDictation.ts @@ -0,0 +1,48 @@ +import type { TurboModule } from 'react-native'; +import { TurboModuleRegistry } from 'react-native'; + +/** + * Codegen spec of the native dictation module. + * + * This file is duplicated (byte for byte) in `package/native-package/src/native/NativeAIDictation.ts` + * and `package/expo-package/src/native/NativeAIDictation.ts`, which are the copies codegen reads. + * `yarn validate-entry-isolation` fails if they drift apart. + * + * Events are delivered through `addListener`/`removeListeners` and a `NativeEventEmitter` (see + * `../native/AIDictation.ts`) rather than codegen `EventEmitter` properties, since the latter are + * not understood by the codegen of every React Native version the SDK supports (>= 0.76). + */ + +export type DictationStartOptions = { + language?: string; + interimResults?: boolean; + silenceTimeoutMs?: number; +}; + +export type DictationResult = { + text: string; + isFinal: boolean; +}; + +export type DictationStateValue = 'idle' | 'starting' | 'listening' | 'stopping'; +export type DictationState = { state: DictationStateValue }; + +export type DictationError = { + code: string; + message: string; +}; + +export interface Spec extends TurboModule { + start(options?: DictationStartOptions): void; + stop(): void; + cancel(): void; + isRecording(): boolean; + + addListener(eventName: string): void; + removeListeners(count: number): void; +} + +// `get` rather than `getEnforcing`: the module is only compiled in when dictation is enabled +// (always on Android, opt-in on iOS), and importing the AI entry point +// must not throw when it is absent. +export default TurboModuleRegistry.get('AIDictation'); diff --git a/package/src/ai-components/native/AIDictation.ts b/package/src/ai-components/native/AIDictation.ts new file mode 100644 index 0000000000..450abae7c8 --- /dev/null +++ b/package/src/ai-components/native/AIDictation.ts @@ -0,0 +1,77 @@ +import { NativeEventEmitter, Platform } from 'react-native'; + +import NativeAIDictation, { + type DictationError, + type DictationResult, + type DictationStartOptions, + type DictationState, +} from '../native-specs/NativeAIDictation'; + +// Event names emitted by the native module (see `AIDictation.mm` and `AIDictationModule.kt`). +const ON_STATE_EVENT = 'AIDictationOnState'; +const ON_RESULT_EVENT = 'AIDictationOnResult'; +const ON_ERROR_EVENT = 'AIDictationOnError'; + +let eventEmitter: NativeEventEmitter | undefined; +const getEventEmitter = () => { + if (!eventEmitter && NativeAIDictation) { + eventEmitter = new NativeEventEmitter(NativeAIDictation); + } + return eventEmitter; +}; + +const noop = () => {}; + +/** + * Whether the native dictation module is part of this build. It is always compiled in on Android, + * while on iOS it is opt-in: add `$StreamChatEnableDictation = true` to your + * `Podfile` (or use the `stream-chat-expo` config plugin) and run `pod install`. + */ +export const isDictationAvailable = () => NativeAIDictation != null; + +let hasWarnedAboutMissingModule = false; +export const warnIfDictationUnavailable = () => { + if (!__DEV__ || hasWarnedAboutMissingModule || isDictationAvailable()) { + return; + } + hasWarnedAboutMissingModule = true; + console.warn( + Platform.OS === 'ios' + ? '[stream-chat-react-native/ai-components] Dictation is not compiled into this build, so the speech-to-text button is hidden. Add `$StreamChatEnableDictation = true` to your Podfile (Expo: add the `stream-chat-expo` config plugin) and run `pod install`.' + : '[stream-chat-react-native/ai-components] Dictation is not available in this build, so the speech-to-text button is hidden.', + ); +}; + +// Low-level API +export const AIDictation = { + start(options?: DictationStartOptions) { + NativeAIDictation?.start(options); + }, + stop() { + NativeAIDictation?.stop(); + }, + cancel() { + NativeAIDictation?.cancel(); + }, + isRecording() { + return NativeAIDictation?.isRecording() ?? false; + }, + addResultListener(listener: (result: DictationResult) => void) { + const sub = getEventEmitter()?.addListener(ON_RESULT_EVENT, (event) => + listener(event as DictationResult), + ); + return sub ? () => sub.remove() : noop; + }, + addStateListener(listener: (payload: DictationState) => void) { + const sub = getEventEmitter()?.addListener(ON_STATE_EVENT, (event) => + listener(event as DictationState), + ); + return sub ? () => sub.remove() : noop; + }, + addErrorListener(listener: (payload: DictationError) => void) { + const sub = getEventEmitter()?.addListener(ON_ERROR_EVENT, (event) => + listener(event as DictationError), + ); + return sub ? () => sub.remove() : noop; + }, +}; diff --git a/package/src/ai-components/native/PerfText.tsx b/package/src/ai-components/native/PerfText.tsx new file mode 100644 index 0000000000..3f28396a78 --- /dev/null +++ b/package/src/ai-components/native/PerfText.tsx @@ -0,0 +1,32 @@ +import React from 'react'; +import { type StyleProp, type ViewStyle } from 'react-native'; + +import { NativeHandlers } from '../../native'; +import type { ColorRange } from '../syntax-highlighting'; + +export type PerfTextProps = { + text: string; + ranges: ColorRange[]; + style?: StyleProp; + selectable?: boolean; + fontSize?: number; + lineHeight?: number; +}; + +export const PerfText = ({ style, fontSize, lineHeight, ranges, text }: PerfTextProps) => { + const PerfTextNative = NativeHandlers.PerfText; + if (!PerfTextNative) { + return null; + } + + return ( + + ); +}; diff --git a/package/src/ai-components/services/clipboard-service/expo.ts b/package/src/ai-components/services/clipboard-service/expo.ts new file mode 100644 index 0000000000..4d72eee31f --- /dev/null +++ b/package/src/ai-components/services/clipboard-service/expo.ts @@ -0,0 +1,11 @@ +let Clipboard: { setString: (string: string) => void } | undefined; + +try { + Clipboard = require('expo-clipboard'); +} catch (_) { + // do nothing +} + +export const setClipboardString = Clipboard + ? (string: string) => Clipboard?.setString(string) + : undefined; diff --git a/package/src/ai-components/services/clipboard-service/index.ts b/package/src/ai-components/services/clipboard-service/index.ts new file mode 100644 index 0000000000..b95fe374c5 --- /dev/null +++ b/package/src/ai-components/services/clipboard-service/index.ts @@ -0,0 +1,4 @@ +import { setClipboardString as expoSetClipboardString } from './expo'; +import { setClipboardString as rncliSetClipboardString } from './rncli'; + +export const setClipboardString = rncliSetClipboardString ?? expoSetClipboardString; diff --git a/package/src/ai-components/services/clipboard-service/rncli.ts b/package/src/ai-components/services/clipboard-service/rncli.ts new file mode 100644 index 0000000000..a7d7ff705d --- /dev/null +++ b/package/src/ai-components/services/clipboard-service/rncli.ts @@ -0,0 +1,11 @@ +let Clipboard: { setString: (string: string) => void } | undefined; + +try { + Clipboard = require('@react-native-clipboard/clipboard').default; +} catch (_) { + // do nothing +} + +export const setClipboardString = Clipboard + ? (string: string) => (Clipboard ? Clipboard.setString(string) : {}) + : undefined; diff --git a/package/src/ai-components/services/index.ts b/package/src/ai-components/services/index.ts new file mode 100644 index 0000000000..bd463035d1 --- /dev/null +++ b/package/src/ai-components/services/index.ts @@ -0,0 +1,2 @@ +export * from './media-picker-service'; +export * from './clipboard-service'; diff --git a/package/src/ai-components/services/media-picker-service/AbstractMediaPickerService.ts b/package/src/ai-components/services/media-picker-service/AbstractMediaPickerService.ts new file mode 100644 index 0000000000..759c500f64 --- /dev/null +++ b/package/src/ai-components/services/media-picker-service/AbstractMediaPickerService.ts @@ -0,0 +1,62 @@ +import { StateStore } from '@stream-io/state-store'; + +export type MediaAsset = { + uri: string; + width?: number; + height?: number; + name?: string; + size?: number; + type?: string; +}; + +export type MediaPickerState = { + assets: MediaAsset[]; +}; + +export type PickMediaOpts = { maxNumberOfFiles?: number }; +export type PickMediaReturnType = { + assets?: MediaAsset[]; + cancelled?: boolean; + askToOpenSettings?: boolean; +}; +export type PickMediaApi = (opts: PickMediaOpts) => Promise; + +export type TakeMediaOpts = { compressionQuality?: number }; +export type TakeMediaReturnType = { + asset?: MediaAsset; + cancelled?: boolean; + askToOpenSettings?: boolean; +}; +export type TakeMediaApi = (opts: TakeMediaOpts) => Promise; + +export abstract class AbstractMediaPickerService { + public state: StateStore; + + constructor() { + this.state = new StateStore({ + assets: [], + }); + } + + abstract pickMedia: PickMediaApi; + + abstract takeMedia: TakeMediaApi; + + appendAssets = (assets: MediaAsset[]) => { + this.state.next((prevState) => ({ + ...prevState, + assets: [...prevState.assets, ...assets], + })); + }; + + removeAsset = (index: number) => { + this.state.next((prevState) => ({ + ...prevState, + assets: prevState.assets.filter((_, i) => i !== index), + })); + }; + + clearAssets = () => { + this.state.partialNext({ assets: [] }); + }; +} diff --git a/package/src/ai-components/services/media-picker-service/expo.ts b/package/src/ai-components/services/media-picker-service/expo.ts new file mode 100644 index 0000000000..53a99ff975 --- /dev/null +++ b/package/src/ai-components/services/media-picker-service/expo.ts @@ -0,0 +1,148 @@ +import { Image, Platform } from 'react-native'; + +import type { ImagePickerAsset } from 'expo-image-picker'; + +import { + AbstractMediaPickerService, + type MediaAsset, + type PickMediaOpts, +} from './AbstractMediaPickerService'; + +let internalService = undefined; + +try { + internalService = require('expo-image-picker'); +} catch (_error) { + /* do nothing */ +} + +type Size = { + height?: number; + width?: number; +}; + +class ExpoMediaPickerService extends AbstractMediaPickerService { + pickMedia = async ({ maxNumberOfFiles = 3 }: PickMediaOpts = {}) => { + try { + let permissionGranted = true; + if (Platform.OS === 'ios') { + const permissionCheck = await internalService!.getMediaLibraryPermissionsAsync(); + const canRequest = permissionCheck.canAskAgain; + permissionGranted = permissionCheck.granted; + if (!permissionGranted) { + if (canRequest) { + const response = await internalService!.requestMediaLibraryPermissionsAsync(); + permissionGranted = response.granted; + } else { + return { askToOpenSettings: true, cancelled: true }; + } + } + } + if (permissionGranted) { + const result = await internalService!.launchImageLibraryAsync({ + allowsMultipleSelection: true, + mediaTypes: 'images', + preferredAssetRepresentationMode: 'current', + selectionLimit: maxNumberOfFiles, + }); + + const canceled = result.canceled; + + if (!canceled) { + const assets = result.assets.map((asset: ImagePickerAsset) => ({ + ...asset, + name: asset.fileName, + size: asset.fileSize, + type: asset.mimeType, + uri: asset.uri, + })); + this.appendAssets(assets); + return { assets, cancelled: false }; + } else { + return { cancelled: true }; + } + } + return { cancelled: true }; + } catch (error) { + console.log('Error while picking image', error); + return { cancelled: true }; + } + }; + + takeMedia = async ({ compressionQuality = 1 }) => { + try { + const permissionCheck = await internalService!.getCameraPermissionsAsync(); + const canRequest = permissionCheck.canAskAgain; + let permissionGranted = permissionCheck.granted; + if (!permissionGranted) { + if (canRequest) { + const response = await internalService!.requestCameraPermissionsAsync(); + permissionGranted = response.granted; + } else { + return { askToOpenSettings: true, cancelled: true }; + } + } + + if (permissionGranted) { + const result = await internalService!.launchCameraAsync({ + mediaTypes: 'images', + quality: Math.min(Math.max(0, compressionQuality), 1), + }); + if (!result || !result.assets || !result.assets.length || result.canceled) { + return { cancelled: true }; + } + // since we only support single photo upload for now we will only be focusing on 0'th element. + const photo = result.assets[0]; + if (!photo) { + return { cancelled: true }; + } + if (photo && photo.mimeType.includes('image') && photo.height && photo.width && photo.uri) { + let size: Size = {}; + if (Platform.OS === 'android') { + const getSize = (): Promise => + new Promise((resolve) => { + Image.getSize(photo.uri, (width, height) => { + resolve({ height, width }); + }); + }); + + try { + const { height, width } = await getSize(); + size.height = height; + size.width = width; + } catch (e) { + console.warn('Error get image size of picture caputred from camera ', e); + } + } else { + size = { + height: photo.height, + width: photo.width, + }; + } + const clearFilter = new RegExp('[.:]', 'g'); + const date = new Date().toISOString().replace(clearFilter, '_'); + const asset: MediaAsset = { + name: 'image_' + date + '.' + photo.uri.split('.').pop(), + size: photo.fileSize, + type: photo.mimeType, + uri: photo.uri, + height: size.height!, + width: size.width!, + }; + this.appendAssets([asset]); + return { + cancelled: false, + asset, + }; + } + } + } catch (error) { + console.log(error); + } + return { cancelled: true }; + }; +} + +const MediaPickerService = internalService ? ExpoMediaPickerService : undefined; + +export { MediaPickerService }; diff --git a/package/src/ai-components/services/media-picker-service/hooks/useMediaPickerState.ts b/package/src/ai-components/services/media-picker-service/hooks/useMediaPickerState.ts new file mode 100644 index 0000000000..a03731ede3 --- /dev/null +++ b/package/src/ai-components/services/media-picker-service/hooks/useMediaPickerState.ts @@ -0,0 +1,14 @@ +import { useStateStore } from '@stream-io/state-store/react-bindings'; + +import type { AbstractMediaPickerService } from '../AbstractMediaPickerService'; +import { type MediaPickerState } from '../AbstractMediaPickerService'; + +export const selector = (nextState: MediaPickerState) => ({ + attachments: nextState.assets, +}); + +export const useMediaPickerState = ({ + service, +}: { + service: AbstractMediaPickerService | undefined; +}) => useStateStore(service?.state, selector); diff --git a/package/src/ai-components/services/media-picker-service/index.ts b/package/src/ai-components/services/media-picker-service/index.ts new file mode 100644 index 0000000000..95edc835d6 --- /dev/null +++ b/package/src/ai-components/services/media-picker-service/index.ts @@ -0,0 +1,4 @@ +import { MediaPickerService as ExpoMediaPickerService } from './expo'; +import { MediaPickerService as RNCLIMediaPickerService } from './rncli'; + +export const MediaPickerService = RNCLIMediaPickerService ?? ExpoMediaPickerService; diff --git a/package/src/ai-components/services/media-picker-service/optional-modules.d.ts b/package/src/ai-components/services/media-picker-service/optional-modules.d.ts new file mode 100644 index 0000000000..03823285c8 --- /dev/null +++ b/package/src/ai-components/services/media-picker-service/optional-modules.d.ts @@ -0,0 +1,25 @@ +// The image picker libraries are optional peer dependencies (`react-native-image-picker` for React +// Native CLI apps, `expo-image-picker` for Expo apps) and are not installed in this package. Only +// the asset shapes the media picker services read are declared here. + +declare module 'react-native-image-picker' { + export type Asset = { + fileName?: string; + fileSize?: number; + height?: number; + type?: string; + uri?: string; + width?: number; + }; +} + +declare module 'expo-image-picker' { + export type ImagePickerAsset = { + fileName?: string | null; + fileSize?: number; + height: number; + mimeType?: string; + uri: string; + width: number; + }; +} diff --git a/package/src/ai-components/services/media-picker-service/rncli.ts b/package/src/ai-components/services/media-picker-service/rncli.ts new file mode 100644 index 0000000000..0d89b9efae --- /dev/null +++ b/package/src/ai-components/services/media-picker-service/rncli.ts @@ -0,0 +1,142 @@ +import { AppState, Image, PermissionsAndroid, Platform } from 'react-native'; +import type { Asset } from 'react-native-image-picker'; + +import { + AbstractMediaPickerService, + type MediaAsset, + type PickMediaOpts, +} from './AbstractMediaPickerService'; + +let internalService = undefined; + +try { + internalService = require('react-native-image-picker'); +} catch (_error) { + /* do nothing */ +} + +class RNCLIMediaPickerService extends AbstractMediaPickerService { + pickMedia = async ({ maxNumberOfFiles = 3 }: PickMediaOpts = {}) => { + try { + const result = await internalService!.launchImageLibrary({ + assetRepresentationMode: 'current', + mediaType: 'photo', + selectionLimit: maxNumberOfFiles, + }); + const canceled = result.didCancel; + const errorCode = result.errorCode; + + if (Platform.OS === 'ios' && errorCode === 'permission') { + return { askToOpenSettings: true, cancelled: true }; + } + if (!canceled) { + const assets = result.assets.map((asset: Asset) => ({ + ...asset, + name: asset.fileName, + size: asset.fileSize, + type: asset.type, + uri: asset.uri, + })); + this.appendAssets(assets); + return { assets, cancelled: false }; + } else { + return { cancelled: true }; + } + } catch (error) { + console.log('Error picking image: ', error); + return { cancelled: true }; + } + }; + + takeMedia = async ({ compressionQuality = Platform.OS === 'ios' ? 0.8 : 1 }) => { + if (Platform.OS === 'android') { + const cameraPermissions = await PermissionsAndroid.check( + PermissionsAndroid.PERMISSIONS.CAMERA, + ); + if (!cameraPermissions) { + const androidPermissionStatus = await PermissionsAndroid.request( + PermissionsAndroid.PERMISSIONS.CAMERA, + ); + if (androidPermissionStatus === PermissionsAndroid.RESULTS.DENIED) { + return { cancelled: true }; + } else if (androidPermissionStatus === PermissionsAndroid.RESULTS.NEVER_ASK_AGAIN) { + return { askToOpenSettings: true, cancelled: true }; + } + } + } + try { + const result = await internalService!.launchCamera({ + mediaType: 'photo', + quality: Math.min(Math.max(0, compressionQuality), 1), + }); + if (!result || !result.assets || !result.assets.length || result.didCancel) { + return { + cancelled: true, + }; + } + const asset = result.assets[0]; + if (!asset) { + return { + cancelled: true, + }; + } + if (asset.height && asset.width && asset.uri) { + let size: { height?: number; width?: number } = {}; + if (Platform.OS === 'android') { + // Height and width returned by ImagePicker are incorrect on Android. + const getSize = (): Promise<{ height: number; width: number }> => + new Promise((resolve) => { + Image.getSize(asset.uri, (width, height) => { + resolve({ height, width }); + }); + }); + + try { + const { height, width } = await getSize(); + size.height = height; + size.width = width; + } catch (e) { + // do nothing + console.warn('Error while getting image size of picture captured from camera ', e); + } + } else { + size = { + height: asset.height, + width: asset.width, + }; + } + const clearFilter = new RegExp('[.:]', 'g'); + const date = new Date().toISOString().replace(clearFilter, '_'); + const localAsset: MediaAsset = { + name: 'image_' + date + '.' + asset.uri.split('.').pop(), + size: asset.fileSize, + type: asset.type, + uri: asset.uri, + ...size, + }; + this.appendAssets([localAsset]); + return { + cancelled: false, + asset: localAsset, + }; + } + } catch (e: unknown) { + if (e instanceof Error) { + // on iOS: if it was in inactive state, then the user had just denied the permissions + if (Platform.OS === 'ios' && AppState.currentState === 'active') { + const cameraPermissionDeniedMsg = 'User did not grant camera permission.'; + // Open settings when the user did not allow camera permissions + if (e.message === cameraPermissionDeniedMsg) { + return { askToOpenSettings: true, cancelled: true }; + } + } + } + } + + return { cancelled: true }; + }; +} + +const MediaPickerService = internalService ? RNCLIMediaPickerService : undefined; + +export { MediaPickerService }; diff --git a/package/src/ai-components/store/bottom-sheet/store.ts b/package/src/ai-components/store/bottom-sheet/store.ts new file mode 100644 index 0000000000..f1433d6cf2 --- /dev/null +++ b/package/src/ai-components/store/bottom-sheet/store.ts @@ -0,0 +1,30 @@ +import { Keyboard } from 'react-native'; + +import { StateStore } from '@stream-io/state-store'; + +export type BottomSheetState = { + open: boolean; + height: number; +}; + +const DEFAULT_STATE: BottomSheetState = { + open: false, + height: Number.MAX_SAFE_INTEGER, +}; + +export const sheetStore = new StateStore(DEFAULT_STATE); + +const openSheet = () => { + Keyboard.dismiss(); + sheetStore.partialNext({ open: true }); +}; + +const closeSheet = () => sheetStore.partialNext(DEFAULT_STATE); + +const setHeight = (height: number) => sheetStore.partialNext({ height }); + +export const sheetStoreApi = { + openSheet, + closeSheet, + setHeight, +}; diff --git a/package/src/ai-components/store/bottom-sheet/useBottomSheetState.ts b/package/src/ai-components/store/bottom-sheet/useBottomSheetState.ts new file mode 100644 index 0000000000..2d5f57ac4d --- /dev/null +++ b/package/src/ai-components/store/bottom-sheet/useBottomSheetState.ts @@ -0,0 +1,10 @@ +import { useStateStore } from '@stream-io/state-store/react-bindings'; + +import { type BottomSheetState, sheetStore } from './store'; + +const selector = ({ open, height }: BottomSheetState) => ({ + open, + height, +}); + +export const useBottomSheetState = () => useStateStore(sheetStore, selector); diff --git a/package/src/ai-components/store/composer/store.ts b/package/src/ai-components/store/composer/store.ts new file mode 100644 index 0000000000..fa6e1a6fc1 --- /dev/null +++ b/package/src/ai-components/store/composer/store.ts @@ -0,0 +1,12 @@ +import { StateStore } from '@stream-io/state-store'; + +export type ComposerState = { + text: string; + custom?: Record; +}; + +const DEFAULT_STATE: ComposerState = { + text: '', +}; + +export const createNewComposerStore = () => new StateStore(DEFAULT_STATE); diff --git a/package/src/ai-components/store/composer/useComposerHasText.ts b/package/src/ai-components/store/composer/useComposerHasText.ts new file mode 100644 index 0000000000..003de50f73 --- /dev/null +++ b/package/src/ai-components/store/composer/useComposerHasText.ts @@ -0,0 +1,14 @@ +import { useStateStore } from '@stream-io/state-store/react-bindings'; + +import { type ComposerState } from './store'; + +import { useMessageComposerContext } from '../../contexts'; + +const selector = ({ text }: ComposerState) => ({ + hasText: !!text && text.length > 0, +}); + +export const useComposerHasText = () => { + const { state } = useMessageComposerContext(); + return useStateStore(state, selector); +}; diff --git a/package/src/ai-components/store/composer/useComposerText.ts b/package/src/ai-components/store/composer/useComposerText.ts new file mode 100644 index 0000000000..0b84583a9b --- /dev/null +++ b/package/src/ai-components/store/composer/useComposerText.ts @@ -0,0 +1,14 @@ +import { useStateStore } from '@stream-io/state-store/react-bindings'; + +import { type ComposerState } from './store'; + +import { useMessageComposerContext } from '../../contexts'; + +const selector = ({ text }: ComposerState) => ({ + text, +}); + +export const useComposerText = () => { + const { state } = useMessageComposerContext(); + return useStateStore(state, selector); +}; diff --git a/package/src/ai-components/store/dictation/store.ts b/package/src/ai-components/store/dictation/store.ts new file mode 100644 index 0000000000..9c3c0514d1 --- /dev/null +++ b/package/src/ai-components/store/dictation/store.ts @@ -0,0 +1,23 @@ +import { StateStore } from '@stream-io/state-store'; + +import type { DictationError, DictationStateValue } from '../../native-specs/NativeAIDictation'; + +export type DictationStoreState = { + state: DictationStateValue; + transcript: string; + isRecording: boolean; + error?: DictationError | null; +}; + +const DEFAULT_STATE: DictationStoreState = { + state: 'idle', + transcript: '', + isRecording: false, + error: null, +}; + +export const dictationStore = new StateStore(DEFAULT_STATE); + +const reset = () => dictationStore.next(DEFAULT_STATE); + +export const dictationStoreApi = { reset }; diff --git a/package/src/ai-components/store/dictation/useDictationState.ts b/package/src/ai-components/store/dictation/useDictationState.ts new file mode 100644 index 0000000000..a40627005d --- /dev/null +++ b/package/src/ai-components/store/dictation/useDictationState.ts @@ -0,0 +1,11 @@ +import { useStateStore } from '@stream-io/state-store/react-bindings'; + +import { dictationStore, type DictationStoreState } from './store'; + +const selector = ({ state, isRecording, error }: DictationStoreState) => ({ + state, + isRecording, + error, +}); + +export const useDictationState = () => useStateStore(dictationStore, selector); diff --git a/package/src/ai-components/store/dictation/useDictationTranscript.ts b/package/src/ai-components/store/dictation/useDictationTranscript.ts new file mode 100644 index 0000000000..6704d72e9f --- /dev/null +++ b/package/src/ai-components/store/dictation/useDictationTranscript.ts @@ -0,0 +1,9 @@ +import { useStateStore } from '@stream-io/state-store/react-bindings'; + +import { dictationStore, type DictationStoreState } from './store'; + +const selector = ({ transcript }: DictationStoreState) => ({ + transcript, +}); + +export const useDictationTranscript = () => useStateStore(dictationStore, selector); diff --git a/package/src/ai-components/store/index.ts b/package/src/ai-components/store/index.ts new file mode 100644 index 0000000000..f737510a79 --- /dev/null +++ b/package/src/ai-components/store/index.ts @@ -0,0 +1,11 @@ +// bottom sheet +export * from './bottom-sheet/store'; +export * from './bottom-sheet/useBottomSheetState'; + +// message composer +export * from './composer/store'; +export * from './composer/useComposerHasText'; +export * from './composer/useComposerText'; + +// dictation +export * from './dictation/store'; diff --git a/package/src/ai-components/syntax-highlighting/SyntaxHighlighter.tsx b/package/src/ai-components/syntax-highlighting/SyntaxHighlighter.tsx new file mode 100644 index 0000000000..9ca9131d6c --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/SyntaxHighlighter.tsx @@ -0,0 +1,66 @@ +import React, { type PropsWithChildren, useMemo } from 'react'; +import { Platform, type ProcessedColorValue } from 'react-native'; + +import './prism-config'; + +// Imported by file: Metro does not tree-shake, so the package index (every highlighter variant, +// including the async ones) and the style indexes (every bundled theme) would all be bundled. +import SyntaxHighlighterHljs from 'react-syntax-highlighter/dist/esm/default-highlight'; +import SyntaxHighlighterPrism from 'react-syntax-highlighter/dist/esm/prism'; +import defaultStyle from 'react-syntax-highlighter/dist/esm/styles/hljs/default-style'; +import prismDefaultStyle from 'react-syntax-highlighter/dist/esm/styles/prism/prism'; + +import type { NativeSyntaxHighlighterProps, SyntaxHighlighterStylesheet } from './types'; +import { DEFAULT_FONT_SIZE, generateNewStylesheet, nativeRenderer } from './utils'; + +import { MarkdownReactiveScrollView } from '../components'; + +export type ColorRange = { + start: number; + end: number; + color: ProcessedColorValue; +}; + +export const SyntaxHighlighter = ({ + fontFamily = Platform.OS === 'ios' ? 'Courier' : 'Monospace', + fontSize = DEFAULT_FONT_SIZE, + children, + highlighter = 'highlightjs', + style = highlighter === 'prism' ? prismDefaultStyle : defaultStyle, + PreTag = MarkdownReactiveScrollView, + CodeTag = MarkdownReactiveScrollView, + ...rest +}: PropsWithChildren) => { + const { transformedStyle, defaultColor } = useMemo( + () => + generateNewStylesheet({ + stylesheet: style, + highlighter, + }), + [highlighter, style], + ); + const renderer = useMemo( + () => + nativeRenderer({ + defaultColor: defaultColor as string, + fontFamily, + fontSize, + }), + [defaultColor, fontFamily, fontSize], + ); + + const Highlighter = highlighter === 'prism' ? SyntaxHighlighterPrism : SyntaxHighlighterHljs; + + return ( + + {children} + + ); +}; diff --git a/package/src/ai-components/syntax-highlighting/components/AndroidNativeRenderer.tsx b/package/src/ai-components/syntax-highlighting/components/AndroidNativeRenderer.tsx new file mode 100644 index 0000000000..498adc198b --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/components/AndroidNativeRenderer.tsx @@ -0,0 +1,29 @@ +import React from 'react'; + +import { PerfText } from '../../components'; +import { flattenRowsToTextAndColorRanges } from '../utils'; + +export const AndroidNativeRenderer = ({ + rows, + stylesheet, + defaultColor, + fontFamily, + fontSize, + lineHeight = 17, +}: { + rows: rendererNode[]; + stylesheet: rendererProps['stylesheet']; + defaultColor: string; + fontFamily?: string; + fontSize: number; + lineHeight: number; +}) => { + const { text, ranges } = flattenRowsToTextAndColorRanges(rows, { + stylesheet, + defaultColor, + fontFamily, + fontSize, + }); + + return ; +}; diff --git a/package/src/ai-components/syntax-highlighting/components/index.ts b/package/src/ai-components/syntax-highlighting/components/index.ts new file mode 100644 index 0000000000..294c4c33c8 --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/components/index.ts @@ -0,0 +1 @@ +export * from './AndroidNativeRenderer'; diff --git a/package/src/ai-components/syntax-highlighting/index.ts b/package/src/ai-components/syntax-highlighting/index.ts new file mode 100644 index 0000000000..f48e59c232 --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/index.ts @@ -0,0 +1,11 @@ +/** + * This sub-library is a port/took inspiration from https://github.com/conorhastings/react-native-syntax-highlighter. + * We create our own since we don't want the version of https://github.com/react-syntax-highlighter/react-syntax-highlighter + * to be pinned to version 6 but rather use whatever we see fit. + */ + +export * from './SyntaxHighlighter'; +export * from './prism-config'; +export * from './utils'; + +export * from './types'; diff --git a/package/src/ai-components/syntax-highlighting/prism-config.ts b/package/src/ai-components/syntax-highlighting/prism-config.ts new file mode 100644 index 0000000000..d0bc5f9e0c --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/prism-config.ts @@ -0,0 +1,4 @@ +// @ts-expect-error This negates the global variable set by react-syntax-highlighter +global.Prism = { disableWorkerMessageHandler: true }; + +export {}; diff --git a/package/src/ai-components/syntax-highlighting/types.ts b/package/src/ai-components/syntax-highlighting/types.ts new file mode 100644 index 0000000000..0cce244fe6 --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/types.ts @@ -0,0 +1,26 @@ +import type React from 'react'; +import type { TextStyle } from 'react-native'; +import type { SyntaxHighlighterProps } from 'react-syntax-highlighter'; + +export type RNStyle = TextStyle; +export type RNSheet = Record; + +export type SyntaxHighlighterStylesheet = NonNullable; + +export type ExtraSyntaxHighlighterProps = { + fontFamily?: string; + fontSize?: number; + highlighter?: 'highlightjs' | 'prism'; +}; + +export type NativeSyntaxHighlighterProps = SyntaxHighlighterProps & ExtraSyntaxHighlighterProps; + +export type CSSP = React.CSSProperties; + +/** Options for unit conversion */ +export type ConvertOpts = { + /** base for `em` (defaults to 16) */ + baseFontSize?: number; + /** root base for `rem` (defaults to baseFontSize) */ + rootFontSize?: number; +}; diff --git a/package/src/ai-components/syntax-highlighting/utils/converter.ts b/package/src/ai-components/syntax-highlighting/utils/converter.ts new file mode 100644 index 0000000000..b211d82d81 --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/utils/converter.ts @@ -0,0 +1,145 @@ +import type { TextStyle } from 'react-native'; + +import type { ConvertOpts, CSSP } from '../types'; + +const EM_DEFAULT = 16; + +// React Native's strict TypeScript API declares style properties as readonly. +type MutableTextStyle = { -readonly [K in keyof TextStyle]: TextStyle[K] }; + +/** Parse CSS lengths like 14, "14", "14px", "1.25em", "0.875rem" */ +const len = (v: unknown, { baseFontSize, rootFontSize }: ConvertOpts): number | undefined => { + if (v == null) return undefined; + if (typeof v === 'number') return v; + if (typeof v !== 'string') return undefined; + + const s = v.trim().toLowerCase(); + if (s.endsWith('px')) { + const n = Number(s.slice(0, -2)); + return Number.isFinite(n) ? n : undefined; + } + if (s.endsWith('em')) { + const n = Number(s.slice(0, -2)); + return Number.isFinite(n) ? n * (baseFontSize ?? EM_DEFAULT) : undefined; + } + if (s.endsWith('rem')) { + const n = Number(s.slice(0, -3)); + const root = rootFontSize ?? baseFontSize ?? EM_DEFAULT; + return Number.isFinite(n) ? n * root : undefined; + } + // plain number-like string + const n = Number(s); + return Number.isFinite(n) ? n : undefined; +}; + +/** Font shorthand: e.g. "italic small-caps 700 14px/20px Menlo, monospace" */ +const parseFontShorthand = (v: string, opts: ConvertOpts) => { + const out: MutableTextStyle = {}; + const parts = v.split(/\s+/); + // Very light parser: pick out style, weight, size[/lineHeight], and family + // Strategy: + // - style: "normal|italic" + // - weight: "normal|bold|100..900" + // - when we hit a token like "14px" or "14px/20px" โ†’ size & optional lineHeight + // - everything after size token is font family (may contain commas/spaces) + let i = 0; + + // style + if (parts[i] === 'italic' || parts[i] === 'normal') { + if (parts[i] !== 'normal') out.fontStyle = 'italic'; + i++; + } + + // (skip small-caps if present; RN uses fontVariant which is separate) + if (parts[i] === 'small-caps') { + out.fontVariant = ['small-caps']; + i++; + } + + // weight + if (/^(normal|bold|[1-9]00)$/.test(parts[i] ?? '')) { + const token = parts[i]!; + out.fontWeight = (token === 'normal' ? undefined : token) as TextStyle['fontWeight']; + i++; + } + + // size[/lineHeight] + const sizeToken = parts[i]; + if ( + sizeToken && + (/.+(px|em|rem)$/.test(sizeToken) || /^[\d.]+(\/[\d.]+)?(px|em|rem)?$/.test(sizeToken)) + ) { + const [fs, lh] = sizeToken.split('/'); + const fontSize = len(fs, opts); + if (fontSize != null) out.fontSize = fontSize; + if (lh) { + const lineHeight = len(lh, opts); + if (lineHeight != null) out.lineHeight = lineHeight; + } + i++; + } + + // family + const family = parts.slice(i).join(' ').replace(/^,|,$/g, '').trim(); + if (family) { + // basic cleanup: take first family; strip quotes + const first = family + .split(',')[0] + ?.trim() + .replace(/^["']|["']$/g, ''); + if (first) out.fontFamily = first; + } + + return out; +}; + +/** map CSS (web) to RN TextStyle */ +export const cssToRNTextStyle = (css: Partial, options: ConvertOpts = {}): TextStyle => { + const opts: ConvertOpts = { + baseFontSize: options.baseFontSize ?? EM_DEFAULT, + rootFontSize: options.rootFontSize ?? options.baseFontSize ?? EM_DEFAULT, + }; + + const out: MutableTextStyle = {}; + + // Simple 1:1 or unit-converted mappings + if (css.color) out.color = String(css.color); + + if (css.fontFamily) out.fontFamily = String(css.fontFamily); + if (css.fontStyle) out.fontStyle = css.fontStyle as TextStyle['fontStyle']; + if (css.fontWeight) out.fontWeight = String(css.fontWeight) as TextStyle['fontWeight']; + + if (css.fontSize != null) { + const n = len(css.fontSize, opts); + if (n != null) out.fontSize = n; + } + + if (css.lineHeight != null) { + const n = len(css.lineHeight, opts); + if (n != null) out.lineHeight = n; + } + + if (css.letterSpacing != null) { + const n = len(css.letterSpacing, opts); + if (n != null) out.letterSpacing = n; + } + + if (css.textAlign) out.textAlign = css.textAlign as TextStyle['textAlign']; + + if (css.font && typeof css.font === 'string') { + Object.assign(out, parseFontShorthand(css.font, opts)); + } + + // Simulating the closest we can get to a CSS overflow-like functionality + const overflowLike = css.overflow ?? css.overflowX ?? css.overflowY; + if (typeof overflowLike === 'string') { + out.overflow = + overflowLike === 'hidden' + ? 'hidden' + : overflowLike === 'scroll' || overflowLike === 'auto' + ? 'scroll' + : 'visible'; + } + + return out as TextStyle; +}; diff --git a/package/src/ai-components/syntax-highlighting/utils/flatten-nodes.ts b/package/src/ai-components/syntax-highlighting/utils/flatten-nodes.ts new file mode 100644 index 0000000000..0fa33f1e06 --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/utils/flatten-nodes.ts @@ -0,0 +1,150 @@ +import { processColor } from 'react-native'; + +// @ts-expect-error createStyleObject is not available in the type exports, it is still exported though +import { createStyleObject } from 'react-syntax-highlighter/dist/esm/create-element'; + +import type { ColorRange } from '../SyntaxHighlighter'; +import type { SyntaxHighlighterStylesheet } from '../types'; + +type FlattenOptions = { + nodes: rendererNode | rendererNode[]; + stylesheet: SyntaxHighlighterStylesheet; + defaultColor: string; + fontFamily?: string; + fontSize?: number; +}; + +export const flattenNodesToTextAndColorRanges = ({ + nodes, + stylesheet, + defaultColor, + fontFamily, + fontSize = 13, +}: FlattenOptions): { text: string; ranges: ColorRange[] } => { + let text = ''; + const rawRanges: { start: number; end: number; color: string | number }[] = []; + + const startingStyle = { fontFamily, fontSize }; + + const walk = (node: rendererNode | null | undefined, inheritedColor: string | number) => { + if (!node) return; + + const { type, tagName, value, properties } = node; + + // leaf node + if (type === 'text') { + if (value == null || value === '') return; + const chunk = String(value); + const start = text.length; + text += chunk; + + const color = inheritedColor ?? defaultColor; + if (color != null && chunk.length) { + rawRanges.push({ start, end: start + chunk.length, color }); + } + return; + } + + // node with children + if (tagName) { + const baseStyle = + properties && properties.className + ? createStyleObject( + properties.className, + Object.assign({ color: defaultColor }, properties.style, startingStyle), + stylesheet, + ) + : { ...startingStyle, color: defaultColor }; + + // we take the node's color if present, then its inherited color + // if present and finally the block's default color if all else + // fails. + const effectiveColor = + (baseStyle as { color?: string | number }).color ?? inheritedColor ?? defaultColor; + + const children = node.children || []; + const childArray = Array.isArray(children) ? children : [children]; + for (let i = 0; i < childArray.length; i++) { + walk(childArray[i], effectiveColor); + } + return; + } + + // fallback + if (value != null) { + const chunk = String(value); + const start = text.length; + text += chunk; + const color = inheritedColor ?? defaultColor; + if (color != null && chunk.length) { + rawRanges.push({ start, end: start + chunk.length, color }); + } + } + }; + + // Recursively flatten nodes + const nodeArray = Array.isArray(nodes) ? nodes : [nodes]; + for (let i = 0; i < nodeArray.length; i++) { + walk(nodeArray[i], defaultColor); + } + + // Merge adjacent ranges with same color and convert them to integers + rawRanges.sort((a, b) => a.start - b.start); + const merged: ColorRange[] = []; + for (let i = 0; i < rawRanges.length; i++) { + const r = rawRanges[i]!; + // in case the color is malformed for some reason, use the defaultColor + // as this one should always be non-nullish. + const colorInt = processColor(r.color)!; // ?? processColor(defaultColor))!; + + if (!merged.length) { + merged.push({ start: r.start, end: r.end, color: colorInt }); + continue; + } + + const last = merged[merged.length - 1]; + if (last && last.end === r.start && last.color === colorInt) { + last.end = r.end; + } else { + merged.push({ start: r.start, end: r.end, color: colorInt }); + } + } + + return { text, ranges: merged }; +}; + +export const flattenRowsToTextAndColorRanges = ( + rows: rendererNode[], + opts: Omit, +): { text: string; ranges: ColorRange[] } => { + let fullText = ''; + const fullRanges: ColorRange[] = []; + + for (let rowIndex = 0; rowIndex < rows.length; rowIndex++) { + const rowNodes = rows[rowIndex]; + + const { text, ranges } = flattenNodesToTextAndColorRanges({ + ...opts, + nodes: rowNodes ?? [], + }); + + const offset = fullText.length; + fullText += text; + + // Add newline between rows (no color) + if (rowIndex < rows.length - 1) { + fullText += '\n'; + } + + // Shift ranges by current offset & push + for (const r of ranges) { + fullRanges.push({ + start: r.start + offset, + end: r.end + offset, + color: r.color, + }); + } + } + + return { text: fullText, ranges: fullRanges }; +}; diff --git a/package/src/ai-components/syntax-highlighting/utils/index.ts b/package/src/ai-components/syntax-highlighting/utils/index.ts new file mode 100644 index 0000000000..8a1144deb4 --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/utils/index.ts @@ -0,0 +1,3 @@ +export * from './converter'; +export * from './flatten-nodes'; +export * from './renderer'; diff --git a/package/src/ai-components/syntax-highlighting/utils/renderer.tsx b/package/src/ai-components/syntax-highlighting/utils/renderer.tsx new file mode 100644 index 0000000000..bce647e88e --- /dev/null +++ b/package/src/ai-components/syntax-highlighting/utils/renderer.tsx @@ -0,0 +1,151 @@ +import React from 'react'; +import { Platform, Text } from 'react-native'; + +import type { SyntaxHighlighterProps } from 'react-syntax-highlighter'; + +// @ts-expect-error createStyleObject is not available in the type exports, it is still exported though +import { createStyleObject } from 'react-syntax-highlighter/dist/esm/create-element'; + +import { cssToRNTextStyle } from './converter'; + +import { AndroidNativeRenderer } from '../components'; +import type { + NativeSyntaxHighlighterProps, + RNSheet, + RNStyle, + SyntaxHighlighterStylesheet, +} from '../types'; + +export const DEFAULT_FONT_SIZE = 13; + +export const generateNewStylesheet = ({ + stylesheet, + highlighter, +}: { + stylesheet: SyntaxHighlighterStylesheet; + highlighter: NativeSyntaxHighlighterProps['highlighter']; +}) => { + stylesheet = Array.isArray(stylesheet) ? stylesheet[0] : stylesheet; + + const transformedStyle = Object.entries(stylesheet ?? {}).reduce( + (newStylesheet, [className, style]) => { + const rn = cssToRNTextStyle(style); + + newStylesheet[className] = rn as RNStyle; + return newStylesheet; + }, + {}, + ); + + const topLevel = ( + highlighter === 'prism' ? transformedStyle['pre[class*="language-"]'] : transformedStyle.hljs + ) as RNStyle; + + const defaultColor = (topLevel && topLevel.color) || '#000'; + + return { transformedStyle, defaultColor }; +}; + +export const createChildren = ({ + stylesheet, + fontSize, + fontFamily, +}: { + stylesheet: SyntaxHighlighterStylesheet; + fontSize?: number; + fontFamily?: string; +}) => { + let childrenCount = 0; + return (children: rendererNode['children'], defaultColor: string) => { + childrenCount += 1; + return (children ?? []).map((child, i) => + createNativeElement({ + node: child, + stylesheet, + key: `code-segment-${childrenCount}-${i}`, + defaultColor, + fontSize, + fontFamily, + }), + ); + }; +}; + +export const createNativeElement = ({ + node, + stylesheet, + key, + defaultColor, + fontFamily, + fontSize = DEFAULT_FONT_SIZE, +}: { + node: rendererNode; + stylesheet: SyntaxHighlighterStylesheet; + key: string; + defaultColor: string; + fontFamily?: string; + fontSize?: number; +}) => { + const { properties, type, tagName: TagName, value } = node; + const startingStyle = { fontFamily, fontSize }; + if (type === 'text') { + return ( + + {value} + + ); + } else if (TagName) { + const childrenFactory = createChildren({ + stylesheet, + fontSize, + fontFamily, + }); + const style = properties + ? createStyleObject( + properties.className, + Object.assign({ color: defaultColor }, properties.style, startingStyle), + stylesheet, + ) + : {}; + const children = childrenFactory(node.children, style.color || defaultColor); + return ( + + {children} + + ); + } + return null; +}; + +export const nativeRenderer = ({ + defaultColor, + fontFamily, + fontSize = 13, +}: { + defaultColor: string; + fontFamily?: string; + fontSize?: number; +}): SyntaxHighlighterProps['renderer'] => { + return ({ rows, stylesheet }) => + Platform.OS === 'android' ? ( + + ) : ( + rows.map((node, i) => + createNativeElement({ + node, + stylesheet, + key: `code-segment-${i}`, + defaultColor, + fontFamily, + fontSize, + }), + ) + ); +}; diff --git a/package/src/ai-components/theme.ts b/package/src/ai-components/theme.ts new file mode 100644 index 0000000000..8d61874359 --- /dev/null +++ b/package/src/ai-components/theme.ts @@ -0,0 +1,138 @@ +import type { ImageStyle, TextStyle, ViewStyle } from 'react-native'; + +import type { MarkdownStyle } from './markdown'; + +export const colors = { + accent_blue: '#005FFF', + accent_red: '#FF3742', + black: '#000000', + code_block: '#DDDDDD', + grey: '#7A7A7A', + grey_neutral: '#9E9E9E', + grey_dark: '#72767E', + grey_gainsboro: '#DBDBDB', + grey_whisper: '#ECEBEB', + overlay: '#000000CC', + transparent: 'transparent', + white: '#FFFFFF', + white_smoke: '#F2F2F2', + shimmer: '#FFFFFF', +}; + +export type Theme = { + colors: typeof colors; + markdown: MarkdownStyle; + composer: { + container: ViewStyle; + containerRow: ViewStyle; + inputPillHeight: number; + inputPillContainer: ViewStyle; + inputPill: ViewStyle; + textInput: TextStyle; + roundButton: ViewStyle; + attachIcon: TextStyle; + mediaPreviewStyle: ViewStyle; + mediaPreviewContentContainerStyle: ViewStyle; + mediaPreviewImage: ImageStyle; + mediaPreviewRemoveButton: ViewStyle; + micIcon: ViewStyle; + sendIcon: ViewStyle; + stopGeneratingIcon: ViewStyle; + iconButton: ViewStyle; + }; + sheet: { + contentContainer: ViewStyle; + quickActionsCard: ViewStyle; + divider: ViewStyle; + listSection: ViewStyle; + quickAction: ViewStyle; + quickActionLabel: TextStyle; + listItem: ViewStyle; + listIcon: ViewStyle; + listTextContainer: ViewStyle; + listTitle: TextStyle; + listSubtitle: TextStyle; + }; + aiTypingIndicatorView: { + text: TextStyle; + }; +}; + +export const defaultTheme: Theme = { + colors, + markdown: { + autolink: {}, + blockQuoteSection: {}, + blockQuoteSectionBar: {}, + blockQuoteText: {}, + strong: {}, + codeBlock: {}, + codeBlockWrapper: {}, + codeBlockContainer: {}, + codeBlockHeaderContainer: {}, + codeBlockHeaderTitle: {}, + em: {}, + heading: {}, + heading1: {}, + heading2: {}, + heading3: {}, + heading4: {}, + heading5: {}, + heading6: {}, + hr: {}, + inlineCode: {}, + br: {}, + list: {}, + listItem: {}, + listItemBullet: {}, + listItemNumber: {}, + listItemText: {}, + listRow: {}, + sublist: {}, + url: {}, + noMargin: {}, + paragraph: {}, + paragraphCenter: {}, + del: {}, + text: {}, + table: {}, + tableRow: {}, + tableRowCell: {}, + tableHeader: {}, + tableHeaderCell: {}, + }, + composer: { + container: {}, + containerRow: {}, + inputPillHeight: 48, + inputPillContainer: {}, + inputPill: {}, + textInput: {}, + roundButton: {}, + attachIcon: {}, + mediaPreviewStyle: {}, + mediaPreviewContentContainerStyle: {}, + mediaPreviewImage: {}, + mediaPreviewRemoveButton: {}, + micIcon: {}, + sendIcon: {}, + stopGeneratingIcon: {}, + iconButton: {}, + }, + sheet: { + contentContainer: {}, + quickActionsCard: {}, + divider: {}, + listSection: {}, + quickAction: {}, + quickActionLabel: {}, + listItem: {}, + listIcon: {}, + listTextContainer: {}, + listTitle: {}, + listSubtitle: {}, + }, + aiTypingIndicatorView: { + text: {}, + }, +}; diff --git a/package/src/ai-components/transcription/useDictation.ts b/package/src/ai-components/transcription/useDictation.ts new file mode 100644 index 0000000000..eb3e3c9522 --- /dev/null +++ b/package/src/ai-components/transcription/useDictation.ts @@ -0,0 +1,106 @@ +import { useCallback, useEffect } from 'react'; +import { PermissionsAndroid, Platform } from 'react-native'; + +import { useStateStore } from '@stream-io/state-store/react-bindings'; + +import { useStableCallback } from '../internal/hooks/useStableCallback'; +import { AIDictation } from '../native/AIDictation'; +import type { DictationResult, DictationStartOptions } from '../native-specs/NativeAIDictation'; +import { dictationStore, type DictationStoreState } from '../store'; + +const DEFAULT_OPTIONS = { + language: 'en-US', + interimResults: true, + silenceTimeoutMs: 2500, +}; + +const ensureAndroidRecordingPermissions = async (): Promise => { + if (Platform.OS !== 'android') return true; + + const granted = await PermissionsAndroid.request(PermissionsAndroid.PERMISSIONS.RECORD_AUDIO); + + return granted === PermissionsAndroid.RESULTS.GRANTED; +}; + +const selector = ({ isRecording }: DictationStoreState) => ({ + isRecording, +}); + +// TODO: add proper jsdocs +export const useDictation = (defaultOptions: DictationStartOptions = DEFAULT_OPTIONS) => { + const { isRecording } = useStateStore(dictationStore, selector); + + useEffect(() => { + const unsubscribeResult = AIDictation.addResultListener((result: DictationResult) => { + dictationStore.partialNext({ + transcript: result.text, + ...(result.isFinal ? { isRecording: false, state: 'idle' } : {}), + }); + }); + + const unsubscribeState = AIDictation.addStateListener(({ state: next }) => { + dictationStore.partialNext({ + state: next, + isRecording: next === 'listening' || next === 'starting', + }); + }); + + const unsubscribeError = AIDictation.addErrorListener((error) => { + dictationStore.partialNext({ error, isRecording: false, state: 'idle' }); + }); + + return () => { + unsubscribeResult(); + unsubscribeState(); + unsubscribeError(); + }; + }, []); + + const start = useCallback( + async (override?: DictationStartOptions) => { + const hasPermission = await ensureAndroidRecordingPermissions(); + if (!hasPermission) { + dictationStore.partialNext({ + error: { + code: 'NO_PERMISSION', + message: 'Microphone permission not granted', + }, + }); + return; + } + + dictationStore.partialNext({ error: null, transcript: '' }); + + const opts: DictationStartOptions = { + ...defaultOptions, + ...override, + }; + + AIDictation.start(opts); + }, + [defaultOptions], + ); + + const stop = useCallback(() => { + AIDictation.stop(); + }, []); + + const cancel = useCallback(() => { + AIDictation.cancel(); + }, []); + + const toggle = useStableCallback(() => { + if (isRecording) { + stop(); + } else { + void start(); + } + }); + + return { + start, + stop, + toggle, + cancel, + }; +}; diff --git a/package/src/native.ts b/package/src/native.ts index b1a19e39f4..f039ca9c80 100644 --- a/package/src/native.ts +++ b/package/src/native.ts @@ -334,6 +334,18 @@ export type NativeShimmerViewProps = ViewProps & { gradientColor?: ColorValue; }; +/** + * Props of the Android-only native `PerfText` component used by `stream-chat-react-native/ai-components` + * to render syntax highlighted code as a single native text view with colour spans. + */ +export type NativePerfTextProps = ViewProps & { + text: string; + colorRanges?: ReadonlyArray<{ color: number; end: number; start: number }>; + fontFamily?: string; + fontSize?: number; + lineHeight?: number; +}; + type Handlers = { Audio?: AudioType; compressImage?: CompressImage; @@ -355,6 +367,7 @@ type Handlers = { shareImage?: ShareImage; Sound?: SoundType; NativeShimmerView?: React.ComponentType; + PerfText?: React.ComponentType; takePhoto?: TakePhoto; triggerHaptic?: TriggerHaptic; Video?: React.ComponentType; @@ -362,7 +375,7 @@ type Handlers = { export const NativeHandlers: Pick< Handlers, - 'Audio' | 'FlatList' | 'NativeShimmerView' | 'Video' | 'Sound' + 'Audio' | 'FlatList' | 'NativeShimmerView' | 'PerfText' | 'Video' | 'Sound' > & Required< Pick< @@ -401,6 +414,7 @@ export const NativeHandlers: Pick< shareImage: fail, Sound: undefined, NativeShimmerView: undefined, + PerfText: undefined, takePhoto: fail, triggerHaptic: fail, Video: undefined, @@ -478,6 +492,10 @@ export const registerNativeHandlers = (handlers: Handlers) => { NativeHandlers.NativeShimmerView = handlers.NativeShimmerView; } + if (handlers.PerfText !== undefined) { + NativeHandlers.PerfText = handlers.PerfText; + } + if (handlers.takePhoto !== undefined) { NativeHandlers.takePhoto = handlers.takePhoto; } diff --git a/yarn.lock b/yarn.lock index ba43028db5..4fe768d1b0 100644 --- a/yarn.lock +++ b/yarn.lock @@ -1546,6 +1546,13 @@ __metadata: languageName: node linkType: hard +"@babel/runtime@npm:^7.28.4": + version: 7.29.10 + resolution: "@babel/runtime@npm:7.29.10" + checksum: 10c0/32f975da8e9c2cae6b33e1a477f8fe8bbff6722c982b5ce35f99affaf3a7a1b612c8272b43afa0e51d3000928d6f3ba38f9296301115db5640264907c027b0cd + languageName: node + linkType: hard + "@babel/template@npm:^7.0.0, @babel/template@npm:^7.27.2, @babel/template@npm:^7.28.6, @babel/template@npm:^7.29.7, @babel/template@npm:^7.3.3": version: 7.29.7 resolution: "@babel/template@npm:7.29.7" @@ -3917,6 +3924,25 @@ __metadata: languageName: node linkType: hard +"@khanacademy/perseus-utils@npm:2.1.5": + version: 2.1.5 + resolution: "@khanacademy/perseus-utils@npm:2.1.5" + checksum: 10c0/d6d511f546024f2e2468a27cb57883706ac599b3855bb02abb8d46fbd67be12956a2571dc95bea1cf6c432f4c5246c1f0d241d47729a5053a2d4522e6619ae5f + languageName: node + linkType: hard + +"@khanacademy/simple-markdown@npm:^2.1.0": + version: 2.2.3 + resolution: "@khanacademy/simple-markdown@npm:2.2.3" + dependencies: + "@khanacademy/perseus-utils": "npm:2.1.5" + peerDependencies: + react: ^18.2.0 + react-dom: ^18.2.0 + checksum: 10c0/cdcee61e95f6e01df298d265edb882a76248694cc24cdd1d55f335083ac4d3508579f3d90710e316e8c7926baff1a69187063d6ffaa62528f5162763c98103bd + languageName: node + linkType: hard + "@napi-rs/wasm-runtime@npm:^0.2.11": version: 0.2.12 resolution: "@napi-rs/wasm-runtime@npm:0.2.12" @@ -5940,6 +5966,34 @@ __metadata: languageName: node linkType: hard +"@shopify/react-native-skia@npm:^2.13.0": + version: 2.14.0 + resolution: "@shopify/react-native-skia@npm:2.14.0" + dependencies: + canvaskit-wasm: "npm:0.41.0" + react-native-skia-android: "npm:154.0.0" + react-native-skia-apple-ios: "npm:154.0.0" + react-native-skia-apple-macos: "npm:154.0.0" + react-native-skia-apple-tvos: "npm:154.0.0" + react-reconciler: "npm:0.31.0" + peerDependencies: + react: ">=19.0" + react-native: ">=0.78" + react-native-reanimated: ">=4.0.0" + react-native-worklets: ">=0.7.0" + peerDependenciesMeta: + react-native: + optional: true + react-native-reanimated: + optional: true + react-native-worklets: + optional: true + bin: + setup-skia-web: scripts/setup-canvaskit.js + checksum: 10c0/1e2741835bb6f6506a42040d48541f72905081c4ac14b4ac210d120d163bd47edad5a8ec077c9dabb38f9148484a86f97a68996c997e5e5b6dc143a5e4b5b284 + languageName: node + linkType: hard + "@sideway/address@npm:^4.1.5": version: 4.1.5 resolution: "@sideway/address@npm:4.1.5" @@ -6092,6 +6146,21 @@ __metadata: languageName: node linkType: hard +"@stream-io/state-store@npm:^1.1.6": + version: 1.1.6 + resolution: "@stream-io/state-store@npm:1.1.6" + peerDependencies: + react: ^17 || ^18 || ^19 + use-sync-external-store: ^1.5.0 + peerDependenciesMeta: + react: + optional: true + use-sync-external-store: + optional: true + checksum: 10c0/42927cdb6a5bf6ba5f4ec11bb5e530db0a04edc65abd5348eb423780e40a3e53f57125e328d0318a0dc6b24ca0fcb28e8b91895a14a62ed8d82645f67ebcee5a + languageName: node + linkType: hard + "@stream-io/typescript-config@workspace:^, @stream-io/typescript-config@workspace:configs/typescript-config": version: 0.0.0-use.local resolution: "@stream-io/typescript-config@workspace:configs/typescript-config" @@ -6433,6 +6502,15 @@ __metadata: languageName: node linkType: hard +"@types/hast@npm:^3.0.0": + version: 3.0.5 + resolution: "@types/hast@npm:3.0.5" + dependencies: + "@types/unist": "npm:*" + checksum: 10c0/f3af8594a6903a507ed191eda944af18099198d6708c29102ae17118c3e20779f6929e91ed37e033541fd28d58055d8a5910a4366dbdf976cf81b13a464741fb + languageName: node + linkType: hard + "@types/istanbul-lib-coverage@npm:*, @types/istanbul-lib-coverage@npm:^2.0.0, @types/istanbul-lib-coverage@npm:^2.0.1, @types/istanbul-lib-coverage@npm:^2.0.6": version: 2.0.6 resolution: "@types/istanbul-lib-coverage@npm:2.0.6" @@ -6561,6 +6639,31 @@ __metadata: languageName: node linkType: hard +"@types/prismjs@npm:^1.0.0": + version: 1.26.6 + resolution: "@types/prismjs@npm:1.26.6" + checksum: 10c0/152a27500cb32b114edfb77f9d0dccd03bebc84828d1e92abacaf212b22d3ccdde041ce421dd58b6ec8461bbec7cd76ed5ee773cae4be7ca36a6dd4ddcf0f9e7 + languageName: node + linkType: hard + +"@types/react-reconciler@npm:^0.28.0": + version: 0.28.9 + resolution: "@types/react-reconciler@npm:0.28.9" + peerDependencies: + "@types/react": "*" + checksum: 10c0/9fe71fa856aab2cd4742fe0416bdd4f5c82ecc05ef6451ee7fcb65a5efdf5fa588f5820fbe2a665b15371b0da3bfc4097f28bb6d450b9a834af2d0fc00f403bd + languageName: node + linkType: hard + +"@types/react-syntax-highlighter@npm:^15.5.13": + version: 15.5.13 + resolution: "@types/react-syntax-highlighter@npm:15.5.13" + dependencies: + "@types/react": "npm:*" + checksum: 10c0/e3bca325b27519fb063d3370de20d311c188ec16ffc01e5bc77bdf2d7320756725ee3d0246922cd5d38b75c5065a1bc43d0194e92ecf6556818714b4ffb0967a + languageName: node + linkType: hard + "@types/react-test-renderer@npm:19.1.0, @types/react-test-renderer@npm:^19.1.0": version: 19.1.0 resolution: "@types/react-test-renderer@npm:19.1.0" @@ -6593,7 +6696,14 @@ __metadata: languageName: node linkType: hard -"@types/unist@npm:^2, @types/unist@npm:^2.0.2": +"@types/unist@npm:*": + version: 3.0.3 + resolution: "@types/unist@npm:3.0.3" + checksum: 10c0/2b1e4adcab78388e088fcc3c0ae8700f76619dbcb4741d7d201f87e2cb346bfc29a89003cfea2d76c996e1061452e14fcd737e8b25aacf949c1f2d6b2bc3dd60 + languageName: node + linkType: hard + +"@types/unist@npm:^2, @types/unist@npm:^2.0.0, @types/unist@npm:^2.0.2": version: 2.0.11 resolution: "@types/unist@npm:2.0.11" checksum: 10c0/24dcdf25a168f453bb70298145eb043cfdbb82472db0bc0b56d6d51cd2e484b9ed8271d4ac93000a80da568f2402e9339723db262d0869e2bf13bc58e081768d @@ -6923,6 +7033,13 @@ __metadata: languageName: node linkType: hard +"@webgpu/types@npm:0.1.21": + version: 0.1.21 + resolution: "@webgpu/types@npm:0.1.21" + checksum: 10c0/74de9683b70064aeadcdd31381747650b323e7e2183fb38f90a35f1a79f80ad267c0a2ee5010c264befb3d31ad213dbdffa04897e2808f6b04feff8d66bc53e0 + languageName: node + linkType: hard + "@xmldom/xmldom@npm:^0.8.8": version: 0.8.13 resolution: "@xmldom/xmldom@npm:0.8.13" @@ -8217,6 +8334,15 @@ __metadata: languageName: node linkType: hard +"canvaskit-wasm@npm:0.41.0": + version: 0.41.0 + resolution: "canvaskit-wasm@npm:0.41.0" + dependencies: + "@webgpu/types": "npm:0.1.21" + checksum: 10c0/a55eb2cb627f2cdf017e05dba794c35225725af13c223c573ffb04f9f783d16e6c80fa0eb21a712c7b0a56f29dc0a693aee1eee61fe75e2b74e821a10236d8ef + languageName: node + linkType: hard + "chalk@npm:^2.0.1, chalk@npm:^2.3.2, chalk@npm:^2.4.2": version: 2.4.2 resolution: "chalk@npm:2.4.2" @@ -8259,6 +8385,13 @@ __metadata: languageName: node linkType: hard +"character-entities-legacy@npm:^3.0.0": + version: 3.0.0 + resolution: "character-entities-legacy@npm:3.0.0" + checksum: 10c0/ec4b430af873661aa754a896a2b55af089b4e938d3d010fad5219299a6b6d32ab175142699ee250640678cd64bdecd6db3c9af0b8759ab7b155d970d84c4c7d1 + languageName: node + linkType: hard + "character-entities@npm:^1.0.0": version: 1.2.4 resolution: "character-entities@npm:1.2.4" @@ -8266,6 +8399,13 @@ __metadata: languageName: node linkType: hard +"character-entities@npm:^2.0.0": + version: 2.0.2 + resolution: "character-entities@npm:2.0.2" + checksum: 10c0/b0c645a45bcc90ff24f0e0140f4875a8436b8ef13b6bcd31ec02cfb2ca502b680362aa95386f7815bdc04b6464d48cf191210b3840d7c04241a149ede591a308 + languageName: node + linkType: hard + "character-reference-invalid@npm:^1.0.0": version: 1.1.4 resolution: "character-reference-invalid@npm:1.1.4" @@ -8273,6 +8413,13 @@ __metadata: languageName: node linkType: hard +"character-reference-invalid@npm:^2.0.0": + version: 2.0.1 + resolution: "character-reference-invalid@npm:2.0.1" + checksum: 10c0/2ae0dec770cd8659d7e8b0ce24392d83b4c2f0eb4a3395c955dce5528edd4cc030a794cfa06600fcdd700b3f2de2f9b8e40e309c0011c4180e3be64a0b42e6a1 + languageName: node + linkType: hard + "chardet@npm:^2.1.1": version: 2.1.1 resolution: "chardet@npm:2.1.1" @@ -8617,6 +8764,13 @@ __metadata: languageName: node linkType: hard +"comma-separated-tokens@npm:^2.0.0": + version: 2.0.3 + resolution: "comma-separated-tokens@npm:2.0.3" + checksum: 10c0/91f90f1aae320f1755d6957ef0b864fe4f54737f3313bd95e0802686ee2ca38bff1dd381964d00ae5db42912dd1f4ae5c2709644e82706ffc6f6842a813cdd67 + languageName: node + linkType: hard + "command-exists@npm:^1.2.8": version: 1.2.9 resolution: "command-exists@npm:1.2.9" @@ -8950,6 +9104,151 @@ __metadata: languageName: node linkType: hard +"d3-array@npm:2 - 3, d3-array@npm:2.10.0 - 3": + version: 3.2.4 + resolution: "d3-array@npm:3.2.4" + dependencies: + internmap: "npm:1 - 2" + checksum: 10c0/08b95e91130f98c1375db0e0af718f4371ccacef7d5d257727fe74f79a24383e79aba280b9ffae655483ffbbad4fd1dec4ade0119d88c4749f388641c8bf8c50 + languageName: node + linkType: hard + +"d3-color@npm:1 - 3": + version: 3.1.0 + resolution: "d3-color@npm:3.1.0" + checksum: 10c0/a4e20e1115fa696fce041fbe13fbc80dc4c19150fa72027a7c128ade980bc0eeeba4bcf28c9e21f0bce0e0dbfe7ca5869ef67746541dcfda053e4802ad19783c + languageName: node + linkType: hard + +"d3-dispatch@npm:1 - 3": + version: 3.0.1 + resolution: "d3-dispatch@npm:3.0.1" + checksum: 10c0/6eca77008ce2dc33380e45d4410c67d150941df7ab45b91d116dbe6d0a3092c0f6ac184dd4602c796dc9e790222bad3ff7142025f5fd22694efe088d1d941753 + languageName: node + linkType: hard + +"d3-drag@npm:2 - 3": + version: 3.0.0 + resolution: "d3-drag@npm:3.0.0" + dependencies: + d3-dispatch: "npm:1 - 3" + d3-selection: "npm:3" + checksum: 10c0/d2556e8dc720741a443b595a30af403dd60642dfd938d44d6e9bfc4c71a962142f9a028c56b61f8b4790b65a34acad177d1263d66f103c3c527767b0926ef5aa + languageName: node + linkType: hard + +"d3-ease@npm:1 - 3": + version: 3.0.1 + resolution: "d3-ease@npm:3.0.1" + checksum: 10c0/fec8ef826c0cc35cda3092c6841e07672868b1839fcaf556e19266a3a37e6bc7977d8298c0fcb9885e7799bfdcef7db1baaba9cd4dcf4bc5e952cf78574a88b0 + languageName: node + linkType: hard + +"d3-format@npm:1 - 3": + version: 3.1.2 + resolution: "d3-format@npm:3.1.2" + checksum: 10c0/0de452ae07585238e7f01607a7e0066665c34609652188b6ac7dc9f424f69465a425e07d16d79bd0e5955202ac7f241c66d0c76f68a79fc6f4857c94cf420652 + languageName: node + linkType: hard + +"d3-interpolate@npm:1 - 3, d3-interpolate@npm:1.2.0 - 3": + version: 3.0.1 + resolution: "d3-interpolate@npm:3.0.1" + dependencies: + d3-color: "npm:1 - 3" + checksum: 10c0/19f4b4daa8d733906671afff7767c19488f51a43d251f8b7f484d5d3cfc36c663f0a66c38fe91eee30f40327443d799be17169f55a293a3ba949e84e57a33e6a + languageName: node + linkType: hard + +"d3-path@npm:^3.1.0": + version: 3.1.0 + resolution: "d3-path@npm:3.1.0" + checksum: 10c0/dc1d58ec87fa8319bd240cf7689995111a124b141428354e9637aa83059eb12e681f77187e0ada5dedfce346f7e3d1f903467ceb41b379bfd01cd8e31721f5da + languageName: node + linkType: hard + +"d3-scale@npm:^4.0.2": + version: 4.0.2 + resolution: "d3-scale@npm:4.0.2" + dependencies: + d3-array: "npm:2.10.0 - 3" + d3-format: "npm:1 - 3" + d3-interpolate: "npm:1.2.0 - 3" + d3-time: "npm:2.1.1 - 3" + d3-time-format: "npm:2 - 4" + checksum: 10c0/65d9ad8c2641aec30ed5673a7410feb187a224d6ca8d1a520d68a7d6eac9d04caedbff4713d1e8545be33eb7fec5739983a7ab1d22d4e5ad35368c6729d362f1 + languageName: node + linkType: hard + +"d3-selection@npm:2 - 3, d3-selection@npm:3": + version: 3.0.0 + resolution: "d3-selection@npm:3.0.0" + checksum: 10c0/e59096bbe8f0cb0daa1001d9bdd6dbc93a688019abc97d1d8b37f85cd3c286a6875b22adea0931b0c88410d025563e1643019161a883c516acf50c190a11b56b + languageName: node + linkType: hard + +"d3-shape@npm:^3.2.0": + version: 3.2.0 + resolution: "d3-shape@npm:3.2.0" + dependencies: + d3-path: "npm:^3.1.0" + checksum: 10c0/f1c9d1f09926daaf6f6193ae3b4c4b5521e81da7d8902d24b38694517c7f527ce3c9a77a9d3a5722ad1e3ff355860b014557b450023d66a944eabf8cfde37132 + languageName: node + linkType: hard + +"d3-time-format@npm:2 - 4": + version: 4.1.0 + resolution: "d3-time-format@npm:4.1.0" + dependencies: + d3-time: "npm:1 - 3" + checksum: 10c0/735e00fb25a7fd5d418fac350018713ae394eefddb0d745fab12bbff0517f9cdb5f807c7bbe87bb6eeb06249662f8ea84fec075f7d0cd68609735b2ceb29d206 + languageName: node + linkType: hard + +"d3-time@npm:1 - 3, d3-time@npm:2.1.1 - 3": + version: 3.1.0 + resolution: "d3-time@npm:3.1.0" + dependencies: + d3-array: "npm:2 - 3" + checksum: 10c0/a984f77e1aaeaa182679b46fbf57eceb6ebdb5f67d7578d6f68ef933f8eeb63737c0949991618a8d29472dbf43736c7d7f17c452b2770f8c1271191cba724ca1 + languageName: node + linkType: hard + +"d3-timer@npm:1 - 3": + version: 3.0.1 + resolution: "d3-timer@npm:3.0.1" + checksum: 10c0/d4c63cb4bb5461d7038aac561b097cd1c5673969b27cbdd0e87fa48d9300a538b9e6f39b4a7f0e3592ef4f963d858c8a9f0e92754db73116770856f2fc04561a + languageName: node + linkType: hard + +"d3-transition@npm:2 - 3": + version: 3.0.1 + resolution: "d3-transition@npm:3.0.1" + dependencies: + d3-color: "npm:1 - 3" + d3-dispatch: "npm:1 - 3" + d3-ease: "npm:1 - 3" + d3-interpolate: "npm:1 - 3" + d3-timer: "npm:1 - 3" + peerDependencies: + d3-selection: 2 - 3 + checksum: 10c0/4e74535dda7024aa43e141635b7522bb70cf9d3dfefed975eb643b36b864762eca67f88fafc2ca798174f83ca7c8a65e892624f824b3f65b8145c6a1a88dbbad + languageName: node + linkType: hard + +"d3-zoom@npm:^3.0.0": + version: 3.0.0 + resolution: "d3-zoom@npm:3.0.0" + dependencies: + d3-dispatch: "npm:1 - 3" + d3-drag: "npm:2 - 3" + d3-interpolate: "npm:1 - 3" + d3-selection: "npm:2 - 3" + d3-transition: "npm:2 - 3" + checksum: 10c0/ee2036479049e70d8c783d594c444fe00e398246048e3f11a59755cd0e21de62ece3126181b0d7a31bf37bcf32fd726f83ae7dea4495ff86ec7736ce5ad36fd3 + languageName: node + linkType: hard + "d@npm:1, d@npm:^1.0.1, d@npm:^1.0.2": version: 1.0.2 resolution: "d@npm:1.0.2" @@ -9044,6 +9343,15 @@ __metadata: languageName: node linkType: hard +"decode-named-character-reference@npm:^1.0.0": + version: 1.3.0 + resolution: "decode-named-character-reference@npm:1.3.0" + dependencies: + character-entities: "npm:^2.0.0" + checksum: 10c0/787f4c87f3b82ea342aa7c2d7b1882b6fb9511bb77f72ae44dcaabea0470bacd1e9c6a0080ab886545019fa0cb3a7109573fad6b61a362844c3a0ac52b36e4bb + languageName: node + linkType: hard + "decode-uri-component@npm:^0.2.2": version: 0.2.2 resolution: "decode-uri-component@npm:0.2.2" @@ -10623,6 +10931,40 @@ __metadata: languageName: node linkType: hard +"expochatgptsample@workspace:examples/ExpoChatGPTSample": + version: 0.0.0-use.local + resolution: "expochatgptsample@workspace:examples/ExpoChatGPTSample" + dependencies: + "@babel/plugin-transform-export-namespace-from": "npm:^7.27.1" + "@react-native-community/netinfo": "npm:12.0.1" + "@rnx-kit/metro-config": "npm:^2.1.2" + "@shopify/react-native-skia": "npm:^2.13.0" + "@stream-io/typescript-config": "workspace:^" + "@types/react": "npm:~19.2.10" + expo: "npm:^57.0.7" + expo-clipboard: "npm:~57.0.1" + expo-constants: "npm:~57.0.6" + expo-image-manipulator: "npm:~57.0.5" + expo-image-picker: "npm:~57.0.5" + expo-linking: "npm:~57.0.3" + expo-router: "npm:~57.0.7" + react: "npm:19.2.3" + react-native: "npm:0.86.0" + react-native-gesture-handler: "npm:~2.32.0" + react-native-reanimated: "npm:4.5.2" + react-native-safe-area-context: "npm:~5.7.0" + react-native-screens: "npm:4.26.2" + react-native-svg: "npm:15.15.4" + react-native-teleport: "npm:^1.1.12" + react-native-worklets: "npm:0.11.1" + stream-chat: "npm:^9.54.0" + stream-chat-expo: "workspace:^" + stream-chat-react-native-core: "workspace:^" + typescript: "npm:6.0.3" + victory-native: "npm:^42.0.1" + languageName: unknown + linkType: soft + "exponential-backoff@npm:^3.1.1": version: 3.1.1 resolution: "exponential-backoff@npm:3.1.1" @@ -10761,6 +11103,15 @@ __metadata: languageName: node linkType: hard +"fault@npm:^1.0.0": + version: 1.0.4 + resolution: "fault@npm:1.0.4" + dependencies: + format: "npm:^0.2.0" + checksum: 10c0/c86c11500c1b676787296f31ade8473adcc6784f118f07c1a9429730b6288d0412f96e069ce010aa57e4f65a9cccb5abee8868bbe3c5f10de63b20482c9baebd + languageName: node + linkType: hard + "faye-websocket@npm:0.11.4": version: 0.11.4 resolution: "faye-websocket@npm:0.11.4" @@ -11067,6 +11418,13 @@ __metadata: languageName: node linkType: hard +"format@npm:^0.2.0": + version: 0.2.2 + resolution: "format@npm:0.2.2" + checksum: 10c0/6032ba747541a43abf3e37b402b2f72ee08ebcb58bf84d816443dd228959837f1cddf1e8775b29fa27ff133f4bd146d041bfca5f9cf27f048edf3d493cf8fee6 + languageName: node + linkType: hard + "fresh@npm:~0.5.2": version: 0.5.2 resolution: "fresh@npm:0.5.2" @@ -11601,6 +11959,28 @@ __metadata: languageName: node linkType: hard +"hast-util-parse-selector@npm:^4.0.0": + version: 4.0.0 + resolution: "hast-util-parse-selector@npm:4.0.0" + dependencies: + "@types/hast": "npm:^3.0.0" + checksum: 10c0/5e98168cb44470dc274aabf1a28317e4feb09b1eaf7a48bbaa8c1de1b43a89cd195cb1284e535698e658e3ec26ad91bc5e52c9563c36feb75abbc68aaf68fb9f + languageName: node + linkType: hard + +"hastscript@npm:^9.0.0": + version: 9.0.1 + resolution: "hastscript@npm:9.0.1" + dependencies: + "@types/hast": "npm:^3.0.0" + comma-separated-tokens: "npm:^2.0.0" + hast-util-parse-selector: "npm:^4.0.0" + property-information: "npm:^7.0.0" + space-separated-tokens: "npm:^2.0.0" + checksum: 10c0/18dc8064e5c3a7a2ae862978e626b97a254e1c8a67ee9d0c9f06d373bba155ed805fc5b5ce21b990fb7bc174624889e5e1ce1cade264f1b1d58b48f994bc85ce + languageName: node + linkType: hard + "hermes-compiler@npm:250829098.0.14": version: 250829098.0.14 resolution: "hermes-compiler@npm:250829098.0.14" @@ -11711,13 +12091,20 @@ __metadata: languageName: node linkType: hard -"highlight.js@npm:^10.7.1": +"highlight.js@npm:^10.4.1, highlight.js@npm:^10.7.1, highlight.js@npm:~10.7.0": version: 10.7.3 resolution: "highlight.js@npm:10.7.3" checksum: 10c0/073837eaf816922427a9005c56c42ad8786473dc042332dfe7901aa065e92bc3d94ebf704975257526482066abb2c8677cc0326559bb8621e046c21c5991c434 languageName: node linkType: hard +"highlightjs-vue@npm:^1.0.0": + version: 1.0.0 + resolution: "highlightjs-vue@npm:1.0.0" + checksum: 10c0/9be378c70b864ca5eee87b07859222e31c946a8ad176227e54f7006a498223974ebe19fcce6e38ad5eb3c1ed0e16a580c4edefdf2cb882b6dfab1c3866cc047a + languageName: node + linkType: hard + "hoist-non-react-statics@npm:^3.3.0": version: 3.3.2 resolution: "hoist-non-react-statics@npm:3.3.2" @@ -12175,6 +12562,13 @@ __metadata: languageName: node linkType: hard +"internmap@npm:1 - 2": + version: 2.0.3 + resolution: "internmap@npm:2.0.3" + checksum: 10c0/8cedd57f07bbc22501516fbfc70447f0c6812871d471096fad9ea603516eacc2137b633633daf432c029712df0baefd793686388ddf5737e3ea15074b877f7ed + languageName: node + linkType: hard + "intl-pluralrules@npm:^2.0.1": version: 2.0.1 resolution: "intl-pluralrules@npm:2.0.1" @@ -12232,6 +12626,13 @@ __metadata: languageName: node linkType: hard +"is-alphabetical@npm:^2.0.0": + version: 2.0.1 + resolution: "is-alphabetical@npm:2.0.1" + checksum: 10c0/932367456f17237533fd1fc9fe179df77957271020b83ea31da50e5cc472d35ef6b5fb8147453274ffd251134472ce24eb6f8d8398d96dee98237cdb81a6c9a7 + languageName: node + linkType: hard + "is-alphanumerical@npm:^1.0.0": version: 1.0.4 resolution: "is-alphanumerical@npm:1.0.4" @@ -12242,6 +12643,16 @@ __metadata: languageName: node linkType: hard +"is-alphanumerical@npm:^2.0.0": + version: 2.0.1 + resolution: "is-alphanumerical@npm:2.0.1" + dependencies: + is-alphabetical: "npm:^2.0.0" + is-decimal: "npm:^2.0.0" + checksum: 10c0/4b35c42b18e40d41378293f82a3ecd9de77049b476f748db5697c297f686e1e05b072a6aaae2d16f54d2a57f85b00cbbe755c75f6d583d1c77d6657bd0feb5a2 + languageName: node + linkType: hard + "is-array-buffer@npm:^3.0.4, is-array-buffer@npm:^3.0.5": version: 3.0.5 resolution: "is-array-buffer@npm:3.0.5" @@ -12352,6 +12763,13 @@ __metadata: languageName: node linkType: hard +"is-decimal@npm:^2.0.0": + version: 2.0.1 + resolution: "is-decimal@npm:2.0.1" + checksum: 10c0/8085dd66f7d82f9de818fba48b9e9c0429cb4291824e6c5f2622e96b9680b54a07a624cfc663b24148b8e853c62a1c987cfe8b0b5a13f5156991afaf6736e334 + languageName: node + linkType: hard + "is-directory@npm:^0.3.1": version: 0.3.1 resolution: "is-directory@npm:0.3.1" @@ -12454,6 +12872,13 @@ __metadata: languageName: node linkType: hard +"is-hexadecimal@npm:^2.0.0": + version: 2.0.1 + resolution: "is-hexadecimal@npm:2.0.1" + checksum: 10c0/3eb60fe2f1e2bbc760b927dcad4d51eaa0c60138cf7fc671803f66353ad90c301605b502c7ea4c6bb0548e1c7e79dfd37b73b632652e3b76030bba603a7e9626 + languageName: node + linkType: hard + "is-interactive@npm:^1.0.0": version: 1.0.0 resolution: "is-interactive@npm:1.0.0" @@ -12830,6 +13255,17 @@ __metadata: languageName: node linkType: hard +"its-fine@npm:^1.2.5": + version: 1.2.5 + resolution: "its-fine@npm:1.2.5" + dependencies: + "@types/react-reconciler": "npm:^0.28.0" + peerDependencies: + react: ">=18.0" + checksum: 10c0/5618955ac7ae0d7788580186f91ac9989301ead2c87250883258cd7ee32f9a4e08574728506d1deb90477090d7a688ba19571126aae9fb25661e90057dd772df + languageName: node + linkType: hard + "jackspeak@npm:^3.1.2": version: 3.4.3 resolution: "jackspeak@npm:3.4.3" @@ -14241,7 +14677,7 @@ __metadata: languageName: node linkType: hard -"lodash@npm:^4.17.15, lodash@npm:^4.17.4": +"lodash@npm:4.18.1, lodash@npm:^4.17.15, lodash@npm:^4.17.4": version: 4.18.1 resolution: "lodash@npm:4.18.1" checksum: 10c0/757228fc68805c59789e82185135cf85f05d0b2d3d54631d680ca79ec21944ec8314d4533639a14b8bcfbd97a517e78960933041a5af17ecb693ec6eecb99a27 @@ -14308,6 +14744,16 @@ __metadata: languageName: node linkType: hard +"lowlight@npm:^1.17.0": + version: 1.20.0 + resolution: "lowlight@npm:1.20.0" + dependencies: + fault: "npm:^1.0.0" + highlight.js: "npm:~10.7.0" + checksum: 10c0/728bce6f6fe8b157f48d3324e597f452ce0eed2ccff1c0f41a9047380f944e971eb45bceb31f08fbb64d8f338dabb166f10049b35b92c7ec5cf0241d6adb3dea + languageName: node + linkType: hard + "lru-cache@npm:^10.0.1, lru-cache@npm:^10.2.0": version: 10.4.3 resolution: "lru-cache@npm:10.4.3" @@ -16458,6 +16904,21 @@ __metadata: languageName: node linkType: hard +"parse-entities@npm:^4.0.0": + version: 4.0.2 + resolution: "parse-entities@npm:4.0.2" + dependencies: + "@types/unist": "npm:^2.0.0" + character-entities-legacy: "npm:^3.0.0" + character-reference-invalid: "npm:^2.0.0" + decode-named-character-reference: "npm:^1.0.0" + is-alphanumerical: "npm:^2.0.0" + is-decimal: "npm:^2.0.0" + is-hexadecimal: "npm:^2.0.0" + checksum: 10c0/a13906b1151750b78ed83d386294066daf5fb559e08c5af9591b2d98cc209123103016a01df776f65f8219ad26652d6d6b210d0974d452049cddfc53a8916c34 + languageName: node + linkType: hard + "parse-json@npm:^4.0.0": version: 4.0.0 resolution: "parse-json@npm:4.0.0" @@ -16818,6 +17279,13 @@ __metadata: languageName: node linkType: hard +"prismjs@npm:^1.30.0": + version: 1.30.0 + resolution: "prismjs@npm:1.30.0" + checksum: 10c0/f56205bfd58ef71ccfcbcb691fd0eb84adc96c6ff21b0b69fc6fdcf02be42d6ef972ba4aed60466310de3d67733f6a746f89f2fb79c00bf217406d465b3e8f23 + languageName: node + linkType: hard + "proc-log@npm:^4.0.0": version: 4.2.0 resolution: "proc-log@npm:4.2.0" @@ -16939,6 +17407,13 @@ __metadata: languageName: node linkType: hard +"property-information@npm:^7.0.0": + version: 7.2.0 + resolution: "property-information@npm:7.2.0" + checksum: 10c0/03662c8f9e1544510914c5e594ae72963f67d261027a5fdc06c4134742584fe40dd49b959470d30e757e9fb70b297d8546ab1362a040364144029926ad4e07c4 + languageName: node + linkType: hard + "proto-list@npm:~1.2.1": version: 1.2.4 resolution: "proto-list@npm:1.2.4" @@ -17496,6 +17971,34 @@ __metadata: languageName: node linkType: hard +"react-native-skia-android@npm:154.0.0": + version: 154.0.0 + resolution: "react-native-skia-android@npm:154.0.0" + checksum: 10c0/e0968826abfbc4e5513bd2b97cd4fa1101ff90c5abbb3d7600517939cf34a70b2f2ace3257dc7753674f76f6245c472a497ba5e4f93ecbb3e80edca3c042dd83 + languageName: node + linkType: hard + +"react-native-skia-apple-ios@npm:154.0.0": + version: 154.0.0 + resolution: "react-native-skia-apple-ios@npm:154.0.0" + checksum: 10c0/bf0a833660f0f46cfb34cd72aac9960370bf120acac7d27c141ee48fdaf33e79a15939bc03de4bec50c7baa61ba8c6f345497032247bab5facabd0b046ef5079 + languageName: node + linkType: hard + +"react-native-skia-apple-macos@npm:154.0.0": + version: 154.0.0 + resolution: "react-native-skia-apple-macos@npm:154.0.0" + checksum: 10c0/f5b87ce78ad8665d12e28e9b2552dba7d28eb1a8b764610944808555d7583e5e60e3853ff6ea5fae0ae4c79088f48222fc4c52d8104b5cb6fbb03ae9e8281346 + languageName: node + linkType: hard + +"react-native-skia-apple-tvos@npm:154.0.0": + version: 154.0.0 + resolution: "react-native-skia-apple-tvos@npm:154.0.0" + checksum: 10c0/5b728e46fbbf6295c0cc50c1e1261a609a9f6036da64c0581ec2ae1528b4052a24240bad8dea53cf092eaee0d541f154d5cb6f1bb4242b7a1b05829e8820ecba + languageName: node + linkType: hard + "react-native-svg@npm:15.15.4": version: 15.15.4 resolution: "react-native-svg@npm:15.15.4" @@ -17725,6 +18228,17 @@ __metadata: languageName: node linkType: hard +"react-reconciler@npm:0.31.0": + version: 0.31.0 + resolution: "react-reconciler@npm:0.31.0" + dependencies: + scheduler: "npm:^0.25.0" + peerDependencies: + react: ^19.0.0 + checksum: 10c0/97920e1866c7206e200c3920c133c2e85f62a3c54fd9bc4b83c10c558d83d98eb378caab4fe37498e0cc1b1b2665d898627f2ae2537b29c8ab295ec8abc0c580 + languageName: node + linkType: hard + "react-refresh@npm:^0.14.0, react-refresh@npm:^0.14.2": version: 0.14.2 resolution: "react-refresh@npm:0.14.2" @@ -17783,6 +18297,22 @@ __metadata: languageName: node linkType: hard +"react-syntax-highlighter@npm:^16.1.0": + version: 16.1.2 + resolution: "react-syntax-highlighter@npm:16.1.2" + dependencies: + "@babel/runtime": "npm:^7.28.4" + highlight.js: "npm:^10.4.1" + highlightjs-vue: "npm:^1.0.0" + lowlight: "npm:^1.17.0" + prismjs: "npm:^1.30.0" + refractor: "npm:^5.0.0" + peerDependencies: + react: ">= 0.14.0" + checksum: 10c0/88e44964f202def4e086c8e5d455e53cce8f32f88f977ac5bad76a82054b5e308fc0f3cd5c5d85221623c287756f9d0ef169042570d0abdd7d7919341a9a0d24 + languageName: node + linkType: hard + "react-test-renderer@npm:19.2.3": version: 19.2.3 resolution: "react-test-renderer@npm:19.2.3" @@ -17942,6 +18472,18 @@ __metadata: languageName: node linkType: hard +"refractor@npm:^5.0.0": + version: 5.0.0 + resolution: "refractor@npm:5.0.0" + dependencies: + "@types/hast": "npm:^3.0.0" + "@types/prismjs": "npm:^1.0.0" + hastscript: "npm:^9.0.0" + parse-entities: "npm:^4.0.0" + checksum: 10c0/7b69a3b525f88de4adae3467f30ff7e80513fa53e65d85656f169ded16e314f471dfa936ccc545ae604e178ee7a108cbd94e6144a5bd541ed8d20626cc75e7b8 + languageName: node + linkType: hard + "regenerate-unicode-properties@npm:^10.2.2": version: 10.2.2 resolution: "regenerate-unicode-properties@npm:10.2.2" @@ -18411,6 +18953,13 @@ __metadata: languageName: node linkType: hard +"scheduler@npm:^0.25.0": + version: 0.25.0 + resolution: "scheduler@npm:0.25.0" + checksum: 10c0/a4bb1da406b613ce72c1299db43759526058fdcc413999c3c3e0db8956df7633acf395cb20eb2303b6a65d658d66b6585d344460abaee8080b4aa931f10eaafe + languageName: node + linkType: hard + "semantic-release@npm:^25.0.3": version: 25.0.3 resolution: "semantic-release@npm:25.0.3" @@ -18873,6 +19422,13 @@ __metadata: languageName: node linkType: hard +"space-separated-tokens@npm:^2.0.0": + version: 2.0.2 + resolution: "space-separated-tokens@npm:2.0.2" + checksum: 10c0/6173e1d903dca41dcab6a2deed8b4caf61bd13b6d7af8374713500570aa929ff9414ae09a0519f4f8772df993300305a395d4871f35bc4ca72b6db57e1f30af8 + languageName: node + linkType: hard + "spawn-error-forwarder@npm:~1.0.0": version: 1.0.0 resolution: "spawn-error-forwarder@npm:1.0.0" @@ -19047,6 +19603,7 @@ __metadata: stream-chat-react-native-core: "workspace:^" typescript: "npm:6.0.3" peerDependencies: + "@shopify/react-native-skia": ">=2.0.0" expo: ">=52.0.0" expo-audio: "*" expo-av: "*" @@ -19060,7 +19617,10 @@ __metadata: expo-sharing: "*" expo-video: "*" react-native: ">=0.76.0" + victory-native: ">=41.0.0" peerDependenciesMeta: + "@shopify/react-native-skia": + optional: true expo-audio: optional: true expo-av: @@ -19083,6 +19643,8 @@ __metadata: optional: true expo-video: optional: true + victory-native: + optional: true languageName: unknown linkType: soft @@ -19093,11 +19655,14 @@ __metadata: "@babel/core": "npm:^7.29.0" "@babel/runtime": "npm:^7.29.2" "@gorhom/bottom-sheet": "npm:^5.2.14" + "@khanacademy/simple-markdown": "npm:^2.1.0" "@op-engineering/op-sqlite": "npm:^18.1.0" "@react-native-community/netinfo": "npm:^12.0.1" "@react-native/babel-preset": "npm:0.87.0" "@react-native/jest-preset": "npm:0.87.0" "@shopify/flash-list": "npm:^2.3.3" + "@shopify/react-native-skia": "npm:^2.13.0" + "@stream-io/state-store": "npm:^1.1.6" "@stream-io/typescript-config": "workspace:^" "@testing-library/jest-native": "npm:^5.4.3" "@testing-library/react-native": "npm:13.2.0" @@ -19109,6 +19674,7 @@ __metadata: "@types/lodash": "npm:4.17.16" "@types/mime-types": "npm:2.1.4" "@types/react": "npm:^19.2.0" + "@types/react-syntax-highlighter": "npm:^15.5.13" "@types/react-test-renderer": "npm:19.1.0" "@types/ungap__structured-clone": "npm:^1.2.0" "@types/use-sync-external-store": "npm:^1.5.0" @@ -19125,6 +19691,7 @@ __metadata: intl-pluralrules: "npm:^2.0.1" jest: "npm:^30.4.2" linkifyjs: "npm:^4.3.2" + lodash: "npm:4.18.1" lodash-es: "npm:4.18.1" mime-types: "npm:^2.1.35" moment-timezone: "npm:^0.6.0" @@ -19140,17 +19707,21 @@ __metadata: react-native-teleport: "npm:^1.1.12" react-native-url-polyfill: "npm:^2.0.0" react-native-worklets: "npm:^0.12.1" + react-syntax-highlighter: "npm:^16.1.0" react-test-renderer: "npm:19.2.3" rimraf: "npm:^6.0.1" stream-chat: "npm:^9.54.0" typescript: "npm:6.0.3" use-sync-external-store: "npm:^1.5.0" uuid: "npm:^11.1.0" + victory-native: "npm:^42.0.1" + zod: "npm:^4.1.12" peerDependencies: "@emoji-mart/data": ">=1.1.0" "@op-engineering/op-sqlite": ">=14.0.0" "@react-native-community/netinfo": ">=11.3.1" "@shopify/flash-list": ">=2.1.0" + "@shopify/react-native-skia": ">=2.0.0" emoji-mart: ">=5.4.0" react-native: ">=0.76.0" react-native-gesture-handler: ">=2.18.0" @@ -19158,6 +19729,7 @@ __metadata: react-native-safe-area-context: ">=5.4.1" react-native-svg: ">=15.8.0" react-native-teleport: ">=0.5.4" + victory-native: ">=41.0.0" peerDependenciesMeta: "@emoji-mart/data": optional: true @@ -19165,8 +19737,12 @@ __metadata: optional: true "@shopify/flash-list": optional: true + "@shopify/react-native-skia": + optional: true emoji-mart: optional: true + victory-native: + optional: true languageName: unknown linkType: soft @@ -19184,6 +19760,7 @@ __metadata: "@react-native-camera-roll/camera-roll": ">=7.9.0" "@react-native-clipboard/clipboard": ">=1.16.0" "@react-native-documents/picker": ">=10.1.1" + "@shopify/react-native-skia": ">=2.0.0" react-native: ">=0.76.0" react-native-audio-recorder-player: ">=3.6.13" react-native-blob-util: ">=0.22.0" @@ -19192,6 +19769,7 @@ __metadata: react-native-nitro-sound: ">=0.2.9" react-native-share: ">=11.0.0" react-native-video: ">=6.18.0" + victory-native: ">=41.0.0" peerDependenciesMeta: "@react-native-camera-roll/camera-roll": optional: true @@ -19199,6 +19777,8 @@ __metadata: optional: true "@react-native-documents/picker": optional: true + "@shopify/react-native-skia": + optional: true react-native-audio-recorder-player: optional: true react-native-blob-util: @@ -19213,6 +19793,8 @@ __metadata: optional: true react-native-video: optional: true + victory-native: + optional: true languageName: unknown linkType: soft @@ -20461,6 +21043,25 @@ __metadata: languageName: node linkType: hard +"victory-native@npm:^42.0.1": + version: 42.0.1 + resolution: "victory-native@npm:42.0.1" + dependencies: + d3-scale: "npm:^4.0.2" + d3-shape: "npm:^3.2.0" + d3-zoom: "npm:^3.0.0" + its-fine: "npm:^1.2.5" + react-fast-compare: "npm:^3.2.2" + peerDependencies: + "@shopify/react-native-skia": ">=2.6.0 <3.0.0" + react: "*" + react-native: "*" + react-native-gesture-handler: ">=2.0.0" + react-native-reanimated: ">=3.19.1" + checksum: 10c0/6d27b9c912bf6da5272f0a08c9379e7dd0d703841176f3cef970ee5b62fd59d7de548c03bd7bbdd890f3fc34d8fe4a0246f36d2dfd6b8617a6200caece7f0783 + languageName: node + linkType: hard + "vlq@npm:^1.0.0": version: 1.0.1 resolution: "vlq@npm:1.0.1" @@ -21034,3 +21635,10 @@ __metadata: checksum: 10c0/5718ec35e3c40b600316c5b4c5e4976f7fee68151bc8f8d90ec18a469be9571f072e1bbaace10f1e85cf8892ea12d90821b200e980ab46916a6166a4260a983c languageName: node linkType: hard + +"zod@npm:^4.1.12": + version: 4.6.5 + resolution: "zod@npm:4.6.5" + checksum: 10c0/32eb6c5e98becbe045675b2141c3611451c16ba567f9779231d79af41c4d651ca1c22f154aef7e6c6e77b929f23e43f1154ca05b82e119e922353725d6776af5 + languageName: node + linkType: hard From f2569ab25897dc032131281a7ba7c319b7169767 Mon Sep 17 00:00:00 2001 From: Ivan Sekovanikj <31964049+isekovanic@users.noreply.github.com> Date: Wed, 14 Oct 2026 13:54:07 +0200 Subject: [PATCH 2/2] chore: report ai-components size separately (#3858) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## ๐ŸŽฏ Goal ## ๐Ÿ›  Implementation details ## ๐ŸŽจ UI Changes
iOS
Before After
Android
Before After
## ๐Ÿงช Testing ## โ˜‘๏ธ Checklist - [ ] I have signed the [Stream CLA](https://docs.google.com/forms/d/e/1FAIpQLScFKsKkAJI7mhCr7K9rEIOpqIDThrWxuvxnwUq2XkHyG154vQ/viewform) (required) - [ ] PR targets the `develop` branch - [ ] Documentation is updated - [ ] New code is tested in main example apps, including all possible scenarios - [ ] SampleApp iOS and Android - [ ] Expo iOS and Android --- examples/SampleApp/Gemfile.lock | 4 ++-- examples/SampleApp/fastlane/Fastfile | 12 ++++++++---- examples/SampleApp/fastlane/Pluginfile | 2 +- 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/examples/SampleApp/Gemfile.lock b/examples/SampleApp/Gemfile.lock index c6b4b5bee8..398fa7a87f 100644 --- a/examples/SampleApp/Gemfile.lock +++ b/examples/SampleApp/Gemfile.lock @@ -186,7 +186,7 @@ GEM google-apis-firebaseappdistribution_v1 (~> 0.3.0) google-apis-firebaseappdistribution_v1alpha (~> 0.2.0) fastlane-plugin-load_json (0.0.1) - fastlane-plugin-stream_actions (0.3.73) + fastlane-plugin-stream_actions (0.4.14) xctest_list (= 1.2.1) fastlane-sirp (1.0.0) sysrandom (~> 1.0) @@ -356,7 +356,7 @@ DEPENDENCIES fastlane fastlane-plugin-firebase_app_distribution fastlane-plugin-load_json - fastlane-plugin-stream_actions (= 0.3.73) + fastlane-plugin-stream_actions (= 0.4.14) logger mutex_m nkf diff --git a/examples/SampleApp/fastlane/Fastfile b/examples/SampleApp/fastlane/Fastfile index f7993958b3..f2680950ee 100644 --- a/examples/SampleApp/fastlane/Fastfile +++ b/examples/SampleApp/fastlane/Fastfile @@ -205,14 +205,18 @@ lane :frameworks_sizes do sh('yarn minify-bundle') sh('yarn minify-ai-components-bundle') - js_bundle_size = file_size(path: 'package/lib/module/bundle.min.js') - UI.message("ai_components_bundle_size: #{file_size(path: 'package/lib/ai-components.min.js').round} KB") - { js_bundle_size: js_bundle_size } + { + js_bundle_size: file_size(path: 'package/lib/module/bundle.min.js'), + ai_components_size: file_size(path: 'package/lib/ai-components.min.js') + } end end lane :show_frameworks_sizes do |options| - next unless is_check_required(sources: ['package/'], force_check: @force_check) + # Besides the SDK itself, changes to how the size is measured (this lane, its plugin, the root + # `minify-*` scripts) re-run the check too. + sources = ['package/', 'package.json', 'examples/SampleApp/fastlane/', 'examples/SampleApp/Gemfile.lock'] + next unless is_check_required(sources: sources, force_check: @force_check) sizes = options[:sizes] || frameworks_sizes show_sdk_size(branch_sizes: sizes, github_repo: github_repo, size_ext: sdk_size_ext) diff --git a/examples/SampleApp/fastlane/Pluginfile b/examples/SampleApp/fastlane/Pluginfile index 5cb59d45fd..280e94a8aa 100644 --- a/examples/SampleApp/fastlane/Pluginfile +++ b/examples/SampleApp/fastlane/Pluginfile @@ -4,4 +4,4 @@ gem 'fastlane-plugin-firebase_app_distribution' gem 'fastlane-plugin-load_json' -gem 'fastlane-plugin-stream_actions', '0.3.73' +gem 'fastlane-plugin-stream_actions', '0.4.14'