[A11y] Add semantics tree UI - #9982
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements the accessibility semantics tree in DevTools, adding the SemanticsNodeModel to represent nodes, updating AccessibilityController to load and parse the tree via service extensions, and introducing the AccessibilitySemanticsTreePane UI along with comprehensive tests. The review feedback highlights several critical issues: a recursive parsing bug in _parseSemanticsNode that leads to duplicate child nodes, the use of a non-existent disposeSemantics service extension (which should be replaced with disabling enableSemantics), and the use of an invalid isCheckable flag which should be updated to hasCheckedState.
| final json = (nodesMap[nodeId] as Map<String, dynamic>?) ?? | ||
| <String, dynamic>{'id': nodeId}; | ||
| final node = _parseSemanticsNode(json); |
There was a problem hiding this comment.
[MUST-FIX] When building the tree from a flat map of nodes, calling _parseSemanticsNode will recursively parse and add children if they are present in the children list of the JSON. Since _buildTreeFromNodesMap also manually resolves and adds children from the nodesMap, this leads to duplicate children being added to the SemanticsNodeModel (one partially parsed, one fully parsed). Adding a parseChildren parameter to _parseSemanticsNode and setting it to false in _buildTreeFromNodesMap prevents this duplication.
| final json = (nodesMap[nodeId] as Map<String, dynamic>?) ?? | |
| <String, dynamic>{'id': nodeId}; | |
| final node = _parseSemanticsNode(json); | |
| final json = (nodesMap[nodeId] as Map<String, dynamic>?) ?? | |
| <String, dynamic>{'id': nodeId}; | |
| final node = _parseSemanticsNode(json, parseChildren: false); |
| SemanticsNodeModel _parseSemanticsNode(Map<String, dynamic> json) { | ||
| final rect = json['rect'] as Map<String, dynamic>?; | ||
| final rectString = rect != null | ||
| ? 'rect: Rect.fromLTWH(${rect['left']}, ${rect['top']}, ${rect['width']}, ${rect['height']})' | ||
| : 'Rect.zero'; | ||
|
|
||
| final flags = (json['flags'] as List?)?.cast<String>() ?? const []; | ||
|
|
||
| final node = SemanticsNodeModel( | ||
| id: json['id']?.toString() ?? '', | ||
| label: json['label']?.toString() ?? '', | ||
| value: json['value']?.toString() ?? '', | ||
| hint: json['hint']?.toString() ?? '', | ||
| flags: flags, | ||
| widgetName: json['widgetName']?.toString() ?? '', | ||
| rectString: rectString, | ||
| ); | ||
|
|
||
| final children = json['children'] as List?; | ||
| if (children != null) { | ||
| for (final childJson in children.whereType<Map<String, dynamic>>()) { | ||
| final child = _parseSemanticsNode(childJson); | ||
| node.addChild(child); | ||
| } | ||
| } | ||
| return node; | ||
| } |
There was a problem hiding this comment.
[MUST-FIX] Update _parseSemanticsNode to accept an optional parseChildren parameter (defaulting to true) to allow disabling recursive child parsing when resolving children from a flat map of nodes.
SemanticsNodeModel _parseSemanticsNode(
Map<String, dynamic> json, {
bool parseChildren = true,
}) {
final rect = json['rect'] as Map<String, dynamic>?;
final rectString = rect != null
? 'rect: Rect.fromLTWH(${rect['left']}, ${rect['top']}, ${rect['width']}, ${rect['height']})'
: 'Rect.zero';
final flags = (json['flags'] as List?)?.cast<String>() ?? const [];
final node = SemanticsNodeModel(
id: json['id']?.toString() ?? '',
label: json['label']?.toString() ?? '',
value: json['value']?.toString() ?? '',
hint: json['hint']?.toString() ?? '',
flags: flags,
widgetName: json['widgetName']?.toString() ?? '',
rectString: rectString,
);
if (parseChildren) {
final children = json['children'] as List?;
if (children != null) {
for (final childJson in children.whereType<Map<String, dynamic>>()) {
final child = _parseSemanticsNode(childJson);
node.addChild(child);
}
}
}
return node;
}| Future<void> _disposeSemanticsOnApp() async { | ||
| try { | ||
| if (serviceConnection.serviceManager.connectedState.value.connected) { | ||
| await serviceConnection.serviceManager | ||
| .callServiceExtensionOnMainIsolate( | ||
| 'ext.flutter.accessibility.disposeSemantics', | ||
| args: {'enabled': 'false'}, | ||
| ); | ||
| } | ||
| } catch (_) { | ||
| // Ignore errors if the app or isolate connection is already closed. | ||
| } | ||
| } |
There was a problem hiding this comment.
[MUST-FIX] The service extension ext.flutter.accessibility.disposeSemantics does not exist in the Flutter framework. To disable semantics on the connected application, you should call ext.flutter.accessibility.enableSemantics with enabled: false.
| Future<void> _disposeSemanticsOnApp() async { | |
| try { | |
| if (serviceConnection.serviceManager.connectedState.value.connected) { | |
| await serviceConnection.serviceManager | |
| .callServiceExtensionOnMainIsolate( | |
| 'ext.flutter.accessibility.disposeSemantics', | |
| args: {'enabled': 'false'}, | |
| ); | |
| } | |
| } catch (_) { | |
| // Ignore errors if the app or isolate connection is already closed. | |
| } | |
| } | |
| Future<void> _disposeSemanticsOnApp() async { | |
| try { | |
| if (serviceConnection.serviceManager.connectedState.value.connected) { | |
| await serviceConnection.serviceManager | |
| .callServiceExtensionOnMainIsolate( | |
| 'ext.flutter.accessibility.enableSemantics', | |
| args: {'enabled': 'false'}, | |
| ); | |
| } | |
| } catch (_) { | |
| // Ignore errors if the app or isolate connection is already closed. | |
| } | |
| } |
| static IconData _iconForNode(SemanticsNodeModel node) { | ||
| if (node.flags.contains('isButton')) return Icons.smart_button_rounded; | ||
| if (node.flags.contains('isTextField')) return Icons.text_fields_rounded; | ||
| if (node.flags.contains('isHeader')) return Icons.title_rounded; | ||
| if (node.flags.contains('isSlider')) return Icons.linear_scale_rounded; | ||
| if (node.flags.contains('isCheckable')) return Icons.check_box_outlined; | ||
| return Icons.widgets_outlined; | ||
| } |
There was a problem hiding this comment.
[CONCERN] There is no isCheckable flag in Flutter's SemanticsFlag enum. To check if a semantics node is checkable, you should check for the hasCheckedState flag instead.
| static IconData _iconForNode(SemanticsNodeModel node) { | |
| if (node.flags.contains('isButton')) return Icons.smart_button_rounded; | |
| if (node.flags.contains('isTextField')) return Icons.text_fields_rounded; | |
| if (node.flags.contains('isHeader')) return Icons.title_rounded; | |
| if (node.flags.contains('isSlider')) return Icons.linear_scale_rounded; | |
| if (node.flags.contains('isCheckable')) return Icons.check_box_outlined; | |
| return Icons.widgets_outlined; | |
| } | |
| static IconData _iconForNode(SemanticsNodeModel node) { | |
| if (node.flags.contains('isButton')) return Icons.smart_button_rounded; | |
| if (node.flags.contains('isTextField')) return Icons.text_fields_rounded; | |
| if (node.flags.contains('isHeader')) return Icons.title_rounded; | |
| if (node.flags.contains('isSlider')) return Icons.linear_scale_rounded; | |
| if (node.flags.contains('hasCheckedState')) return Icons.check_box_outlined; | |
| return Icons.widgets_outlined; | |
| } |
srawlins
left a comment
srawlins
left a comment
There was a problem hiding this comment.
Nice tests; I haven't quite looked at everything yet, but do you want to look at gemini's feedback and give each a thumbs up or down?
| /// The current value of this node (e.g. the text in a text field). | ||
| final String value; | ||
|
|
||
| /// Additional hint text spoken after a delay (maps to `SemanticsData.hint`). |
There was a problem hiding this comment.
Can we link with [SemanticsData.hint]? If it is not imported, we can /// @docImport it.
2 participants
List which issues are fixed by this PR.
Replace this paragraph with a description of what this PR is changing or adding, and why. If your PR is updating any UI functionality, please include include before/after screenshots and/or a gif of the UI interaction.
Pre-launch Checklist
General checklist
///).Issues checklist
contributions-welcomeorgood-first-issuelabel.contributions-welcomeorgood-first-issuelabel. I understand this means my PR might take longer to be reviewed.Tests checklist
AI-tooling checklist
Feature-change checklist
release-notes-not-requiredlabel or left a comment requesting the label be added.packages/devtools_app/release_notes/NEXT_RELEASE_NOTES.md.If you need help, consider asking for help on Discord.