Repository navigation
SONARJAVA-7341 Simplify the parser interface - #6517
aurelien-coet-sonarsource wants to merge 3 commits into
Conversation
Code Review
|
| Auto-apply | Compact | Unblock |
|
|
|
Was this helpful? React with 👍 / 👎 | Gitar
| * `org.sonar.java.model.JParser` is now an interface, independent of the underlying parser implementation, and replaces `JParserConfig`. | ||
| Create parsers with `JParser.Mode.BATCH.create(JavaVersion, List<File>)` or `JParser.Mode.FILE_BY_FILE.create(JavaVersion, List<File>)`. Both factories also accept an optional third `boolean` argument to ignore the unnamed module for split packages. | ||
| The string-based `parse(String version, String unitName, String source)` overload has been removed. Use `JParser#parse(Iterable<InputFile>, BooleanSupplier, AnalysisProgress, BiConsumer<InputFile, JParseResult>)` for all parsing, including a single file supplied as `List.of(inputFile)`. | ||
| The Java version and classpath are set when creating the parser; source contents, filename, and charset come from each `InputFile`. | ||
| The input parameter now takes `Iterable<InputFile>` instead of `Iterable<? extends InputFile>`; callers must supply an `Iterable<InputFile>`, and implementations must update their override signatures. Code compiled against `JParserConfig` must be recompiled. | ||
| Parsing callbacks now receive `org.sonar.java.model.JParseResult` instead of `JParserConfig.Result`. `JParseResult` is final and provides `success(tree)` and `failure(exception)` factories; `get()` returns the tree or rethrows the parsing exception. |
There was a problem hiding this comment.
💡 Quality: 8.47 notes skip removed parser entry points and misstate the migration
The rewritten 8.47 Breaking notes only mention that the parse(String version, String unitName, String source) overload is gone. This PR also removes other parser entry points that the earlier 8.47 notes advertised: the static JParser.parse(JParser, String, String, String), JParserConfig#astParser(), JParserConfig#parse(String unitName, String source), and the ParserConfigConstructor interface. MAXIMUM_SUPPORTED_JAVA_VERSION has moved to JParser, and the notes don't say that either. Line 162 also tells readers that "Code compiled against JParserConfig must be recompiled", but JParserConfig has been deleted, so that code has to be migrated, and recompiling alone fails. Existing consumers of these calls, such as JParserTestUtils.parse in sonar-java-symbolic-execution, are covered by the SE JParser dependency and the JParserConfig SE dependency notes, and they get no migration guidance. Fix: list the removed symbols, name JParser.MAXIMUM_SUPPORTED_JAVA_VERSION and JParser.Mode#create(...) plus the Iterable<InputFile> parse as the replacements, and replace the 'recompiled' sentence with a migration instruction.
Name the removed entry points and give accurate migration guidance:
The input parameter now takes `Iterable<InputFile>` instead of `Iterable<? extends InputFile>`; callers must supply an `Iterable<InputFile>`, and implementations must update their override signatures.
`JParserConfig` has been removed together with `JParserConfig#astParser()`, `JParserConfig#parse(String, String)` and the static `JParser.parse(JParser, String, String, String)`; migrate to `JParser.Mode#create(...)` and the `Iterable<InputFile>` `parse`. `MAXIMUM_SUPPORTED_JAVA_VERSION` is now `JParser.MAXIMUM_SUPPORTED_JAVA_VERSION`.
- Apply fix
Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎
| public interface JParser { | ||
|
|
||
| JavaTree.CompilationUnitTreeImpl parse(String version, String unitName, String source); | ||
| JavaVersion MAXIMUM_SUPPORTED_JAVA_VERSION = new JavaVersionImpl(JavaVersionImpl.MAX_SUPPORTED, true); | ||
|
|
||
| enum Mode { | ||
| BATCH, | ||
| FILE_BY_FILE; | ||
|
|
||
| static JavaTree.CompilationUnitTreeImpl parse(JParser parser, String version, String unitName, String source) { | ||
| return parser.parse(version, unitName, source); | ||
| public JParser create(JavaVersion javaVersion, List<File> classpath) { | ||
| return create(javaVersion, classpath, false); | ||
| } | ||
|
|
||
| public JParser create(JavaVersion javaVersion, List<File> classpath, boolean shouldIgnoreUnnamedModuleForSplitPackage) { | ||
| return JDTParser.create(this, javaVersion, classpath, shouldIgnoreUnnamedModuleForSplitPackage); | ||
| } |
There was a problem hiding this comment.
⚠️ Cross-Repo: SE test helper JParserTestUtils.parse stops compiling on java-frontend bump
Fix here and in SonarSource/sonar-java-symbolic-execution
Breaks in: SonarSource/sonar-java-symbolic-execution - java-symbolic-execution/java-symbolic-execution-plugin/src/test/java/org/sonar/java/se/utils/JParserTestUtils.java - JParserTestUtils.parse
This PR deletes org.sonar.java.model.JParserConfig, which takes astParser() and MAXIMUM_SUPPORTED_JAVA_VERSION with it. It also removes the static JParser.parse(JParser, String, String, String) helper and the instance method parse(String, String, String). JParser is left with only the multi-file parse(Iterable<InputFile>, BooleanSupplier, AnalysisProgress, BiConsumer<InputFile, JParseResult>).
In SonarSource/sonar-java-symbolic-execution, JParserTestUtils.parse (java-symbolic-execution/java-symbolic-execution-plugin/src/test/java/org/sonar/java/se/utils/JParserTestUtils.java) still uses JParserConfig.MAXIMUM_SUPPORTED_JAVA_VERSION, JParserConfig.Mode.FILE_BY_FILE.create(...).astParser() and the static JParser.parse(parser, ...). When that repo upgrades its pinned java-frontend to a build that includes this change, the helper no longer compiles, and neither does any SE test that depends on it.
The removal is deliberate, so the SE repo is the side that has to move. Because only the InputFile-based API is left, its test module also needs a test-scoped sensor-test-fixtures dependency (the same one this PR adds to java-checks-common/pom.xml) or some other way to get an InputFile. The 8.47 notes in documentation.md don't name the removed JParserConfig#astParser(), the static JParser.parse(JParser, …), or the move of MAXIMUM_SUPPORTED_JAVA_VERSION.
Here: In the 8.47 Breaking notes, list the removed JParserConfig#astParser(), JParserConfig#parse(String, String) and static JParser.parse(JParser, String, String, String), and say that MAXIMUM_SUPPORTED_JAVA_VERSION now lives on JParser. Then tell sonar-java-symbolic-execution so it can migrate to the InputFile-based parse when it bumps java-frontend.
There: In JParserTestUtils.parse, remove the JParserConfig, astParser() and static JParser.parse usage. Wrap the source in a TestInputFileBuilder InputFile and call the multi-file parse on JParser.Mode.FILE_BY_FILE.create(version, classpath). Then add a test-scoped org.sonarsource.scanner.engine:sensor-test-fixtures dependency to the plugin's pom.
In SonarSource/sonar-java-symbolic-execution - java-symbolic-execution/java-symbolic-execution-plugin/src/test/java/org/sonar/java/se/utils/JParserTestUtils.java:
public static CompilationUnitTree parse(String unitName, String source, List<File> classpath) {
JavaVersion version = JParser.MAXIMUM_SUPPORTED_JAVA_VERSION;
InputFile inputFile = new TestInputFileBuilder("", unitName)
.setContents(source).setCharset(StandardCharsets.UTF_8).build();
List<JParseResult> results = new ArrayList<>();
JParser.Mode.FILE_BY_FILE.create(version, classpath)
.parse(List.of(inputFile), () -> false, new AnalysisProgress(1), (input, result) -> results.add(result));
try {
return results.getFirst().get();
} catch (RuntimeException e) {
throw e;
} catch (Exception e) {
throw new IllegalStateException("Unable to parse source", e);
}
}
Was this helpful? React with 👍 / 👎
| public static JParser create(JParser.Mode mode, JavaVersion javaVersion, List<File> classpath, boolean shouldIgnoreUnnamedModuleForSplitPackage) { | ||
| if (shouldIgnoreUnnamedModuleForSplitPackage) { | ||
| LOG.info("The Java analyzer will ignore the unnamed module for split packages."); | ||
| } | ||
| return ((JavaTree) node).getChildren().iterator(); | ||
| return switch (mode) { | ||
| case BATCH -> new Batch(javaVersion, classpath, shouldIgnoreUnnamedModuleForSplitPackage); | ||
| case FILE_BY_FILE -> new FileByFile(javaVersion, classpath, shouldIgnoreUnnamedModuleForSplitPackage); | ||
| }; | ||
| } | ||
|
|
||
| @VisibleForTesting | ||
| static List<Token> lex(String version, String unitName, char[] sourceChars) { | ||
| List<Token> tokens = new ArrayList<>(); | ||
| Scanner scanner = new Scanner( | ||
| true, | ||
| false, | ||
| false, | ||
| CompilerOptions.versionToJdkLevel(version), | ||
| null, | ||
| null, | ||
| false | ||
| ); | ||
| scanner.fakeInModule = "module-info.java".equals(unitName); | ||
| scanner.setSource(sourceChars); | ||
| while (true) { | ||
| try { | ||
| TerminalToken tokenType = scanner.getNextToken(); | ||
| Token token = Token.fromCurrent(scanner, tokenType); | ||
| tokens.add(token); | ||
| if (tokenType == TerminalToken.TokenNameEOF) { | ||
| break; | ||
| } | ||
| } catch (InvalidInputException e) { | ||
| throw new IllegalStateException(e); | ||
| } | ||
| } | ||
| return tokens; | ||
| } | ||
|
|
||
| private CompilationUnit compilationUnit; | ||
|
|
||
| private TokenManager tokenManager; | ||
| private LineColumnConverter lineColumnConverter; | ||
|
|
||
| private JDTSema sema; | ||
|
|
||
| private final Deque<JLabelSymbol> labels = new LinkedList<>(); | ||
|
|
||
| private void declaration(@Nullable IBinding binding, Tree node) { | ||
| if (binding == null) { | ||
| return; | ||
| } | ||
| sema.declarations.put(binding, node); | ||
| static boolean shouldEnablePreviewFlag(JavaVersion currentVersion) { | ||
| return currentVersion.arePreviewFeaturesEnabled(); | ||
| } | ||
|
|
There was a problem hiding this comment.
⚠️ Cross-PR: JDK release-option support from #6459 is lost after JParserConfig removal
Breaks when this PR and #6459 both merge into the same line. Whichever lands second either fails to compile or silently drops the sonar.java.useJdkReleaseOption behaviour.
How
- #6459 adds a
shouldUseJdkReleaseOptionfield, a four-argumentMode.create(...)anduseReleaseApiOfConfiguredJdk(...)toJParserConfig(java-frontend/src/main/java/org/sonar/java/model/JParserConfig.java:141). It calls them fromJavaFrontend.scanBatchandJavaAstScanner.scan. - This PR deletes
JParserConfig. Parser construction moves toJParser.Mode.create(JavaVersion, List<File>, boolean)(java-frontend/src/main/java/org/sonar/java/model/JParser.java:36-42) andJDTParser.create. JDTParser.jdtAstParser()(java-frontend/src/main/java/org/sonar/java/jdt/JDTParser.java:84-112) builds the ECJ environment from the raw classpath and never setsSystemReleaseClasspath.OPTION_RELEASE. If #6459's call sites are only ported toJParser.Mode, the parser keeps using the running JDK's API instead of the target release's.
Fix
- This PR: If #6459 lands first, add the
shouldUseJdkReleaseOptionflag toJParser.Mode.create,JDTParser.createand theJDTParserconstructor. MoveuseReleaseApiOfConfiguredJdkintoJDTParser.jdtAstParser()and keep theJavaFrontend/JavaAstScannerwiring. - #6459: If this PR lands first, rebase onto
JParser.Mode/JDTParser. Put the release-option logic inJDTParser.jdtAstParser()instead ofJParserConfig.astParser(), and move theJParserConfigTestcases toJParserModeTest/JDTParserTest.
Compared 5006a8c with #6459 at 71f4165
Was this helpful? React with 👍 / 👎
|



Summary by Gitar
JParserConfigwith aJParserinterface andJParser.Modefactories inJParser.javaIterable<InputFile>and reports results throughJParseResultJDTParserConfigintoJDTParser(batch and file-by-file) andJDTAstConverterdocumentation.mdThis will update automatically on new commits.