Repository navigation
SONARJAVA-7339 [DRAFT][PoC] Serve SonarJava from udg-gen --server get-file/get-types (SemSitter PR #4952) - #6514
Conversation
|
Warning Gitar could not complete this run. Comment Verification Rules 2/2 metFunctional validation No results Rules 2 passed 2 not applicable
Auto-approval Not enabledFor more verification details, comment OptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
| synchronized JsonObject analyze(JsonObject request) throws IOException { | ||
| String root = projectRoot(request); | ||
| if (process == null) { | ||
| start(root); | ||
| } | ||
| if (!root.equals(scannedKey)) { | ||
| scan(request); | ||
| scannedKey = root; | ||
| } | ||
| JsonObject result = new JsonObject(); | ||
| result.addProperty("ok", true); | ||
| result.addProperty("root", root); | ||
| result.add("files", request.getAsJsonArray("files")); | ||
| return result; | ||
| } |
There was a problem hiding this comment.
💡 Cross-PR: Sourcepath files outside the module base dir are never scanned · #6501
Breaks when #6501's sourcepath experiment is combined with this PR's udg-gen session, and a sonar.java.internal.sourcepath directory lies outside the module base directory.
How
- #6501 adds the sourcepath files to the file list passed to
bridge.analyzeinJParserConfig.openBridge(java-frontend/src/main/java/org/sonar/java/model/JParserConfig.java:213). AnalyzerSession.analyzescans only the module root, which is thebaseDirset byJavaFrontend. It echoes the requestfilesback without ever sending them to the server (java-frontend/src/main/java/org/sonar/java/semsitter/AnalyzerSession.java:93-107).- The server only indexes
--src root(AnalyzerSession.java:136-137), so sourcepath types outside that root never resolve, and no error is reported. #6501's Rust-side changes insemsitter-bridge/src/types.rsare deleted by this PR.
Fix
- This PR: Forward the extra source directories to the server (for example as an additional
--srcor a settings key), or fail loudly when a requested file is outside the scanned root. - #6501: When rebasing, pass the sourcepath directories through the udg-gen server settings instead of the deleted bridge file list.
Compared 5e3aebe with #6501 at 4be1331
Was this helpful? React with 👍 / 👎
|
/pvf |
|
📊 Performance Validation complete Projects with unexpected PVF results:
|
# Conflicts: # java-frontend/src/main/resources/udg-gen/udg-gen-linux-aarch64.xz
…he bridge process
…down on a failed handshake, and resolve a packaged analyzer by platform
…scan early, read declaration annotations and thrown types from the type answers, and decode the flat file frame of the analyzer protocol
…types, imports and constructor choice with the compiler
…ation views, overload choice, static imports, enum constants and generic calls from their targets
…quest on its own, and leave members of undeclared types unknown without libraries
…ild the tree of a file with syntax errors from the recovered answer
…pe-use and package annotations, and keep nullability of local classes local
…mbdas, arrays, instanceof patterns, casts, annotations and import uses with the compiler
…he frontend tests compile against the analyzer-based model
…nd document how to run with it
…uper as keywords, keep intersection bounds, report masked catch blocks and align symbol identity with the compiler
…s and constructor invocations, and read default values of annotation members
…ldcard equality and array element types, and align test expectations with the analyzer-based model
…thods when looking for overridden ones, annotate wildcards, type assignment targets and describe unresolved types
…the record, describe parameters of unseen methods and align frontend test expectations
…variables with the analyzer model
…ons and type arguments where the compiler had them, and align frontend test expectations with the analyzer-based model
…library table out of check tests and align type tests
…ith inferred parameters, type getClass calls and return targets of lambdas, and keep fields of local classes
…supertypes, widen redundant casts in assignments and align non-compiling check tests
…d qualified names unbound in locals, leave lambda parameters without a target unbound and give unresolved annotations no member values
…rameterization with an undeclared argument as unknown
…members of an unknown parameterization unresolved
…arity type variable
…class-file annotation class values as type symbols
…turn type shared by overriding members
3da4910 to
4b017f2
Compare
Draft PoC. Do not merge. Built on SemSitter's
--servermode (get-file/get-types; the first measurements used PR #4952 ofSonarSource/semsitter, head22bbb0bd1f) and on top of #6415 (Romain's ECJ replacement; this PR's base is his branch so the diff shows only the client change). No Jira key.Update (client rewritten against the current analyzer)
This push replaces the earlier head. The client now talks to a Java-only SemSitter analyzer (a trimmed build of the server that only analyzes Java) instead of the full
udg-gen; the measurement tables below were taken with the earlier build and client and are kept only as history, they do not describe this head./semsitter/analyzer-<platform>.xz(linux-x64,linux-aarch64,macos-aarch64andwindows-x64are packaged; other platforms need-Dsonar.java.semsitter.executable=<path>orSEMSITTER_ANALYZER, and fail with an explicit message otherwise). The packaged build is the optimized, stripped Java-only analyzer.JAVA_HOMEon JDK 25):java-frontend2,039 tests, 0 failures (3 skipped).java-checks5 failures out of 2,492:ClassBuilderWithMethodCheckTestcases: lambda parameters of an overloaded JDK 25 method are not resolved by the analyzer yet.CollectorsToListCheckTestcases: need full inference of a nested generic call from its return target.SpringAntMatcherOrderCheckTestcase: a member type writtenOuter<X>.Inneris substituted by a newer analyzer protocol that the client does not consume yet.The two
TryWithResourcesCheckTestcases need a JDK 25 test JVM (Reader.of) and pass withJAVA_HOMEon JDK 25. The analyzer-side items are being fixed; this description will be updated when they land.java-checkstests run with the built-in library table off (sonar.java.semsitter.frameworkTypes=false) so that they see what the test classpath provides.What changes
semsitter-bridgeprocess, its prebuilt binaries and the shm/JSONL sessions are replaced by oneudg-gen --serverper scanner JVM (AnalyzerSession):hello,set-settings+full-genonce per module,get-fileper file,get-typeswith a session cache,release+ counters at shutdown.sonar.java.libraries/binaries/test.*becomejvm.*, JDK home fromsonar.java.jdkHomeor the scanner JVM.-Dsonar.java.semsitter.udgorSEMSITTER_UDG_GEN. Not done: single-sourceJParser.parse, a shared JVM-side type table.5faffac831unmodified.How to read the table
What is measured. One run scans one whole project with the complete SonarJava sensor: parse the sources, build the semantic model, run all 780 java-checks rules, report the issues. Each cell is a single cold run: one fresh JVM, no warm-up iteration, one measured iteration, no cache from a previous run (PVF
SensorBenchmark, JMH non-forked,-wi 0 -i 1). JIT warm-up is therefore inside the numbers. Both sides get exactly the same projects, rules and settings.The two sides.
5faffac831, unmodified. The scanner JVM parses with ECJ itself. No other process.udg-gen --serverprocess (SemSitter PR SONARJAVA-5134 S2245 considers RandomStringUtils.secure|secureStrong as safe #4952). So a SemSitter run is two processes: the scanner JVM and the server.Columns.
nomeans no libraries or compiled classes at all (source-only checkout), so types from third-party libraries are unresolved on both sides.312 jars + classesmeans OFBiz was built and its 312 dependency jars and compiled classes were given to both sides.(rule, file, line)entries. In brackets: how many more (or fewer) issues SemSitter reports than ECJ, in %.G= 10^9; 1,743 means 1.743 trillion). It includes everything the process did: parsing, semantic model, the 780 rules, issue serialisation and the benchmark harness. Instructions are used instead of wall time because the host was loaded (1-minute load 31 to 89, threshold 5), which makes wall time invalid; instruction counts do not depend on the load. Measured with/usr/bin/time -l.scanner JVM + udg-gen server = total./usr/bin/time -ldoes not count the instructions of a Java child process, so the server was measured under its owntime -land added. The JVM part is the same work as ECJ's column except that parsing is replaced by asking the server (get-file,get-types) and ingesting the answers. The server part is the wholeudg-genscan of the module plus answering those requests.1.30x (+30%)means 30 % more instructions than ECJ; below 1.0x would be a win.JVM alonecompares only the scanner-JVM part of SemSitter with the whole ECJ run (so it shows the cost on the client side before adding the server);serveris the part of the SemSitter total spent inudg-gen.udg-genserver. The server value comes from 1-second sampling and can under-read short peaks. SemSitter's total memory is the JVM plus the server. Heap was-Xmx4geverywhere except lucene on SemSitter (-Xmx8g).Server counters line under the table (SemSitter only):
1 scan per run: the server scanned the module once, whatever the batch size.own_parses/shared_parses: files the server parsed for itself versus files it served from the scan it already had.own_parses0 means no file was parsed twice;shared_parsesequals the number of files.type requests: how manyget-typescalls the client made.type bytes: total size of the answers.Caveats. One run per cell, instruction counts vary about ±5 % between runs, overlapping runs on a loaded host. Only the large gaps mean something.
Measurement (PVF,
SensorBenchmarkJMH non-forked, 780 rules, one run per cell)Wall time invalid everywhere (1-minute load 31 to 89, threshold 5). Instructions retired from
/usr/bin/time -l; the server runs under its owntime -lbecause the parent's figure excludes a Java child (verified). Full method, caveats and classification:semsitter-dre/java-bridge/08-sonarjava-semsitter-pvf.md(local notes).guava/src(no)-Xmx8g; 4g ran out of memory)java.basesources (no; the JDK home is given to both sides)guava/src(5 jars + classes)-Xmx8g)Server counters: 1 scan per run,
own_parses0,shared_parses= file count (1,261 / 7,576 / 1,473 / 611 / 9,516 / 6,085); type requests 10,583 / 43,615 / 13,289 / 1,874 / 34,191 / 17,648; type bytes 19 / 52 / 18 / 14 / 90 / 62 MB (OFBiz with classpath: 4,363 requests, 34 MB).Added rows: the JDK and projects with jars + classes
Measured later with the same method (one cold run per cell, ECJ and SemSitter interleaved, each run started only with 1-minute load below 8 and no other server/PVF/build process;
-Xmx4g, lucene on SemSitter-Xmx8g). Differences from the first rows:udg-genbuilt from SemSitter master15a8a877f7(PR SONARJAVA-5134 S2245 considers RandomStringUtils.secure|secureStrong as safe #4952 is merged) instead of22bbb0bd1f. A re-run of OFBiz with classpath on it gave 2,189 + 318 = 2,507 G (table: 2,521 G) and 168,446 issues (table: 168,359), inside the noise. The client jar and the ECJ jar are the same as in the first rows, i.e. measured before the Gitar fixes and the embedded binaries of this PR.lib/src.zip,java.base/*only (3,477 files, 1,307,374 lines, ncloc 621,975 ECJ / 623,668 SemSitter). The JDK home is passed to both sides, so the platform classes are already available; a second JDK row with a classpath would only duplicate the scanned sources. The sources that definejava.lang.*are scanned next to the same classes in the JDK home, which may affect override and type rules.test-suitemodule (which needs themaven-openmrs-pluginextension, not on Repox) was left out of the build copy, so its sources are scanned without classes. Lucene: a build-copy tweak of the minimum Gradle version.What the new rows show:
throws; lucene 8,769, spring 2,591): the thrown types of inherited or generic cross-file methods seem to be lost, confirmed by reading two cases; S2175 and S9396/S2387 (generic substitution, member hiding); S3878 (probably varargs resolution); on the JDK S1226 (4,482, probably true positives that ECJ misses) and S1161 (override detection differs).ReflectionOnNonRuntimeAnnotationCheck) throwsArrayIndexOutOfBoundsExceptionon 1 to 6 files per project under SemSitter (JDK 3, guava 1, openmrs 1, SonarQube 1, spring 4, lucene 6) and never under ECJ.Reading
ReflectionOnNonRuntimeAnnotationCheck(S2109) throwsArrayIndexOutOfBoundsExceptionon 1 to 5 files per project under SemSitter, none under ECJ.Test plan:
mvn test -pl java-frontend,java-checkswith the packaged analyzer (macOS aarch64) or-Dsonar.java.semsitter.executable; expected: java-frontend green, java-checks with the failures listed in the Update section.TryWithResourcesCheckTestneedsJAVA_HOMEon JDK 25.