ROX-36784: Mount sysimage RPM DB for V4 node indexing #22669
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| import static util.Helpers.waitForTrue | ||
| import static util.Helpers.withRetry | ||
|
|
||
| import orchestratormanager.OrchestratorTypes | ||
|
|
||
| import io.stackrox.proto.storage.NodeOuterClass.Node | ||
|
|
||
| import org.junit.Assume | ||
|
|
||
| import common.Constants | ||
| import services.BaseService | ||
| import services.NodeService | ||
| import util.Env | ||
|
|
||
| import spock.lang.IgnoreIf | ||
| import spock.lang.Requires | ||
| import spock.lang.Tag | ||
|
|
||
| // Central's scanner-v4-matcher is required to enrich index reports into node scans. | ||
| @IgnoreIf({ Env.ONLY_SECURED_CLUSTER == "true" }) | ||
| // V4 node index reports are produced on OpenShift (RHCOS/RHEL RPM DBs). | ||
| @Requires({ Env.mustGetOrchestratorType() == OrchestratorTypes.OPENSHIFT }) | ||
| @Tag("PZ") | ||
| class NodeIndexTest extends BaseSpecification { | ||
| def setupSpec() { | ||
| BaseService.useBasicAuth() | ||
| } | ||
|
|
||
| @Tag("BAT") | ||
| def "Verify node index scans"() { | ||
| given: | ||
| "Scanner V4 is enabled and the cluster has nodes" | ||
| Assume.assumeTrue("Scanner V4 node indexing is required", scannerV4Enabled) | ||
| List<Node> nodes = NodeService.getNodes() | ||
| assert nodes.size() > 0 | ||
|
|
||
| when: | ||
| "scanner-v4-matcher is ready so index reports can be enriched" | ||
| log.info("Waiting for scanner-v4-matcher deployment to be ready") | ||
| waitForTrue(20, 6) { | ||
| orchestrator.deploymentReady(Constants.STACKROX_NAMESPACE, "scanner-v4-matcher") | ||
| } | ||
| // Matcher-not-ready drops the first index report as unretryable, so force a | ||
| // prompt rescan after matcher is up. The default first-scan delay is 5m. | ||
| // Sometimes one pod in the daemon set misses the env variable despite updating. | ||
| waitForTrue(2, 20) { | ||
| log.info("Setting collector.compliance ROX_NODE_SCANNING_MAX_INITIAL_WAIT to 1s") | ||
| orchestrator.updateDaemonSetEnv(Constants.STACKROX_NAMESPACE, Constants.COLLECTOR_DS, "compliance", | ||
|
Contributor
There was a problem hiding this comment. can we change this setting on cluster provision?
Contributor
Author
There was a problem hiding this comment. Such a low value is needed only in the node scanning tests, but I think it does not hurt to set it for all openshift clusters from the very beginning. |
||
| "ROX_NODE_SCANNING_MAX_INITIAL_WAIT", "1s") | ||
| try { | ||
| log.info("Wait for collector DS to be restarted with new values") | ||
| waitForTrue(20, 10) { | ||
| orchestrator.daemonSetEnvVarUpdated(Constants.STACKROX_NAMESPACE, Constants.COLLECTOR_DS, | ||
| "compliance", "ROX_NODE_SCANNING_MAX_INITIAL_WAIT", "1s") | ||
| } | ||
|
|
||
| log.info("Wait for collector DS to be ready") | ||
| waitForTrue(20, 10) { | ||
| orchestrator.daemonSetReady(Constants.STACKROX_NAMESPACE, Constants.COLLECTOR_DS) | ||
| } | ||
| } | ||
| catch (Exception ignored) { | ||
| log.info("Unable to bring collector ds to the desired state") | ||
| return false | ||
| } | ||
| return true | ||
| } | ||
|
|
||
| then: | ||
| "each OpenShift node scan has RPM packages, not only kubelet/kernel/runtime" | ||
| withRetry(12, 30) { | ||
| nodes = NodeService.getNodes() | ||
| assert nodes.size() > 0, "Expected to find at least one node" | ||
| nodes.each { node -> | ||
| assert node.getScan(), "Expected to find a nodeScan on the node" | ||
| int n = node.getScan().getComponentsList().size() | ||
| log.info("Node ${node.getName()} scan contains ${n} components") | ||
| assert n > 4, "Expected to find more than 4 components on OpenShift node" | ||
| } | ||
| } | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this will modify pkgs, was that intended?
https://go.dev/play/p/5qIEiXdhpNf
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will double check whether it works as I planned (deduplication).
The intention is to remove duplicates. If a node filesystem has symlinks leading to the same RPM database, the indexer will find it twice and all packages will be duplicated. Here, I added deduplication to avoid that issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, this is intentional :) Look at the old code - it did the same.
We want to copy packages from
pkgsintooutbut filtered and without duplicates. Thus we don't start with emptyout, but we start with len=1, so that the first element frompkgsis already inout. Later, we keep adding packages to out - if filtering applies and there are not duplicates.This works as intended.