Repository navigation
test(sensors): share the lidar model between tests of a parameterization - #211
Merged
Merged
Conversation
Bazel runs tests in parallel, and torch / numpy start one OpenMP / BLAS thread per core in every test process, so the processes oversubscribe the CPU. With the 4 python versions of the lidar test running at once on 16 cores, each run took ~390s instead of ~50s with a single thread, and the full test suite took ~530s instead of ~175s.
Every test of the parameterized row-offset lidar model tests created a new model, so the angles to columns map (a KD-tree query over all elements, the most expensive part of these tests) was computed once per test instead of once per parameterization. Create the model and the element grid once per test class instead, and give the test that moves the model between devices a model of its own. This halves the run time of the lidar tests (82s -> 39s with all python versions in parallel on 16 cores).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every test of the parameterized
TestRowOffsetStructuredSpinningLidarModelclasses created a new lidar model, so the angles to columns map (a KD-tree query over all elements, the most expensive part of these tests) was computed per test rather than per parameterization.setUpClass).test_model_parameters_roundtripmoves the model between devices, so it creates its own.test_angles_to_columns_map_valuesreuses the map if it was already computed.Stacked on #210 (single OpenMP thread for tests); only the last commit belongs to this PR.
Measurements
All 4 python versions of
pytest_lidarin parallel, restricted to 16 cores, withOMP_NUM_THREADS=1: slowest target 82s -> 39s.On CI (
build-and-test, all 71 tests), compared to #210 alone:bazel test ...pytest_lidar)pytest_lidar)build-and-testjobTesting
row_elevations_rad) makes the subsequent tests of that parameterization fail, so tests do not silently pass on stale shared state.OMP_NUM_THREADS=1it brings no further gain (26s vs 23s for the lidar targets alone, at 4x the processes), so it is not included.