Read the external id map alongside the mmapped CSR - #43
Conversation
Signed-off-by: Zirui Song <zrsong@amazon.com>
chishui
left a comment
chishui
left a comment
There was a problem hiding this comment.
swig file also needs to be updated with read_csr_and_read_id, add one test in python
| void add_with_ids(idx_t n, const idx_t* indptr, const term_t* indices, | ||
| const float* values, const idx_t* ids) override; | ||
|
|
||
| void read_csr_and_read_id(const char* csr_path, const char* id_path, |
There was a problem hiding this comment.
what about read_csr_and_ids?
There was a problem hiding this comment.
Renamed to read_csr_and_ids throughout
| // the count-vs-CSR-row check can fail once the delegate has ingested; on | ||
| // ANY throw from this method the half-built index must be discarded, not | ||
| // reused. | ||
| if (!std::filesystem::exists(id_path)) { |
There was a problem hiding this comment.
add a check_if_file_valid in check.h to include null pointer check and file exist check
There was a problem hiding this comment.
Great call. Ack.
| std::to_string(delegate_size) + "): " + id_path); | ||
| } | ||
|
|
||
| internal_to_external_ = std::move(internal_to_external); |
There was a problem hiding this comment.
from 167-172 is shared logic with other functions, make it a function
| set_id_map(std::move(internal_to_external)); | ||
| } | ||
|
|
||
| void IDMapIndex::set_id_map(std::vector<idx_t> internal_to_external) { |
There was a problem hiding this comment.
void IDMapIndex::set_id_map(std::vector<idx_t> &&internal_to_external)
| // the count-vs-CSR-row check can fail once the delegate has ingested; on | ||
| // ANY throw from this method the half-built index must be discarded, not | ||
| // reused. | ||
| std::ifstream in(id_path, std::ios::binary); |
There was a problem hiding this comment.
what about moving ids read to a private function and called from here
Signed-off-by: Zirui Song <zrsong@amazon.com>
ed1a675 to
4bf121b
Compare
2 participants
Description
Add
IDMapIndex::read_csr_and_read_id: the delegate ingests vectors from the CSR (borrowed via mmap) and the Lucene -> internal id map is read from a companion[int64 count][idx_t x count]file, row-aligned with the CSR. The file-based analog ofadd_with_ids. The id file is fully validated before the CSR is ingested, so a bad file leaves the index untouched.Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.