Keep a rebased object out of the null page

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

_find_safe_rebase_addr starts its search at address 0 whenever the architecture is narrower than 32 bits or the main image reaches into the top half of the address space. The extern object and the other objects CLE invents then land on the null page, so a null or uninitialized pointer in the target reads as an address inside a real object and find_object_containing(0) answers with the extern object instead of nothing.

Try the space above the image first, then the space at or above the null page, and fall back to the null page only when nothing else fits. Only the branch that used to start at 0 changes; every other placement is byte-identical.

Some images still map 0 afterwards and should: an ELFCore whose own first segment starts at 0 never goes through this function, and a 16-bit blob that runs to the top of its address space has nowhere else to go. That last case is the one the issue names as blocking a fix.

Fixes #745.

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Regression. tests/test_rebase.py fails without the loader change: with an ARMEL blob at base_addr=0x90000000, loader.memory.load(0, 8) returns zeros and find_object_containing(0) answers with the extern object. With the change the extern object is placed above the image and address 0 is unmapped.

Images that still map 0, and why that is correct. On the placement sample, 14 of 198 measured still map address 0: 12 are ELFCore images whose own first segment starts at 0, and the main object's placement never goes through _find_safe_rebase_addr; 2 are 16-bit blobs based at 0x210 and 0x2cd that run to 0xffff, where nothing fits above the image and nothing fits at or above the null page, so the ladder's last step behaves exactly as before. That second case is the one #745 names as blocking a fix.

Headline object. c299158142da0d55, an ARM blob at base_addr=0x90000000, use_sim_procedures=False, CFGFast(normalize=True, data_references=False, resolve_indirect_jumps=True). Each arm ran in its own forked child, because angr keeps process-global state between projects and scoring several objects in one interpreter does not give the same answer as scoring each alone.

arm	blocks	functions	in invented memory	fabricated	invented functions	bytes of the object covered	maps 0	peak RSS
stock	76,856	14,821	8,192	8,187	4,096	32,762 of 32,768	yes	657 MB
loader change	68,666	10,727	2	0	2	2	no	579 MB
CFGFast change	68,669	10,730	5	0	5	5	yes	578 MB
both	68,666	10,727	2	0	2	2	no	579 MB

Every block remaining in an invented object is a hook node. Peak RSS falls 12%. The two changes are independently sufficient here and are not additive.

A 96-byte MIPS:BE:64:R6 blob at 0x1000, digest 315a2e40e3ba032b, is the same pathology in miniature: 1,523 blocks, 1,522 of them inside the extern object, all reached from a single Ijk_Boring edge out of the image.

Scale. Over the scored sample, 1,966 objects contribute 144,547 blocks decoded inside invented memory, split 922 objects / 53,507 blocks where the invented object sits below the image and 1,044 / 91,040 where it sits above:

backend	placement	objects	blocks
blob	below	916	52,003
blob	above	114	45,294
ihex	above	461	21,606
elf	above	211	15,859
coff	above	217	3,342
mach-o	above	5	3,051
elf	below	6	1,504
hex	above	27	1,176

Noise floor. CFGFast is not reproducible on every input (angr/angr#6840), so the comparison is over sorted block sets rather than counts, and the stock arm was repeated in a second fresh child wherever it was cheap. Across 508 placement, 260 control and 33 synthetic objects the two stock runs disagreed 0 times. The one control-group difference observed is shown directly to be that nondeterminism rather than an effect of either change.

Full local gate. cle: 228 passed, 9 skipped. angr: 2,480 passed, 46 skipped, 2 xfailed, 260 subtests passed. angr Rust: 35 passed. Workspace checks, test-inputs and all pre-commit hooks passed with no rewrite.

Lint and type checks, merge-base relative, over the changed files:

ok   cle/loader.py: 9.93 -> 9.93                            badness 0.029049 -> 0.028902
ok   cle/tests/test_rebase.py: 10.00 -> 10.00               badness 0.0 -> 0.0
ok   angr/analyses/cfg/cfg_fast.py: 9.90 -> 9.90            badness 0.157150 -> 0.157076
ok   angr/tests/analyses/cfg/test_cfgfast.py: 9.91 -> 9.91  badness 0.086873 -> 0.084906

COFF is excluded from the credit. #761 makes a COFF object's undefined externals into imports, so angr hooks them and stops decoding them; a hooked address is exactly what the CFGFast guard lets through. On COFF the two reach the same end by different routes and are not additive, so the 41 COFF objects among the 4,327 measured are broken out and measured again with cle#761 applied to both sides rather than counted here.

Rebase. Rebased onto 5125f1ba0 with no conflict; git range-diff reports the commit identical. The previous head sat five commits back, before #762 added the Mach-O section-attribute predicates, so angr's tests/analyses/cfg/test_cfg_macho_sections.py — which master's angr now carries and which asserts __cstring and the ObjC string sections are not scanned — failed against it. Those two failures were the stale base, not this change: the diff touches only _find_safe_rebase_addr. On the rebased head pytest tests/test_rebase.py gives 6 passed. The population figures above were taken on 45c6509 and are carried forward; none of the five intervening commits touches cle/loader.py.

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_765