MotifBench benchmark reproducibility issue

[paoloc03]

Hello,

Thank you for your work, truly appreciated.

I am trying to reproduce the MotifBench benchmark data as in issue #5 but I end up getting 0 unique successful designs for each protein listed in the .yaml file.

I used the default config 03_motifbench.yaml, run with:
python3 -m protpardelle.sample 03_motifbench.yaml --motif-dir ../examples/motifs/motif_bench --num-samples 100 --num-mpnn-seqs 8

This is an example of the output file I get for 05_5ZE9, for which I should obtain several unique solutions:
INFO:main:Sampling 05_5ZE9 took 32.79 seconds.
INFO:main:Redundant success rate: 0/100
INFO:main:Non-redundant success rate: 0/100
INFO:main:Sampling concluded after 1528.71 seconds.
INFO:main:Of this, 32.79 seconds were for actual sampling.
INFO:main:100 total samples were drawn.

Can you help me identifying the problem?

[tianyu-lu]

Hi,

Thank you for your interest and apologies for the late response.

I would recommend evaluating the samples with MotifBench code directly for reproducibility. The sampling success rates here are specific to the La-Proteina motif scaffolding benchmark which are not the same as the definition of a success by MotifBench:

Adjusting these thresholds to backbone-only self-consistency success thresholds should change the results. Let me know if you run into any issues!