Clarification on noise_dropout Argument

[LEEYOONHYUNG]

Hi,

Thank you for your excellent work! I have a question regarding the noise_dropout argument of DitheredFSQ module (code).

From my understanding, there is no explicit mention of using "unquantized" input in your original paper, but it appears to be the case based on your code. Also, it seems that the stochasticity of choosing between "unquantized," "quantized," and "pseudo-quantized" techniques is solely determined by the noise_dropout argument. Moreover, considering the order of random masking, it appears that pseudo-quantization is used most frequently (about 50% of the time if the noise_dropout is 0.5 as in your paper), while the other techniques are used 25% of the time each.

Is my understanding correct, and was this your intention? If so, could you please share your experience and intuition behind this approach?

Lastly, it seems that the current code does not include the part of randomly changing the quantization levels to achieve the effects of quantizer-dropout. Is this correct as well?

Thank you for your great work!

[julian-parker]

HI @LEEYOONHYUNG, thanks for your interest!

You are correct about the dropout. This was an oversight in the paper, which has already been fixed for the camera-ready version.

You are also correct that the changing of quantization levels is not in the current codebase. I'll look at adding this in the next update.

[LEEYOONHYUNG]

@julian-parker

Thank you for the clarification. I look forward to seeing the updates to the codebase regarding the quantization level changes.

Additionally, I believe the order of applying the Bernoulli mask can lead to different results, and there seems to be an inconsistency between the codebase and your camera-ready version. If possible, you may want to consider making changes.