[BUG][pt_expt][DPA1] Automatic graph-lower export changes predictions for dense-trained se_atten_v2 models with nonzero davg

[hcustc]

Bug summary

I trained a DPA1-L0 model with the standard PyTorch backend (dp --pt train) and converted the
frozen .pth model to .pt2 using dp convert-backend. The descriptor used the default
set_davg_zero: false and has nonzero davg.

The automatically selected graph-lower model loads and runs, but it is not numerically
equivalent to the source model. On the same six-frame system, the force-component difference
is:

• MAE: 1.1381067 eV/Angstrom
• RMSE: 1.7619205 eV/Angstrom
• Maximum absolute difference: 6.7789032 eV/Angstrom

In contrast, a manually exported nlist/dense-lower .pt2 model reproduces the source model
with a force-component MAE of 5.13e-6 eV/Angstrom and a maximum difference of 1.91e-5 eV/ Angstrom.

The discrepancy is reproduced by an uncompiled CPU eager comparison between
forward_common_lower and forward_common_lower_graph, and is already present in the
descriptor output. As a diagnostic, setting the descriptor mean (davg) to zero in memory
reduces the descriptor discrepancy to numerical noise.

I also independently retrained four models with set_davg_zero: true. After automatic graph-
lower conversion, their force-component MAE differences are 1.76e-5–1.84e-5 eV/Angstrom,
with a maximum observed difference of 1.33e-4 eV/Angstrom.

These results indicate that automatic conversion accepts a dense-trained se_atten_v2 model
with nonzero davg, even though its graph-lower representation is not functionally equivalent.

DeePMD-kit Version

0.1.dev1+gbc902daf7.d20260809

Backend and its version

PyTorch 2.11.0+cu128

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.

A minimal reproducer archive is attached containing:

reproducer/
├── frozen_model.pth
├── input.json
├── compare_predictions.py
└── test_system/
    ├── type.raw
    ├── type_map.raw
    ├── nopbc
    └── set.000/
        ├── box.npy
        ├── coord.npy
        ├── energy.npy
        └── force.npy

The test system is nonperiodic and contains six frames with seven atoms per frame.

Relevant model settings:

{
  "type_map": ["C", "Cl", "H", "O"],
  "descriptor": {
    "type": "se_atten_v2",
    "sel": 250,
    "rcut": 6.0,
    "attn": 128,
    "attn_layer": 0,
    "tebd_input_mode": "strip",
    "precision": "float32"
  }
}

set_davg_zero was not specified during training, so its effective value is false. The trained
descriptor has nonzero davg; its largest absolute value is approximately 0.01418.

Environment:

DeePMD-kit: 0.1.dev1+gbc902daf7.d20260809
Git commit: bc902daf747d93c5d7314338d287fd2dd0a7e32c
PyTorch: 2.11.0+cu128
Python: 3.12.12
OS: Linux x86_64

The build is based on the official master branch. The only local source change preserves an
explicitly requested `CMAKE_CUDA_ARCHITECTURES=120`; no Python, model, descriptor,
serialization, or inference code was modified.

There is no exception or error traceback. Conversion and inference both exit successfully. This
is a silent numerical correctness failure.

Observed source .pth versus automatic graph-lower .pt2 difference:

Energy MAE:                  1.3291797 eV/frame
Energy maximum difference:  2.0507687 eV/frame
Force-component MAE:        1.1381067 eV/Angstrom
Force-component RMSE:       1.7619205 eV/Angstrom
Force maximum difference:   6.7789032 eV/Angstrom

For comparison, manually exporting the same model through nlist/dense lower gives a force-
component MAE of 5.13e-6 eV/Angstrom and a maximum difference of 1.91e-5 eV/Angstrom.

### Steps to Reproduce

[deepmd-dpa1-graph-lower-reproducer.zip](https://github.com/user-attachments/files/31117915/deepmd-dpa1-graph-lower-reproducer.zip)

1. Extract the attached minimal reproducer and enter its directory:

   ```bash
   unzip deepmd-dpa1-graph-lower-reproducer.zip
   cd deepmd-dpa1-graph-lower-reproducer

2. Convert the supplied source model. dp convert-backend automatically selects graph lower:

   dp convert-backend frozen_model.pth model_graph.pt2

3. Evaluate the source and converted models on the same six-frame system:

  dp --pt test -m frozen_model.pth -s test_system -n 32 -d pth
  dp --pt-expt test -m model_graph.pt2 -s test_system -n 32 -d graph

4. Compare their predictions:

   python compare_predictions.py pth graph

   Expected reproduction output:

   Energy MAE difference:              1.3291797 eV/frame
   Energy maximum absolute difference: 2.0507687 eV/frame
   Force-component MAE:                1.1381067 eV/Angstrom
   Force-component RMSE:               1.7619205 eV/Angstrom
   Force-component maximum difference: 6.7789032 eV/Angstrom

No retraining, LAMMPS, or Kokkos is required. The same underlying dense-versus-graph
discrepancy was also reproduced independently in uncompiled CPU eager execution.

### Further Information, Files, and Links

_No response_

[dosubot]

Hi @hcustc! I'm Dosu and I'm helping the DeepModeling team.

This is a variant of a previously reported bug. The root cause is the same phantom padding-neighbor residual documented in deepmd/dpmodel/descriptor/dpa1.py: the dense lower path (forward_common_lower) retains a -davg/dstd contribution for every vacant padding slot in the fixed-capacity neighbor list, while the graph lower path (forward_common_lower_graph) omits padding neighbors entirely [1]. When davg ≠ 0, the two paths are not numerically equivalent.

Your case differs from issue #5862 (which covers .pt checkpoint loading): here the discrepancy appears during dp convert-backend to .pt2. The export gate model_uses_graph_lower in deepmd/pt_expt/utils/serialization.py does not check whether the model has nonzero davg, so a dense-trained model with set_davg_zero: false gets silently exported into a graph-lower .pt2 whose descriptor semantics differ from the source.

Your set_davg_zero: true retrained models round-trip cleanly (MAE ~1.8×10⁻⁵ eV/Å) because with davg = 0 the padding residual vanishes and the two paths produce identical outputs [2].

Two potential fixes to consider:

1. Block automatic graph-lower export when davg ≠ 0 — make dp convert-backend fall back to the nlist/dense-lower format and warn the user, rather than silently producing an inequivalent model.
2. Zero out davg in the graph-lower export path — mathematically valid if the graph path has already absorbed the normalization elsewhere, but this needs verification against the DPA1 aggregation semantics.

The companion issue #5824 proposes decoupling DPA1 graph normalization and statistics from the dense sel-based representation [3], which is the structural prerequisite for option 2. Issue #5862 tracks the related .pt checkpoint path mismatch [1].

Immediate workaround: Manually export via the nlist/dense-lower path (as you've already done), which you confirmed gives a force-component MAE of ~5×10⁻⁶ eV/Å. Until the export gate logic is fixed, models trained with set_davg_zero: false should not use the automatically selected graph-lower .pt2 format.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[njzjz-bot]

Dosu's root-cause diagnosis is correct, but one suggested fix needs correction.

The source .pth model was trained with the legacy dense-lower semantics. With nonzero davg, converting it to a graph lower changes the represented function because the dense path retains the historical padding contribution while the graph path has no padding edges.

Do not zero davg in an already trained model. The normalization statistics are part of the trained function; changing them after training changes the descriptor inputs and generally changes the model predictions. set_davg_zero: true is a valid workaround only when used for training/retraining, as the reporter tested.

The conversion-side fix is PR #5975: preserve the source artifact's concrete lower-input semantics instead of automatically reinterpreting every graph-capable reconstructed model as graph-native. For a legacy dense .pth source, conversion should therefore emit the nlist/dense lower. Until that fix is merged, the reporter's manual dense-lower export is the correct workaround.

Coding agent: Codex
Codex version: codex-cli 0.149.0
Model: gpt-5.6-sol
Reasoning effort: xhigh