[Bug]: ExecutionConfig retry settings are ignored for tool failures

[kevinyang03]

Describe the bug

When a tool call fails with an exception, the retry policy configured in ExecutionConfig (maxAttempts, retryOn, backoff) is never applied. The tool is invoked exactly once and the failure is immediately returned as a ToolResultBlock error result, even for clearly retryable errors such as IOException (network/transport failures).

Root cause: ToolExecutor.executeWithInfrastructure(...) runs the compatibility entry execute(...), which converts tool failures into ToolResultBlock error results inside executeCore(...) before the scheduling/timeout/retry layers run. By the time applyRetry(...) executes, failures are already completed results, so retryWhen never sees an error signal. Only the error emitted by the timeout(...) operator can still reach the retry layer.

To Reproduce

Steps to reproduce the behavior:

1. Add the following test class to agentscope-core/src/test/java/io/agentscope/core/tool/RetryReproTest.java:

package io.agentscope.core.tool;

import io.agentscope.core.message.ToolResultBlock;
import io.agentscope.core.message.ToolUseBlock;
import io.agentscope.core.model.ExecutionConfig;
import java.io.IOException;
import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.Test;
import reactor.core.publisher.Mono;

/** Minimal reproduction for tool retry behavior. */
class RetryReproTest {

    static final AtomicInteger CALLS = new AtomicInteger(0);

    static class FlakyTools {
        @Tool(name = "flaky", description = "Fails once, then recovers")
        public String flaky() throws IOException {
            if (CALLS.incrementAndGet() == 1) {
                throw new IOException("Network down");
            }
            return "recovered";
        }
    }

    @Test
    void reproduce() {
        Toolkit toolkit = new Toolkit();
        toolkit.registerTool(new FlakyTools());

        ExecutionConfig config =
                ExecutionConfig.builder()
                        .maxAttempts(3) // 1 initial attempt + up to 2 retries
                        .initialBackoff(Duration.ofMillis(100))
                        .maxBackoff(Duration.ofMillis(200))
                        .retryOn(ExecutionConfig.RETRYABLE_ERRORS) // IOException is retryable
                        .build();

        ToolUseBlock call =
                ToolUseBlock.builder()
                        .id("call-1")
                        .name("flaky")
                        .input(Map.of())
                        .content("{}") // required: schema validation reads ToolUseBlock.content()
                        .build();

        List<ToolResultBlock> results =
                toolkit.callTools(List.of(call), config, null, null).block();

        System.out.println("REPRO attempts = " + CALLS.get());
        System.out.println("REPRO state    = " + results.get(0).getState());
        System.out.println("REPRO output   = " + results.get(0).getOutput());
    }

    @Test
    void timeoutComparison() {
        AtomicInteger timeoutCalls = new AtomicInteger(0);
        Toolkit toolkit = new Toolkit();
        toolkit.registerTool(
                new AgentTool() {
                    @Override
                    public String getName() {
                        return "never";
                    }

                    @Override
                    public String getDescription() {
                        return "never completes";
                    }

                    @Override
                    public Map<String, Object> getParameters() {
                        return Map.of();
                    }

                    @Override
                    public Mono<ToolResultBlock> callAsync(ToolCallParam param) {
                        timeoutCalls.incrementAndGet();
                        return Mono.never();
                    }
                });

        ExecutionConfig config =
                ExecutionConfig.builder()
                        .timeout(Duration.ofMillis(100))
                        .maxAttempts(3)
                        .initialBackoff(Duration.ofMillis(50))
                        .maxBackoff(Duration.ofMillis(100))
                        .build();

        ToolUseBlock call =
                ToolUseBlock.builder()
                        .id("call-2")
                        .name("never")
                        .input(Map.of())
                        .content("{}")
                        .build();

        List<ToolResultBlock> results =
                toolkit.callTools(List.of(call), config, null, null).block();

        System.out.println("REPRO timeout attempts = " + timeoutCalls.get());
        System.out.println("REPRO timeout state    = " + results.get(0).getState());
        System.out.println("REPRO timeout output   = " + results.get(0).getOutput());
    }
}

2. Run it from the repository root:

mvn -pl agentscope-core test -Dtest=RetryReproTest -Dsurefire.failIfNoSpecifiedTests=false

3. See the printed output and the surefire report (see Error messages).

Expected behavior

The first failure should surface as an error signal to the retry layer, so the tool is re-invoked and recovers:

REPRO attempts = 2
REPRO state    = RUNNING
REPRO output   = ["recovered"]

(RUNNING is the framework's default state for a plain successful tool result.) In the timeout comparison, the tool should be re-invoked on every attempt and the final error result should keep the root cause:

REPRO timeout attempts = 3
REPRO timeout state    = ERROR
REPRO timeout output   = [Error: Tool execution failed: Tool execution timeout after PT0.1S]

Error messages

No exception is thrown; the symptom is the missing retry. Actual output on the current main:

REPRO attempts = 1
REPRO state    = ERROR
REPRO output   = [Error: Tool execution failed: Network down]
REPRO timeout attempts = 1
REPRO timeout state    = ERROR
REPRO timeout output   = [Error: Tool execution failed: Retries exhausted: 2/2]

Observations:

• For the flaky tool, no Retrying tool call 'flaky' ... warning ever appears: the retry layer never runs for tool exceptions.
• The timeout comparison isolates the two halves of the bug: the timeout error signal is retried (the Retries exhausted: 2/2 message proves 3 attempts), but the tool is still only invoked once (REPRO timeout attempts = 1) — retries merely re-subscribe the already-assembled chain instead of re-invoking the tool. Additionally, the exhausted-retry message replaces the real cause (Tool execution timeout after PT0.1S) with the generic Retries exhausted: 2/2.

Environment (please complete the following information):

• AgentScope-Java Version: 2.0.3-SNAPSHOT (reproduced on main at commit ddad42e3; the same code path exists in earlier 2.0.x)
• Java Version: 17
• OS: macOS (the issue is OS-independent)

Additional context

• Relevant code: ToolExecutor.executeCore(...) converts failures with ToolResultBlock.error(...); ToolExecutor.executeWithInfrastructure(...) then pipes that chain through applyScheduling / applyTimeout / applyRetry / applyShutdownGuard, so applyRetry only ever sees completed results (except timeouts).
• A fix needs the tool invocation to keep failures as reactive error signals until after the timeout/retry layers (for example, a separate execution channel that is re-invoked per attempt), while preserving the existing Toolkit.callTool(...) contract that direct calls still receive error results.
• The fix must also keep ToolSuspendException (external-execution suspension) from ever being retried, even when it is wrapped in reflection/future/arbitrary exceptions, since replaying a suspension duplicates side effects.
• I am preparing a fix and will open a PR referencing this issue.