Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions docs/portal/operate/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,44 @@ Confirm the client's `taskQueue` exactly matches the worker's queue. Then confir

If the workflow scheduled an activity, repeat the same check for the activity type. A workflow worker can be healthy while the activity capacity it needs is absent.

## Inspect a durable service call

Use an operator credential in the namespace you are investigating. The
service catalog read methods return bounded pages and the original durable
call, including its deadline, attempts and terminal cause.

```php
$page = $client->listServiceCatalogCalls([
'scope' => 'caller',
'status_bucket' => 'failed',
'per_page' => 50,
]);
foreach ($page['data'] as $row) {
$call = $client->describeServiceCatalogCall($row['id']);
}
```

Browse registered endpoints, services and operations with
`listServiceCatalogEndpoints()`, `listServiceCatalogServices()` and
`listServiceCatalogOperations()`. Their corresponding
`describeServiceCatalogEndpoint()`, `describeServiceCatalogService()` and
`describeServiceCatalogOperation()` methods use catalog IDs. Lists accept
`page` and `per_page`. Filter services by `endpoint_id`, operations by
`endpoint_id` or `service_id`, and calls by those IDs or `operation_id`.

The methods require the backend's advertised `service_catalog_read_contract`
version 1. An absent or incompatible capability raises `ServerException` with
`reason: backend_capability_unavailable` and the required/advertised contract.
A discovery outage keeps its actual connection or backend failure. It does not
become an unsupported-capability message.

Foreign linked runs remain opaque references. Foreign input, output and failure
payload references are null, with `payloads_in_observer_namespace: false`.
Authorized payload references remain opaque runtime objects. A visible call
does not authorize browsing another namespace. See the
[Server read contract](https://github.com/durable-workflow/server/blob/main/docs/contracts/service-catalog-reads.md)
for scopes, filters, pagination and detail bounds.

## A result wait timed out

`WorkflowHandle::result()` polls for a bounded caller duration. When it times out, the durable execution may still be running. Call `describe()` and inspect its status before starting anything again. Reuse the original workflow ID or an application idempotency key; do not create duplicates just because the client lost patience.
Expand Down
94 changes: 94 additions & 0 deletions src/Client.php
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,100 @@ public function clusterInfo(bool $includeDiagnostics = false): ClusterInfo
return ClusterInfo::fromArray($this->control('GET', $path));
}

/**
* @param array<string, int|string> $filters
* @return array<string, mixed>
*/
public function listServiceCatalogEndpoints(array $filters = []): array
{
return $this->serviceCatalogRead('/service-catalog/endpoints', $filters);
}

/** @return array<string, mixed> */
public function describeServiceCatalogEndpoint(string $endpointId): array
{
return $this->serviceCatalogRead('/service-catalog/endpoints', id: $endpointId);
}

/**
* @param array<string, int|string> $filters
* @return array<string, mixed>
*/
public function listServiceCatalogServices(array $filters = []): array
{
return $this->serviceCatalogRead('/service-catalog/services', $filters);
}

/** @return array<string, mixed> */
public function describeServiceCatalogService(string $serviceId): array
{
return $this->serviceCatalogRead('/service-catalog/services', id: $serviceId);
}

/**
* @param array<string, int|string> $filters
* @return array<string, mixed>
*/
public function listServiceCatalogOperations(array $filters = []): array
{
return $this->serviceCatalogRead('/service-catalog/operations', $filters);
}

/** @return array<string, mixed> */
public function describeServiceCatalogOperation(string $operationId): array
{
return $this->serviceCatalogRead('/service-catalog/operations', id: $operationId);
}

/**
* @param array<string, int|string> $filters
* @return array<string, mixed>
*/
public function listServiceCatalogCalls(array $filters = []): array
{
return $this->serviceCatalogRead('/service-catalog/calls', $filters);
}

/** @return array<string, mixed> */
public function describeServiceCatalogCall(string $callId): array
{
return $this->serviceCatalogRead('/service-catalog/calls', id: $callId);
}

/**
* @param array<string, int|string> $filters
* @return array<string, mixed>
*/
private function serviceCatalogRead(string $path, array $filters = [], ?string $id = null): array
{
if ($id !== null) {
if (trim($id) === '') {
throw new InvalidArgumentException('Catalog detail requires a non-empty ID.');
}
$path .= '/'.$this->segment($id);
}
$schema = 'durable-workflow.v2.service-catalog-read.contract';
$contract = $this->clusterInfo()->raw['service_catalog_read_contract'] ?? null;
if (!is_array($contract) || ($contract['schema'] ?? null) !== $schema || ($contract['version'] ?? null) !== 1) {
throw new ServerException(
'The connected backend does not advertise service catalog operator reads with contract version 1.',
501,
'backend_capability_unavailable',
[
'capability' => 'service_catalog_reads',
'required_contract' => $schema,
'required_contract_version' => 1,
'advertised_contract' => $contract,
],
);
}
if ($filters !== []) {
$path .= '?'.http_build_query($filters, '', '&', PHP_QUERY_RFC3986);
}

return $this->control('GET', $path);
}

/** Discover ordinary runtime capabilities using the worker credential. */
public function workerClusterInfo(): ClusterInfo
{
Expand Down
121 changes: 121 additions & 0 deletions tests/ServiceCatalogReadClientTest.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
<?php

declare(strict_types=1);

namespace DurableWorkflow\Tests;

use DurableWorkflow\Client;
use DurableWorkflow\Exception\ServerException;
use DurableWorkflow\Exception\TransportException;
use DurableWorkflow\Tests\Support\FakeTransport;
use PHPUnit\Framework\Attributes\DataProvider;
use PHPUnit\Framework\TestCase;

final class ServiceCatalogReadClientTest extends TestCase
{
#[DataProvider('readMethods')]
public function testReadsUseDiscoveredContractAndScopedControlPlane(string $method, string $path, bool $detail): void
{
$surface = ['data' => [['id' => 'call-1']], 'call_deadline_at' => '2026-10-11T19:00:01.123456Z',
'failure_type' => 'ServiceCallTimedOut', 'input_payload_reference' => ['reference_id' => 'opaque-1']];
$transport = new FakeTransport([$this->discovery(), $surface]);
$client = new Client('https://server.example', token: 'operator-token', namespace: 'tenant-a', transport: $transport);
$result = $detail ? $client->{$method}('call/1') : $client->{$method}(['page' => 2, 'per_page' => 10]);

self::assertSame($surface, $result);
self::assertCount(2, $transport->requests);
self::assertSame('https://server.example/api/cluster/info', $transport->requests[0]['uri']);
self::assertSame('https://server.example/api/service-catalog/'.$path.($detail ? '/call%2F1' : '?page=2&per_page=10'), $transport->requests[1]['uri']);
foreach ($transport->requests as $request) {
self::assertSame('GET', $request['method']);
self::assertNull($request['body']);
self::assertSame('tenant-a', $request['headers']['X-Namespace']);
self::assertSame('2', $request['headers']['X-Durable-Workflow-Control-Plane-Version']);
self::assertSame('Bearer operator-token', $request['headers']['Authorization']);
}
}

public static function readMethods(): array
{
return [
['listServiceCatalogEndpoints', 'endpoints', false], ['describeServiceCatalogEndpoint', 'endpoints', true],
['listServiceCatalogServices', 'services', false], ['describeServiceCatalogService', 'services', true],
['listServiceCatalogOperations', 'operations', false], ['describeServiceCatalogOperation', 'operations', true],
['listServiceCatalogCalls', 'calls', false], ['describeServiceCatalogCall', 'calls', true],
];
}

public function testDescribeRejectsBlankIdentitiesBeforeDiscovery(): void
{
$transport = new FakeTransport();
$client = new Client('https://server.example', transport: $transport);
foreach (self::readMethods() as [$method, , $detail]) {
if (!$detail) {
continue;
}
foreach (['', ' '] as $id) {
try {
$client->{$method}($id);
self::fail('Blank identity must not turn a detail lookup into a collection read.');
} catch (\InvalidArgumentException $exception) {
self::assertStringContainsString('non-empty ID', $exception->getMessage());
self::assertCount(0, $transport->requests);
}
}
}
}

#[DataProvider('unsupportedContracts')]
public function testUnsupportedBackendIsExplainedBeforeAttemptingARead(mixed $contract): void
{
$transport = new FakeTransport([['service_catalog_read_contract' => $contract]]);
$client = new Client('https://server.example', transport: $transport);
try {
$client->listServiceCatalogCalls();
self::fail('Missing or unsupported read capability must be diagnosed.');
} catch (ServerException $exception) {
self::assertSame(501, $exception->status);
self::assertSame('backend_capability_unavailable', $exception->reason);
self::assertSame('service_catalog_reads', $exception->details['capability']);
self::assertSame(1, $exception->details['required_contract_version']);
self::assertSame($contract, $exception->details['advertised_contract']);
self::assertCount(1, $transport->requests);
}
}

public static function unsupportedContracts(): array
{
return [[null], ['legacy'], [[]], [['schema' => 'other', 'version' => 1]],
[['schema' => 'durable-workflow.v2.service-catalog-read.contract', 'version' => 2]],
[['schema' => 'durable-workflow.v2.service-catalog-read.contract', 'version' => '1']]];
}

public function testActualReadRefusalsRetainTheServerReasonAndDiscoveryFailuresAreNotUnsupported(): void
{
$transport = new FakeTransport([$this->discovery(), new TransportException('Forbidden', 403, ['reason' => 'forbidden'])]);
$client = new Client('https://server.example', transport: $transport);
try {
$client->describeServiceCatalogCall('call-1');
self::fail('Worker role must remain refused by the backend.');
} catch (ServerException $exception) {
self::assertSame(403, $exception->status);
self::assertSame('forbidden', $exception->reason);
}

$transport = new FakeTransport([new TransportException('Backend unavailable', 503, ['reason' => 'backend_unavailable'])]);
$client = new Client('https://server.example', transport: $transport);
try {
$client->listServiceCatalogCalls();
self::fail('Discovery failure must retain the backend failure.');
} catch (ServerException $exception) {
self::assertSame(503, $exception->status);
self::assertSame('backend_unavailable', $exception->reason);
self::assertCount(1, $transport->requests);
}
}

private function discovery(): array
{
return ['service_catalog_read_contract' => ['schema' => 'durable-workflow.v2.service-catalog-read.contract', 'version' => 1]];
}
}
Loading