[integrations][java] Add OpenAI embedding model

[purushah]

Closes #1103.

Adds a Java OpenAI embedding model integration, closing the Java/Python parity gap for embeddings: Python already had OpenAIEmbeddingModelConnection/OpenAIEmbeddingModelSetup; Java had only Ollama and Bedrock, so a Java agent needed the Python wrapper (and a Python runtime on the TaskManagers) for a plain HTTPS call.

What's in the change

• New module integrations/embedding-models/openai with OpenAIEmbeddingModelConnection and OpenAIEmbeddingModelSetup, built on com.openai:openai-java (already a dependency of the OpenAI chat module; no new third-party dependency).
• Arguments mirror the Python classes one for one. Connection: api_key (required), base_url, request_timeout (seconds, 0 disables), max_retries, organization, project. Setup: connection, model, encoding_format (float or base64), dimensions, user, additional_kwargs (forwarded as extra body properties; keys that repeat a typed request field are rejected, as in the Watsonx chat connection).
• A batch of texts is sent as one request; results are placed by the response index so input order is preserved even if the API reorders. embedWithUsage reports prompt_tokens/total_tokens, so the embedding token metrics from [api][python][java] Track embedding token usage metrics #870 cover this provider.
• Registration: ResourceName.EmbeddingModel.OPENAI_CONNECTION/OPENAI_SETUP, the Python ResourceName.EmbeddingModel.Java mirror, YAML aliases (openai) in both the Java Aliases table and aliases.py, dist, ide-support, and the e2e integration module.
• Docs: embedding_models.md OpenAI section gains Java tabs (usage, connection and setup parameters); the Python-only hint, the FAQ support matrix, the cross-language example, and the YAML alias table are updated.
• Tests: OpenAIEmbeddingModelTest (29 tests, SDK client mocked, following BedrockEmbeddingModelTest): client wiring for defaults and explicit options (base URL, timeout, retries, organization, project, blank strings), missing api_key, timeout/retry parsing bounds, setup parameters and argument-type validation (including a missing or non-string connection/model and an additional_kwargs value the SDK cannot serialize), encoding_format validation, reserved additional_kwargs keys, single and batch embeddings with token usage, a response without usage, with partial usage, or with decimal-string counts, out-of-order, index-less, consistent and contradictory partially indexed, non-integer-index, and out-of-range-index response handling, null input rejection with its position, dimensions range validation per call, parameter forwarding (encoding format, dimensions, user, additional_kwargs as body properties, other per-call keys not forwarded), a base64 response decoded to floats (with a truncated payload rejected rather than silently shortened), a malformed vector reported with its position, an empty batch validated without a request, a response without data, with a malformed data field, with a null item, or with a string vector when float was requested, mismatched response size, missing model. AliasesTest and the Python test_aliases.py cover the new openai embedding alias in both languages. The Python test_openai_embedding_model.py gains mocked tests for request_timeout: 0, invalid timeout/retry bounds, additional_kwargs sent as extra_body, reserved-key rejection, encoding_format/dimensions validation, base64 decoding with index ordering and the prompt-token fallback, short/contradictory/duplicate-index, malformed or corrupted-base64 vector, unusable-usage and missing-data responses, empty batches, and blank/null-string default handling. EmbeddingIntegrationTest gains an OPENAI provider (gated on OPENAI_API_KEY, like the chat model integration test).

Review follow-ups deliberately left out of this PR: chunking batches at batch_size (the OpenAI per-request input cap applies equally to the Python connection and to how the vector stores call embed), unwrapping InvocationTargetException in JavaResourceProvider so constructor errors surface their message, validating connection/model in BaseEmbeddingModelSetup for every setup, shared numeric-argument, reserved-key and token-usage helpers for the OpenAI family, a provider-to-required-env-vars table for the integration test gates, coercing quoted numeric YAML scalars in Java (or rejecting them in the Python connection's request_timeout/max_retries, which still use pydantic's lax coercion), probing additional_kwargs values for JSON-serializability in the Python setup as the Java setup does, and rejecting fractional response indices that the Java SDK's Jackson mapper truncates to integers. Each is a cross-cutting change beyond #1103. One review suggestion was declined on purpose: requesting base64 on the wire by default would cut response size, but both connections send the configured encoding_format verbatim (default float) so the wire format is the same in both languages and for OpenAI-compatible servers. Another: falling back to response order when a server returns duplicate or shifted index values (some proxies do) would silently risk attributing vectors to the wrong texts, so both connections fail the call instead.

Naming follows the Python embedding connection (base_url, request_timeout) rather than the Java chat connection (api_base_url, timeout), as proposed in the issue, so the two embedding implementations align across languages.

Design notes from self-review: only the setup's additional_kwargs reach the request body, and other per-call parameters such as a caller's timeout are ignored. Per-call parameters override setup parameters entry by entry (the base class contract), so a per-call additional_kwargs map replaces the setup's map; the javadoc says so. The Python OpenAIEmbeddingModelConnection is aligned in this PR: it now sends additional_kwargs as extra_body, rejects keys that repeat the typed fields, bounds request_timeout/max_retries like the Python OpenAI chat models, validates encoding_format and dimensions, decodes base64 responses (previously the characters of the base64 string were returned; both languages now reject a payload whose length is not a multiple of four instead of returning a short vector), and applies the same response size and index checks as Java. A usage block missing prompt_tokens reports prompt = total in both languages, as the Tongyi connection does.

Compatibility impact (Python OpenAIEmbeddingModelSetup): additional_kwargs entries that are not typed fields were previously dropped and are now sent in the request body; entries that repeat a typed field (for example {"dimensions": 8}) previously overrode it silently and now fail validation; encoding_format outside float/base64, non-integer or non-positive dimensions, a blank connection, non-string typed arguments, malformed additional_kwargs, and non-string batch elements now fail validation instead of reaching the API. The docs state the current contract only; this description carries the compatibility notes for the release notes. Blank user/encoding_format/base_url are treated as absent/default as in Java (a blank Python api_key stays accepted for unauthenticated OpenAI-compatible servers, as before this PR; Java rejects it like the Java chat connection); the resolved encoding_format is always sent on the wire in both languages. No Java compatibility impact (new module). Responses without usage, with partial usage, or without per-item index (OpenAI-compatible servers) are tolerated; an item without an index keeps its response position and every slot must be filled exactly once, so out-of-range, duplicated or contradictory indices fail the call rather than being guessed (same rule in Python). request_timeout: 0 disables the timeout like the Java and Python OpenAI chat connections; the Python embedding connection gets the same one-line 0 -> None mapping (with a test) so both languages agree. Numeric parsing follows the Java chat connection's exact-BigDecimal rules (sub-millisecond values round up, integrality via toBigIntegerExact). Argument validation has one owner, package-private parsers in the connection, used both by the setup at construction and per call, so per-call overrides cannot bypass it. The OPENAI case of EmbeddingIntegrationTest is gated on OPENAI_API_KEY like the chat provider cases; CI does not supply that secret, so it is manual-only, and the results below come from a local run.

Test commands and results

Unit tests for the new module:

mvn -pl integrations/embedding-models/openai test
# Tests run: 29, Failures: 0, Errors: 0, Skipped: 0 -- BUILD SUCCESS

Complete Java unit suite (all non-e2e modules, as tools/ut.sh runs it):

mvn test -pl '!e2e-test/flink-agents-end-to-end-tests-integration,!e2e-test/flink-agents-end-to-end-tests-resource-cross-language'
# 19 modules, 221 test classes: Tests run: 2286, Failures: 0, Errors: 0, Skipped: 42 -- BUILD SUCCESS
# (earlier revisions of this description double-counted the per-class lines as 3575)

Complete Python unit suite (e2e tests excluded, as tools/ut.sh does) plus the cross-language resource-name consistency check:

cd python && uv sync --extra test && uv pip install 'apache-flink~=2.3.0'
uv run --no-sync pytest flink_agents -k "not e2e_tests" -m "not integration"
# 1356 passed, 13 skipped, 171 deselected
uv run --no-sync pytest flink_agents/integrations/embedding_models/tests/test_openai_embedding_model.py
# 11 passed, 1 skipped (the live-API test needs TEST_API_KEY; the mocked tests are part of the unit suite above)
uv run --no-sync python ../e2e-test/test-scripts/check_resource_consistency.py
# ResourceName Cross-language consistency check successful

End-to-end, real OpenAI API, embedded Flink 2.3 cluster (OPENAI_API_KEY exported from a local file, never printed):

mvn -pl e2e-test/flink-agents-end-to-end-tests-integration test -Dtest=EmbeddingIntegrationTest -Pflink-2.3
# Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 -- BUILD SUCCESS
# [OLLAMA] 10 embeddings, Dimension=768; [OPENAI] 10 embeddings, Dimension=256 (text-embedding-3-small, dimensions=256)

End-to-end, real OpenAI API, standalone local Flink 2.3.0 cluster (start-cluster.sh, flink-agents-dist-flink-2.3 in lib/, throwaway job submitted with flink run, results written by a FileSink):

text=Apache Flink processes streams|dim=256|norm=1.000|batch=2|batchDim=256|promptTokens=5|totalTokens=13
text=Embeddings map text to vectors|dim=256|norm=1.000|batch=2|batchDim=256|promptTokens=6|totalTokens=15
text=Model routing picks a model per request|dim=256|norm=1.000|batch=2|batchDim=256|promptTokens=7|totalTokens=17
# job openai-embedding-e2e FINISHED, no exceptions in JobManager/TaskManager logs

The requested dimensions are honored, OpenAI's unit-normalized vectors come back with norm 1.000, batch calls return one vector per input, and token usage is reported for single and batch calls.

The key-gate fix in OllamaPreparationUtils.hasApiKey also re-arms the OPENAI_RESPONSES case of ChatModelIntegrationTest (it was gated on a variable nothing sets); that test was run once locally with OPENAI_API_KEY exported:

mvn -pl e2e-test/flink-agents-end-to-end-tests-integration test -Dtest=ChatModelIntegrationTest -Pflink-2.3
# Tests run: 5, Failures: 0, Errors: 0, Skipped: 2 -- BUILD SUCCESS (OLLAMA, OPENAI, OPENAI_RESPONSES ran; ANTHROPIC and AZURE_OPENAI skipped, no keys)

Also run: ruff check and ruff format --check on the touched Python files, Hugo build of the docs, spotless on the touched modules.

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Claude Code 2.1.273 (Claude Fable 5.1)

🤖 Generated with Claude Code

[wenjin272]

Thanks for adding the Java OpenAI embedding model and for the thorough tests.

The overall scope is acceptable to keep in this PR, but could you clean up the commit history before merging? Please consolidate the review-round commits into a few meaningful commits, ordered as follows:

1. Chat E2E API-key gating and test cleanup.
2. Python OpenAI embedding refactoring and behavioral hardening.
3. Java OpenAI embedding implementation, registration, documentation, and tests.

This keeps the preparatory fixes and refactoring before the main Java feature, making the change history easier to review and trace.

[purushah]

Commit history consolidated as requested, in this order: (1) [e2e] chat/embedding API-key gating and test cleanup, (2) [integrations][python] OpenAI embedding refactoring and hardening, (3) [integrations][java] OpenAI embedding implementation, registration, documentation and tests. The tree is unchanged apart from the docs wording above. Full Java and Python suites plus the local Flink E2E were rerun on the squashed tree before pushing.

[wenjin272]

Thanks for cleaning up the commit history and updating the documentation—the provider reference now focuses on the current contract.

There is one remaining commit-ordering issue: the first [e2e] commit adds the OpenAI embedding module dependency and references ResourceName.EmbeddingModel.OPENAI_*, but those are introduced only by the third Java commit. As a result, the first commit cannot build independently. Could you keep only the Chat E2E API-key gating/helper change in the first commit and move the embedding E2E files and POM dependency into the Java feature commit (or a later commit)?

I also left one inline comment about keeping malformed partial Base64 padding rejected consistently in Java and Python.