fix: resolve batched service header columns

[Lenin Mookiah (leninworld)]

Related Issues/PRs

Fixes #2064

What changes are proposed in this pull request?

Automatically batched Cognitive Service transformers now resolve request-header service parameters from the batched values instead of casting the entire Spark array to a scalar.

• Resolve string credentials (subscription key, AAD token, and custom authorization header) from the first non-blank value in a batch.
• Resolve custom and telemetry header maps from the first non-empty sanitized map.
• Apply the same batch-aware path to the Fabric fallback credential check.
• Keep payload parameters such as text and language on the existing typed path so document-aligned arrays remain intact.
• Raise a parameter-specific error when a header column has an incompatible Spark element type.
• Document that one credential authenticates each HTTP batch; users who require per-row credential selection should set batch size to 1.

Batching behavior is otherwise unchanged. No warning is emitted for multiple distinct credentials within one batch.

How is this patch tested?

• I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Local validation with JDK 11 and Spark 3.5.0:

• cognitive/Test/compile passed.
• Cognitive production and test scalastyle passed with zero errors and warnings.
• CognitiveServiceBaseSuite: 12/12 passed, covering batched string/map headers, null and blank credentials, Fabric fallback, invalid element types, public batching/flattening, and payload isolation.

Docker integration used Spark 3.5.1, Java 11.0.22, and Linux x86-64. A real TextSentiment stage loaded the branch-built Cognitive JAR and called a credential-free local mock endpoint. Two rows were combined into one request, the first usable subscription key authenticated the batch, and FlattenBatch preserved both original row keys.

Validation evidence

Local compilation, style, and focused Scala regression suite:

Docker Spark end-to-end regression using the branch-built artifact:

Repaired TextSentiment function loaded and transformed the automatically batched subscriptionKeyCol:

Does this PR change any dependencies?

• No. You can skip this section.

Does this PR add a new feature? If so, have you added samples on website?

• No. You can skip this section.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Lenin Mookiah (@leninworld) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Copilot]

Copilot review overview

🟢 Approval recommended

No unresolved review issues remain, and all assessments indicate approval readiness.

Review effort: Lite
Findings: None

What changed in this PR

Fixes batched Cognitive Service authentication by resolving credentials and headers from batched values while preserving payload arrays.

Changes:

• Adds batch-aware credential and header-map resolution.
• Updates Fabric fallback handling and validates incompatible types.
• Adds regression tests and documents batch credential behavior.

File	Description
docs/​Explore Algorithms/​AI Services/​Advanced Usage - Async, Batching, and Multi-Key.ipynb	Documents per-batch credential behavior.
cognitive/​src/​test/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​CognitiveServiceBaseSuite.scala	Tests batching, fallback, validation, and payload preservation.
cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​ServiceHeaderValues.scala	Resolves and validates batched header values.
cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​CognitiveServiceBase.scala	Applies batch-aware authentication and header resolution.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.