Gemma 2 annotation field rename: regenerate the SDK (migration, not a hotfix) - #44
Open
ppavlidis wants to merge 2 commits into
Open
Gemma 2 annotation field rename: regenerate the SDK (migration, not a hotfix)#44ppavlidis wants to merge 2 commits into
ppavlidis wants to merge 2 commits into
Conversation
Gemma 2's REST API renames four fields on the annotation value objects: className -> category, classUri -> categoryUri, termName -> value, termUri -> valueUri. CategoryWithUsageStatisticsValueObject renames its className/classUri the same way. sdk-gen/openapi.yaml here is the Gemma 1.0 spec with those ten lines renamed in place, not Gemma 2's spec re-downloaded; gemmapy/sdk was then regenerated from it with the sdk-gen codegen, so the three model files are generated output rather than hand edits. Note that running sdk-gen/generate-sdk.sh unmodified will curl the spec back from gemma.msl.ubc.ca and undo the renames, because that host is Gemma 1.0. Renaming the wire fields renames the Python attributes with them: AnnotationValueObject.class_name is now .category, .term_name is now .value, and likewise for the URI fields.
process_annotations now reads category / category_uri / value / value_uri off the regenerated AnnotationValueObject. The DataFrame column names (class_name, class_URI, term_name, term_URI) are left alone, so get_dataset_annotations still returns a six-column frame. The get_dataset_annotations docstring records that on a factor-value row the term field is now the term's label rather than a composed sentence such as "wild type genotype has background APP/PS1".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing released is broken — this is migration work, not a hotfix
Released gemmapy targets Gemma 1.0. The default host is
https://gemma.msl.ubc.ca/rest/v2/, the 1.32.x deployment, and that host stillserves the old field names. I fetched
/rest/v2/openapi.yamlfrom both hostswhile preparing this:
gemma.msl.ubc.castill declaresclassName/classUri/
termName/termUri. So there is no urgency here and nothing on PyPI needsa rescue release.
What changed on the server
Gemma 2 (
gemma2.msl.ubc.ca, buildb5c6747f68) renamed four fields on theannotation value object:
classNamecategoryclassUricategoryUritermNamevaluetermUrivalueUriThat affects
GET/PUT /datasets/{id}/annotations,GET /datasets/annotations,the sample-characteristics routes, and the experiment-tag add/remove responses.
GET /datasets/categoriesrenames its ownclassName/classUritocategory/categoryUrithe same way.Second change, shipped with it: on a factor-value row the term field used to
hold a composed sentence; it is now the term's own label. Plain tags, subset tags
and sample characteristics are unchanged. Same dataset, same row, both hosts:
The "acute" qualifier did not disappear — it moved into new structured
predicate/objectfields. See "What this PR does not do" below.Regenerating the SDK renames the Python attribute names, not just the wire
keys:
AnnotationValueObject.class_name.categoryAnnotationValueObject.class_uri.category_uriAnnotationValueObject.term_name.valueAnnotationValueObject.term_uri.value_uriSame four on
AnnotationWithUsageStatisticsValueObject;CategoryWithUsageStatisticsValueObjectgets.category/.category_uri.Any user code reading those attributes off the raw SDK breaks.
The high-level DataFrame API is unaffected:
get_dataset_annotationsstillreturns the same six columns under the same names (
class_name,class_URI,term_name,term_URI,object_class,evidence_code). The resultingattribute-vs-column mismatch inside
_processors.process_annotationsisdeliberate and is commented as such.
Whether gemmapy absorbs the attribute rename now, ships aliases for a release, or
waits is your call — I have not invented a compatibility mechanism.
The question this PR is really putting to you
Does gemmapy want to target Gemma 2 yet, and if so, one server or both?
As it stands this branch is a hard switch. The costs, as far as I can see them:
output. But anyone pointed at Gemma 1.0 — which is the default host, so that is
everyone by default — silently gets four empty columns. It does not raise; the
deserializer just finds no matching keys. Demonstrated below.
living in
gemmapy/sdk/, whichsdk-gen/generate-sdk.shoverwrites wholesaleon the next regeneration, or a translation layer above the SDK.
this then. Costs nothing today; the branch is here when you want it.
I have deliberately not picked one.
Regenerated, not hand-edited — and how I know
gemmapy/sdk/**is genuine codegen output, not three hand-patched attribute maps.I did not take that on trust; I re-ran the codegen
(
swagger-codegen-cli3.0.73,-t sdk-gen/templ, pergenerate-sdk.sh) againstthe
openapi.yamlon this branch and diffed the result against the working tree.All 593 files matched byte for byte, with one exception:
api_client.py'suser-agent line, which differs only because
generate-sdk.shextracts the versionwith a
sedthat leaves the quotes in (gemmapy/"2.0.7"). That line is unchangedfrom
master, so it is pre-existing and not something this branch introduces.What was hand-edited is
sdk-gen/openapi.yaml, and only the ten lines thatspell those field names. It is still the Gemma 1.0 spec, not Gemma 2's spec
re-downloaded — I diffed
master's copy against the livegemma.msl.ubc.caspec and they agree except for 36 lines of property/enumordering noise.
🛑 A trap worth knowing about: running
sdk-gen/generate-sdk.shunmodifiedwill
curlthe spec back fromgemma.msl.ubc.caand silently undo these renames,because that host is Gemma 1.0. This is noted in the first commit message.
What I verified
pytest: 11 passed on this branch, 11 passed onmaster. Python 3.10.6,pytest 9.0.3, against the live default host (Gemma 1.0). No change in outcome.
tox: not run — not installed on this machine, and only 3.10 of the3.10, 3.11, 3.12, 3.13matrix is available locally.tox.inionly runspytest, and CI covers the matrix.The suite passing does not mean the change is exercised. The tests hit a
live Gemma, and
test_get_dataset_annotationsasserts only the column count(
res.shape[1] == 6), so it stays green while the contents go empty. Actualget_dataset_annotations(1)output on this branch:class_nameclass_URIterm_nameterm_URIgemma.msl.ubc.ca(default)gemma2.msl.ubc.caCI will be green on this PR for the same reason — the workflow runs bare
pytestagainst the default host. Please do not read that green tick asevidence the rename works.
I have left the test alone: making it assert non-empty data would turn it red
against Gemma 1.0, and which server it should be asserting against is exactly
the question above.
Confirmed no stragglers: zero occurrences of
class_name/class_uri/term_name/term_uriremain anywhere ingemmapy/sdk/, and none of the oldspellings remain in
sdk-gen/openapi.yaml. The two survivingterm_namereferences in
_processors.pyandgemmapy_api.pybelong toprocess_GOandare a different value object.
process_search_annotationsneeded no change —AnnotationSearchResultValueObjectalready used
category/valueon Gemma 1.0.What this PR does not do
(
predicate,predicateUri,object,objectUri,secondPredicate,secondPredicateUri,secondObject,secondObjectUri,supportingEvidence).The spec on this branch has none of them, so the SDK drops them. Concretely,
the
acutequalifier shown in the payloads above is not reachable fromgemmapy on this branch — against Gemma 2 the
term_namecolumn loses it withno warning. That is the part of the term-field change that is a real data
loss, and it wants a decision rather than a quiet fix from me.
gemma2_compaton gemma.R, which is being handled separately.Files changed
🤖 Generated with Claude Code
https://claude.ai/code/session_01P2yKU9FAQmF6T99v23vWQt