Skip to content

fix(django-google-spanner): preserve JsonObject in JSONField adaptation and deserialization - #18989

Open
sakthivelmanii wants to merge 3 commits into
mainfrom
fix/django-spanner-jsonfield-adapt
Open

sakthivelmanii wants to merge 3 commits into
mainfrom
fix/django-spanner-jsonfield-adapt

Conversation

@sakthivelmanii

Copy link
Copy Markdown
Contributor

Problem

  • Inserting or updating a JSONField against live Cloud Spanner fails with ProgrammingError: 400 Value has type STRING which cannot be inserted into column <col>, which has type JSON.
  • Although django_spanner registers json_field_get_prep_value to wrap values in JsonObject, Django's JSONField.get_db_prep_value subsequently passes that value to connection.ops.adapt_json_value(value, self.encoder). BaseDatabaseOperations.adapt_json_value calls json.dumps(value), serializing JsonObject back into a Python str (TypeCode.STRING).
  • When reading JSON columns back from Spanner, the Python Spanner client may already return deserialized JsonObject, dict, or list instances, which causes JSONField.from_db_value (json.loads(value)) to raise TypeError.

Solution

  • Override DatabaseOperations.adapt_json_value in django_spanner/operations.py to preserve JsonObject and wrap dict/list values in JsonObject so DB-API binds the parameter with Spanner's JSON type.
  • Register json_field_from_db_value on JSONField.from_db_value in django_spanner/__init__.py to return already-deserialized JsonObject, dict, or list values directly before falling back to json.loads.
  • Add unit tests in tests/unit/django_spanner/test_operations.py covering adapt_json_value and JSONField.from_db_value.

@sakthivelmanii
sakthivelmanii requested a review from a team as a code owner November 4, 2026 17:09

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for JsonObject in Django Spanner's JSONField handling by implementing adapt_json_value in DatabaseOperations and updating from_db_value in JSONField. However, several issues were identified in the review: adapt_json_value bypasses custom encoders and causes type mismatches for primitive JSON values, which can be resolved by serializing and deserializing the value before wrapping it in JsonObject. Additionally, a missing import of JsonObject in __init__.py will cause a runtime NameError, and the unit tests should be expanded to cover primitive values and custom encoder serialization.

Comment on lines +775 to +782
def adapt_json_value(self, value, encoder):
if value is None:
return None
if isinstance(value, JsonObject):
return value
if isinstance(value, (dict, list)):
return JsonObject(value)
return super().adapt_json_value(value, encoder)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The current implementation of adapt_json_value has two critical issues:

  1. Custom Encoder Bypass: Directly wrapping dict or list values in JsonObject bypasses Django's custom encoder (e.g., DjangoJSONEncoder). If the dictionary or list contains custom types like UUID, Decimal, or datetime, the Spanner client library's internal serializer will fail with a TypeError because it does not know how to serialize them.
  2. Primitive Values Type Mismatch: If value is a primitive JSON value (e.g., str, int, float, bool), it bypasses the dict/list check and falls back to super().adapt_json_value, which returns a serialized JSON string. Because this is a Python str, the Spanner DB-API driver binds it as TypeCode.STRING instead of TypeCode.JSON, leading to a ProgrammingError: 400 Value has type STRING which cannot be inserted into column <col>, which has type JSON.

To resolve both issues, serialize the value using the custom encoder first, deserialize it back to standard Python types, and then wrap it in JsonObject.

Suggested change
def adapt_json_value(self, value, encoder):
if value is None:
return None
if isinstance(value, JsonObject):
return value
if isinstance(value, (dict, list)):
return JsonObject(value)
return super().adapt_json_value(value, encoder)
def adapt_json_value(self, value, encoder):
if value is None:
return None
if isinstance(value, JsonObject):
return value
serialized = json.dumps(value, cls=encoder)
return JsonObject(json.loads(serialized))
References
  1. Do not import modules or classes inside functions if they are already imported at the module level.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated adapt_json_value to serialize via super().adapt_json_value(value, encoder) and construct JsonObject.from_str(...) so custom encoders and primitive JSON scalars are handled properly.

Comment on lines +99 to +101
def jsonfield_from_db_value(self, value, expression, connection):
if isinstance(value, (dict, list, JsonObject)):
return value

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The JsonObject class is used in the isinstance check but is not imported in this file, which will raise a NameError at runtime when from_db_value is called. Import JsonObject from google.cloud.spanner_v1 inside the function to avoid circular dependencies and ensure runtime correctness.

Suggested change
def jsonfield_from_db_value(self, value, expression, connection):
if isinstance(value, (dict, list, JsonObject)):
return value
def jsonfield_from_db_value(self, value, expression, connection):
from google.cloud.spanner_v1 import JsonObject
if isinstance(value, (dict, list, JsonObject)):
return value

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JsonObject is already imported at the module level in django_spanner/__init__.py (line 30). Also updated jsonfield_from_db_value to unwrap JsonObject null, array, and scalar values into their native Python equivalents.

Comment on lines +351 to +368
def test_adapt_json_value(self):
from django.db.models import JSONField
from google.cloud.spanner_v1 import JsonObject

self.assertIsNone(self.db_operations.adapt_json_value(None, None))
obj = JsonObject({"a": 1})
self.assertIs(self.db_operations.adapt_json_value(obj, None), obj)
adapted_dict = self.db_operations.adapt_json_value({"a": 1}, None)
self.assertIsInstance(adapted_dict, JsonObject)
adapted_list = self.db_operations.adapt_json_value([1, 2], None)
self.assertIsInstance(adapted_list, JsonObject)
self.assertEqual(self.db_operations.adapt_json_value("str", None), '"str"')

field = JSONField()
self.assertIs(field.from_db_value(obj, None, None), obj)
self.assertEqual(field.from_db_value({"a": 1}, None, None), {"a": 1})
self.assertEqual(field.from_db_value('{"a": 1}', None, None), {"a": 1})
self.assertIsNone(field.from_db_value(None, None, None))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Update the unit tests to verify that primitive JSON values are correctly wrapped in JsonObject and to cover custom encoder serialization (e.g., using uuid.UUID and DjangoJSONEncoder).

    def test_adapt_json_value(self):
        import uuid
        from django.core.serializers.json import DjangoJSONEncoder
        from django.db.models import JSONField
        from google.cloud.spanner_v1 import JsonObject

        self.assertIsNone(self.db_operations.adapt_json_value(None, None))
        obj = JsonObject({"a": 1})
        self.assertIs(self.db_operations.adapt_json_value(obj, None), obj)
        adapted_dict = self.db_operations.adapt_json_value({"a": 1}, None)
        self.assertIsInstance(adapted_dict, JsonObject)
        self.assertEqual(adapted_dict.value, {"a": 1})
        adapted_list = self.db_operations.adapt_json_value([1, 2], None)
        self.assertIsInstance(adapted_list, JsonObject)
        self.assertEqual(adapted_list.value, [1, 2])
        adapted_str = self.db_operations.adapt_json_value("str", None)
        self.assertIsInstance(adapted_str, JsonObject)
        self.assertEqual(adapted_str.value, "str")

        # Test custom encoder serialization (e.g., UUID)
        u = uuid.UUID("12345678-1234-5678-1234-567812345678")
        adapted_uuid = self.db_operations.adapt_json_value({"u": u}, DjangoJSONEncoder)
        self.assertIsInstance(adapted_uuid, JsonObject)
        self.assertEqual(adapted_uuid.value, {"u": "12345678-1234-5678-1234-567812345678"})

        field = JSONField()
        self.assertIs(field.from_db_value(obj, None, None), obj)
        self.assertEqual(field.from_db_value({"a": 1}, None, None), {"a": 1})
        self.assertEqual(field.from_db_value('{"a": 1}', None, None), {"a": 1})
        self.assertIsNone(field.from_db_value(None, None, None))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated test_adapt_json_value to cover primitive JSON values, custom encoder serialization (UUID with DjangoJSONEncoder), and JsonObject unwrapping in from_db_value.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant