Proto Serializers

Proto Serializers are used to convert Django database data into protobuf messages that can be sent via gRPC and vice versa.

There are four types of proto serializers available:

BaseProtoSerializer : Base class for all proto serializers
ProtoSerializer : Same as BaseProtoSerializer but inherit from DRF Serializer class
ListProtoSerializer : Base class for all proto serializers that use DRF fields and have kwargs many=True
ModelProtoSerializer : Class that allow to generate serializer field from Model fields

They work exactly in the same way as DRF serializer. You just have to inherit from a the corresponding DSG class (see mapping below) and add two meta attributes proto_class and proto_class_list (s. examples).

Mapping between DRF and DSG

DRF to DSG Class Mapping
DRF Class	DSG class
`rest_framework.serializers.BaseSerializer`	`django_socio_grpc.proto_serializers.BaseProtoSerializer()`
`rest_framework.serializers.Serializer`	`django_socio_grpc.proto_serializers.ProtoSerializer()`
`rest_framework.serializers.ListSerializer`	`django_socio_grpc.proto_serializers.ListProtoSerializer()`
`rest_framework.serializers.ModelSerializer`	`django_socio_grpc.proto_serializers.ModelProtoSerializer()`

BaseProtoSerializer

BaseProtoSerializer is the base class for all proto serializers. It doesn’t have any fields and is used to convert data into a gRPC message.

It needs to define the method to_proto_message to be able to correctly generate proto file. See Proto generation for generation and Request/Response format of grpc_action for expected return format.

from django_socio_grpc import proto_serializers

class BaseProtoSerializer(proto_serializers.BaseProtoSerializer):
    def to_representation(self, el):
        return {
            "uuid": str(el.uuid),
            "number_of_elements": el.number_of_elements,
            "is_archived": el.is_archived,
        }

    def to_internal_value(self, data):
        return {
            "uuid": UUID(data["uuid"]),
            "number_of_elements": data["number_of_elements"],
            "is_archived": data["is_archived"],
        }

    def to_proto_message(self):
        return [
            {"name": "uuid", "type": "string"},
            {"name": "number_of_elements", "type": "int32"},
            {"name": "is_archived", "type": "bool"},
        ]

ProtoSerializer

ProtoSerializer is the same as BaseProtoSerializer except it inherit from rest_framework.Serializer instead of rest_framework.BaseSerializer.

You can find more information on the DRF documentation

It also need to define the method to_proto_message to be able to correctly generate proto file. See Proto generation for generation and Request/Response format of grpc_action for expected return format.

ListProtoSerializer

The ListProtoSerializer class provides the behavior for serializing and validating multiple objects at once. You won’t typically need to use ListProtoSerializer directly, but should instead simply pass many=True when instantiating a serializer.

When a serializer is instantiated and many=True is passed, a ListSerializer instance will be created. The serializer class then becomes a child of the parent ListSerializer

The following argument can also be passed to a ListSerializer field or a serializer that is passed many=True:

allow_empty: This is True by default, but can be set to False if you want to disallow empty lists as valid input.

max_length: This is None by default, but can be set to a positive integer if you want to validate that the list contains no more than this number of elements.

min_length: This is None by default, but can be set to a positive integer if you want to validate that the list contains no fewer than this number of elements.

ModelProtoSerializer

Often you’ll want serializer classes that map closely to Django model definitions.

The ModelProtoSerialize class provides a shortcut that lets you automatically create a Serializer class with fields that correspond to the Model fields.

The ModelProtoSerialize class is the same as a regular Serializer class, except that:

It will automatically generate a set of fields for you, based on the model.

It will automatically generate validators for the serializer, such as unique_together validators.

It includes simple default implementations of .create() and .update().

Example of a ModelProtoSerializer

This Example will only focus on ModelProtoSerializer.

First, we will use our Post model used in the Getting started.

class Post(models.Model):
    pub_date = models.DateField()
    headline = models.CharField(max_length=200)
    content = models.TextField()
    user = models.ForeignKey(User, on_delete=models.CASCADE)

Then we generate the proto file for this model. See Proto Generation for more information. Be sure you completed all the step before the Generate proto quickstart step

You can now define your serializer like this:

#quickstart/serializers.py
from django_socio_grpc import proto_serializers
from rest_framework import serializers
from quickstart.models import Post

from quickstart.grpc.quickstart_pb2 import (
    PostResponse,
    PostListResponse,
)

class PostProtoSerializer(proto_serializers.ModelProtoSerializer):
    pub_date = serializers.DateTimeField(read_only=True)

    class Meta:
        model = Post
        proto_class = PostResponse
        proto_class_list = PostListResponse
        fields = "__all__"

proto_class and proto_class_list

proto_class and proto_class_list will be used to convert incoming gRPC messages or Python data into gRPC messages.

proto_class_list is used when the parameter many=True is passed to the serializer. It allows us to have two different proto messages with the same models for list and retrieve methods in a ModelService.

If the message received in the request is different than the one used in the response, then you will have to create two serializers.

serializer.data vs serializer.message

DSG supports retro compatibility, so serializer.data is still accessible and still in dictionary format. However, it’s recommended to use serializer.message that is in the gRPC message format and should always return serializer.message as response data.

Note that async method serializer.adata and serializer.amessage exist. See Sync vs Async page

Extra kwargs options

Extra kwargs options are used like this: serializer_instance = SerializerClass(**extra_kwras_options)

stream <Boolean>: Return the message as a list of proto_class instead of an instance of proto_class_list to be used in stream. See Stream example
message_list_attr <String>: Change the attribute name for the list of instances returned by a proto_class_list (default is results). See Customizing the Name of the Field in the ListResponse
proto_comment <ProtoComment or string>: Add to the model (message) comment in the output PROTO file. ProtoComment class is declared in django_socio_grpc.protobuf and helps to have multi-line comments. See Add comments to fields

Use Cases

Required, Nullable and default values for fields (optional)

In gRPC, all fields have a default value. For example, if you have a field of type int32 and you don’t set a value, the default value will be 0. To know if this field was set (so its value is actually 0) or not, the field needs to be declared as optional (see proto3 documentation).

To work with this different behavior between REST and gRPC we use the combination of required, allow_null and default field parameters to find the adapted behavior.

There are multiple ways to have proto fields with optional:

In ProtoSerializer, you can use allow_null=True, required=False or default=<xxxx> in the field kwargs. Note that default should not be None or rest_framework.fields.empty. If default is None just set allow_null to True
In SerializerMethodField, you can use the return annotation Optional[...] or ... | None for Python 3.10+.
In ModelProtoSerializer, model fields with null=True will be converted to optional fields.
In GRPCAction you can set cardinality to optional in the request or response FieldDict.

How the values are choosen by DSG when values are not set in the message (this behavior is possible only if field is optional and should be automatic depending of what you specified in your model or serializer but can be customized depending on your need):

If create or update, and the field has required=True: Set to the default grpc value for the type (”” for string, 0 for int, False for boolean, …)
If create or update, and the field has required=True and a default in the serializer field : Set to None

If update, and the field has allow_null=True: Set to None
If create, and the field has allow_null=True and the field have a default in the model: Set to the model field default
If create, and the field has allow_null=True and the field have a default in the serializer: Set to serializer field default
If create, and the field has allow_null=True and the field haven’t a default in the model or the serializer: Set to None

If partial update, and the field is not in the list of field to update: delete the field
If partial update, and the field is in the list of field to update and allow_null=True: Set to None
If partial update, and the field is in the list of field to update and the field have a default in the serializer: Set to serializer field default
If partial update, and the field is in the list of field to update and none of the above option: Set to the default grpc value for the type

How the values are choosen by DSG when values are set to default gRPC values:

If create or update, and the field has required=True: Use the default grpc value
Else use same logic than value not set.

Note

To see real examples of this behavior please see Model, Serializer and Tests

Read-Only and Write-Only Props

If the setting SEPARATE_READ_WRITE_MODEL is True, DSG will automatically use read_only and write_only field kwargs to generate fields only in the request or response message. This is also true for Django fields with specific values (e.g., editable=False).

Warning

This setting is deprecated. See setting documentation

Example:

from django_socio_grpc import proto_serializers

class BasicLoginServiceSerializer(proto_serializers.ProtoSerializer):

    user_name = serializers.CharField(read_only=True)
    email = serializers.CharField()
    password = serializers.CharField(write_only=True)

    class Meta:
        fields = ["user_name", "email", "password"]

Will result in the following proto after generation:

message BasicLoginServiceRequest {
    string user_name = 1;
    string password = 2;
}

message BasicLoginServiceResponse {
    string user_name = 1;
    string email = 2;
}

Nested Serializer

DSG supports nested serializers without any extra work. Just try it.

You can see full example of it in our app used for unit testing DSG. Extract from it:

from django_socio_grpc import proto_serializers

class ExampleRelatedFieldModelSerializer(proto_serializers.ModelProtoSerializer):

    # foreign_obj id the name of the foreign key in RelatedFieldModel and ForeignModelSerializer it's serializer. These are only example taken from unit test of DSG.
    foreign_obj = ForeignModelSerializer(read_only=True)
    # many_many_obj id the name of the many to many key in RelatedFieldModel and ManyManyModelSerializer it's serializer. These are only example taken from unit test of DSG.
    many_many_obj = ManyManyModelSerializer(read_only=True, many=True)

    class Meta:
        # RelatedFieldModel is the model that have foreign_obj and many_many_obj attributes
        model = RelatedFieldModel
        fields = ["uuid", "foreign_obj", "many_many_obj"]

Will result in the following proto after generation:

message ExampleRelatedFieldModelResponse {
    string uuid = 1;
    ForeignModelResponse foreign_obj = 2;
    repeated ManyManyModelResponse many_many_obj = 3;
}

Special Case of BaseProtoSerializer

As BaseProtoSerializer doesn’t have fields but only to_representation and to_internal_value, we can’t automatically introspect code to find the correct proto type.

To address this issue, you have to manually declare the name and protobuf type of the BaseProtoSerializer in a to_proto_message method.

This to_proto_message needs to return a list of dictionaries in the same format as grpc action request or response as a list input.

from django_socio_grpc import proto_serializers

class BaseProtoExampleSerializer(proto_serializers.BaseProtoSerializer):
    def to_representation(self, el):
        return {
            "uuid": str(el.uuid),
            "number_of_elements": el.number_of_elements,
            "is_archived": el.is_archived,
        }

    def to_proto_message(self):
        return [
            {"name": "uuid", "type": "string"},
            {"name": "number_of_elements", "type": "int32"},
            {"name": "is_archived", "type": "bool"},
        ]

Generated Proto:

message BaseProtoExampleResponse {
    string uuid = 1;
    int32 number_of_elements = 2;
    bool is_archived = 3;
}

Special Case of SerializerMethodField

DRF SerializerMethodField class is a field type that returns the result of a method. So there is no possibility to automatically find the type of this field. To circumvent this problem, DSG introduces function introspection where we are looking for return annotation in the method to find the prototype.

from typing import List, Dict
from django_socio_grpc import proto_serializers
from rest_framework import serializers

class ExampleSerializer(proto_serializers.ProtoSerializer):

    default_method_field = serializers.SerializerMethodField()
    custom_method_field = serializers.SerializerMethodField(method_name="custom_method")

    def get_default_method_field(self, obj) -> int:
        return 3

    def custom_method(self, obj) -> List[Dict]:
        return [{"test": "test"}]

    class Meta:
        fields = ["default_method_field", "custom_method_field"]

Generated Proto:

message ExampleResponse {
    int32 default_method_field = 2;
    repeated google.protobuf.Struct custom_method_field = 3;
}

Customizing the Name of the Field in the ListResponse

By default, the name of the field used for the list response is results. You can override it in the meta of your serializer:

from django_socio_grpc import proto_serializers
from rest_framework import serializers

class ExampleSerializer(proto_serializers.ProtoSerializer):

    uuid = serializers.CharField()
    name = serializers.CharField()

    class Meta:
        message_list_attr = "list_custom_field_name"
        fields = ["uuid", "name"]

Generated Proto:

message ExampleResponse {
    string uuid = 1;
    string name = 2;
}

message ExampleListResponse {
    repeated ExampleResponse list_custom_field_name = 1;
    int32 count = 2;
}

Adding Comments to Fields

You could specify comments for fields in your model (proto message) via help_text attribute and django_socio_grpc.protobuf.ProtoComment() class:

from django_socio_grpc import proto_serializers
from rest_framework import serializers
from django_socio_grpc.protobuf import ProtoComment

class ExampleSerializer(proto_serializers.ProtoSerializer):

    name = serializers.CharField(help_text=ProtoComment(["Comment for the name field"]))
    value = serializers.CharField(help_text=ProtoComment(["Multiline comment", "for the value field"]))

    class Meta:
        fields = ["name", "value"]

Generated Proto:

message ExampleResponse {
    // Comment for the name field
    string name = 1;
    // Multiline comment
    // for the value field
    string value = 2;
}

Choosing cardinality of a field

Protobuf has different cardinality key words to specify behavior of a field such as optional or repeated.

It’s what’s coming before the type of a field in a proto message:

message MyMessage {
    // optional     \ is field cardinality
    // string       \ is field type
    // my_variable  \ is field name
    // 1 is field   \ position
    optional string my_variable = 1;
}

See FieldCardinality for exhaustive list of cardinality DSG support.

It is actually not possible to specifically choose cardinality for a serializer field for now. optional cardinality is set following what is described here. repeated cardinality is set when using ListField, ListSerializer or Serializer with many=true argument.

We started discussions about adding more cardinality options and let field set them. You are welcome for contribution in this issue.