Proto generation

DSG provides an automatic way to generate proto files from your django models and services with the generateproto command.

To be able to generate proto files you need to register your service first. To do so please refer to:

Getting started: Register Services for quick registration
Services Registry for more understanding

Description

Protobuf is a serialization format developed by Google and used in gRPC. It is a binary format that is optimized to be sent over the network. It is also used to generate code for the client and server side.

This format is developed in a large community in its own project. The following documentation expects that you are at least familiar with the concepts of Protobuf. If you are not, please read their documentation <https://protobuf.dev/getting-started/pythontutorial/>_ first.

Proto files contain the classes, descriptors and controller logic (pb2.py files) and proto message syntax (.proto file) necessary to run a grpc server.

In DSG, proto files are generated from a grpc_action request / response contents (see grpc action for more use cases).

To simplify usage grpc_action are automatically generated from the Proto Serializer when using Generic Mixins.

In order to generate these files and its contents, there is a django command to run whenever you add a grpc_action, a Service or modify your request / response:

Usage

python manage.py generateproto

Options Available:
Option	Shortcut	Default value	Description
–project	-p	Use DJANGO_SETTINGS_MODULE first folder	Name of the django project that is use in the proto package name.
–dry-run	-dr	False	Print in terminal the protofile content without writing it to a file or generate new python code.
–no-generate-pb2	-nopb2	False	Avoid generating python file. Only proto.
–check	-c	False	Check if the current protofile is the same that one that will be generated by a new command to be sur your api is sync with your models.
–custom-verbose	-cv	0	Number from 1 to 4 indicating the verbose level of the generation.
–directory	-d	None	Directory where the proto files will be generated. Default will be in the apps directories
–override-fields-number	-ofn	False	Do not follow old field number when generating. /!\ this can lead to API breaking change.
–extra-args	-a	None	Add extra arguments to the protoc command (generateproto -a –mypy_out=./ –mypy_grpc_out=./)

Example

# quickstart/models.py
from django.db import models


class User(models.Model):
    full_name = models.CharField(max_length=70)

    def __str__(self):
        return self.full_name

# quickstart/serializers.py
from django_socio_grpc import proto_serializers
from rest_framework import serializers
from quickstart.models import User, Post, Comment


class UserProtoSerializer(proto_serializers.ModelProtoSerializer):
    # This line is written here as an example,
    # but can be removed as the serializer integrates all the fields in the model
    full_name = serializers.CharField(allow_blank=True)
    class Meta:
        model = User
        fields = "__all__"

# Service
from django_socio_grpc import generics
from django_socio_grpc.decorators import grpc_action
from ..models import User
from ..serializers import UserProtoSerializer

# inherits from AsyncModelService, therefore will register all default CRUD actions.
class UserService(generics.AsyncModelService):
    queryset = User.objects.all()
    serializer_class = UserProtoSerializer

    @grpc_action
    async def SomeCustomMethod(
        request=[{"name": "foo", "type": "string"}],
        response=[{"name": "bar", "type": "string"}],
        response_stream=True
    ):
        # logic here
        pass

# quickstart/handlers.py
from django_socio_grpc.services.app_handler_registry import AppHandlerRegistry
from quickstart.services import UserService

def grpc_handlers(server):
    app_registry = AppHandlerRegistry("quickstart", server)
    app_registry.register(UserService)

At the root of your project, run:

python manage.py generateproto

If command executed successfully, you will see inside your user app, a grpc folder with two .py files, (user_pb2.py and user_pb2_grpc.py) and a user.proto file. user.proto file should contain these lines:

syntax = "proto3";

package doc_example.generate_proto_doc;

import "google/protobuf/empty.proto";

service UserController {
    rpc List(UserListRequest) returns (UserListResponse) {}
    rpc Create(UserRequest) returns (UserResponse) {}
    rpc Retrieve(UserRetrieveRequest) returns (UserResponse) {}
    rpc Update(UserRequest) returns (UserResponse) {}
    rpc Destroy(UserDestroyRequest) returns (google.protobuf.Empty) {}
    rpc SomeCustomMethod(SomeCustomMethodRequest) returns (stream SomeCustomMethodResponse) {}
}

message UserResponse {
    string id = 1;
    string full_name = 2;
}

message UserListRequest {
}

message UserListResponse {
    repeated UserResponse results = 1;
}

message UserRequest {
    string id = 1;
    string full_name = 2;
}

message UserRetrieveRequest {
    string id = 1;
}

message UserDestroyRequest {
    string id = 1;
}

message SomeCustomMethodRequest {
    string foo = 1;
}

message SomeCustomMethodResponse {
    string bar = 1;
}

Note: these files are meant to be read only, please do not modify, since they might be overwritten by a next generation call. You can use the .proto file as a reference to verify whether or not your serializer fields were correctly mapped but you should not try to modify them manually.

For more example and use case go to Generic Mixins and grpc action

Proto Generation Plugins

DSG allow you to customize the ProtoMessage generated with a Plugin system. A plugin is a class that inherit from BaseGenerationPlugin. It will expose 4 methods that you can override to customize behavior: check_condition, transform_request_message, transform_response_message and run_validation_and_transform

Basically run_validation_and_transform will call check_condition and if the return value of this call is True it will then call transform_request_message and transform_response_message.

check_condition and run_validation_and_transform will take the same arguments that are: - service: That is the instance of the service that being transformed into protobuf format. - request_message (can also be an str if request_name is set): That is the proto message as a python object of the request - response_message): That is the proto message as a python object of the response - message_name_constructor: That is the instance of the NameConstructor class used to generate the request and response proto name. It is usefull if you need a plugin that need to transform the name of the proto message. By default the class used is DefaultMessageNameConstructor

Some helper class for transforming message to list, adding field and other exist. Please refer to the list of existing plugin

Example of a plugin that change the type to all responses fields to string:

from django_socio_grpc.protobuf.generation_plugin import BaseGenerationPlugin

class ReplaceAllTypeToStringGenerationPlugin(BaseGenerationPlugin):

    def __init__(self, type_to_put="string"):
        self.type_to_put = type_to_put

    def transform_response_message(
        self,
        service,
        proto_message,
        message_name_constructor,
    ):
        # proto_message can be a string if the response_name is set. Be carreful to handle this case in your plugin
        if isinstance(proto_message, str):
            return proto_message
        for field in proto_message.fields:
            field.field_type = self.type_to_put
        return proto_message

Then you can use it for a specific action or globally.

Globally: See setting doc

For a specific action:

# quickstart/services.py
from django_socio_grpc import generics
from quickstart.models import Post
from quickstart.serializer import PostProtoSerializer
from django_socio_grpc.decorators import grpc_action


# This service will have all the CRUD actions
class PostService(generics.GenericService):
    queryset = Post.objects.all()
    serializer_class = PostProtoSerializer

    @grpc_action(
        request=[],
        response=PostProtoSerializer,
        use_generation_plugins=[ReplaceAllTypeGenerationPlugin(type_to_put="string")],
    )
    async def CustomActionWithPlugin(self, request, context):
        ...

MessageNameConstructor

DSG uses a MessageNameConstructor to construct the names of unspecified messages.

By default DefaultMessageNameConstructor() is used, it follow this logic:

If a specific request or response name is set we use it.
If not
- If the message is a string we use it
- If the message is a Serializer we use the name of the Serializer without the “Serializer” or “ProtoSerializer” ending
- If the message is a list of FieldDict we use the concatenation of the service name and the action name.
If SEPARATE_READ_WRITE_MODEL settings is True we add the “REQUEST” or “RESPONSE” suffix.

If you want to change this behavior by your own you can inherit from MessageNameConstructor and override it’s methods.

Once you have your own message name constructor class you pass it to your specific grpc_action or change it globally

Field number attribution

COMING SOON