How to generate python class files from protobuf
Question:
I am trying to transfer large amounts of structured data from Java to Python. That includes many objects that are related to each other in some form or another. When I receive them in my Python code, it’s quiet ugly to work with the types that are provided by protobuf. My VIM IDE crashed when trying to use autocomplete on the types, PyCharm doesn’t complete anything and generally it just seems absurd that they don’t provide some clean class definition for the different types.
Is there a way to get IDE support while working with protobuf messages in python? I’m looking at 20+ methods handling complex messages and without IDE support I might as well code with notepad.
I understand that protobuf is using metaclasses (although I don’t know why they do that). Maybe there is a way to generate python class files from that data or maybe there is something similar to typescript typing files.
Did I maybe misuse protobuf? I believed I would describe my domain model in a way that may be used across languages. In Java I am happy with the generated classes and I can use them easily. Should I maybe have used something like swagger.io instead?
Answers:
As of now, nothing like that is available. You might want to follow this issue: https://github.com/google/protobuf/issues/2638 to be up to date.
mypy-protobuf generates the type hint files. But as discussed here this works only from protobuf 3.0 and python 2.7 onwards.
If you are using a recent Python (3.7+) then https://github.com/danielgtaylor/python-betterproto (disclaimer: I’m the author) will generate very clean Python dataclasses as output which will give you proper typing and IDE completion support.
For example, this input:
syntax = "proto3";
package hello;
// Greeting represents a message you can tell a user.
message Greeting {
string message = 1;
}
Would generate the following output:
# Generated by the protocol buffer compiler. DO NOT EDIT!
# sources: hello.proto
# plugin: python-betterproto
from dataclasses import dataclass
import betterproto
@dataclass
class Hello(betterproto.Message):
"""Greeting represents a message you can tell a user."""
message: str = betterproto.string_field(1)
In general the output of this plugin mimics the *.proto input and is very easy to read if you happen to jump to definition on a message or field. It’s been a huge improvement for me personally over the official Google compiler plugin, and supports async gRPC out of the box as well.
What "-Zaffy" said worked for me. (–pyi_out)
EXAMPLE
Note : Protobuf filename example = addressbook.proto
CLI Cmd to create intellsense info (compiles in this same dir)
protoc -I . --pyi_out=. addressbook.proto
Then create python code. (compiles in this same dir)
protoc -I . --python_out=. addressbook.proto
This will create a addressbook_pb2.py and addressbook_pb2.pyi
then VS code away!
I am trying to transfer large amounts of structured data from Java to Python. That includes many objects that are related to each other in some form or another. When I receive them in my Python code, it’s quiet ugly to work with the types that are provided by protobuf. My VIM IDE crashed when trying to use autocomplete on the types, PyCharm doesn’t complete anything and generally it just seems absurd that they don’t provide some clean class definition for the different types.
Is there a way to get IDE support while working with protobuf messages in python? I’m looking at 20+ methods handling complex messages and without IDE support I might as well code with notepad.
I understand that protobuf is using metaclasses (although I don’t know why they do that). Maybe there is a way to generate python class files from that data or maybe there is something similar to typescript typing files.
Did I maybe misuse protobuf? I believed I would describe my domain model in a way that may be used across languages. In Java I am happy with the generated classes and I can use them easily. Should I maybe have used something like swagger.io instead?
As of now, nothing like that is available. You might want to follow this issue: https://github.com/google/protobuf/issues/2638 to be up to date.
mypy-protobuf generates the type hint files. But as discussed here this works only from protobuf 3.0 and python 2.7 onwards.
If you are using a recent Python (3.7+) then https://github.com/danielgtaylor/python-betterproto (disclaimer: I’m the author) will generate very clean Python dataclasses as output which will give you proper typing and IDE completion support.
For example, this input:
syntax = "proto3";
package hello;
// Greeting represents a message you can tell a user.
message Greeting {
string message = 1;
}
Would generate the following output:
# Generated by the protocol buffer compiler. DO NOT EDIT!
# sources: hello.proto
# plugin: python-betterproto
from dataclasses import dataclass
import betterproto
@dataclass
class Hello(betterproto.Message):
"""Greeting represents a message you can tell a user."""
message: str = betterproto.string_field(1)
In general the output of this plugin mimics the *.proto input and is very easy to read if you happen to jump to definition on a message or field. It’s been a huge improvement for me personally over the official Google compiler plugin, and supports async gRPC out of the box as well.
What "-Zaffy" said worked for me. (–pyi_out)
EXAMPLE
Note : Protobuf filename example = addressbook.proto
CLI Cmd to create intellsense info (compiles in this same dir)
protoc -I . --pyi_out=. addressbook.proto
Then create python code. (compiles in this same dir)
protoc -I . --python_out=. addressbook.proto
This will create a addressbook_pb2.py and addressbook_pb2.pyi
then VS code away!