Add code generator for otel proto

[sfc-gh-jopel]

This is the first of a few PRs to add custom marshaling logic in place of opentelemetry-exporter-otlp-proto-common. The objective is to remove the hard runtime dependency on protobuf from snowflake-telemetry-python. This first PR

• Adds a protoc plugin to generate marshaling code for custom proto message types
• Adds a script to compile opentelemetry-proto spec
• Adds handwritten serialization code for protobuf primitive types used in opentelemetry-proto

More detailed explanation:
src/snowflake/telemetry/_internal/opentelemetry/proto/* is the code generated by running ./scripts/proto_codegen.sh. scripts/plugin.py is a plugin for the protoc compiler. It uses protoc to generate marshallng code for the custom message types defined in opentelemetry-proto. The responsibility of this generated code is to produce the same serialized bytestrings as pb2 generated structure would, given the same inputs. Additionally, handwritten util code located in src/snowflake/telemetry/_internal/serialize and is used to serialize primitive protobuf types, like varint, string, bytes, ints, etc.

[sfc-gh-sili]

Thanks for the work!
Can you add a section in the README that uses the proto generating script?

[sfc-gh-sili]

This might be a bit hard to understand without context. Can you add some explanation about this plug in and the standard protoc generator in the PR description? Is it simply a slimmed down version of protoc or does it do anything special?

[sfc-gh-sili]

Would it be easier if this is a seperate file called, say, tests/data/simple.proto

[sfc-gh-sili]

It might make more sense to have this file (and template) under scripts since it is only used when we generate from proto.
Unless that stops tests from reading this file.

[sfc-gh-sili]

In general, I am convinced that this is a solid approach. We just want to make sure we have enough doc/comment to understand this a year from now.

[sfc-gh-sili]

Looks good! Thanks

[sfc-gh-sili]

Keeping a note here for future reference: Before these gets into a new release, we need a test that validates this serialization against the official version. Psudo-code:

ourPyObj = snowflake.telemetry.SomeTelemetryObject()
ourSerialized = ourPyObj.serializeToString()
deserializedPyObj = protobuf.fromString(ourSerialized)

officialPyObj = protobuf.SomeTelemetryObject()

assert officialPyObj.serializeToString(deterministic=True) = deserializedPyObj.serializeToString(deterministic=True)
// alternatively, assert officialPyObj = deserializedPyObj if comparator is implemented

[sfc-gh-tmonk]

This looks good to me. I like the use of the custom plugin to generate the pure Python serialization code we need.

Could we create a Jenkins job that regularly tests the full build flow, now that it is getting more complex?

• code generation
• build the package (.tar.bz2 and .conda)
• running tests against the built packages, including, like Sindy mentioned a comparison against serialized messages generated by official Python protobuf packages?