Practical .NET

Reusing and Recycling Data Structures in gRPC Services in .NET Core

Here's everything you need to know to create a standard set of reusable message formats to use with your gRPC services.

• By Peter Vogel
• 01/10/2020

If you're building a gRPC service you could create an entire set of message formats for communicating with your clients, with each message specially tailored for the service. I think that's an excellent strategy if (a) you're paid by the hour and (b) want to make life more difficult for the developers building the clients. The smart thing to do is to consider each of your message formats as a reusable object that you can combine as needed. The Protocol Buffer (ProtoBuf) specification that you use to define your messages lets you do that.

Simple Reuse
If, for example, I wanted to define a SalesOrder message that included Customer information, I might begin by defining a message that holds Customer information (for background on this definition see my earlier column on designing messages for gRPC services):

message CustomerResponse {
  int32 custId = 1;
  string firstName = 2;
  string lastName = 3;
  int32 age = 4;
}

Now that I've defined that CustomerResponse, I can reuse it when defining my SalesOrder message because a ProtoBuf message format can be used as a type:

message SalesOrder {
  int32 SoId = 1;
  int32 TotalValue = 2;
  string DeliveryDate = 3;
  CustomerResponse custInfo = 4;

Code to load data into this structure might look like this:

SalesOrder so = new SalesOrder();
so.SoId = 1;
so.CustInfo = new CustomerResponse();
so.CustInfo.CreditLimit = 5000;

There are some problems with this sales order structure, though: DeliveryDate is stored as a string and TotalValue (which might include dollars and cents) is an integer. The solution to those problems are Google's "well known types" which I'll cover in a later column.

Reuse in Collections and Dictionaries
I can also reuse structures as repeated fields. So, for example, a sales order will be associated with a potentially unlimited number of addresses (billing, shipping, contact, etc.). To handle that, I might start with an AddressFormat message:

message AddressFormat
{
  string City = 1;
  string Street = 2;
  string Country = 3;
}

I could then extend that address to create a TypedAddress that flags whether it's a shipping or billing address. All I have to do is add a type field and re-use my AddressFormat, like this:

message TypedAddress
{
  string Type = 1;
  AddressFormat Address = 2;
}

And, finally, I can incorporate that TypedAddress into my sales order as a repeated field so that I can include as many different types of addresses as a SalesOrder needs:

message SalesOrder 
{
  int32 SoId = 1;
  repeated TypedAddress Addresses = 2;

The code to create the sales order with addresses would look something like this:

SalesOrder so = new SalesOrder();
so.SoId = 1;
AddressFormat af = new AddressFormat();
af.City = "Regina";
TypedAddress ta = new TypedAddress();
ta.Type = "Shipping";
ta.Address = af;
so.Addresses.Add(ta);

Actually, though, this is a pretty poor design because a client that needs the shipping address would have to loop the Addresses collection looking for it or use a LINQ query. A better design would be to define the Addresses as a ProtoBuf map (effectively, a dictionary -- again, see that earlier article on defining gRPC messages).

My SalesOrder now uses this design which allows me to store addresses by key which will be "Shipping," "Billing" and so on:

message SalesOrder 
{
  int32 SoId = 1;
  map<string, TypedAddress> AddressFormat Addresses = 2;

Now, I can add a new address to my sales order with this line of code (and I don't need the TypedAddress definition at all):

so.Addresses.Add("Shipping", af);

A client can now check to see if a shipping address exists and retrieve it with code like this:

if (so.Addresses.ContainsKey("Shipping"))
{
  AddressFormat afRetrieved = so.Addresses["Shipping"];
}

Obviously, I should be using an enumerated value to set the Type property rather than using a string like "Shipping." ProtoBuf supports enumerated values and I'll cover how to do that in an upcoming column, too.

Referencing Formats
Of course, that assumes that my AddressFormat definitions are separate structures -- that probably requires more planning than I'm capable of. Fortunately, even if I've buried my AddressFormat in another definition, I can still reuse it. For example, I can define my AddressFormat inside my CustomerResponse definition and then use that definition inside the CustomerResponse. That code looks like this:

message CustomerResponse {
  int32 custId = 1;
  message AddressFormat
  {
    string City = 1;
    string Street = 2;
    string Country = 3;
  }
  AddressFormat Address = 2;

Setting the Address in the CustomerResponse now looks like this:

CustomerResponse cr2 = new CustomerResponse();

cr2.CustId = 1;
cr2.Address = new Types.AddressFormat();
cr2.Address.City = "Regina";

As this code shows, formats defined inside another format (like my AddressFormat) are put into a namespace called Types within the namespace of the message they are part of. The fullname of my AddressFormat is gRPCV3.Protos.CustomerResponse.Types.AddressFormat.

You can avoid using that Types namespace inline by adding (or, more accurately, having Visual Studio add) a static namespace. For my CustomerResponse message that namespace looks like this:

using static gRPCV3.Protos.CustomerResponse.Types;

Even though my AddressFormat is defined inside another message, I can still use it in other messages definined in my .proto file -- I just have to prefix the message type with the message type that it's nested inside of.

That means that my SalesOrder definition can pick up the addressFormat in my CustomerResponse with this code in my .proto file:

message SalesOrder 
{
  int32 SoId = 1;  
  map<string, CustomerResponse.AddressFormat> Addresses = 5;
}

Importing Formats
However, if I'm willing to do a little planning, I should probably define my AddressFormat separately from any other definition. That's especially true if I suspect that I'm going to use that AddressFormat in a lot of places -- in fact, I will probably want to use AddressFormat in several different .proto files.

I can do that, too. First, of course, I move my AddressFormat to a file called AddressFormat.proto. I'm going to need to add some additional content to that file, though. Because I'm using the proto3 syntax, I have to specify that at the start of my file. I'm also going to give this file a package name (effectively a ProtoBuf namespace) to prevent name collisions among ProtoBuf definitions. Finally, I'm going to define a C# namespace to use with the AddressFormat class that's generated from this ProtoBuf definition. For simplicity's sake I'll use "common" as both my ProtoBuf and C# namespace (though I'll uppercase the "C" in "common" in the C# namespace to show where I'm using each).

With all of that in place, my AddressFormat.proto file ends up looking like this:

syntax = "proto3";
package common;
option csharp_namespace = "Common";

message AddressFormat
{
  string City = 1;
  string Street = 2;
  string Country = 3;
}

In order to get an AddressFormat class file generated from that definition, I also need to reference that AddressFormat.proto file in my project's csproj file (the gRPC tools I've added to my project through NuGet -- see that first article -- will take care of generating the related C# code).

In the project file, I've already referenced the proto file that defines my service so I just need to add my AddressFormat file to the already existing ItemGroup. Since I keep all my .proto files in a folder called Protos off my project root, the result looks like this:

<ItemGroup>
  <Protobuf Include="Protos\CustomerService.proto" GrpcServices="Client" />
  <Protobuf Include="Protos\AddressFormat.proto" GrpcServices="Client" />
</ItemGroup>

With all that done, I can now use my AddressFormat in the .proto file that includes my CustomerResponse and SalesOrder definitions. I just add this line that points to my AddressFormat.proto file to the top of my existing .proto file (notice that this is a relative address from my project's root):

import "Protos/AddressFormat.proto";

In my definitions, I have to refer to my AddressFormat using its ProtoBuf namespace:

message SalesOrder 
{
  int32 SoId = 1;
  map<string, common.AddressFormat> Addresses = 5;

Similarly, in my code, I need to use the C# namespace either inline as I do here or with a using statement:

Common.AddressFormat af = new Common.AddressFormat();

These tools give you a lot of flexibility in defining and, more importantly, re-using your message definitions. It's the Don't Repeat Yourself principle in action and that's always a good thing.