Protobuf

protobuf 即 Protocol Buffers，是一种轻便高效的结构化数据存储格式，与语言、平台无关，可扩展可序列化。protobuf 性能和效率大幅度优于 JSON、XML 等其他的结构化数据格式。protobuf 是以二进制方式存储的，占用空间小，但也带来了可读性差的缺点。protobuf 在通信协议和数据存储等领域应用广泛。Protobuf 在 .proto 定义需要处理的结构化数据，可以通过 protoc 工具，将 .proto 文件转换为 C、C++、Golang、Java、Python 等多种语言的代码，兼容性好，易于使用。

Installation

环境: WSL2 unbuntu 20.04.1，

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# install protobuf-compiler 
sudo apt install protobuf-compiler
# libprotoc 3.6.1
protoc --version

# install protoc-gen-go
go get -u github.com/golang/protobuf/protoc-gen-go

# protoc-gen-go 会自动安装到 $GOPATH/bin 之下，需要加 PATH 里
vim ~/.bashrc

# 复制如下加在文件末尾
export PATH="$GOPATH/bin:$PATH"

# 生效
source ~/.bashrc

小插曲是把 PATH 弄乱了，vi 出现了报错

1
2
3
4
5
6
# PATH 乱了
The command could not be located because '/usr/bin' is not included in the PATH environment variable.
gedit: command not found

# 解决方法
export PATH=/usr/bin

Example 0

protubuf 官方给了 example 和 tutorial，可以通过这个来比较好地开始对 protobuf 有初步了解。

1
2
3
4
5
6
7
8
9
10
11
12
# 下载
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf/examples

# readme 里可以用 bazel 或者 Makefile，我们用 Makefile 就可以。
make go

# 输入数据
./add_person_go addressbook.data

# 展示数据
./add_person_go addressbook.data

Makefile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# See README.txt.

go:     add_person_go     list_people_go
gotest: add_person_gotest list_people_gotest

clean:
	rm -f add_person_cpp list_people_cpp add_person_java list_people_java add_person_python list_people_python
	rm -f javac_middleman AddPerson*.class ListPeople*.class com/example/tutorial/*.class
	rm -f protoc_middleman addressbook.pb.cc addressbook.pb.h addressbook_pb2.py com/example/tutorial/AddressBookProtos.java
	rm -f *.pyc
	rm -f protoc_middleman_go tutorial/*.pb.go add_person_go list_people_go go.mod go.sum
	rm -f protoc_middleman_dart dart_tutorial/*.pb*.dart
	rmdir dart_tutorial 2>/dev/null || true
	rmdir tutorial 2>/dev/null || true
	rmdir com/example/tutorial 2>/dev/null || true
	rmdir com/example 2>/dev/null || true
	rmdir com 2>/dev/null || true

# .proto -> .go

protoc_middleman_go: addressbook.proto
	mkdir -p tutorial # make directory for go package，defined in .proto 
	protoc $$PROTO_PATH --go_out=tutorial addressbook.proto
	@touch protoc_middleman_go

go_mod:
	go mod init github.com/protocolbuffers/protobuf/examples
	go mod tidy

add_person_go: add_person.go protoc_middleman_go go_mod
	go build -o add_person_go add_person.go

add_person_gotest: add_person_test.go add_person_go go_mod
	go test add_person.go add_person_test.go

list_people_go: list_people.go protoc_middleman_go go_mod
	go build -o list_people_go list_people.go

list_people_gotest: list_people.go list_people_go go_mod
	go test list_people.go list_people_test.go

Protobuf Definition

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
// See README.txt for information and build instructions.
//
// Note: START and END tags are used in comments to define sections used in
// tutorials.  They are not part of the syntax for Protocol Buffers.
//
// To get an in-depth walkthrough of this file and the related examples, see:
// https://developers.google.com/protocol-buffers/docs/tutorials

// [START declaration]
syntax = "proto3"; // 
package tutorial; // 

// import 其他会使用到的 proto 定义
import "google/protobuf/timestamp.proto"; // import 其他会使用到的 proto 定义
// [END declaration]

// [START go_declaration]
// go_package 表示生成的 go source code 存储的地址
// 需要 go_package 的理由, protobuf import：[https://stackoverflow.com/questions/66321215/what-is-the-go-package-option-used-for-in-a-protocol-buffer-file](https://stackoverflow.com/questions/66321215/what-is-the-go-package-option-used-for-in-a-protocol-buffer-file)
option go_package = "../tutorial";      
// [END go_declaration]

// [START messages]
message Person {
  string name = 1; // "=1" 1 表示唯一表示的标识符
  int32 id = 2;  // Unique ID number for this person.
  string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }

  repeated PhoneNumber phones = 4;

  google.protobuf.Timestamp last_updated = 5;
}

// Our address book file is just one of these.
message AddressBook {
  repeated Person people = 1;
}
// [END messages]

Protobuf API

  1. 基本使用

    ref：https://developers.google.com/protocol-buffers/docs/gotutorial#the_protocol_buffer_api

    参考官方 tutorial，使用 proto.Marshal 将 数据编码，使用 UnMarshel 将数据解码。

ref

  1. Go 简明教程：https://geektutu.com/post/quick-golang.html
  2. 官方文档 Tutorial：https://developers.google.com/protocol-buffers/docs/gotutorial
  3. protobuf import 可能的问题： https://segmentfault.com/a/1190000021456180
  4. Docs：https://developers.google.com/protocol-buffers/

Others

bazel 安装

  1. 官方文档里说支持 ubuntu 16 和 18，但我的环境是 wls2 ubuntu 20
  2. 安装方式：https://stackoverflow.com/questions/61982500/cannot-install-bazel-on-ubuntu-20-04-invalid-expkeysig