🌜
🌞
google-protobuf

google-protobuf

v3.21.0

Protocol Buffers for JavaScript

npm install google-protobuf

README

Protocol Buffers - Google's data interchange format

Copyright 2008 Google Inc.

This directory contains the JavaScript Protocol Buffers runtime library.

The library is currently compatible with:

  1. CommonJS-style imports (eg. var protos = require('my-protos');)
  2. Closure-style imports (eg. goog.require('my.package.MyProto');)

Support for ES6-style imports is not implemented yet. Browsers can be supported by using Browserify, webpack, Closure Compiler, etc. to resolve imports at compile time.

To use Protocol Buffers with JavaScript, you need two main components:

  1. The protobuf runtime library. You can install this with npm install google-protobuf, or use the files in this directory. If npm is not being used, as of 3.3.0, the files needed are located in binary subdirectory; arith.js, constants.js, decoder.js, encoder.js, map.js, message.js, reader.js, utils.js, writer.js
  2. The Protocol Compiler protoc. This translates .proto files into .js files. The compiler is not currently available via npm, but you can download a pre-built binary on GitHub (look for the protoc-*.zip files under Downloads).

Project Status as of July 2022

This project is currently in a somewhat broken state that we are working to rectify. We expect to have updated releases by the end of July 2022.

Support Status

Protobuf JavaScript is widely used and well maintained internally at Google but does not currently have staffing for more than minimal support for this open source project.

Contributing

Due to our limited support resources, we are also limited in the kinds of contributions we can accept:

  • Small bug fixes may be accepted where they have limited compatibility concerns.
  • Feature requests will generally not be accepted, except for those referring to build or tooling integration.
  • Most other PRs will be rejected until this policy is revisited.

Setup

First, obtain the Protocol Compiler. The easiest way is to download a pre-built binary from https://github.com/protocolbuffers/protobuf/releases.

If you want, you can compile protoc from source instead. To do this follow the instructions in the top-level README.

Once you have protoc compiled, you can run the tests provided along with our project to examine whether it can run successfully. In order to do this, you should download the Protocol Buffer source code from the release page with the link above. Then extract the source code and navigate to the folder named js containing a package.json file and a series of test files. In this folder, you can run the commands below to run the tests automatically.

$ npm install
$ PROTOC_INC=/usr/include/google/protobuf npm test

PROTOC_INC specifies the protobuf include path. By default, we use protoc located from PATH. Optionally, you can use the PROTOC enviroment variable to specify an alternative protoc.

This will run two separate copies of the tests: one that uses Closure Compiler style imports and one that uses CommonJS imports. You can see all the CommonJS files in commonjs_out/. If all of these tests pass, you know you have a working setup.

Using Protocol Buffers in your own project

To use Protocol Buffers in your own project, you need to integrate the Protocol Compiler into your build system. The details are a little different depending on whether you are using Closure imports or CommonJS imports:

Closure Imports

If you want to use Closure imports, your build should run a command like this:

$ protoc --js_out=library=myproto_libs,binary:. messages.proto base.proto

For Closure imports, protoc will generate a single output file (myproto_libs.js in this example). The generated file will goog.provide() all of the types defined in your .proto files. For example, for the unit tests the generated files contain many goog.provide statements like:

goog.provide('proto.google.protobuf.DescriptorProto');
goog.provide('proto.google.protobuf.DescriptorProto.ExtensionRange');
goog.provide('proto.google.protobuf.DescriptorProto.ReservedRange');
goog.provide('proto.google.protobuf.EnumDescriptorProto');
goog.provide('proto.google.protobuf.EnumOptions');

The generated code will also goog.require() many types in the core library, and they will require many types in the Google Closure library. So make sure that your goog.provide() / goog.require() setup can find all of your generated code, the core library .js files in this directory, and the Google Closure library itself.

Once you've done this, you should be able to import your types with statements like:

goog.require('proto.my.package.MyMessage');

var message = proto.my.package.MyMessage();

If unfamiliar with Closure or its compiler, consider reviewing Closure documentation.

CommonJS imports

If you want to use CommonJS imports, your build should run a command like this:

$ protoc --js_out=import_style=commonjs,binary:. messages.proto base.proto

For CommonJS imports, protoc will spit out one file per input file (so messages_pb.js and base_pb.js in this example). The generated code will depend on the core runtime, which should be in a file called google-protobuf.js. If you are installing from npm, this file should already be built and available. If you are running from GitHub, you need to build it first by running:

$ gulp dist

Once you've done this, you should be able to import your types with statements like:

var messages = require('./messages_pb');

var message = new messages.MyMessage();

The --js_out flag

The syntax of the --js_out flag is:

--js_out=[OPTIONS:]output_dir

Where OPTIONS are separated by commas. Options are either opt=val or just opt (for options that don't take a value). The available options are specified and documented in the GeneratorOptions struct in src/google/protobuf/compiler/js/js_generator.h.

Some examples:

  • --js_out=library=myprotos_lib.js,binary:.: this contains the options library=myprotos.lib.js and binary and outputs to the current directory. The import_style option is left to the default, which is closure.
  • --js_out=import_style=commonjs,binary:protos: this contains the options import_style=commonjs and binary and outputs to the directory protos. import_style=commonjs_strict doesn't expose the output on the global scope.

API

The API is not well-documented yet. Here is a quick example to give you an idea of how the library generally works:

var message = new MyMessage();

message.setName("John Doe");
message.setAge(25);
message.setPhoneNumbers(["800-555-1212", "800-555-0000"]);

// Serializes to a UInt8Array.
var bytes = message.serializeBinary();

var message2 = MyMessage.deserializeBinary(bytes);

For more examples, see the tests. You can also look at the generated code to see what methods are defined for your generated messages.

Release Notes

3.20.1
By Joshua Haberman • Published on April 21, 2022

PHP

  • Fix building packaged PHP extension (#9727)
  • Fixed composer.json to only advertise compatibility with PHP 7.0+. (#9819)

Ruby

  • Disable the aarch64 build on macOS until it can be fixed. (#9816)

Other

  • Fix versioning issues in 3.20.0
3.20.0
By Darly Paredes • Published on April 1, 2022

2022-03-25 version 3.20.0 (C++/Java/Python/PHP/Objective-C/C#/Ruby/JavaScript)

Ruby

  • Dropped Ruby 2.3 and 2.4 support for CI and releases. (#9311)
  • Added Ruby 3.1 support for CI and releases (#9566).
  • Message.decode/encode: Add recursion_limit option (#9218/#9486)
  • Allocate with xrealloc()/xfree() so message allocation is visible to the Ruby GC. In certain tests this leads to much lower memory usage due to more frequent GC runs (#9586).
  • Fix conversion of singleton classes in Ruby (#9342)
  • Suppress warning for intentional circular require (#9556)
  • JSON will now output shorter strings for double and float fields when possible without losing precision.
  • Encoding and decoding of binary format will now work properly on big-endian systems.
  • UTF-8 verification was fixed to properly reject surrogate code points.
  • Unknown enums for proto2 protos now properly implement proto2's behavior of putting such values in unknown fields.

Java

  • Revert "Standardize on Array copyOf" (#9400)
  • Resolve more java field accessor name conflicts (#8198)
  • Don't support map fields in DynamicMessage.Builder.{getFieldBuilder,getRepeatedFieldBuilder}
  • Fix parseFrom to only throw InvalidProtocolBufferException
  • InvalidProtocolBufferException now allows arbitrary wrapped Exception types.
  • Fix bug in FieldSet.Builder.mergeFrom
  • Flush CodedOutputStream also flushes underlying OutputStream
  • When oneof case is the same and the field type is Message, merge the subfield. (previously it was replaced.)’
  • Add @CheckReturnValue to some protobuf types
  • Report original exceptions when parsing JSON
  • Add more info to @deprecated javadoc for set/get/has methods
  • Fix initialization bug in doc comment line numbers
  • Fix comments for message set wire format.

Kotlin

  • Add test scope to kotlin-test for protobuf-kotlin-lite (#9518)
  • Add orNull extensions for optional message fields.
  • Add orNull extensions to all proto3 message fields.

Python

  • Dropped support for Python < 3.7 (#9480)
  • Protoc is now able to generate python stubs (.pyi) with --pyi_out
  • Pin multibuild scripts to get manylinux1 wheels back (#9216)
  • Fix type annotations of some Duration and Timestamp methods.
  • Repeated field containers are now generic in field types and could be used in type annotations.

*[Breaking change] Protobuf python generated codes are simplified. Descriptors and message classes' definitions are now dynamic created in internal/builder.py. Insertion Points for messages classes are discarded.

  • has_presence is added for FieldDescriptor in python
  • Loosen indexing type requirements to allow valid index() implementations rather than only PyLongObjects.
  • Fix the deepcopy bug caused by not copying message_listener.
  • Added python JSON parse recursion limit (default 100)
  • Path info is added for python JSON parse errors
  • Pure python repeated scalar fields will not able to pickle. Convert to list first.
  • Timestamp.ToDatetime() now accepts an optional tzinfo parameter. If specified, the function returns a timezone-aware datetime in the given time zone. If omitted or None, the function returns a timezone-naive UTC datetime (as previously).
  • Adds client_streaming and server_streaming fields to MethodDescriptor.
  • Add "ensure_ascii" parameter to json_format.MessageToJson. This allows smaller JSON serializations with UTF-8 or other non-ASCII encodings.
  • Added experimental support for directly assigning numpy scalars and array.
  • Improve the calculation of public_dependencies in DescriptorPool.
  • [Breaking Change] Disallow setting fields to numpy singleton arrays or repeated fields to numpy multi-dimensional arrays. Numpy arrays should be indexed or flattened explicitly before assignment.

Compiler

  • Migrate IsDefault(const std::string*) and UnsafeSetDefault(const std::string*)
  • Implement strong qualified tags for TaggedPtr
  • Rework allocations to power-of-two byte sizes.
  • Migrate IsDefault(const std::string*) and UnsafeSetDefault(const std::string*)
  • Implement strong qualified tags for TaggedPtr
  • Make TaggedPtr Set...() calls explicitly spell out the content type.
  • Check for parsing error before verifying UTF8.
  • Enforce a maximum message nesting limit of 32 in the descriptor builder to guard against stack overflows
  • Fixed bugs in operators for RepeatedPtrIterator
  • Assert a maximum map alignment for allocated values
  • Fix proto1 group extension protodb parsing error
  • Do not log/report the same descriptor symbol multiple times if it contains more than one invalid character.
  • Add UnknownFieldSet::SerializeToString and SerializeToCodedStream.
  • Remove explicit default pointers and deprecated API from protocol compiler

Arenas

  • Change Repeated*Field to reuse memory when using arenas.
  • Implements pbarenaz for profiling proto arenas
  • Introduce CreateString() and CreateArenaString() for cleaner semantics
  • Fix unreferenced parameter for MSVC builds
  • Add UnsafeSetAllocated to be used for one-of string fields.
  • Make Arena::AllocateAligned() a public function.
  • Determine if ArenaDtor related code generation is necessary in one place.
  • Implement on demand register ArenaDtor for InlinedStringField

C++

  • Enable testing via CTest (#8737)
  • Add option to use external GTest in CMake (#8736)
  • CMake: Set correct sonames for libprotobuf-lite.so and libprotoc.so (#8635) (#9529)
  • Add cmake option protobuf_INSTALL to not install files (#7123)
  • CMake: Allow custom plugin options e.g. to generate mocks (#9105)
  • CMake: Use linker version scripts (#9545)
  • Manually *struct Cord fields to work better with arenas.
  • Manually destruct map fields.
  • Generate narrower code
  • Fix https://github.com/protocolbuffers/protobuf/issues/9378 by removing shadowed cached_size field
  • Remove GetPointer() and explicit nullptr defaults.
  • Add proto_h flag for speeding up large builds
  • Add missing overload for reference wrapped fields.
  • Add MergedDescriptorDatabase::FindAllFileNames()
  • RepeatedField now defines an iterator type instead of using a pointer.
  • Remove obsolete macros GOOGLE_PROTOBUF_HAS_ONEOF and GOOGLE_PROTOBUF_HAS_ARENAS.

PHP

  • Fix: add missing reserved classnames (#9458)
  • PHP 8.1 compatibility (#9370)

C#

  • Fix trim warnings (#9182)
  • Fixes NullReferenceException when accessing FieldDescriptor.IsPacked (#9430)
  • Add ToProto() method to all descriptor classes (#9426)
  • Add an option to preserve proto names in JsonFormatter (#6307)

Objective-C

  • Add prefix_to_proto_package_mappings_path option. (#9498)
  • Rename proto_package_to_prefix_mappings_path to package_to_prefix_mappings_path. (#9552)
  • Add a generation option to control use of forward declarations in headers. (#9568)

General

License
BSD-3-Clause
Typescript Types
Tree-shakeable
No

Popularity

GitHub Stargazers
56.4K
Community Interest
66.9K
Number of Forks
14.4K

Maintenance

Commits
10/219/220200
Last Commit
Open Issues
573
Closed Issues
4,607
Open Pull Requests
41
Closed Pull Requests
1,282

Versions

Versions Released
10/219/2205
Latest Version Released
Jul 28, 2022
Current Tags
latest3.21.0

Contributors

jskeet
jskeet
Commits: 547
haberman
haberman
Commits: 460
TeBoring
TeBoring
Commits: 452
liujisi
liujisi
Commits: 323
xfxyjwf
xfxyjwf
Commits: 317
acozzette
acozzette
Commits: 297
jtattermusch
jtattermusch
Commits: 292
thomasvl
thomasvl
Commits: 269
csharptest
csharptest
Commits: 227
deannagarcia
deannagarcia
Commits: 100
BSBandme
BSBandme
Commits: 98
ObsidianMinor
ObsidianMinor
Commits: 85
rafi-kamal
rafi-kamal
Commits: 68
anandolee
anandolee
Commits: 65
ArnoldZokas
ArnoldZokas
Commits: 46