1 // Copyright 2009 The Go Authors. All rights reserved.
2 // Use of this source code is governed by a BSD-style
3 // license that can be found in the LICENSE file.
6 Package gob manages streams of gobs - binary values exchanged between an
7 Encoder (transmitter) and a Decoder (receiver). A typical use is transporting
8 arguments and results of remote procedure calls (RPCs) such as those provided by
11 The implementation compiles a custom codec for each data type in the stream and
12 is most efficient when a single Encoder is used to transmit a stream of values,
13 amortizing the cost of compilation.
17 A stream of gobs is self-describing. Each data item in the stream is preceded by
18 a specification of its type, expressed in terms of a small set of predefined
19 types. Pointers are not transmitted, but the things they point to are
20 transmitted; that is, the values are flattened. Nil pointers are not permitted,
21 as they have no value. Recursive types work fine, but
22 recursive values (data with cycles) are problematic. This may change.
24 To use gobs, create an Encoder and present it with a series of data items as
25 values or addresses that can be dereferenced to values. The Encoder makes sure
26 all type information is sent before it is needed. At the receive side, a
27 Decoder retrieves values from the encoded stream and unpacks them into local
32 The source and destination values/types need not correspond exactly. For structs,
33 fields (identified by name) that are in the source but absent from the receiving
34 variable will be ignored. Fields that are in the receiving variable but missing
35 from the transmitted type or value will be ignored in the destination. If a field
36 with the same name is present in both, their types must be compatible. Both the
37 receiver and transmitter will do all necessary indirection and dereferencing to
38 convert between gobs and actual Go values. For instance, a gob type that is
43 can be sent from or received into any of these Go types:
45 struct { A, B int } // the same
46 *struct { A, B int } // extra indirection of the struct
47 struct { *A, **B int } // extra indirection of the fields
48 struct { A, B int64 } // different concrete value type; see below
50 It may also be received into any of these:
52 struct { A, B int } // the same
53 struct { B, A int } // ordering doesn't matter; matching is by name
54 struct { A, B, C int } // extra field (C) ignored
55 struct { B int } // missing field (A) ignored; data will be dropped
56 struct { B, C int } // missing field (A) ignored; extra field (C) ignored.
58 Attempting to receive into these types will draw a decode error:
60 struct { A int; B uint } // change of signedness for B
61 struct { A int; B float } // change of type for B
62 struct { } // no field names in common
63 struct { C, D int } // no field names in common
65 Integers are transmitted two ways: arbitrary precision signed integers or
66 arbitrary precision unsigned integers. There is no int8, int16 etc.
67 discrimination in the gob format; there are only signed and unsigned integers. As
68 described below, the transmitter sends the value in a variable-length encoding;
69 the receiver accepts the value and stores it in the destination variable.
70 Floating-point numbers are always sent using IEEE-754 64-bit precision (see
73 Signed integers may be received into any signed integer variable: int, int16, etc.;
74 unsigned integers may be received into any unsigned integer variable; and floating
75 point values may be received into any floating point variable. However,
76 the destination variable must be able to represent the value or the decode
79 Structs, arrays and slices are also supported. Structs encode and decode only
80 exported fields. Strings and arrays of bytes are supported with a special,
81 efficient representation (see below). When a slice is decoded, if the existing
82 slice has capacity the slice will be extended in place; if not, a new array is
83 allocated. Regardless, the length of the resulting slice reports the number of
86 In general, if allocation is required, the decoder will allocate memory. If not,
87 it will update the destination variables with values read from the stream. It does
88 not initialize them first, so if the destination is a compound value such as a
89 map, struct, or slice, the decoded values will be merged elementwise into the
92 Functions and channels will not be sent in a gob. Attempting to encode such a value
93 at the top level will fail. A struct field of chan or func type is treated exactly
94 like an unexported field and is ignored.
96 Gob can encode a value of any type implementing the GobEncoder or
97 encoding.BinaryMarshaler interfaces by calling the corresponding method,
98 in that order of preference.
100 Gob can decode a value of any type implementing the GobDecoder or
101 encoding.BinaryUnmarshaler interfaces by calling the corresponding method,
102 again in that order of preference.
106 This section documents the encoding, details that are not important for most
107 users. Details are presented bottom-up.
109 An unsigned integer is sent one of two ways. If it is less than 128, it is sent
110 as a byte with that value. Otherwise it is sent as a minimal-length big-endian
111 (high byte first) byte stream holding the value, preceded by one byte holding the
112 byte count, negated. Thus 0 is transmitted as (00), 7 is transmitted as (07) and
113 256 is transmitted as (FE 01 00).
115 A boolean is encoded within an unsigned integer: 0 for false, 1 for true.
117 A signed integer, i, is encoded within an unsigned integer, u. Within u, bits 1
118 upward contain the value; bit 0 says whether they should be complemented upon
119 receipt. The encode algorithm looks like this:
123 u = (^uint(i) << 1) | 1 // complement i, bit 0 is 1
125 u = (uint(i) << 1) // do not complement i, bit 0 is 0
129 The low bit is therefore analogous to a sign bit, but making it the complement bit
130 instead guarantees that the largest negative integer is not a special case. For
131 example, -129=^128=(^256>>1) encodes as (FE 01 01).
133 Floating-point numbers are always sent as a representation of a float64 value.
134 That value is converted to a uint64 using math.Float64bits. The uint64 is then
135 byte-reversed and sent as a regular unsigned integer. The byte-reversal means the
136 exponent and high-precision part of the mantissa go first. Since the low bits are
137 often zero, this can save encoding bytes. For instance, 17.0 is encoded in only
138 three bytes (FE 31 40).
140 Strings and slices of bytes are sent as an unsigned count followed by that many
141 uninterpreted bytes of the value.
143 All other slices and arrays are sent as an unsigned count followed by that many
144 elements using the standard gob encoding for their type, recursively.
146 Maps are sent as an unsigned count followed by that many key, element
147 pairs. Empty but non-nil maps are sent, so if the receiver has not allocated
148 one already, one will always be allocated on receipt unless the transmitted map
149 is nil and not at the top level.
151 In slices and arrays, as well as maps, all elements, even zero-valued elements,
152 are transmitted, even if all the elements are zero.
154 Structs are sent as a sequence of (field number, field value) pairs. The field
155 value is sent using the standard gob encoding for its type, recursively. If a
156 field has the zero value for its type (except for arrays; see above), it is omitted
157 from the transmission. The field number is defined by the type of the encoded
158 struct: the first field of the encoded type is field 0, the second is field 1,
159 etc. When encoding a value, the field numbers are delta encoded for efficiency
160 and the fields are always sent in order of increasing field number; the deltas are
161 therefore unsigned. The initialization for the delta encoding sets the field
162 number to -1, so an unsigned integer field 0 with value 7 is transmitted as unsigned
163 delta = 1, unsigned value = 7 or (01 07). Finally, after all the fields have been
164 sent a terminating mark denotes the end of the struct. That mark is a delta=0
165 value, which has representation (00).
167 Interface types are not checked for compatibility; all interface types are
168 treated, for transmission, as members of a single "interface" type, analogous to
169 int or []byte - in effect they're all treated as interface{}. Interface values
170 are transmitted as a string identifying the concrete type being sent (a name
171 that must be pre-defined by calling Register), followed by a byte count of the
172 length of the following data (so the value can be skipped if it cannot be
173 stored), followed by the usual encoding of concrete (dynamic) value stored in
174 the interface value. (A nil interface value is identified by the empty string
175 and transmits no value.) Upon receipt, the decoder verifies that the unpacked
176 concrete item satisfies the interface of the receiving variable.
178 If a value is passed to Encode and the type is not a struct (or pointer to struct,
179 etc.), for simplicity of processing it is represented as a struct of one field.
180 The only visible effect of this is to encode a zero byte after the value, just as
181 after the last field of an encoded struct, so that the decode algorithm knows when
182 the top-level value is complete.
184 The representation of types is described below. When a type is defined on a given
185 connection between an Encoder and Decoder, it is assigned a signed integer type
186 id. When Encoder.Encode(v) is called, it makes sure there is an id assigned for
187 the type of v and all its elements and then it sends the pair (typeid, encoded-v)
188 where typeid is the type id of the encoded type of v and encoded-v is the gob
189 encoding of the value v.
191 To define a type, the encoder chooses an unused, positive type id and sends the
192 pair (-type id, encoded-type) where encoded-type is the gob encoding of a wireType
193 description, constructed from these types:
195 type wireType struct {
201 type arrayType struct {
206 type CommonType struct {
207 Name string // the name of the struct type
208 Id int // the id of the type, repeated so it's inside the type
210 type sliceType struct {
214 type structType struct {
216 Field []*fieldType // the fields of the struct.
218 type fieldType struct {
219 Name string // the name of the field.
220 Id int // the type id of the field, which must be already defined
222 type mapType struct {
228 If there are nested type ids, the types for all inner type ids must be defined
229 before the top-level type id is used to describe an encoded-v.
231 For simplicity in setup, the connection is defined to understand these types a
232 priori, as well as the basic gob types int, uint, etc. Their ids are:
242 // gap for reserved ids.
249 // 22 is slice of fieldType.
252 Finally, each message created by a call to Encode is preceded by an encoded
253 unsigned integer count of the number of bytes remaining in the message. After
254 the initial type name, interface values are wrapped the same way; in effect, the
255 interface value acts like a recursive invocation of Encode.
257 In summary, a gob stream looks like
259 (byteCount (-type id, encoding of a wireType)* (type id, encoding of a value))*
261 where * signifies zero or more repetitions and the type id of a value must
262 be predefined or be defined before the value in the stream.
264 Compatibility: Any future changes to the package will endeavor to maintain
265 compatibility with streams encoded using previous versions. That is, any released
266 version of this package should be able to decode data written with any previously
267 released version, subject to issues such as security fixes. See the Go compatibility
268 document for background: https://golang.org/doc/go1compat
270 See "Gobs of data" for a design discussion of the gob wire format:
271 https://blog.golang.org/gobs-of-data
278 Tokens starting with a lower case letter are terminals; int(n)
279 and uint(n) represent the signed/unsigned encodings of the value n.
284 uint(lengthOfMessage) Message
286 TypeSequence TypedValue
288 (TypeDefinition DelimitedTypeDefinition*)?
289 DelimitedTypeDefinition:
290 uint(lengthOfTypeDefinition) TypeDefinition
294 int(-typeId) encodingOfWireType
296 SingletonValue | StructValue
300 builtinValue | ArrayValue | MapValue | SliceValue | StructValue | InterfaceValue
302 NilInterfaceValue | NonNilInterfaceValue
305 NonNilInterfaceValue:
306 ConcreteTypeName TypeSequence InterfaceContents
308 uint(lengthOfName) [already read=n] name
310 int(concreteTypeId) DelimitedValue
314 uint(n) FieldValue*n [n elements]
316 uint(n) (FieldValue FieldValue)*n [n (key, value) pairs]
318 uint(n) FieldValue*n [n elements]
320 (uint(fieldDelta) FieldValue)*
324 For implementers and the curious, here is an encoded example. Given
325 type Point struct {X, Y int}
328 the bytes transmitted that encode p will be:
329 1f ff 81 03 01 01 05 50 6f 69 6e 74 01 ff 82 00
330 01 02 01 01 58 01 04 00 01 01 59 01 04 00 00 00
331 07 ff 82 01 2c 01 42 00
332 They are determined as follows.
334 Since this is the first transmission of type Point, the type descriptor
335 for Point itself must be sent before the value. This is the first type
336 we've sent on this Encoder, so it has type id 65 (0 through 64 are
339 1f // This item (a type descriptor) is 31 bytes long.
340 ff 81 // The negative of the id for the type we're defining, -65.
341 // This is one byte (indicated by FF = -1) followed by
342 // ^-65<<1 | 1. The low 1 bit signals to complement the
343 // rest upon receipt.
345 // Now we send a type descriptor, which is itself a struct (wireType).
346 // The type of wireType itself is known (it's built in, as is the type of
347 // all its components), so we just need to send a *value* of type wireType
348 // that represents type "Point".
349 // Here starts the encoding of that value.
350 // Set the field number implicitly to -1; this is done at the beginning
351 // of every struct, including nested structs.
352 03 // Add 3 to field number; now 2 (wireType.structType; this is a struct).
353 // structType starts with an embedded CommonType, which appears
354 // as a regular structure here too.
355 01 // add 1 to field number (now 0); start of embedded CommonType.
356 01 // add 1 to field number (now 0, the name of the type)
357 05 // string is (unsigned) 5 bytes long
358 50 6f 69 6e 74 // wireType.structType.CommonType.name = "Point"
359 01 // add 1 to field number (now 1, the id of the type)
360 ff 82 // wireType.structType.CommonType._id = 65
361 00 // end of embedded wiretype.structType.CommonType struct
362 01 // add 1 to field number (now 1, the field array in wireType.structType)
363 02 // There are two fields in the type (len(structType.field))
364 01 // Start of first field structure; add 1 to get field number 0: field[0].name
366 58 // structType.field[0].name = "X"
367 01 // Add 1 to get field number 1: field[0].id
368 04 // structType.field[0].typeId is 2 (signed int).
369 00 // End of structType.field[0]; start structType.field[1]; set field number to -1.
370 01 // Add 1 to get field number 0: field[1].name
372 59 // structType.field[1].name = "Y"
373 01 // Add 1 to get field number 1: field[1].id
374 04 // struct.Type.field[1].typeId is 2 (signed int).
375 00 // End of structType.field[1]; end of structType.field.
376 00 // end of wireType.structType structure
377 00 // end of wireType structure
379 Now we can send the Point value. Again the field number resets to -1:
381 07 // this value is 7 bytes long
382 ff 82 // the type number, 65 (1 byte (-FF) followed by 65<<1)
383 01 // add one to field number, yielding field 0
384 2c // encoding of signed "22" (0x22 = 44 = 22<<1); Point.x = 22
385 01 // add one to field number, yielding field 1
386 42 // encoding of signed "33" (0x42 = 66 = 33<<1); Point.y = 33
387 00 // end of structure
389 The type encoding is long and fairly intricate but we send it only once.
390 If p is transmitted a second time, the type is already known so the
393 07 ff 82 01 2c 01 42 00
395 A single non-struct value at top level is transmitted like a field with
396 delta tag 0. For instance, a signed integer with value 3 presented as
397 the argument to Encode will emit:
403 03 // this value is 3 bytes long
404 04 // the type number, 2, represents an integer