JSON for Modern C++  3.5.0
binary_formats.md
1 # Binary formats
2 
3 ![conversion between JSON and binary formats](images/binary.png)
4 
5 Several formats exist that encode JSON values in a binary format to reduce the size of the encoded value as well as the required effort to parse encoded value. The library implements three formats, namely
6 
7 - [CBOR](https://tools.ietf.org/html/rfc7049) (Concise Binary Object Representation)
8 - [MessagePack](https://msgpack.org)
9 - [UBJSON](http://ubjson.org) (Universal Binary JSON)
10 
11 ## Interface
12 
13 ### JSON to binary format
14 
15 For each format, the `to_*` functions (i.e., `to_cbor`, `to_msgpack`, and `to_ubjson`) convert a JSON value into the respective binary format. Taking CBOR as example, the concrete prototypes are:
16 
17 ```cpp
18 static std::vector<uint8_t> to_cbor(const basic_json& j); // 1
19 static void to_cbor(const basic_json& j, detail::output_adapter<uint8_t> o); // 2
20 static void to_cbor(const basic_json& j, detail::output_adapter<char> o); // 3
21 ```
22 
23 The first function creates a byte vector from the given JSON value. The second and third function writes to an output adapter of `uint8_t` and `char`, respectively. Output adapters are implemented for strings, output streams, and vectors.
24 
25 Given a JSON value `j`, the following calls are possible:
26 
27 ```cpp
28 std::vector<uint8_t> v;
29 v = json::to_cbor(j); // 1
30 
31 json::to_cbor(j, v); // 2
32 
33 std::string s;
34 json::to_cbor(j, s); // 3
35 
36 std::ostringstream oss;
37 json::to_cbor(j, oss); // 3
38 ```
39 
40 ### Binary format to JSON
41 
42 Likewise, the `from_*` functions (i.e, `from_cbor`, `from_msgpack`, and `from_ubjson`) convert a binary encoded value into a JSON value. Taking CBOR as example, the concrete prototypes are:
43 
44 ```cpp
45 static basic_json from_cbor(detail::input_adapter i, const bool strict = true); // 1
46 static basic_json from_cbor(A1 && a1, A2 && a2, const bool strict = true); // 2
47 ```
48 
49 Both functions read from an input adapter: the first function takes it directly form argument `i`, whereas the second function creates it from the provided arguments `a1` and `a2`. If the optional parameter `strict` is true, the input must be read completely (or a parse error exception is thrown). If it is false, parsing succeeds even if the input is not completely read.
50 
51 Input adapters are implemented for input streams, character buffers, string literals, and iterator ranges.
52 
53 Given several inputs (which we assume to be filled with a CBOR value), the following calls are possible:
54 
55 ```cpp
56 std::string s;
57 json j1 = json::from_cbor(s); // 1
58 
59 std::ifstream is("somefile.cbor", std::ios::binary);
60 json j2 = json::from_cbor(is); // 1
61 
62 std::vector<uint8_t> v;
63 json j3 = json::from_cbor(v); // 1
64 
65 const char* buff;
66 std::size_t buff_size;
67 json j4 = json::from_cbor(buff, buff_size); // 2
68 ```
69 
70 ## Details
71 
72 ### CBOR
73 
74 The mapping from CBOR to JSON is **incomplete** in the sense that not all CBOR types can be converted to a JSON value. The following CBOR types are not supported and will yield parse errors (parse_error.112):
75 
76 - byte strings (0x40..0x5F)
77 - date/time (0xC0..0xC1)
78 - bignum (0xC2..0xC3)
79 - decimal fraction (0xC4)
80 - bigfloat (0xC5)
81 - tagged items (0xC6..0xD4, 0xD8..0xDB)
82 - expected conversions (0xD5..0xD7)
83 - simple values (0xE0..0xF3, 0xF8)
84 - undefined (0xF7)
85 
86 CBOR further allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps with keys other than UTF-8 strings are rejected (parse_error.113).
87 
88 The mapping from JSON to CBOR is **complete** in the sense that any JSON value type can be converted to a CBOR value.
89 
90 If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the dump() function which serializes NaN or Infinity to null.
91 
92 The following CBOR types are not used in the conversion:
93 
94 - byte strings (0x40..0x5F)
95 - UTF-8 strings terminated by "break" (0x7F)
96 - arrays terminated by "break" (0x9F)
97 - maps terminated by "break" (0xBF)
98 - date/time (0xC0..0xC1)
99 - bignum (0xC2..0xC3)
100 - decimal fraction (0xC4)
101 - bigfloat (0xC5)
102 - tagged items (0xC6..0xD4, 0xD8..0xDB)
103 - expected conversions (0xD5..0xD7)
104 - simple values (0xE0..0xF3, 0xF8)
105 - undefined (0xF7)
106 - half and single-precision floats (0xF9-0xFA)
107 - break (0xFF)
108 
109 ### MessagePack
110 
111 The mapping from MessagePack to JSON is **incomplete** in the sense that not all MessagePack types can be converted to a JSON value. The following MessagePack types are not supported and will yield parse errors:
112 
113 - bin 8 - bin 32 (0xC4..0xC6)
114 - ext 8 - ext 32 (0xC7..0xC9)
115 - fixext 1 - fixext 16 (0xD4..0xD8)
116 
117 The mapping from JSON to MessagePack is **complete** in the sense that any JSON value type can be converted to a MessagePack value.
118 
119 The following values can not be converted to a MessagePack value:
120 
121 - strings with more than 4294967295 bytes
122 - arrays with more than 4294967295 elements
123 - objects with more than 4294967295 elements
124 
125 The following MessagePack types are not used in the conversion:
126 
127 - bin 8 - bin 32 (0xC4..0xC6)
128 - ext 8 - ext 32 (0xC7..0xC9)
129 - float 32 (0xCA)
130 - fixext 1 - fixext 16 (0xD4..0xD8)
131 
132 Any MessagePack output created `to_msgpack` can be successfully parsed by `from_msgpack`.
133 
134 If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the `dump()` function which serializes NaN or Infinity to `null`.
135 
136 ### UBJSON
137 
138 The mapping from UBJSON to JSON is **complete** in the sense that any UBJSON value can be converted to a JSON value.
139 
140 The mapping from JSON to UBJSON is **complete** in the sense that any JSON value type can be converted to a UBJSON value.
141 
142 The following values can not be converted to a UBJSON value:
143 
144 - strings with more than 9223372036854775807 bytes (theoretical)
145 - unsigned integer numbers above 9223372036854775807
146 
147 The following markers are not used in the conversion:
148 
149 - `Z`: no-op values are not created.
150 - `C`: single-byte strings are serialized with S markers.
151 
152 Any UBJSON output created to_ubjson can be successfully parsed by from_ubjson.
153 
154 If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the `dump()` function which serializes NaN or Infinity to null.
155 
156 The optimized formats for containers are supported: Parameter `use_size` adds size information to the beginning of a container and removes the closing marker. Parameter `use_type` further checks whether all elements of a container have the same type and adds the type marker to the beginning of the container. The `use_type` parameter must only be used together with `use_size = true`. Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the receiving side is immediately informed on the number of elements of the container.
157 
158 ## Size comparison examples
159 
160 The following table shows the size compared to the original JSON value for different files from the repository for the different formats.
161 
162 | format | sample.json | all_unicode.json | floats.json | signed_ints.json | jeopardy.json | canada.json |
163 | ----------------------- | -----------:| ----------------:| -----------:| ----------------:| -------------:| -----------:|
164 | JSON | 100.00 % | 100.00 % | 100.00 % | 100.00 % | 100.00 % | 100.00 % |
165 | CBOR | 87.21 % | 71.18 % | 48.20 % | 44.16 % | 87.96 % | 50.53 % |
166 | MessagePack | 87.16 % | 71.18 % | 48.20 % | 44.16 % | 87.91 % | 50.56 % |
167 | UBJSON unoptimized | 88.15 % | 100.00 % | 48.20 % | 44.16 % | 96.58 % | 53.20 % |
168 | UBJSON size-optimized | 89.26 % | 100.00 % | 48.20 % | 44.16 % | 97.40 % | 58.56 % |
169 | UBJSON format-optimized | 89.45 % | 100.00 % | 42.85 % | 39.26 % | 94.96 % | 55.93 % |
170 
171 The results show that there does not exist a "best" encoding. Furthermore, it is not always worthwhile to use UBJSON's optimizations.