View source for Software:Cap'n Proto

{{Short description|Serialization format and Remote Procedure Call (RPC) framework}}
{{Infobox software
| author = Kenton Varda
| latest release version = 0.10.4
| latest release date = {{Start date and age|2023|04|13}}
| programming language = [[C++]]
| genre = [[Remote procedure call]] framework, [[Serialization|serialization]] format and library, [[Interface description language|IDL]] compiler
| license = [[Software:MIT License|MIT License]]
| website = {{URL|https://capnproto.org}}
| repo = {{URL|https://github.com/capnproto/capnproto}}
}}

'''Cap’n Proto''' is a [[Serialization|data serialization]] format and [[Remote procedure call|Remote Procedure Call]] (RPC) framework for exchanging data between computer programs.  The high-level design focuses on speed and security, making it suitable for network as well as inter-process communication.  Cap'n Proto was created by the former maintainer of Google's popular [[Software:Protocol Buffers|Protocol Buffers]] framework (Kenton Varda) and was designed to avoid some of its perceived shortcomings.

== Technical overview ==

=== IDL Schema ===

Like most RPC frameworks dating as far back as [[Sun RPC]] and OSF DCE RPC (and their object-based descendants [[Common Object Request Broker Architecture|CORBA]] and [[Distributed Component Object Model|DCOM]]), Cap'n Proto uses an [[Interface description language|Interface Description Language]] (IDL) to generate RPC libraries in a variety of programming languages - automating many low level details such as handling network requests, converting between data types, etc.  The Cap'n Proto interface schema uses a [[List of programming languages by type|C-like syntax]] and supports common [[Primitive data type|primitives data types]] (booleans, integers, floats, etc.), [[Composite data type|compound types]] (structs, lists, enums),  as well as [[Generic programming|generics]] and dynamic types.<ref>{{Cite web|last=Varda|first=Kenton|date=|title=Cap'n Proto Schema Language|url=https://capnproto.org/language.html|url-status=live|archive-url=https://web.archive.org/web/20150317184515/https://capnproto.org/language.html |archive-date=2015-03-17 |access-date=2020-09-05|website=}}</ref> Cap'n Proto also supports object-oriented features such as multiple inheritance, which has been criticized for its complexity.<ref>{{Cite web|last=Denhardt|first=Ian|date=June 2019|title=A Critique of the Cap'n Proto Schema Language.|url=https://zenhack.net/2019/06/25/a-critique-of-the-capnproto-schema-language.html|url-status=live|archive-url=https://web.archive.org/web/20190626022038/https://zenhack.net/2019/06/25/a-critique-of-the-capnproto-schema-language.html |archive-date=2019-06-26 |access-date=2020-10-10|website=zenhack.net}}</ref>   <source lang="capnp">
@0xa558ef006c0c123; #Unique identifiers are manually or automatically assigned to files and compound types

struct Date @0x5c5a558ef006c0c1 {
  year @0 :Int16; #@n marks order values were added to the schema
  month @1 :UInt8;
  day @2 :UInt8;
}

struct Contact @0xf032a54bcb3667e0 {
  name @0 :Text;
  birthday @2 :Date; #fields can be added anywhere in the definition, but their numbering must reflect the order in which they were added

  phones @1 :List(PhoneNumber);

  struct PhoneNumber { #Compound types without an static ID cannot be renamed, as automatic IDs are deterministically generated
    number @0 :Text;
    type @1 :PhoneType = mobile; #Default value

    enum PhoneType {
      mobile @0;
      landline @1;
    }
  }
}
</source>Values in Cap'n Proto messages are represented in [[Binary code|binary]], as opposed to text encoding used by "[[Human-readable medium|human-readable]]" formats such as [[JSON]] or [[XML]].  Cap'n Proto tries to make the storage/network protocol appropriate as an in-memory format, so that no translation step is needed when reading data into memory or writing data out of memory.<ref group="note">Unlike [[Software:Apache Arrow|Apache Arrow]], Cap'n Proto's in-memory values are [https://news.ycombinator.com/item?id=14248780 not suited for sharing mutable data]</ref>  For example, the representation of numbers ([[Endianness|endianness]]) was chosen to match the representation the most popular CPU architectures.<ref>{{Cite web|first=Kenton|last=Varda|date=|title=Cap'n Proto: Introduction|url=https://capnproto.org/index.html|url-status=live|archive-url=https://web.archive.org/web/20150317185148/https://capnproto.org/index.html |archive-date=2015-03-17 |access-date=2020-11-09|website=Cap'n Proto Homepage}}</ref> When the in-memory and wire-protocol representations match, Cap'n Proto can [[Zero-copy|avoid copying and encoding data]] when creating or reading a message and instead [[Pointer (computer programming)|point]] to the location of the value in memory.  Cap'n Proto also supports random access to data, meaning that any field can be read without having to read the entire message.<ref name=":0">{{Cite web|last=Varda|first=Kenton|date=|title=Cap'n Proto: Encoding Spec|url=https://capnproto.org/encoding.html|url-status=live|archive-url=https://web.archive.org/web/20150317202732/https://capnproto.org/encoding.html |archive-date=2015-03-17 |access-date=|website=Cap'n Proto}}</ref>

Unlike other binary serialization protocols such as [[Fast Infoset|XMI]], Cap'n Proto considers fine-grained [[Data validation|data validation]] at the RPC level an anti-feature that limits a protocol's ability to [[Schema evolution|evolve]].  This was informed by experiences at Google where simply changing a field from ''mandatory'' to ''optional'' would cause complex operational failures.<ref>{{Cite web|first=Kenton|last=Varda|date=|title=FAQ § How do I make a field "required", like in Protocol Buffers?|url=https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers|url-status=live|archive-url=https://web.archive.org/web/20150318011733/https://capnproto.org/faq.html |archive-date=2015-03-18 |access-date=2020-09-05|website=Cap'n Proto}}</ref><ref group="note">Marking a field as required was removed from [https://developers.google.com/protocol-buffers/docs/proto3 Protocol Buffers 3].</ref>  Cap'n Proto schemas are designed to be flexible as possible and pushes data validation to the application level, allowing arbitrary renaming of fields, adding new fields, and making concrete types generic.<ref>{{Cite web|title=Cap'n Proto: Schema Language|url=https://capnproto.org/language.html#evolving-your-protocol|access-date=2020-10-10|website=capnproto.org}}</ref> Cap'n Proto does, however, validate pointer bounds and type check individual values when they are first accessed.<ref name=":0" />

Enforcing complex schema constraints would also incur significant overhead,<ref group="note">''Assuming the data has already been allocated'' (e.g. in network buffers, read from disk) access becomes [[Big O notation|O(1)]].  Additional serialization/deserialization steps (as required to inspect values) would limit performance to O(n).</ref> negating the benefits of reusing in-memory data structures and preventing random access to data.<ref>{{Cite web|title=Cap'n Proto: Cap'n Proto, FlatBuffers, and SBE|url=https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html|access-date=2020-10-10|website=capnproto.org}}</ref> Cap'n Proto protocol is ''theoretically'' suitable<ref>{{Cite web|last=Richardson|first=Corey|date=October 2016|title=Robigalia: An Operating System for the Modern Era|url=https://robigalia.gitlab.io/book/rosme.html#_capabilities_in_phoma|url-status=live|archive-url=https://web.archive.org/web/20180915204543/https://robigalia.gitlab.io/book/rosme.html |archive-date=2018-09-15 |access-date=2020-10-10|website=robigalia.gitlab.io}}</ref> for very fast [[Inter-process communication|inter-process communication]] (IPC) via immutable shared memory, but as of October 2020 none of the implementations support data passing via shared memory.<ref>{{Cite web|last=Kenton|first=Varda|date=May 3, 2017|title=Why is not intended that in-memory state be in Cap'n Proto / Protobuf objects?|url=https://news.ycombinator.com/item?id=14252940|access-date=2020-10-10|website=Hacker News (news.ycombinator.com)}}</ref> However, Cap'n Proto is still generally considered faster than Protocol Buffers and similar RPC libraries.<ref>{{Cite web|last=Naughton|first=Chris|date=Aug 24, 2018|title=Protocol Benchmarks|url=https://github.com/ChrisMacNaughton/proto_benchmarks|url-status=live|archive-url=https://web.archive.org/web/20180830220546/https://github.com/ChrisMacNaughton/proto_benchmarks |archive-date=2018-08-30 |access-date=2020-09-05|website=Github}}</ref><ref>{{Cite web|last=Parimi|first=Dinesh|date=2019|title=Datacenter Tax Cuts: Improving WSC Efficiency Through Protocol Buffer Acceleration|url=https://people.eecs.berkeley.edu/~kubitron/courses/cs262a-F19/projects/reports/project4_report_ver2.pdf|url-status=live|archive-url=https://archive.today/20200906032310/https://people.eecs.berkeley.edu/~kubitron/courses/cs262a-F19/projects/reports/project4_report_ver2.pdf|archive-date=2020-09-06|access-date=2020-09-05|website=}}</ref>

=== Networking ===
Cap'n Proto RPC is network aware: supporting both handling of disconnects and [[Futures and promises|promise pipelining]], wherein a server pipes the output of one function into another function. This saves a client a round trip per successive call to the server without having to provide a dedicated API for every possible call graph.  Cap'n Proto can be layered on top of TLS<ref>{{Cite web|title=Cap'n Proto: Road Map|url=https://capnproto.org/roadmap.html|access-date=2020-10-10|website=capnproto.org}}</ref> and support for the Noise Protocol Framework is on the roadmap.<ref name=":1">{{Cite web|date=2021-03-13|title=Roadmap|url=https://capnproto.org/roadmap.html#rpc-protocol-features|url-status=live|website=Cap'n Proto|archive-url=https://web.archive.org/web/20150317192558/https://capnproto.org/roadmap.html |archive-date=2015-03-17 }}</ref>  Cap'n Proto RPC is transport agnostic, with the mainline implementation supporting WebSockets, HTTP, TCP, and UDP.<ref>{{Cite web|title=Cap'n Proto: C++ RPC|url=https://capnproto.org/cxxrpc.html|access-date=2020-10-10|website=capnproto.org}}</ref>

=== Capability security ===
The Cap'n Proto RPC standard has a rich capability security model based on the CapTP protocol used by the [[E (programming language)|E programming language]].<ref>{{Cite web|title=RPC Protocol|url=https://capnproto.org/rpc.html#protocol-features|url-status=live|website=Cap'n Proto|archive-url=https://web.archive.org/web/20150318011754/https://capnproto.org/rpc.html |archive-date=2015-03-18 }}</ref>
As of October 2020, the reference implementation only supports level 2.<ref name=":1" />

== Adoption ==
Cap'n Proto was originally created for Sandstorm.io, a startup offering a web application hosting platform with capability-based security.  After Sandstorm.io failed commercially, the development team was acqui-hired by Cloudflare,<ref>{{Cite web|last=Varda|first=Kenton|date=13 Mar 2017|title=The Sandstorm Team is joining Cloudflare|url=https://sandstorm.io/news/2017-03-13-joining-cloudflare|url-status=live|archive-url=https://web.archive.org/web/20170313180618/https://sandstorm.io/news/2017-03-13-joining-cloudflare |archive-date=2017-03-13 |access-date=2020-09-05|website=Sandstorm.io}}</ref> which uses Cap'n Proto internally.<ref>{{Cite web|last=Zhi|first=Jiale|date=2013|title=Introducing lua-capnproto: better serialization in Lua|url=https://blog.cloudflare.com/introducing-lua-capnproto-better-serialization-in-lua/|url-status=live|archive-url=https://web.archive.org/web/20140306184242/http://blog.cloudflare.com:80/introducing-lua-capnproto-better-serialization-in-lua |archive-date=2014-03-06 |access-date=2020-09-05|website=}}</ref>

== Notes ==
{{reflist|group=note}}

== References ==
<!-- Inline citations added to your article will automatically display here. See en.wikipedia.org/wiki/WP:REFB for instructions on how to add citations. -->
{{reflist}}

{{Data Exchange}}

{{DEFAULTSORT:Capn Proto}}
[[Category:Data serialization formats]]
[[Category:Remote procedure call]]
[[Category:Inter-process communication]]

{{Sourceattribution|Cap'n Proto|1}}