This PR adds a new IPLD spec.

Some things TODO:

• [x] paths: link to path issue in go-ipfs or go-ipld
• [ ] paths: list path resolving restrictions
• [x] paths: show examples of path resolving
• [ ] examples/bitcoin: make this a real txn
• [ ] more examples

@mildred @diasdavid could you review?

This still needs work, but it gives a starting point for people to clarify the discussion. Please submit comments and/or changes.

ref ipfs/ipfs#227 ipfs/in-web-browsers#4 ipfs/in-web-browsers#28 https://github.com/ipfs/specs/pull/139

protocol/network#Encryption:

IPFS uses cyphersuites like TLS.

NOTE: we do not use TLS directly, because we do not want the CA system baggage. Most TLS implementations are very big. Since the IPFS model begins with keys, IPFS only needs to apply ciphers. This is a minimal portion of the whole TLS standard.

So you rolled your own transport encryption? I find this surprising, and I'm… skeptical.

This needs to be a new section (or an entirely separate document) rather than an aside.

(Also, I'll add that if all you want is to connect to something, exchange RSA public keys, and get an encrypted transport stream, TLS is entirely capable of providing that without involving CAs.)

Important note: This is not an entirely new proposal, it has been discussed several times over the course of the last 9 months in different situations with different views and opinions across the IPFS team. This comes to formalize the process so that we can commit and move it forward :)

Several of our users have been mislead to think of IPFS of File System instead of a Graph Database (fair point that is the name of the project after all:D). This mostly comes from the fact that the cat, get and addcommands come as first order commands and are the ones mostly used by demos.

Now, with the addition of the 'Files API', another layer of complexity and indirection was added. The common reaction to it is "Wait, what is Files API, weren't we adding Files all this time?".

With this, we miss the chance to lead our users to understand what a great Graph Database primitives IPFS offers and we also make it really hard for users to understand what Files API is, specially when it has such a generic name.

So, were is the proposal (this is not just me, although I'm the one writing it) that we've discussed through several points in time.

Rename the Files API to mfs, this will enable us to have one non generic keyword to call it that we can use with our users and also something that the community will be able to search for or aspect specially, since it has very technical details.

Move the cat, get and add commands to under the files umbrela.

In practical terms, this means:

# currently, the file manipulation commands are:
ipfs cat
ipfs get
ipfs add
ipfs files mkdir
ipfs files flush
ipfs files read
ipfs files cp
ipfs files ls
ipfs files stat
ipfs files rm
ipfs files write

Next:

# Manipulation of files
ipfs files cat
ipfs files get
ipfs files add

# mfs - Mutable File System
ipfs mfs mkdir
ipfs mfs flush
ipfs mfs read
ipfs mfs cp
ipfs mfs ls
ipfs mfs stat
ipfs mfs rm
ipfs mfs write

This is part of https://github.com/ipfs/specs/issues/257, working code in https://github.com/ipfs/kubo/pull/8890

👀 Preview

• IPIP - 0002-gateway-redirects.md
• Spec - REDIRECTS_FILE.md
• HTTP Gateways - README.md

For reference, working code in Kubo PR: https://github.com/ipfs/kubo/pull/8890

READ THIS PARAGRAPH FIRST

Hey everyone, the below is a proposal for some changes to IPFS, IPLD, and how we link to data structures. It would address a bunch of open problems that have been identified, and improve the use, tooling, and model of IPLD to allow lots of what people have been requesting for months. Please review and leave comments. We feel pretty strongly about this being a good solution, but we're not sure if we're just drinking the koolaid and going to make things worse. Sanity check before we move further pls? Also my apologies, i would spend more time writing up a better version but i just dont have enough time right now and time is of the essence on this.

[EXPERIMENTAL PROPOSAL] CIDv1 -- Important Updates to IPFS, IPLD, Multicodec, and more.

IPFS migration path to IPLD (CBOR) from MerkleDAG (ProtoBuf)

Multicodec Packed Representation

It is useful to have a compact version of mulicodec, for use in small identifiers. This compact identifier will just be a single varint, looked up in a table. Different applications can use different tables. We should probably have one common table for well-known formats.

We will establish a table for common authenticated data structure formats, for example: IPFS v0 Merkledag, CBOR IPLD, Git, Bitcoin, and more. The table is a simple varint lookup.

IPLD Links Updates (new format)

Open Problems (Motivation)

IPLD allows content to be stored in multiple different formats, and thus we need a way to understand what kind of content is being loaded in when traversing a link. A problematic issue is that old ipfs content (protobuf merkledag) does not use multicodec. It makes it difficult to distinguish between the new CBOR IPLD objects and the old Protobuf objects.

It has been proposed earlier that we wrap protobuf objects with a multicodec. But this is a problem, because the protobuf multicodec would not be authenticated. This is further complicated because many people have been requesting the ability to address raw leaf objects directly (that is, a hash linking to raw content, without ipld nor protobuf wrapping). This is a nice thing to have, but introduces difficulty in distinguishing between a protobuf or a raw encoded object, particularly when neither has a multicodec header which is authenticated by the object's hash. This lack of authentication is an attack vector: adversaries may provide protobuf objects with a raw multicodec, and depending on how implementations handle the multicodec, may poison an implementation's object repo.

Another important performance constraint is that multicodec headers are quite large: /ipld/cbor/v0, for example, is 13 bytes, which is way too large for many applications of small data. Instead, we would like to be able to use a compact multicodec representation ("multicodec packed", a single varint) to distinguish the formats. So that encoded objects are wrapped with minimal overhead. Note that this still does not affect protobuf or raw objects because these do not include headers.

Additional complications include how bitswap sends or identifies blocks, how a DagStore can pull out the object for a multihash and know what format encoding to use for it (eg raw vs protobuf), whether to allow linking from one object type to another, support for multiple base encodings for links, among others.

In discussions we (@jbenet, @diasdavid, and @whyrusleeping) reviewed many different possiblities. We considered possibilities and how it affected linking data, wrapping the data with multicodec, storing it that way under the many layers of abstraction (dag store, blockstore, datastore, file systems), fetching and retrieving objects, knowing what format to use when, ensuring values are authenticated and not opening up vectors for attackers to poison repos, and more.

In the end, we came up with a few small changes to how we represent IPLD links that solve all our problems (tm) \o/. These are:

• teach IPLD links to carry data formats (using multicodec)
• teach IPLD links to distinguish base encodings

It is worth crediting many people here that have tirelessly pushed hard to get a bunch of these ideas out. @davidar @mildred @nicola to name a few, but many others too. But they haven't looked at this yet. this first post is the first they'll hear of this construction, and they may very well hate this particular combination of ideas :) please be direct with feedback, the sooner the better.

IPLD Links learn about Base Encoding

We propose adding a multibase prefix to representations of IPLD links. This is particularly important where the encoding is not binary.

At this time, we recommend not including it in direct storage, where it should be binary. However, it may be found during the course of review that it is better to always retain the multibase prefix, even when storing in binary.

This change is a much requested option to support multiple encodings for the hashes. Current links use by default base58, which is perfect for URLs as it doesn't contain any non supported char and can be easily copy-pasted, however, for performance reasons, it is not always the best format. Some users already encode IPFS multihashes in other bases, and therefore it would be ideal to have all IPFS and IPLD tooling support these encodings through multibase, avoiding confusing failures.

IPLD Links acquire a version

The fact we propose here changes to the basic link structure remind us of the basic multiformats principle:

"Never going to change" considered harmful.

therefore we deem it wise to ensure that henceforth we include a version so that evolution can be simple, and not complex. The below changes suggest a way to distinguish between old and new links, but we should avoid such situations in the future, as this approach leverages knowledge about multihash distributions in the wild. This will be less feasible in the future.

IPLD Links learn about Codecs

The most important component of these changes introduces a multicodec-packed varint prefix to the link, to signal the encoding of the linked-to object. This enables the link to carry information about the data it points to, and ensure it is interpreted correctly. This ensures that the multicodec prefix is NOT necessary for interpretation of an IPLD object, as the link to the object carries information for its interpretation.

All proper IPLD formats (cbor and on) should carry the multicodec header at the beginning of their serialized representation, which authenticates the header and ensures clients can interpret the object without even having a link. But, this is not possible with objects of formats created before the IPLD spec, such as the first merkledag protobuf object codec in IPFS (go-ipfs 0.4.x and below). This includes also objects from other authenticated data structure distributed systems, such as Git, Bitcoin, Ethereum, and more. Finally, raw data -- which many hope to be able to address directly in IPLD -- cannot carry an authenticated prefix either.

The approach of adding the multicodec to the link entirely side-steps the problem of not being able to authenticate multicodec headers for protobufs, git, bitcoin, or raw data objects. And this avoids a nasty repo poisoning attack, possible in other proposed suggestions that rely on an unauthenticated multicodec header (carried along with the object) to determine the type of an object.

This also ensures that IPLD objects can still be content-addressed nicely, without needing to also store codec metadata alongside.

This change has been long-proposed in other forms. These other forms usually suggested attaching a @multicodec key to IPLD link objects (as a property on or next to the link), which was cumbersome and introduced complexity in other ways. Specially, it was not easy to carry over this info to a URL or copy-pasted identifier.

This multicodec-packed prefix will be sampled from a special table, maintained along with the IPLD spec. This table is expandable over time. A global multicodec table could grow from this one, or start separately.

Content IDs

This document will use the words Content IDs or CIDs. this abstraction is useful here but may not be useful beyond it. Another word -- albeit much less precise -- may be IPLD Link.

Other options are:

• SID - Self-describing IDentifier
• SSDID - Secure Self Describable Identifier
• IPLD Links -- no fancy name, less abstraction creep. less precise.

Let the old base58 multihash links to protobuf data be called CID Version 0.

CIDs Version 1 (new)

Putting together the IPLD Link update statements above, we can term the new handle for IPLD data CID Version 1, with a multibase prefix, a version, a packed multicodec, and a multihash.

<mbase><version><mcodec><mhash>

Where:

• <mbase> is a multibase prefix describing the base that encodes this CID. If binary, this is omitted.
• <version> is the version number of the cid.
• <mcodec> is a multicodec-packed identifier, from the CID multicodec table
• <mhash> is a cryptographic multihash, including: <mhash-code><mhash-len><mhash-value>

Note that all CIDs v1 and on should always begin with <mbase><version>, this evolving nicely.

Distinguishing v0 and v1 CIDs (old and new)

It is a HARD CONSTRAINT that all IPFS links continue to work. This means we need to continue to support v0 CIDs. This means IPFS APIs must accept both v0 and v1 CIDs. This section defines how to distinguish v0 from v1 CIDs.

Old v0 CIDs are strictly sha2-256 multihashes encoded in base58 -- this is because IPFS tooling only shipped with support for sha2-256. This means the binary versions are 34 bytes long (sha2-256 256 bit multihash), and that the string versions are 46 characters long (base58 encoded). This means we can recognize a v0 CID by ensuring it is a sha256 bit multihash, of length 256 bits, and base58 encoded (when a string). Basically:

• <mbase> is implicitly base58.
• <version> is implicitly 0.
• <mcodec> is implicitly protobuf (todo: add code here)
• <mhash> is a cryptographic multihash, explicit.

We can re-write old v0 CIDs into v1 CIDs, by making the elements explicit. This should be done henceforth to avoid creating more v0 CIDs. But note that many references exist in the wild, and thus we must continue supporting v0 links. In the distant future, we may remove this support after sha2 breaks.

Note we can cleanly distinguish the values, which makes it easy to support both. The code for this check is here: https://gist.github.com/jbenet/bf402718a7955bf636fb47d214bcef8a

IPLD supports non-CID hash links as implicit CIDv1s

Note that raw hash links stored in various data structures (eg Protbouf, Git, Bitcoin, Ethereum, etc) already exist. These links -- when loaded directly as one of these data structures -- can be seen as "linking within a network" whereas proper CIDv1 IPLD links can be seen as linking "across networks" (internet of data! internet of data structures!). Supporting these existing (or even new) raw hash links as a CIDv1 can be done by noting that when on data structure links with just a raw binary link, the rest of the CIDv1 fields are implicit:

• <mbase> is implicitly binary or whatever the format encodes.
• <version> is implicitly 1.
• <mcodec> is implicitly the same as the data structure.
• <mhash> can be determined from the raw hash.

Basically, we construct the corresponding CIDv1 out of the raw hash link because all the other information is in the context of the data structure. This is very useful because it allows:

• more compact encoding of a CIDv1 when linking from one data struct to another
• linking from CBOR IPLD to other CBOR IPLD objects exactly as has been spec-ed out so far, so any IPLD adopters continue working.
• (most important) opens the door for native support of other data structures

IPLD native support for Git, Bitcoin, Ethereum, and other authenticated data structures

Given the above addressing changes, it is now possible to directly address and implement native support for Git, Bitcoin, Ethereum, and other authenticated data structure formats. Such native support would allow resolving through such objects, and treat them as true IPLD objects, instead of needing to wrap them in CBOR or another format. This is the proper merkle-forest. \o/

IPLD addresses raw data

Given the above addressing changes, it is now possible to address raw data directly, as an IPLD node. This node is of course taken to be just a byte buffer, and devoid of links (i.e. a leaf node).

The utility of this is the ability to directly address any object via hashing external to IPLD datastructures, which is a much-requested feature.

Support for multiple binary packed formats

Contrary to existing Merkle objects (e.g IPFS protobuf legacy, git, bitcoin, dat and others), new IPLD ojects are authenticated AND self described data blobs, each IPLD object is serialized and prefixed by a multicodec identifying its format.

Some candidate formats:

• /ipld/cbor
• /ipld/ion/1.0.0
• /ipld/protobuf/3.0.0
• /ipld/protobuf/2.0.0

There is one strong requirement for these formats to work: a format MUST have a 1:1 mapping to the canonical IPLD serialiation format. Today (July 29, 2016), that format is CBOR.

Changes to Interfaces / Specs

Need changes to:

• IPFS specs (addressing in particular) need to support CIDv1
• IPFS interfaces need to support CIDv1
• Add a new, small CIDv1 or "IPLD Links" spec
• IPLD spec is compatible. Can improve in wording. CBOR data format does not change. Pathing does not change. .

Support for CID v0 and v1

It is a HARD CONSTRAINT that all IPFS links continue to work. In order to support both CID v0 paths (/ipfs/<mhash>) and the new CID v1 paths (/ipfs/<mbase><version><mcodec><mhash>, IPFS and other IPLD tooling will detect the version of the CID through a matching function. (See "Distinguishing v0 and v1 CIDs (old and new)" above).

The following interfaces must support both types:

• The IPFS API, which takes CIDs and Paths
  • This includes subprotocols, such as Bitswap
• HTTP-to-IPFS Gateway, for all existing https://ipfs.io/ipfs/... links

Current UnixFSv1 importers do not encode most of the standard file metadata from most file systems.

This has been a particular challenge for package managers since they already rely on some of this metadata.

The goal of this issue is to surface all the necessary discussion points in order to drive a new PR against the unixfs spec.

Potential metadata

• Permissions
  • Executable bit
  • Ownership (user and group)
• Filename in file object
• mtime
• ctime
• atime

Additional considerations

For time stamps (mtime, ctime, atime) we need to decide if we’re going to use high precision times or not. Most systems expect a 32-bit integer (low precision) while other use cases may need a 64-bit integer (high precision).

Do we want to store additional metadata of the directory? How do we handle updating this when someone updates only a single file in the directory?

Where do we store this metadata?

In terms of the data format, should these properties be added to the File message or the Data message?

History

The history of this feature as well as meeting notes where this feature was prioritized are available here.

As currently defined the closest we can come to removing a mode/mtime previously set on a file is to set it's value to 0, which works in that the CID returns to what it would have been pre-metadata, but when combined with default file/dir modes also has the side effect of making it impossible to actually set the file mode to 0.

Worse, if you set the file mode to 0 you actually get the default file mode back of 0644 which is probably not what you intended.

This is because protocol buffers assign default values to all fields so we cannot differentiate between the situation where a user has set a thing to 0 vs when they haven't set anything at all.

The proposed solution is to store the mode/mtime as a string which can then be interpreted as a base 8 number (base 10 for mtime), which is a little icky but does solve the problem:

| Value | Interpretation | | -------- | -------- | | '' | No mode has been set | | '0' | Mode has been set to zero | | '0755' | Mode has been set to 0755 |

mtime is changed too because if I add some metadata to a file, I should be able to remove it - it shouldn't be a one-way thing. Unless we are happy with the mtime of all entries defaulting to 1970.

It also removes the default mode values as otherwise it's not an opt-in upgrade.

This patch fixes some of the inconsistencies in the IPLD spec, as well as resolving some existing issues, namely:

• ipfs/go-ipfs#2053 - allow raw data in leaf nodes
• ipfs/go-ipfs#1582 - use EJSON for encoding (non-UTF8) binary data
• mandate CBOR Strict Mode to avoid having to deal with duplicated keys

Also some possible changes to the examples, but these are more to do with suggested convention rather than IPLD itself:

• change link to content (although we could also use data or something similar), as file content needn't be merkle-links, and can in fact be directly embedded into small files
• removed the extraneous subfiles wrapper, as file content can be either (a merkle-link to) a bytestring, a unicode string, or a list thereof

I can split these into separate PRs if that would be more helpful.

Improves PR #37 and replaces PR #60. The idea is that there is only one kind of merkle-paths that are not to be confused with unixfs paths. These paths are powerful enough to be able to access multiple properties in IPLD objects and resolve merkle links.

In PR #37, we left out an important part of the spec aside: the relationship with protocol buffer serialization. This ought to be described as it has effects that may be far reaching.

TODO items:

• [X] Decide if we choose a format that requires path component escaping: no escaping needed
• [X] Decide which special key to use to avoid conflict with path component (@attrs in one proposition, with @ escaping, . in the other proposition): not needed

Not sure who to tag for review on this but some of these other implementations have out and in use at least some places for a good while now, so I suppose we ought to link to them :)

Another draft.

This one is short and simple. Just describes how one can find a gateway if the multi gateway client isn't used. Consider this spec to be for those apps that want IPFS integration but for whatever reason can't use a multi gateway client library.

A spec to describe how multi gateway clients - formally known as racing gateways - should behave. This is very much a companion spec to https://github.com/ipfs/specs/pull/356.

To refine it's place. This spec describes how multi gateway clients work and should be implemented. https://github.com/ipfs/specs/pull/356 would have this in it's implementation.

The spec docs for Cache-Control: only-if-cached (See https://github.com/ipfs/kubo/pull/9082) don't mention how gateways should handle requests with inline CIDs (whether they should be treated as any other CID, or separately to other CIDs).

Right now, gateways respond with a miss on the initial request, but also, they seem to respond with HTTP/2 200 as well? (I'm pretty sure this is affecting newer gateways as I don't think I saw HTTP/2 412 in the Curl requests once)

Draft for sure!

This spec is intended to specify how a "ipfsclient" works. It relies on 3 different specs:

1. IPIP-0280 (gateways file)
2. IPIP-0000 (multi gateway client. This spec defines how gateways are determined to be used and use them)
3. IPIP-0000 (using gateways. This spec defines how an application can find and use gateways)

The missing specs will be on github later. Will update this message when they are online.