continuation of use string instead of struct backing #71

kevina · 2018-08-25T07:44:06Z

Rebased #47 and made some improvements. Most of the changes where noted in my review of #47.

I tried to keep each change in it's own commit, so if any of my changes are not agreeable I can back them out.

Closes #3. Closes #38.

cid.go

Stebalien · 2018-08-25T18:20:16Z

cid.go

+	if c.Version() == 0 {
+		return DagProtobuf
+	}
+	bytes := c.Bytes()


We may want to either:

Implement a basic non-allocating "byte reader" for strings and use binary.ReadUvarint.

Manually parse these.

Note: for the first one, we can just check c.string[0] == 1 and then skip it.

The ByteReader would probably look like:

type stringReader string func (r *stringReader) ReadByte() (byte, error) { if len(*r) == 0 { return 0, io.EOF } b := r[0] *r = *r[1:] return b, nil }

Stebalien · 2018-08-25T18:21:54Z

cid.go

@@ -159,9 +159,9 @@ func NewCidV1(codecType uint64, mhash mh.Multihash) Cid {
 // - version uvarint
 // - codec uvarint
 // - hash mh.Multihash
-type Cid string
+type Cid struct{ string }


Good idea. This will also allow us to add fields if we need them (as long as we can still put the struct in a map).

Stebalien · 2018-08-25T18:22:31Z

This looks like a good way to do this. We should probably, eventually, do the same with Multihash but we can do that later.

kevina · 2018-08-25T19:42:24Z

This looks like a good way to do this. We should probably, eventually, do the same with Multihash but we can do that later.

The fact that the Cid type change from a pointer to a non-pointer type is going to make this change painful, I would vote to change multihash to a string at the same time and get it over with, but I am not of a strong option on this if you are eager to push this trough now.

Stebalien · 2018-08-27T20:24:16Z

The fact that the Cid type change from a pointer to a non-pointer type is going to make this change painful, I would vote to change multihash to a string at the same time and get it over with, but I am not of a strong option on this if you are eager to push this trough now.

I think the multihash change will be painful either way, will it not?

kevina · 2018-08-27T20:28:28Z

@Stebalien Probably, it might be easier to do it at the same time though.

warpfork · 2018-08-27T22:37:28Z

cid.go

+// - hash mh.Multihash
+type Cid struct{ str string }
+
+var Nil = Cid{}


How about calling this Zero?

"Zero" would more closely match https://golang.org/ref/spec#The_zero_value

Nil is a more intuitive name to me. Zero implies a numeric value to me.

You are correct in that go would call this the "Zero" value. However, I really don't like it.

Arguments against Zero:

To me, "Zero" implies a number.

Zero could mean the CID for the number 0 (z7xsiH4JqWQudech83). I expect that users will define "constant" CIDs all the time although this one might be a stretch.

We've historically used nil for "empty CIDs".

We have a CIDv0 ("version zero").

Zero "looks funny" (entirely subjective) in the return position.

func Foo() (Cid, error) { return Nil, fmt.Errorf("bad stuff") } func Bar() (Cid, error) { return Zero, fmt.Errorf("bad stuff") }

Some other options just to toss out:

cid.Invalid -- this shows up in a lot of other idiomatic Go code... but mostly in enum situations, so maybe less appropriate here.

cid.None -- avoids all the numeric connotations, less "looks funny" in returns I think?

I still like Nil better...

@warpfork @Stebalien instead of Nil how about Undef? I like it a bit better then Invalid and about the same as Nil.

I could go for Undef 👍

👎 on undef and invalid. The value isn't undefined and isn't the only invalid value. I'd prefer None (although I still prefer Nil).

Actually, seeing cid.Defined(), I'm onboard with this (if we return Undef and check with cid.Defined()).

Okay I am going with Undef.

Stebalien · 2018-08-27T23:14:57Z

@kevina up to you.

kevina · 2018-08-28T18:52:11Z

@kevina up to you.

Once we switch to using Multihashes in the blockstore, the cid.Hash() method is going to be called a lot more. To prevent performance regressions it would probably be a good idea if that method doesn't alloc.

Stebalien · 2018-08-28T19:20:21Z

Go ahead.

kevina · 2018-08-29T02:36:54Z

Go ahead.

Done. See multiformats/go-multihash#82

Stebalien · 2018-08-29T21:37:30Z

Probably, it might be easier to do it at the same time though.

Given that this isn't looking to be the case, let's go ahead with this and then try tackling the multihash issue.

kevina · 2018-08-29T22:18:35Z

Given that this isn't looking to be the case, let's go ahead with this and then try tackling the multihash issue.

I left an additional option on the p.r. that might make this easier.

Please let me know what you intended time frame for landing this is. My current plan was to add the --cid-base to go-ipfs and then swing back around to merging this p.r. and fixing any broken code along the way.

Stebalien · 2018-08-29T22:34:53Z

I'd like to get this done sooner or later. It's blocking ipfs/go-ipld-cbor#30.

kevina · 2018-08-29T23:25:10Z

Given that this isn't looking to be the case, let's go ahead with this and then try tackling the multihash issue.

Okay I factored out those changes into a separate p.r.

Stebalien

There are probably some optimizations we can toy with later but this looks good for now (from my end).

cid.go

kevina · 2018-08-30T02:30:49Z

There are probably some optimizations we can toy with later but this looks good for now (from my end)

Exactly what did you have in mind for this?

Stebalien · 2018-08-30T02:45:14Z

Exactly what did you have in mind for this?

We can assume that the version is one byte (and check against the constant 0x1).
We can probably rearrange some of the logic related to Prefix() to avoid parsing varints multiple times.
etc...

But let's handle that later. The key part here is changing the interfaces.

kevina · 2018-08-30T02:51:56Z

We can assume that the version is one byte (and check against the constant 0x1).

Actually I already did this, but it was part of the commit the used the new Multihash. (#73)

We can probably rearrange some of the logic related to Prefix() to avoid parsing varints multiple times.

Ditto. I think.

Stebalien · 2018-08-30T03:16:41Z

@kevina want to go ahead and do the refactor?

kevina · 2018-08-30T03:25:18Z

@kevina want to go ahead and do the refactor?

I'm confused? To make it clear. Most of those optimizations I did as part of the switch to using a string multihash which in its self is an optimization. Did you want me to split #73 to separate out those optimizations?

Stebalien · 2018-08-30T03:44:43Z

I'm confused?

No no, I'm talking about the whole cid type refactor (that is, fixing all the dependent packages). We can do any additional optimizations to this package later.

kevina · 2018-08-30T03:48:10Z

Sure, not sure I will get to it tonight though.

Applied Fix.

kevina · 2018-09-08T00:21:34Z

Ok I bubbled this all up all the way to go-ipfs but created p.r. to give others a chance to review and know what is going on. I will merge within a week.

@Stebalien @warpfork On second thought I am not super happy with using Undef and still think Nil (or maybe None) is better, but I don't want to bikeshed on this.

While bubbling this up I often used cid.Cid{} returning an error from a function that also returns a Cid (i.e. return cid.Cid{}, err). That may be ugly, but it will be immanently obvious what is going on. In a few limited cases I used cid.Undef, in particular when assigning to a variable where we care if the Cid is defined or not. I used Defined() to test if a Cid is defined and never c == Undef.

All the p.r. involved have gx publish commit as part of the p.r. I will rebase that commit out and republish as I merge them.

lanzafame · 2018-09-11T03:08:14Z

cid.go

 	}
+
+	return Cid{string(buf[:n+hashlen])}
 }

 // Cid represents a self-describing content adressed
 // identifier. It is formed by a Version, a Codec (which indicates
 // a multicodec-packed content type) and a Multihash.


@kevina Could you please add a high level description of how the version, codec, and hash are packed into a string, similar to how the multihash docs do, to the godoc.

We decided against that. The format varies according to the CID type (see #71 (comment)). The layout is according to the cid spec. see https://github.com/ipld/cid.

Fair, thoughts on adding a link to the cid spec to the comment then? The location of various specs isn't the easiest thing to deduce, i.e. the cid spec is under the ipld org whereas the implementation is under the ipfs org. Just a suggestion, not a blocker.

There entire documentation could use cleaning up. The format is mentioned in multiple places and the link to the spec is in the documentation for the package. https://godoc.org/github.com/ipfs/go-cid.

kevina · 2018-09-11T18:05:03Z

cid.go

+	}
+	if dec.Code != mh.SHA2_256 || dec.Length != 32 {
+		panic("invalid hash for cidv0")
+	}
 	return Cid{string(mhash)}


@Stebalien I panic here in order to avoid introducing another API change. However, the patch to strict CIDs (#40) does change the return type to (Cid, error) for both NewCidV0 and NewCidV1. Should we just change them and get it now and get it over with?

I don't have strong felling either way?

Let's keep this simple.

Okay, the only reason I suggested doing it now is to avoid another API change in the near future. But we can keep this simple and change the return type later.

I just don't want to drag this out.

Stebalien · 2018-09-11T22:35:49Z

@kevina are we going to need to bubble that multibase update? If so, we should do that first. The list of changes we'll need to make for that will be much larger.

kevina · 2018-09-11T22:41:53Z

@Stebalien actually it won't I checked.

Using my new tool:

go-ipfs$ gx-update-helper preview go-multibase
github.com/multiformats/go-multibase

github.com/ipfs/go-cid :: go-multibase

github.com/ipfs/go-block-format :: go-cid
github.com/ipfs/go-cidutil :: go-cid
github.com/ipfs/go-ipfs-ds-help :: go-cid
github.com/libp2p/go-libp2p-routing :: go-cid
github.com/ipfs/go-verifcid :: go-cid

github.com/ipfs/go-ipfs-blocksutil :: go-block-format
github.com/ipfs/go-ipfs-chunker :: go-block-format
github.com/ipfs/go-ipfs-exchange-interface :: go-block-format
github.com/ipfs/go-ipld-format :: go-block-format
github.com/libp2p/go-libp2p-kad-dht :: go-libp2p-routing
github.com/libp2p/go-libp2p-routing-helpers :: go-libp2p-routing
github.com/ipfs/go-ipfs-blockstore :: go-block-format go-ipfs-ds-help
github.com/ipfs/go-ipfs-routing :: go-ipfs-ds-help go-libp2p-routing

github.com/ipfs/go-ipfs-posinfo :: go-ipld-format
github.com/ipfs/go-ipld-cbor :: go-ipld-format
github.com/ipfs/go-ipld-git :: go-ipld-format
github.com/libp2p/go-libp2p-pubsub-router :: go-libp2p-routing-helpers
github.com/ipfs/go-ipfs-exchange-offline :: go-ipfs-blockstore go-ipfs-blocksutil go-ipfs-exchange-interface
github.com/ipfs/go-bitswap :: go-ipfs-blockstore go-ipfs-blocksutil go-ipfs-exchange-interface go-ipfs-routing

github.com/ipfs/go-blockservice :: go-bitswap go-ipfs-exchange-offline

github.com/ipfs/go-merkledag :: go-blockservice

github.com/ipfs/go-path :: go-merkledag
github.com/ipfs/go-unixfs :: go-merkledag

github.com/ipfs/go-mfs :: go-path go-unixfs

github.com/ipfs/go-ipfs :: go-mfs

Which is saying anything that depends on go-multibase also depends on go-cid.

Stebalien · 2018-09-11T22:46:05Z

Hm. Yeah, you're right. But whatever we do, let's really try to avoid tacking on random additions at the last minute.

ghost assigned kevina Aug 25, 2018

ghost added the status/in-progress In progress label Aug 25, 2018

kevina changed the title ~~use string instead of struct backing~~ continuation of use string instead of struct backing Aug 25, 2018

kevina mentioned this pull request Aug 25, 2018

cid implementation roundup #70

Merged

Stebalien reviewed Aug 25, 2018

View reviewed changes

warpfork reviewed Aug 27, 2018

View reviewed changes

warpfork mentioned this pull request Aug 27, 2018

cid implementation variations++ #72

Merged

kevina force-pushed the feat/string branch from fa4a10b to 70b07e9 Compare August 29, 2018 03:11

kevina force-pushed the feat/string branch from 70b07e9 to 193039c Compare August 29, 2018 23:13

Stebalien reviewed Aug 30, 2018

View reviewed changes

cid.go Outdated Show resolved Hide resolved

kevina requested a review from Stebalien September 8, 2018 00:13

kevina mentioned this pull request Sep 10, 2018

Making GX pleasant to use whyrusleeping/gx#179

Open

lanzafame reviewed Sep 11, 2018

View reviewed changes

lanzafame mentioned this pull request Sep 11, 2018

Update to new Cid type ipfs-cluster/ipfs-cluster#535

Closed

4 tasks

kevina commented Sep 11, 2018

View reviewed changes

kevina added 3 commits September 11, 2018 19:17

Make sure we have a SHA2_256, length 32 hash when creating a CidV0.

dfc48d3

Gx update go-multibase.

f003360

gx publish 0.9.0

6e296c5

kevina force-pushed the feat/string branch from 4ea31fa to 6e296c5 Compare September 11, 2018 23:19

kevina merged commit 6e296c5 into master Sep 12, 2018

ghost removed the status/in-progress In progress label Sep 12, 2018

continuation of use string instead of struct backing #71

continuation of use string instead of struct backing #71

Conversation

kevina commented Aug 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien commented Aug 25, 2018

kevina commented Aug 25, 2018

Stebalien commented Aug 27, 2018

kevina commented Aug 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien commented Aug 27, 2018

kevina commented Aug 28, 2018

Stebalien commented Aug 28, 2018

kevina commented Aug 29, 2018

Stebalien commented Aug 29, 2018

kevina commented Aug 29, 2018

Stebalien commented Aug 29, 2018

kevina commented Aug 29, 2018

Stebalien left a comment

Choose a reason for hiding this comment

kevina commented Aug 30, 2018

Stebalien commented Aug 30, 2018

kevina commented Aug 30, 2018 • edited Loading

Stebalien commented Aug 30, 2018

kevina commented Aug 30, 2018

Stebalien commented Aug 30, 2018

kevina commented Aug 30, 2018

kevina commented Sep 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien commented Sep 11, 2018

kevina commented Sep 11, 2018

Stebalien commented Sep 11, 2018

kevina commented Aug 25, 2018 •

edited

Loading

kevina commented Aug 30, 2018 •

edited

Loading

kevina commented Sep 8, 2018 •

edited

Loading