determine the files to be different sizes #3209

bluelovers · 2020-08-04T18:25:16Z

Version:

Platform:

Subsystem:

Severity:

Description:

https://discuss.ipfs.io/t/why-has-same-sub-cids-but-not-same-root-cid/8810/3

Steps to reproduce the error:

upload file by ipds-desktop and https://share.ipfs.io/

aschmahmann · 2020-08-04T18:47:16Z

@achingbrain @olizilla

I'm not sure if this is because share.ipfs.io is out of date or not. However, I did try adding the same files from the link above into go and js-ipfs ipfs add -r theDirectory and got the same result which is good, and indicates it may be a share.ipfs.io issue (it hasn't been updated in a while).

aschmahmann · 2020-08-04T18:47:38Z

Perhaps an unrelated issue, is that it looks like jsipfs files stat does not seem to be calculating cumulative size correctly for directories. If it's an unrelated issue feel free to delete this comment and put it in a new issue.

PS /home/adin/workspace/tmp/full/QmbHK4rKG1p87DTnemXrhh9QSxZk6xBCSHg5E985MEH1Zu> jsipfs files ls /
hof.zip
php-5.6.32-nts-Win32-VC11-x64.zip
PS /home/adin/workspace/tmp/full/QmbHK4rKG1p87DTnemXrhh9QSxZk6xBCSHg5E985MEH1Zu> jsipfs files stat /       
QmPq9t1UbjT51Y2bo4dc84KKXtHqD5KnD4vDNMqeAertvi
Size: 0
CumulativeSize: 8070
ChildBlocks: 2
Type: directory
Mode: drwxr-xr-x
Mtime: -
PS /home/adin/workspace/tmp/full/QmbHK4rKG1p87DTnemXrhh9QSxZk6xBCSHg5E985MEH1Zu> jsipfs files stat /hof.zip
QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH
Size: 20010278
CumulativeSize: 20015062
ChildBlocks: 77
Type: file
Mode: -rw-r--r--
Mtime: -

bluelovers · 2020-08-04T19:08:25Z

temp/111.png =>
https://explore.ipld.io/#/explore/QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9

import { outputFile } from 'fs-extra';
import fetchIPFS from 'fetch-ipfs';
import { statSync } from 'fs';
import { publishToIPFSAll } from 'fetch-ipfs/lib/put/all';
import { filterList } from 'ipfs-server-list';

console.log(`fs.stat`, statSync('./temp/111.png').size)

fetchIPFS(`QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9`)
	.then(async (buf) => {

		console.log(`fetch ipfs cid`, buf.length, `QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9`)

		await publishToIPFSAll(buf, [
			...filterList('API'),
		])
			.then(result => {
				console.log(`add to ipfs again`, result[0].value[0].size, result[0].value[0].cid.toString());
			})
		;

		return outputFile('./temp/111.png', buf)
	})
	.catch(e => console.trace(e))
;

will see file size is not same 98734 vs 98748

fs.stat 98734
fetch ipfs cid 98734 QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9
add to ipfs again 98748 QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9

bluelovers · 2020-08-04T19:22:29Z

@aschmahmann i think not only dir, also happen on file

achingbrain · 2020-08-05T12:49:10Z

Here is what I see:

Bare filesystem

$ ls -la dir
total 84992
drwxr-xr-x  4 alex  staff       128  5 Aug 11:58 .
drwxr-xr-x  4 alex  staff       128  5 Aug 13:25 ..
-rw-r--r--  1 alex  staff  20010278  5 Aug 11:58 hof.zip
-rw-r--r--  1 alex  staff  23016923  5 Aug 11:58 php-5.6.32-nts-Win32-VC11-x64.zip

go-IPFS

Add the directory

$ ipfs add -r dir
added QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH dir/hof.zip
added QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 dir/php-5.6.32-nts-Win32-VC11-x64.zip
added QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn dir

List the root dag

$ ipfs ls /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH 20010278 hof.zip
QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 23016923 php-5.6.32-nts-Win32-VC11-x64.zip

Stat the root of the dag

$ ipfs files stat /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
Size: 0
CumulativeSize: 43037585
ChildBlocks: 2
Type: directory

js-IPFS

Add the directory

$ jsipfs add -r dir
added QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH dir/hof.zip
added QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 dir/php-5.6.32-nts-Win32-VC11-x64.zip
added QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn dir

List the root dag

$ jsipfs ls /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
-rw-r--r-- - QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH 20010278 hof.zip
-rw-r--r-- - QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 23016923 php-5.6.32-nts-Win32-VC11-x64.zip

Stat the root of the dag

$ jsipfs files stat /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
Size: 0
CumulativeSize: 43037585
ChildBlocks: 2
Type: directory
Mode: drwxr-xr-x
Mtime: -

Everything seems in alignment.

@aschmahmann what is QmPq9t1UbjT51Y2bo4dc84KKXtHqD5KnD4vDNMqeAertvi?

@bluelovers the reason the sizes are different is the first is the size of the file, the second is the size of the dag that represents the file:

$ ipfs files stat /ipfs/QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9
QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9
Size: 98734
CumulativeSize: 98748
ChildBlocks: 0
Type: file

See Size vs CumulativeSize.

I'm not familiar with the modules you are using in your example. If you can put an example together that only uses the ipfs module that shows the problem you're seeing that would be helpful.

aschmahmann · 2020-08-05T14:12:41Z

@aschmahmann what is QmPq9t1UbjT51Y2bo4dc84KKXtHqD5KnD4vDNMqeAertvi?

@achingbrain it's the hash of the MFS root directory after manually copying in the two files into it. See the below walkthrough.

Script of commands

ipfs files ls / # show that it's empty
ipfs files add -r . # add all files (and the directory object) to IPFS noting that hash Qmdvb...
ipfs files cp QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip /hof.zip # add the first file to the MFS root (could've used the direct hash instead, but this shows we're definitely adding the right file with the right name)
ipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip /php-5.6.32-nts-Win32-VC11-x64.zip # add the second file to the MFS root
ipfs files ls / # show the directory has the two files now
ipfs files stat / # shows what the MFS root looks like... in go-ipfs it matches the result of ipfs add -r . but in js-ipfs it does not.

Go-IPFS

(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files ls /
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs add -r .
added QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip
added QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip
added QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
 41.03 MiB / 41.03 MiB [====================================================================] 100.00%(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip /hof.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip /php-5.6.32-nts-Win32-VC11-x64.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files ls
hof.zip
php-5.6.32-nts-Win32-VC11-x64.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files stat /
QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
Size: 0
CumulativeSize: 43037585
ChildBlocks: 2
Type: directory
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs version
ipfs version 0.6.0

JS-IPFS

(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files ls /
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs add -r .
added QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip
added QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip
added QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip /hof.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip /php-5.6.32-nts-Win32-VC11-x64.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files ls
hof.zip
php-5.6.32-nts-Win32-VC11-x64.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files stat /
QmPq9t1UbjT51Y2bo4dc84KKXtHqD5KnD4vDNMqeAertvi
Size: 0
CumulativeSize: 8070
ChildBlocks: 2
Type: directory
Mode: drwxr-xr-x
Mtime: -
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs version
js-ipfs version: 0.48.1

bluelovers · 2020-08-05T14:41:39Z

import ipfsClient from 'ipfs-http-client';
import IPFS from 'ipfs';

const cid = `QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9`;

(async () =>
{

	let ipfs = await ipfsClient({
		port: 5001,
	});

	let ipfs2 = await IPFS.create();

	await doTest(ipfs);

	console.log(`--------------------`)

	await doTest(ipfs2);

})();

async function doTest(ipfs)
{
	let chunks: Buffer[] = [];
	for await (let chunk of ipfs.cat(cid))
	{
		chunks.push(chunk)
	}
	let buf = Buffer.concat(chunks)

	console.dir(buf.length)

	for await (let ret of ipfs.addAll(buf))
	{
		console.log(ret)
	}
}

98734
{
  path: 'QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9',
  cid: CID(QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9),
  size: 98748
}
--------------------
98734
{
  path: 'QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9',
  cid: CID(QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9),
  size: 98748,
  mode: 420,
  mtime: undefined
}

olizilla · 2020-08-05T20:35:35Z

Related ipld/explore.ipld.io#40 - ipld explorer always reports the size for dag-pb nodes as the value of node.size, it doesn't currently try to match the logic of ipfs files stat

SgtPooki · 2022-11-03T16:54:14Z

Per @olizilla in ipld/explore.ipld.io#40 (comment)

Looking at the latest code for js-ipfs that seems like it should be updated to take into account the unixfs data, and either return 0 for directories or the value of unixfs.fileSize()

Next actions:

Ensure sizes are consistently reported, regardless of node/encoding type.
Ensure FAQ/documentation is updated to indicate variances such as these, and why they are not able to be resolved.

SgtPooki · 2023-05-25T23:45:12Z

js-ipfs is being deprecated in favor of Helia. You can learn more about this deprecation and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see #4336).

FYI that share.ipfs.io will be migrating to Helia soon: ipfs-shipyard/ipfs-share-files#136

ipld-explorer-components and explore.ipld.io are already migrating off of js-ipfs: ipfs/ipld-explorer-components#360

bluelovers added the need/triage Needs initial labeling and prioritization label Aug 4, 2020

achingbrain added kind/support A question or request for support and removed need/triage Needs initial labeling and prioritization labels Aug 5, 2020

jacobheun added the need/maintainers-input Needs input from the current maintainer(s) label Aug 20, 2020

baby636 mentioned this issue Jul 8, 2022

[Snyk] Security upgrade lerna from 3.22.1 to 5.1.8 baby636/js-ipfs#307

Open

MarcelRaschke mentioned this issue Aug 20, 2022

[Snyk] Fix for 5 vulnerabilities 47-studio-org/js-ipfs#36

Open

SgtPooki added status/deferred Conscious decision to pause or backlog kind/enhancement A net-new feature or improvement to an existing feature exp/intermediate Prior experience is likely helpful and removed need/maintainers-input Needs input from the current maintainer(s) labels Nov 3, 2022

SgtPooki self-assigned this May 17, 2023

SgtPooki closed this as completed May 25, 2023

SgtPooki added the kind/wontfix label May 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

determine the files to be different sizes #3209

determine the files to be different sizes #3209

bluelovers commented Aug 4, 2020

aschmahmann commented Aug 4, 2020

aschmahmann commented Aug 4, 2020

bluelovers commented Aug 4, 2020 •

edited

Loading

bluelovers commented Aug 4, 2020

achingbrain commented Aug 5, 2020 •

edited

Loading

aschmahmann commented Aug 5, 2020

bluelovers commented Aug 5, 2020

olizilla commented Aug 5, 2020

SgtPooki commented Nov 3, 2022

SgtPooki commented May 25, 2023

determine the files to be different sizes #3209

determine the files to be different sizes #3209

Comments

bluelovers commented Aug 4, 2020

Severity:

Description:

Steps to reproduce the error:

aschmahmann commented Aug 4, 2020

aschmahmann commented Aug 4, 2020

bluelovers commented Aug 4, 2020 • edited Loading

bluelovers commented Aug 4, 2020

achingbrain commented Aug 5, 2020 • edited Loading

Bare filesystem

go-IPFS

js-IPFS

aschmahmann commented Aug 5, 2020

bluelovers commented Aug 5, 2020

olizilla commented Aug 5, 2020

SgtPooki commented Nov 3, 2022

SgtPooki commented May 25, 2023

bluelovers commented Aug 4, 2020 •

edited

Loading

achingbrain commented Aug 5, 2020 •

edited

Loading