[Discussion archive] File Browser

[daviddias]

A pleasing file browsing is essential. People have strong opinions about how they should work and a lot of experience using native desktop file browsers on their OS of choice.

With MFS, IPFS has a filesystem api that we can map to a traditional posix / unix like filesystem. But IPFS also has additional concepts like the CID, immutability, and file sharing, content pinning that we need to find a good way to articulate to the user.

Should all IPFS apps share a common file browser implementation or should they redirect the user to an improved WebUI, so the reuse is at the application level? Should we implement an OS integration, so you see IPFS files in your preferred file browser? What does a great file browsing experience with IPFS super powers looks like?

Project description: https://github.com/protocol/pm-ipfs-gui/blob/master/proj-descriptions/FILE_BROWSER.md

The current state of things is described here: https://github.com/ipfs-shipyard/pm-ipfs-gui/blob/master/research/README.md#browsing-files

[olizilla]

There should be a mechanism to remove files, but it should be undoable, via a trash metaphor.

Note that deleting an item from this view does not necessarily remove it from the users system. That only happens when ipfs.repo.gc gets run. It may be worthwhile to have a UI element (like a trash bin on windows) that contains deleted files, and you can 'clean it out' which runs a gc.

See: https://github.com/ipfs-shipyard/pm-ipfs-gui/blob/master/proj-descriptions/FILE_BROWSER.md#deleting-an-item

[olizilla]

Why do I need a file browser? What am I trying to do when I use it?

WebUI is catering for people who want to administer a remote gateway (security issues aside) but it's also a general interest tool for anyone running an IPFS node on a desktop too (at the moment). For and admin, the file-browser is a way to keep an eye on what content is being hosted on a shared gateway... just a rough heuristic sort of oh look at all those CIDs, with the ability to click on one and have a look.

A desktop user is more likely to be looking for a file they previously shared, and want to share again. As the IPFS repo structure is separated from their OS file-system, and not transparently viewable in their regular file browser, and files added to the IPFS repo are duplicated from the file system, it seems unlikely that people will be integrating IPFS in to their day-to-day file organisation now. IPFS could be the best "~/Public" folder ever, but there is friction around using it in that way. I feel like we have to consider rethinking IPFS Desktop's file browsing feature as a native OS file-browser integration to really remove friction for a casual user.

[lidel]

I may be opinionated here, but I'd say people who administer a gateway will ALWAYS prefer to use CLI tools instead. File Browser listing thousands of raw CIDs is simply a bad UX for such low-level things.

AFAIK the "browser" for low-level things will be "IPLD Explorer", which is tracked in #11.
The "File Browser" should focus on ipfs.files and nothing more.

File Browser provides a great frame for delivering some of key features of IPFS for Desktop users:

• Reusable UI for managing ipfs.files API (tracked in this issue)
  • UI for adding files to IPFS (Make file-adding experience consistent across our tools #35)
  • UI for sharing files from IPFS (Design: Share files via IPFS #34)

My current design sentiments:

• File Browser will be provided with IPFS Desktop
  • focus on making a reusable web-based File Browser that works on all platforms
    • native OS integrations can be added, but as additional features (I am afraid it will be too expensive to make it a seamless experience on all platforms, and we will leave some users with no file browser at all)
  • File Browser should adapt to available space
    eg. be compact by default with an option to be resized to fit more files/directories
    • what if there was a button that expands, or even opens File Browser in a new, bigger window (detached from menubar popup)?
• it should be equally easy to add files via IPFS Path as from user's disk
  • Make file-adding experience consistent across our tools #35 (comment) & [META] Solidify mental model/language for MFS add vs IPFS pin #10 (comment)
• "sharing" a file/directory should be super intuitive (Design: Share files via IPFS #34)
• "deleting" simply moves files to "Trash" folder
• "pinning" should be hidden from UI and implicit ([META] Solidify mental model/language for MFS add vs IPFS pin #10 (comment))
  • all files are effectively pinned until removed from "Trash" directory, which removes them from ipfs.files and runs gc

[olizilla]

I fully agree that WebUI currently doesn't do a great job of visualising files for a gateway maintainer. I agree that most admins would like to use a cli. From comments on security issues where people have wanted to expose their WebUI on a remote system, I think we have to consider that some people are currently trying to use WebUI to see what's on their node, or at least as a reassuring visual counter part to the cli tool. That it doesn't do a great job of it, but people still want to use it is an interesting data point.

I mildly disagree that our first step should be to make a re-usable file browser that works well in all shapes and sizes. I'd like to check if anyone else supports the OS integration route, as I think anything short of that will be significantly limited in it's day-to-day usefulness. We can make a great custom file browser, but I want to triple check that such a thing is the best way to get people using IPFS to host there public files.

I totally take your point about the dangers of embarking on an OS file-system integration effort, but I'd argue that, if it's possible, then it may end up being a similar effort to trying to make a really good alternative file-browser in html, but be of more value.

I could be convinced either way. I think we have to make a really good file html file browser anyway if that is to remain a feature of WebUI. I'd really like to take some time to test some proof-o-concepts for the native integration to see if that could be a thing. @alanshaw has made good progress on OSX flavour https://www.youtube.com/watch?v=jXkTEBdM6aA ...that is the only desktop UX I can see really catching on.

[lidel]

I agree that OS integration is a valuable proposition for managing local files, and I am sure will have it eventually, but in my mind it remains a nice-to-have feature in addition to basic-but-robust web-based File Browser.

My key concern with relying only on OS integration is the difficulty to provide feature-parity across platforms:

• windows does not support FUSE -- will need something else
• FUSE alone may be not enough
  • I imagine we need to add context-actions to files and directories ("Add by IPFS Path", "Share", "Copy IPFS Path")
    • It is virtually impossible to add context actions to the myriad of file managers on Linux, need to target most popular ones first, leaving niche users with bad UX etc
  • On the other hand, if someone is using niche stuff, does not mind CLI tools, or FUSE alone IS enough

[hacdias]

@lidel about FUSE on Windows, there's a nice package called Dokany which allows us to use FUSE bindings on Windows.

[djdv]

Chiming in here, I intend to make go-ipfs's ipfs mount work on Windows (when other issues are resolved first), more than likely with Dokany but possibly with something else, this combined with a mountable MFS seems like it would allow people to extend the environment however they wish by working on top of this foundation.

As for native integration, I wrote something a while ago for myself and put it up here
https://ipfs.io/ipfs/zDMZof1m1fX98cTLyC2VLe9iDQQhWgDLu5foshBSsxSWHQNuiyYV
It's a glorified wrapper for ipfs.exe and a bridge between the Windows shell. I have an unfinished version that takes advantage of the http api instead of calling the binary itself but that's currently not being worked on.

The point I wish to make is that I made that out of necessity, I wanted to be able to link someone a file, using various non-default ipfs arguments, quickly, this worked well enough for me that people asked me how I was linking things out so quickly. I think integration like this is important if not critical for some people. Especially on platforms such as Windows where the stock command line experience is...

[alanshaw]

My thoughts are that we're going to have to build/adapt a decent web based IPFS file browser and a native integration will be the end game.

If you need just one reason then this - I need to see my files in my js-ipfs node running in IPFS Companion. This is simply impossible with native integration.

I would love to see a good web based, OSS, file browser with a pluggable implementation. In the same way that to build a FUSE filesystem you just need to implement a bunch of functions for reading, writing and stat-ing - maybe even that same interface!? Before I got onto building a FUSE I was thinking I wanted to plug IPFS MFS into a web based file browser as a demo for window.ipfs but I couldn't find anything what wasn't payware or old or terrible. [N.B. I'm not very good at finding things so please shout if you know of something already out there.]

FUSE is a good POC but I think it's a non-starter to require people to install FUSE to be able to use IPFS Desktop, unless we can find a way of integrating a FUSE install with our IPFS Desktop install.

That said, we could go a long way into exploring what native integration might look like with that approach, and if we really like it, then we can get serious and remove the FUSE element.

I don't know if your average casual user is comfortable with IPFS being an app they can drag and drop files into but they can't see them on their hard drive anywhere using the native OS file explorer. In the past I've not been comfortable with it and it's only as a developer that I've come to understand that there are different storage formats and you can't just browse everything with the Finder. I bet there's younger generation(s) that are fine with it though and I suspect that it's a mixture.

One parallel is the macOS Photos app. Photos are stored in a "Photos Library" which you can navigate to using the Finder but double clicking on it opens the app. I cannot get to the files and I've definitely seen frustration with people trying to copy files onto a USB drive or select pictures to upload to Facebook.

That being the case, it makes sense to have a native integration to avoid alienating those users who don't understand "where the files are going", but familiarity is the really big one - Using something you already have (the native OS file explorer) but with extra bells and whistles is probably way easier than learning a completely new app. I'm speculating that most people nowadays are comfortable using their native OS file explorer and I reckon that's probably what Dropbox also gambled on (ahem, researched well ;)). Dropbox is pretty popular and in my experience seems to be understood well by most people technical or not.

It's worth pointing out at this juncture that Dropbox has a web based file explorer as well as a native integration.

There's also the reason that IPFS has already done a lot of low level work to expose itself as a regular Unix filesystem. It seems rude to not take advantage of that when it's so easy to do so.

IMHO a native integration and, as @olizilla put it, having the greatest public folder ever is a super valuable goal for anyone wanting to share files easily or build a website that's instantly available to browse.

[lidel]

Ok, I feel we have a consensus that web version and native OS integration solve different problems and BOTH are required for IPFS being successful on "desktop".

Both will happen eventually, but we should decide what has bigger priority in the short term (eg. what to focus on in Q2).

I propose:

• short-term – "focus on low hanging fruits"
  • p0: IPFS Desktop provides a web-based, basic File Browser for ipfs.files with:
    • redesigned UI for adding (Make file-adding experience consistent across our tools #35) and sharing (Design: Share files via IPFS #34)
    • cleaned up "pinning" abstraction ([META] Solidify mental model/language for MFS add vs IPFS pin #10)
  • p1: work towards File Browser being responsive and reusable
    • test use case: we want to use it in a web browser tab for managing ipfs.files of IPFS Companion's embedded js-ipfs
• long-term – work towards native OS integration
  • as it is not "existing feature" we should move it to a new, separate issue
    • DONE: File Browser: Native OS Integration File Browser: Native OS Integration #43

Does this sound good?

ps. @alanshaw ping @hacdias, he did some relevant work at https://filebrowser.github.io (not sure how pluggable it is tho)

[hacdias]

About the File Browser I did, I think that with some work it could easily be used for IPFS. There would be some stuff to do though. Right now, File Browser relies on a WebServer so it is not really a Single Page Application (we don't use hash history) but I think that could be easily reworked.

I could start working on that to try it out and see if I can plug it in 😮

[olizilla]

@hacdias we'll definitely want you try that out soon, but not today. It'd be great to get ipfs/ipfs-desktop#619 ipfs/ipfs-desktop#617 fixed (manually setting the config is fine for now) and get a release of Desktop out with the usability improvements in. Please find some people to test ipfs/ipfs-desktop#613 as well.

I'd like to work on some wireframes with @akrych before we jump to pluging in your file browser. We will absolutely want to use as much of it as makes sense, but give us some time to sketch out how the apps should fit together first.

[hacdias]

Sure, I'll work on fixing those issues right now. Do you have any ETA for those sketches?

[olizilla]

Unradical but worth mentioning. I like OSX finder. Specifically, clean, but not totally minimal. Context menus hide a lot of the work, which may not be appropriate for a web ui version. I use the keyboard navigation and preview function all the time. I dislike multi-pane views, i prefer to focus on one dir at a time but I fear that everyone prefers the file explorer they already know, and that we'll need to offer multiple views.

left-hand side is shortcuts to often used directories, right hand is file list.

but i don't like

[lidel]

(Moved from: #44 (comment))

What I like

Chrome OS file.app

Clean and calm, not much going on (especially in older versions)

• well known abstractions: important directories and foldable tree on left, path-based navigation on the top left, actions on the right top
• personally, I am not a big fan of foldable directory tree on the left , always keep it folded and prefer to use this space for pinning "bookmarks" to special directories

Version with thumbnails may be bit too much tho:

ownCloud

What i like here is that UI is "faded to background" and what "pops" are your files (if you ignore "enterprisey" banner at the top 😅):

What I did not like

Bitcasa

What I did not like:

• sensory overload (tries to be everything at once)
• things feel squeezed together, and at the same time there is a lot of whitespace
• bold design statement that does not play well with window borders of native OS

Mega.nz

What I did not like:

• not intuitive: vague navigation on the left + item on top + hamburger menu in the right top corner
• unnecessary clutter (eg. wrapping of things like folder icons)

[alanshaw]

The file browser needs to show all the contents of the node*. Users need to be able to visualise what their repo is storing. It cannot be just MFS, because there will be many users who have already pinned things in their repo who need to access them, either to view them, share them, or unpin them.

* Is it possible to get a listing of everything in the repo that is not pinned?

It's important to know what else is in my repo because it might be utilising significant disk space and bandwidth. GC is not run automatically so users need to be able to determine if if they should run the collector.

Similar to the "All Mail" filter in Gmail we should be able to browse the flat structure of the IPFS node and drill down into the hashes (if they are directories).

As I understand it, the MFS root is just another object in the repo that is the current root of the IPFS MFS system. It should be labelled appropriately and perhaps brought to the top of the list.

Each item should have a visual indicator as to it's file type. A directory icon for directories and a content specific file icon for files.

Note: it's possible in IPFS for an object to not be fully present locally* (see return value for ipfs.files.stat). So, to keep the UI snappy and to avoid unnecessary network activity we should only use content sniffing if the content is available locally. Otherwise fallback to file extension, if in MFS, and if all else fails then just a generic file icon.

* Is it possible to determine if a DAG node is fully present locally outside of MFS?

Because of the flat structure, we should be able to filter our view on the dataset from all items to just the MFS items or just the pinned items. The root listing (all files) might look something like this:

[All] [MFS] [Pinned]

[i] /ipfs/Qm1
[d] /ipfs/Qm2 {in MFS}
[f] /ipfs/Qm3 {Pinned}
[d] /ipfs/Qm4 {MFS ROOT}
    - [d] /foo
    - [d] /bar
[d] /ipfs/Qm5

• Filters are in [square braces] at the top
• Single letters in [square braces] are file/folder icons i=image, d=directory, f=file
• Labels are in {curly braces}. At the top level we'll see the MFS root node but also other items in MFS, as well as items that we've pinned. We should distinguish between them - labels is one way we could do that
• We could privilege the MFS root as the first item

A tree view might make things super complicated so I suggest we stick to a click through with a breadcrumb (similar to how IPFS Desktop currently works).

The breadcrumb should give an accurate representation of the path we're navigating. Currently in IPFS Desktop an MFS path appears as "ipfs / foo" but this is not a valid IPFS path in either view, mutable or not. We should be consistent when displaying MFS and regular IPFS paths e.g.

/ ipfs / Qm4 {MFS ROOT} / foo / bar
/ ipfs / Qm5 / lala / cats.jpg

When browsing the MFS path we should label or highlight it appropriately.

In a future iteration we might want to add a column with the number of peers seeding a file or downloading a file that might click through to a list of per file peer activity.

[alanshaw]

One interesting UX problem of this combined file browser is adding files.

When deep in a non-MFS path, can I add/remove/edit a file? Are those features are disabled?

[lidel]

Listing raw repo

I hope I am not right, but listing everything that is in repo does not sound feasible.
Remember that there is no "root /ipfs/ directory" in IPFS. Just a lot of "root CIDs", but these are not annotated in any way in the repo. CID of a directory looks the same as CID of a file or a chunk of a file. To simplify, repo is just a store for hashes. Everything is looked up via hash. And as a data point, ipfs refs local -u | wc -l returns over 25k unique objects physically present in my local repo. Listing/sniffing/detecting it all will not translate to a fast and responsive UI.

Think about raw repo as something like "browser cache". It exists as a low-level technical detail not designed to be browsed or micro-managed by a human.

ipfs.files API was created to provide "a mutable, virtual root for humans" and solve these exact limitations.

Exposing low-level pins

Low-level pinning is being redesigned and may even go away. My initial take was that it should not be exposed via GUI for now. MFS provides the same feature, but in more intuitive form.
See #10 (comment) and ipfs/kubo#4763

MFS Root

I don't think we are able to get CID for MFS root: ipfs files ls -l does not include it.
CID of the root of mutable directory tree would always change anyway.
I feel it is okay for breadcrumbs to just start with "home icon" alone, or explicit label like "Local Files" or "ipfs.files".

Communicating disk usage

You pointed out a very important problem of user asking "my ipfs.files is nearly empty – what is eating the rest of my disk space?".

I think we could address that by displaying a pie chart representing all used storage with chunks for "ipfs.files" (MFS) and "node cache" (everything else). It could be located on a dedicated "Stats" screen, or on a tab within "File Browser". And there should be "Run GC" button near it.

[olizilla]

We should be showing the CID for each file in the file browser view. It's a complicated and not particularly readable string, so there is an argument for truncating it by default, but a key aspect of all our guis is to help the user understand what IPFS does. Creating a strong association between file name (link name) aliases and there CID is a must have.

[olizilla]

Users are asking for this too, see: ipfs/dir-index-html#15

[olizilla]

Here is an example of a file in 2 states.

The top line is what the user sees when a file is being added. The add file progress bar is replaced a circular progress wheel, so it takes up less horizontal space and can be put in the same place as the status icons while the file is being added. The peers column isn't relevant while the file add is in progress, so it seems like a good swap.

The second line shows the added file, with it's CID below the filename. The last 4 chars of the CID are subtly highlighted to help the user visually spot a CID that recurs, and give them more memorable slice of the CID for tracking things down.

We don't know the CID while adding a file, so that space is used to inform the user about what is happening while we wait for the CID to be available. The peers icon is initially set to red / danger / low peer count. Asking the dht for providers will take a while, so we play it safe and suggest that this content is not well replicated, until we know better.

@hacdias @lidel @alanshaw you like?

[hacdias]

Uh, I like the idea. I just don't think the last 4 characters should be highlighted with that color. They contrast a lot with the others.

[fsdiogo]

Looks promising!

👍 for the circular progress wheel.

I agree with @hacdias, the highlighted CID looks a bit off. A way to circumvent that would be to highlight it only when hovering that row.

I think that the peer icon in red is a bit too strong too, but maybe that's the idea 🤔 it could have its circles in white to show low peer count, and they would fill out in grey when the peer count goes up. Just a thought to keep it streamlined.

[olizilla]

@fsdiogo we wanted to warn people that "adding to your ipfs repo" is not the same as "uploading to the cloud". I think 3 empty circles that take on a fill as the peer count goes up could work, but then we'd need to communicate the issue of "no other peers have this yet, so it's not 'safe'" some other way.

[fsdiogo]

Hmm, got it. Still not sure if the peer icon in red is clear enough on its own.

[lidel]

A clean and minimalist Web UI + Files screen by @patrykadas from https://www.figma.com/file/YTMV7cOMmDSfqw8BgwoncbJ5/IPFS-Desktop?node-id=0%3A1:

[lidel]

Cool project from ETHDenver'19  includes a tile view of image-heavy file collections stored in IPFS' MFS:

Refs:

• https://discuss.ipfs.io/t/ipfs-chrome-extension-inspiration/4896?u=lidel
  • https://medium.com/twos-complement/ipfs-stash-local-ipfs-for-everyone-ethdenver2019-b5b45ae0d5c9
  • https://github.com/nicovalencia/ipfs-stash

[ericronne]

Incremental work on this is proceeding, eg in this shipyard pr by @hacdias. Is the intent to keep this issue open as a conversational thread?