APFS hiding techniques are not working

[shoaloak]

InodeTable.getAllInodes always returns empty list. Seems like it cannot parse APFS structures correctly.

[MC68]

What type of APFS image are you using? It is currently necessary to remove all non-APFS specific data from the used image. The first bytes read need to be the current container superblock's magic bytes NXSB (besides obviously the objects header).

It is also possible that this issue is related to #41, as larger amounts of inodes will currently stop the program(are you getting some sort of error message?) - a solution for this is in the works, but might take some time.

[MC68]

Running a test program that calls getAllInodes and then prints the gathered data from our test images returns the (partially) pictured list.

[woudtje]

Hi, I'm working together with @shoaloak and the image we're providing to fishy starts with the NXSB's obj_phys, directly followed by the NXSB magic. I can see in the source code that the NXSB parser uses an offset of 32 bytes, which should be where the magic is located in our image.

Additionally, it is an image of only 1GB with very few inodes, since it was newly formatted and only has about 10 files on there.

We're unsure what the cause could be of Fishy not being able to parse our image.

Fishy returns an error that there is not enough space (when there definitely should be) which is because of the earlier mentioned issue of not being able to parse any inodes.

[MC68]

Would it be possible for you to upload the image so I could take a look? As seen in the screenshot above, the function should be able to locate all inodes - I'll try and see over the weekend if I can find any other potential sources for your problem.

If it is not possible to upload the image, I would like to ask you to use a hexeditor such as iBored (can be found here: http://files.tempel.org/iBored/) and give me some info on how your image is structured, e.g. where is your checkpoint area located, how many volume superblocks are present and how do their b-trees look (essentially: can the inodes be found manually by going Container Superblock -> object map -> Volume Superblock -> object map -> volume root b-tree) while I check the code once more.

When using the info switch instead of trying to write, do you also get an error?

[shoaloak]

Here is one of our images.
It is a dd dump of an APFS partition on an USB, with just a few files and folders.
The info argument also fails.

fishy -d /mnt/hdd/apfs-dumps/APFS-Partition-WithDirectories.dump inode_padding --info
0 inodes found.

0 bytes of data usable.

EDIT:
The APFS was created on macOS 10.15.2, we are unsure how to determine the APFS version. Do you look at the APFS driver or is there a field which specifies this?

[MC68]

As I was speculating today, the error lies with the non-consistent way of inode locations in APFS. Usually, the root node of your volume superblock's b-tree contains only references(?) to inodes, which are recognized as inodes but do not have any of the entry values needed to hide data
- which can be seen in this screenshot.

To prevent any hiding techniques from writing to these inodes, we remove the root node from the list gathered by InodeTable.

The commented code is what stops InodeTable from finding your inodes.

Removing this code shows the following inodes in your system:

Which is another concern. I can find your data manually, but I can not find any inodes in your system besides these 2, either manually or via our parser. These 2 are the only inodes connected to the volume in your image.

It is entirely possible that there are further nodes containing inodes in the image that I have overlooked since it is possible for nodes to exist without connection to a b-tree, but finding those nodes is currently not possible for our tool. I will however attempt to refine the InodeTable class(which has some other issues, too) to make smarter decisions on what inodes to exclude. For now, if you want to use the tool with your current image without making any changes to it, you can just comment the lines shown above within the InodeTable class.

[MC68]

I just realized the reason I can not find any inodes for your added files - you must have removed the actual block 0 (a copy of the current container superblock) when creating this image. Your current block 0 is the second container superblock version, whereas the newest container superblock present has a version of 45 - a copy of this superblock should be at block 0.

Edit: The inodes related to this newer version of the superblock are stored in the regular way - it seems you may not have cut your actual block 0 but rather may have unmounted the image in a way that prevented the system from properly updating block 0.

[woudtje]

Thanks for your time and constructive input.

To summarize in my own words: you observed that our image has the wrong block 0 and Fishy does not recognize this as it's not designed for handling of improperly dismounted images. This situation came to be as I assumed simply ejecting the USB through the GUI would suffice as a proper dismount operation.

it is possible for nodes to exist without connection to a b-tree

This confuses me, but we will have to look further into it.

[MC68]

To summarize in my own words: you observed that our image has the wrong block 0 and Fishy does not recognize this as it's not designed for handling of improperly dismounted images. This situation came to be as I assumed simply ejecting the USB through the GUI would suffice as a proper dismount operation.

Correct. Something, possibly the simply the fact that your container was located on a USB drive, impeded your container from updating block 0. I am not too familiar with APFS running on USB drives, but I can imagine APFS not being able to run properly on non-Apple hardware.