file browser can't list directory with non-engish characters content

[michalfratczak]

Version: Gaffer1.4.11.0-windows
Third-party tools: none
Third-party modules: none

Description

File browser can't list folder's content if there is file/dir with non english characters. It stops at last item before the rogue one.

Steps to reproduce

Just create a dir with any of polish letters: á ć ę ł ś ó

Basically non-english characters should be forbidden, but when it happens and you can't rename the folder - there's a problem...

[johnhaddon]

This seems to be OK on Linux, so I assume it's Windows-specific. Would you be able to take a quick look @ericmehl?

[ericmehl]

Windows uses wstring to store its paths internally, but at

gaffer/src/Gaffer/FileSystemPath.cpp

Line 512 in 9599500

children.push_back( new FileSystemPath( it->path().generic_string(), const_cast<PathFilter *>( getFilter() ), m_includeSequences ) );

we're asking std::filesystem::path to give us a std::string. It raises an error with characters like @michalfratczak pointed out.

I think it may be fixed by usingstd::filesystem::path::generic_wstring() and then converting that to a UTF-8 std::string. That would fix the listing widget in the file browser, but I don't know about actually opening the file with the converted name. I'm pretty sure we use std::filesystem throughout Gaffer now, so I'm guessing it will work but I'll have to test that.

It would be nice to be able to use https://en.cppreference.com/w/cpp/locale/wstring_convert for the conversion. But those methods are deprecated as of c++17.

Instead we might need to resort to WideCharToMultiByte(). I'm a bit fuzzy on the whole code page thing, but it might be that method is the most correct on Windows.

I haven't looked into the Cortex side of it yet (we call IECore::ls() a little further down in FileSystemPath), so there may be something there that needs to be adapted as well?

I'll look into that closer and add some tests and a fix.

[ericmehl]

Just noting here based on the discussion in #6012, that we may want to consider adding a test, or at least verify the behavior works, for sorting by extension with troublesome Windows characters like those pointed out above.