SFTP over OpenSSH server doesn't work

[lensaj]

I choose AFM as a client for my smartphone to exchange files with my laptop.
A connection has been created to authenticate with privkey, and when is established, it correctly chroots the folder indicated in sshd_config
Problems come at this point, when I select copy from some file's 3-point menu, and paste it on the other side of the connection, either way.
See screenshot for message error,

The same server/user config is working fine with total command on an android tablet
Anyway, I am going to attach a part of the sshd log since I notice that there is no entry with scp command or other file operation
https://pastebin.com/RSz4EMUZ

• Device: Huawei P40 Pro
• OS: EMUI 12
• Rooted: No
• Version:

Thank you. This app is very nice, I hope to use it for this purpose.

[VishnuSanal]

@TranceLove ping

[TranceLove]

Developer's self-note

From https://pastebin.com/RSz4EMUZ it seems you're using the OpenSSH server provided by Windows. The version string SSH-2.0-OpenSSH_for_Windows_8.6 suggested the repo https://github.com/PowerShell/Win32-OpenSSH

IIRC there were similar reports about using Amaze FM with OpenSSH server on Windows, though that was with 3rd party implementations, and seems problem was with "chrooting" specified directory. This error log seems not the same case 🤔

[KhalilSantana]

Hi,

I'm facing the same issue using a Linux OpenSSH daemon/sftp server. My client device is a Samsung Note 9 (Amaze v3.10, F-Droid), server is NixOS (OpenSSH_9.6p1, OpenSSL 3.0.13 30 Jan 2024).

[KhalilSantana]

On Amaze 3.8.2 it worked exactly once, was very slow (10Mbps download on a 300Mbps link) and then doesn't connect again or display the same error

[TranceLove]

For @KhalilSantana 's report, my quick test of listing files and copy file, using current release/4.0 codebase (which I worked on #3656) against the NixOS 23.11 VM appliance with the same version of OpenSSH server (9.6p1 with OpenSSL 3.0.13), seems working on my side.

I'm using password authentication. May perhaps more details on reproducing your symptom will be more helpful.

[KhalilSantana]

@TranceLove,

I'm using a password-protected, ed25519 SSH private key, which is stored locally on the phone's internal storage

[KhalilSantana]

Also, here's a speed comparison between SFTP in Termux, which basically saturates the wireless connection downloading a file

Then there's Amaze, performing 20x slower

[TranceLove]

@TranceLove,

I'm using a password-protected, ed25519 SSH private key, which is stored locally on the phone's internal storage

My quick test with passphrase protected ed25519 keys against the same NixOS appliance and same OpenSSH server, even with conventional ssh command didn't work. I also tried using ssh-agent, I did got the passphrase prompt to decrypt the key, but upon connecting I ended up getting the password prompt, i.e. server isn't accepting the key.

Upstream sshj library does support passphrase protected ed25519 keys - I did wrote an unit test as well.

So I think it's more a protocol wise problem rather than Amaze's own.

Sorry but if there's a combination that will work, I'll be glad to look into it.

(Answer re: slow I/O problems in next comment)

[TranceLove]

Re @lensaj 's problem, I made a quick test with Windows's own OpenSSH installed via optional features in Windows 11. Copy/move files did worked for me with current release/4.0 codebase, even when there's a colon in the file path, like C:/Documents and Settings

So my next guess would be the code parsing the paths. No need to know what's in the file, but I would like to know the path of the file in question - possible that some special characters I'm unaware of 🥲

[KhalilSantana]

My quick test with passphrase protected ed25519 keys against the same NixOS appliance and same OpenSSH server, even with conventional ssh command didn't work. I also tried using ssh-agent, I did got the passphrase prompt to decrypt the key, but upon connecting I ended up getting the password prompt, i.e. server isn't accepting the key.

Maybe it is only looking at the authorized_keys file in /etc/ssh/authorized_keys.d/$USER?

Maybe try this in your configuration.nix (and then rebuild-switch before testing again)

  users.users.demo = {
    description = "Demo User";
    openssh.authorizedKeys.keys = [
     # paste your SSH public key here, between double quotes, like this:
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMDqtqzdBywVWHbIMv/rOGWHb0xBHisyqsa8ZbITxjY demo@nixos"
    ];
  };

[TranceLove]

My quick test with passphrase protected ed25519 keys against the same NixOS appliance and same OpenSSH server, even with conventional ssh command didn't work. I also tried using ssh-agent, I did got the passphrase prompt to decrypt the key, but upon connecting I ended up getting the password prompt, i.e. server isn't accepting the key.

Maybe it is only looking at the authorized_keys file in /etc/ssh/authorized_keys.d/$USER?

Maybe try this in your configuration.nix (and then rebuild-switch before testing again)

  users.users.demo = {
    description = "Demo User";
    openssh.authorizedKeys.keys = [
     # paste your SSH public key here, between double quotes, like this:
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMDqtqzdBywVWHbIMv/rOGWHb0xBHisyqsa8ZbITxjY demo@nixos"
    ];
  };

If according to https://nixos.wiki/wiki/SSH_public_key_authentication, NixOS does honour $HOME/.ssh/authorized_keys although doing this way would go against their idea of reproducible builds of the OS using centralized config.

I just also tried key-based authentication with ed25519 key without passphrase, and by pasting the public key to $HOME/.ssh/authorized_keys authentication did went through.

[TranceLove]

Re: slow file I/O over SSH, unfortunately I think this is the best we can do given the available resources and knowledge :/

With #3829 I made a few changes apart from fixing problems, also to prevent parallel file I/O over SSH which would made I/O performance even worse. Then the rest, with the current stream-based I/O between upstream sshj and Amaze, IMO I'm seeing little room for improvement, if large scale code optimization is not possible at the moment.

Possible to use native (C/Go/Rust-based?) I/O, but that's completely other story then...

[KhalilSantana]

Looking at the upstream SSHJ issue tracker, it seems others are also observing the same slowness symptoms: SSHJ issue 688

[TranceLove]

I made use of RemoteFile.ReadAheadRemoteFileInputStream too

AmazeFileManager/app/src/main/java/com/amaze/filemanager/filesystem/HybridFile.java

Lines 1098 to 1128 in dd693b7

	public InputStream getInputStream(Context context) {
	InputStream inputStream;
	switch (mode) {
	case SFTP:
	inputStream =
	SshClientUtils.execute(
	new SFtpClientTemplate<InputStream>(getPath(), false) {
	@Override
	public InputStream execute(@NonNull final SFTPClient client) throws IOException {
	final RemoteFile rf =
	SFTPClientExtKt.openWithReadAheadSupport(
	client, NetCopyClientUtils.extractRemotePathFrom(getPath()));
	return rf.new ReadAheadRemoteFileInputStream(READ_AHEAD_MAX_UNCONFIRMED_READS) {
	@Override
	public void close() throws IOException {
	try {
	LOG.debug("Closing input stream for {}", getPath());
	super.close();
	} catch (Throwable e) {
	e.printStackTrace();
	} finally {
	LOG.debug("Closing client for {}", getPath());
	rf.close();
	client.close();
	}
	}
	};
	}
	});
	break;

Perhaps we may adjust the size of BufferedInputStream - from the comment in sshj/688 it makes a buffer of 1024*1024 = 1048576 = 1MB while we have only 64KB 🤦‍♂️ (DEFAULT_TRANSFER_QUANTUM below is 65536)

AmazeFileManager/app/src/main/java/com/amaze/filemanager/filesystem/files/GenericCopyUtil.java

Lines 128 to 130 in dd693b7

	} else if (mSourceFile.isSmb() || mSourceFile.isSftp() || mSourceFile.isFtp()) {
	bufferedInputStream =
	new BufferedInputStream(mSourceFile.getInputStream(mContext), DEFAULT_TRANSFER_QUANTUM);

[AhmedFaisal]

Issue spotted on Samsung A23, Android 14, amaze v3.10
Server: openssh 9.2p1 on Debian bookworm stable.

Error report:

Issue explanation (write below this line)

Exception

• App Name: Amaze File Manager
• Package: com.amaze.filemanager
• Version: 3.10
• User Action: UI Error
• Request: Application crash
• OS: Linux samsung/a23nsxx/a23:14/UP1A.231005.007/A235FXXU4DWL1:user/release-keys 14 - 34
• Device: a23
• Model: SM-A235F
• Product: a23nsxx

Crash log

io.reactivex.exceptions.UndeliverableException: The exception could not be delivered to the consumer because it has already canceled/disposed the flow or the exception has nowhere to go to begin with. Further reading: https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0#error-handling | java.lang.OutOfMemoryError: Failed to allocate a 1073741840 byte allocation with 8790120 free bytes and 247MB until OOM, target footprint 17580241, growth limit 268435456
	at io.reactivex.plugins.RxJavaPlugins.onError(RxJavaPlugins.java:367)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:69)
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
	at java.lang.Thread.run(Thread.java:1012)
Caused by: java.lang.OutOfMemoryError: Failed to allocate a 1073741840 byte allocation with 8790120 free bytes and 247MB until OOM, target footprint 17580241, growth limit 268435456
	at net.schmizz.sshj.common.Buffer.ensureCapacity(Buffer.java:144)
	at net.schmizz.sshj.sftp.PacketReader.readPacket(PacketReader.java:78)
	at net.schmizz.sshj.sftp.SFTPEngine.init(SFTPEngine.java:86)
	at net.schmizz.sshj.SSHClient.newSFTPClient(SSHClient.java:736)
	at com.amaze.filemanager.filesystem.ssh.SFtpClientTemplate.executeWithSSHClient(SFtpClientTemplate.kt:42)
	at com.amaze.filemanager.filesystem.ssh.SshClientTemplate.execute(SshClientTemplate.kt:38)
	at com.amaze.filemanager.filesystem.ftp.NetCopyClientUtils.execute$lambda$1$lambda$0(NetCopyClientUtils.kt:101)
	at com.amaze.filemanager.filesystem.ftp.NetCopyClientUtils.$r8$lambda$OuWcoqR3_mBlkrJUumwfWTouExw(Unknown Source:0)
	at com.amaze.filemanager.filesystem.ftp.NetCopyClientUtils$$ExternalSyntheticLambda0.call(Unknown Source:4)
	at io.reactivex.internal.operators.maybe.MaybeFromCallable.subscribeActual(MaybeFromCallable.java:46)
	at io.reactivex.Maybe.subscribe(Maybe.java:4290)
	at io.reactivex.internal.operators.maybe.MaybeSubscribeOn$SubscribeTask.run(MaybeSubscribeOn.java:54)
	at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
	... 6 more

---

[TranceLove]

Issue spotted on Samsung A23, Android 14, amaze v3.10 Server: openssh 9.2p1 on Debian bookworm stable.

Error report:

Issue explanation (write below this line)

Exception

* **App Name:** Amaze File Manager

* **Package:** com.amaze.filemanager

* **Version:** 3.10

* **User Action:** UI Error

* **Request:** Application crash

* **OS:** Linux samsung/a23nsxx/a23:14/UP1A.231005.007/A235FXXU4DWL1:user/release-keys 14 - 34

* **Device:** a23

* **Model:** SM-A235F

* **Product:** a23nsxx

Crash log

io.reactivex.exceptions.UndeliverableException: The exception could not be delivered to the consumer because it has already canceled/disposed the flow or the exception has nowhere to go to begin with. Further reading: https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0#error-handling | java.lang.OutOfMemoryError: Failed to allocate a 1073741840 byte allocation with 8790120 free bytes and 247MB until OOM, target footprint 17580241, growth limit 268435456
	at io.reactivex.plugins.RxJavaPlugins.onError(RxJavaPlugins.java:367)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:69)
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
	at java.lang.Thread.run(Thread.java:1012)
Caused by: java.lang.OutOfMemoryError: Failed to allocate a 1073741840 byte allocation with 8790120 free bytes and 247MB until OOM, target footprint 17580241, growth limit 268435456
	at net.schmizz.sshj.common.Buffer.ensureCapacity(Buffer.java:144)
	at net.schmizz.sshj.sftp.PacketReader.readPacket(PacketReader.java:78)
	at net.schmizz.sshj.sftp.SFTPEngine.init(SFTPEngine.java:86)
	at net.schmizz.sshj.SSHClient.newSFTPClient(SSHClient.java:736)
	at com.amaze.filemanager.filesystem.ssh.SFtpClientTemplate.executeWithSSHClient(SFtpClientTemplate.kt:42)
	at com.amaze.filemanager.filesystem.ssh.SshClientTemplate.execute(SshClientTemplate.kt:38)
	at com.amaze.filemanager.filesystem.ftp.NetCopyClientUtils.execute$lambda$1$lambda$0(NetCopyClientUtils.kt:101)
	at com.amaze.filemanager.filesystem.ftp.NetCopyClientUtils.$r8$lambda$OuWcoqR3_mBlkrJUumwfWTouExw(Unknown Source:0)
	at com.amaze.filemanager.filesystem.ftp.NetCopyClientUtils$$ExternalSyntheticLambda0.call(Unknown Source:4)
	at io.reactivex.internal.operators.maybe.MaybeFromCallable.subscribeActual(MaybeFromCallable.java:46)
	at io.reactivex.Maybe.subscribe(Maybe.java:4290)
	at io.reactivex.internal.operators.maybe.MaybeSubscribeOn$SubscribeTask.run(MaybeSubscribeOn.java:54)
	at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
	... 6 more

Looks like you're copying a large file?

Just for knowing, to help me on reproducing the issue 🙇