Aim: "Process should be simple and not require a technical background."

[ronste]

Hi all,

we set the aim to develop the importer tool to require no technical background. The tool in the current state does not fullfill this aim since it is a command line tool only employable by admins.

While it is easy enough to put the code into a plugin to make it available in the OJS backend, this approach is limited since we also require to upload a significant number of files which will be limited (on size and/or numbers) on the server side. One can easly reach gigabytes of publication files to be uploaded for a single import process.

What options do we have to address this issue? I would think into directions as e.g.

• queued file uploads
• S3 file storage

What is your opinion on that and how far is the support for those mechanics in OJS 3.4 already?

Ronald

[jonasraoni]

I consider that a basic feature for the plugin is to allow uploading tar/zip files. There are known side issues: a HTTP server is normally configured to disallow both large file uploads and to fail long requests with a timeout.

I see two ways to solve this issue:

• Temporarily modify the server configuration: definitely not interesting as it requires access to the server.
• Upload the file in chunks: at this moment it's possible to read local files through JavaScript, therefore, I'm sure there must be libraries providing chunked uploads (if not, it's not tough to come up with one).

I've seen OJS being used to upload large files. Therefore, such upload component would be useful for any kind of application, especially if it can detect/adapt/retry with smaller chunks when errors/timeouts happen). I mean, it's something that could be moved to upstream... 🤔

---

AFAICR OJS has the Composer package league/flysystem-sftp-v3, which provides SFTP capabilities, also Laravel has a bunch of integrations, that we might explore. These might be useful for higher skilled users.

[jonasraoni]

Just tagging @jardakotesovec @asmecher, to discuss the feasibility of adding a frontend component to upload files by chunks to upstream. I didn't research much, but it looks useful and has its drawbacks (cleaning failed/aborted chunks?).

[asmecher]

I think plupload (which we use for many/most uploads) already handles chunked uploads. This isn't chunked in the sense of using multiple HTTP requests, just breaking the file transmission up into multiple pieces within the same HTTP request.

[jonasraoni]

Thanks @asmecher. I see plupload supports chunking, but it's disabled by default, so some extra work will be needed.

Considering that we're trying to provide good defaults for low-skilled users/limited hosting services, I think it's worth to get it to work (with some attention to avoid abusing).

[ronste]

Given that only admins or journal managers should be able to import issues/submissions I guess the ability to upload files greater than what is allowed by the server should be limited to those roles. The plugin can take care of this.

Could chunking be enabled for an individual dialog or is this an application wide setting?

[jonasraoni]

The chunking requires interaction with the backend, so it's not possible to just enable it. I've created a GitHub issue.