Proposal: add client for NMFMorphing from live input

[elgiano]

Hi and thanks for this amazing library, I'm really into it and reading some of your papers too! Great work!
I wanted to do NMF morphing on some live input, after having pretrained bases for both source and target. Here I wrote an additional client that combines NMFMatch and NMFMorph, so that it can get activations live from an audio input and multiply them with interpolated bases from source and target.

I thought it could be useful to share it, so here it is. If you like it and you want to pull it I could write a SuperCollider help file for it as well.

[g-roma]

Hello @elgiano , thanks for the PR! It looks like a cool idea, so we will review shortly. I take it you have tested this in SuperCollider?

[elgiano]

Yes, I tested it in SuperCollider and it works beautifully.

By the way, one thing that I found also for NMFMorph is that to have good results (aka not so much crackle which I guess comes from phase estimation) I need a hopSize of at least fftSize/8.

And another thing, here is the sc class, which I based on FluidNMFMorph, with two comments:

FluidNMFMorphIn : FluidRTUGen {

	*ar { arg in, source = -1, target = -1, autoassign = 1, interp = 0, windowSize = 1024, hopSize = -1, fftSize = -1, maxFFTSize = 16384;
                // 1. here I use .asUGenInput so that I can pass Buffer objects directly, for instance when using Ndefs for quick testing
                // MultiOutUGen subclasses do it by default on all arguments in *multiNew, normal UGen doesn't
		source = source.asUGenInput ?? {-1};
		target = target.asUGenInput ?? {-1};

		^this.new1('audio', in.asAudioRateInput(this), source, target, autoassign, interp, windowSize, hopSize, fftSize, maxFFTSize);
	}

	init {arg ...theInputs;
		inputs = theInputs;
                // 2. I don't really know what specialIndex is for, but setting it to -1 wouldn't work here (all parameters would be shifted)
		// specialIndex = -1;
	}

	checkInputs {
		if(inputs.last.rate != 'scalar') {
			^(": maxFFTSize cannot be modulated.");
		};
		^this.checkValidInputs;
	}
}

[g-roma]

Nice, thanks. Yes, the phase synthesis works better with a lot of overlap, although so far we tend to have uniform defaults for all spectral processing.

[elgiano]

Fair enough! But I would consider mentioning it in the help files / examples :)

[tremblap]

Thanks indeed for this @elgiano - and yes, optimal FFT settings for each algo, with their trade-offs, for various signal types, is something on our radar for the next year where we try to get these concepts across in creative ways.

[elgiano]

@tremblap
OT:
For next year you mean 2022 or next academic year? Because I'm studying sound and music computing in Copenhagen and I need to do an internship next semester (sep 2021)... and to work on creative applications with flucoma sounds like a great project for me :)

[tremblap]

Oh this idea of internship has a lot of potential indeed! I'll need to speak to the team first, but feel free to contact me on my academic email ([email protected]) with what you have in mind.