(((Signal Feature) to (Audio Generator Parameter Modulation) Map) Chain) Synthesis

(((Signal Feature) to (Audio Generator Parameter Modulation) Map) Chain) Synthesis

=> Input signal

-Features extracted (amplitude, frequency, spectral properties, impulses in a frequency band, amplitude transients, phase, etc.)

-Feature values used to modulate parameters of N generators (oscillators, synths, recursive SFAGPMMC Synthesis modules, anything to generate a signal which has modulation parameters)

-N output signals from said generators, summed into the output signal

<= Output signal

These systems can be combined together into arbitrarily long chains.  Arbitrarily complex features can be extracted.  Will post more information at some future point describing how FM synthesis, Virtual Analog synthesis, and some other synthesis methods can be generalized to this system.  And since it is so generalized, it will be easy to implement more novel synthesis techniques in the future through changing simply:

-Features extracted from input signal

-Mapping between feature values and generator modulation parameters

-Chain organization of these mapping layers

My main worry is this will result in noisy or just unpleasant synthesis techniques, but there may very well be some interesting novel ones which make pleasant sounds.  Since it's so generalized, the entire parameter space of sound synthesis techniques could theoretically be enumerated and fed into a machine learning system to categorize them by interestingness or pleasantness

The coolest part is that theoretically this can automatically create covers of songs using different instruments, if implemented in a sufficiently smart way.  You could hum or beatbox into a microphone and turn it into a track through sufficiently complex mappings

Will post more details in the future.

Very interested, keep us updated

Systems such as these could be automatically generated using various machine learning and AI techniques.  For a given layer, the feature set extracted from the input signal could be calculated ahead of time using a large and representative input signal buffer, and with a sufficiently large "vocabulary" of potentially extractable features, cluster analysis and automatic grouping of extracted features could be used to weight the value of each feature in the vocabulary and choose which ones become members of that layer.  These features could then be manually mapped to signal generator parameter modulations by the user

careful with those triple parenthesis

Caution:  Too many parentheses might give you a lisp

Reminds me of this:

this is a few tier brackets above my level of autism, any sound examples? 

  On 11/13/2016 at 9:09 PM, Zeffolia said:

My main worry is this will result in noisy or just unpleasant synthesis techniques, but there may very well be some interesting novel ones which make pleasant sounds.  Since it's so generalized, the entire parameter space of sound synthesis techniques could theoretically be enumerated and fed into a machine learning system to categorize them by interestingness or pleasantness

If I understand all of this correctmy, my guess is that it'll end up being quite noisy as well - though I'm excited about hearing what it can generate. I'd be interested in seeing what would happen if you switch everything around. Like, set the amplitude to the pitch, room size in a reverb plugin, LFO rate that controls something or another.

Random thoughts:

-My speculation earlier that this can be prettily generalized to FM and virtual analog synthesis is wrong I think.  It would be ugly and complicated.  In general this system would not have the uses I thought it would, but would instead be better for other things, I'm just not sure what those things are yet. 

The system is by itself no more powerful than any other, it can't do anything extra over what plain old Max/MSP can do.  It's just a good mental model for simplifying problems and experimenting and discovering novel things.  It will make complex input->output configurations simple, literally drawing a couple lines of Max/MSP instead of rethinking how to do the same thing over and over

I think of it as the API for a library or programming methodology for Max/MSP, csound, etc.  

The most powerful aspect will be pattern matching signal feature extraction.  

For instance, let's say a signal has amplitude transient starts extracted, and the extractor object has 3 outlets: trigger impulse {0, 1}, attack (0, 1], fundamental frequency (0, SampleRate/2], timbre intensity (0, 1]

Then say the trigger impulse of this extractor is inputted to an impulse pattern matcher matching the ("drum") pattern |_|| as in a beat, followed by another, followed 2 fast beats.  You could hook this feature extractor to output an impulse when that pattern is detected at a certain quantization time level (ms range with jitter allowance).  This trigger could then be sent to a generator creating a quick reverb room size spike on some other signal generator, maybe the lead melody or whatever you want

Anyway this is all pretty straightforward.  But the framework, the word beforehand, the creation of an API to do things like this easily, will result in an extreme ease-of-use for creating complex generative sequencing engines in Max/MSP

Furthermore, keeping the signal output on each layer and mixing them could be really useful for making complex stateful distortion patches.  

All of this being said my main worry is still just noise.  I suspect the more complex this all gets the noisier and less resembling music it will get.  Maybe that's not a bad thing though

One design problem I'm having trouble moving past is: signal property envelope following to detect events vs. using the raw input control signals that generated those events to begin with

A bit of a chicken and the egg problem.  I guess "why not both?" but that makes my inner autist cry inside

The main problem however is: all these stupid ideas and I literally have implemented or tested 0% of this.  Will get around to it once it's done being designed I suppose.