Cascaded Audio Regressor
Neural Networks as Mediators between Voice and Synthesis
Neural networks, MFCC analysis, FluCoMa, SuperCollider
Research project, 2025
Developed in the seminar Programming and Music 2 with Bjarni Gunnarsson, Institute of Sonology
Project files: code, trained models and example audio training sets
Much of my work is about mappings: how one stream of information comes to steer another, and what is gained or lost along the way. Neural networks offer a particular way into this problem. A small network can learn the relationship between an input and a sound from examples, so nobody has to code it by hand. In doing so it can compress a high-dimensional signal into a handful of meaningful dimensions, or expand a few dimensions back out into many.
The voice is a good test case. Its spectrum shifts continuously across many dimensions, while a performer thinks about it in only a few: louder or softer, clearer or breathier, relaxed or strained. A synthesiser runs the other way. The few perceptual qualities a composer cares about are spread across many interdependent parameters. Between these two sits a parametrised space: a low-dimensional map with perceptual labels that both sides can be translated into and out of.
This project explored that idea with a two-stage system. One network compresses a live analysis of the voice into a two-dimensional perceptual space. A second expands positions in that space into eight synthesis parameters. My interest was in testing how such a chain behaves: where it is precise, where it breaks down, and what each of those outcomes might be good for in performance and composition.
System
Both networks are trained interactively. For Network 1, a sound is played and its position in the perceptual space is marked. For Network 2, a position is chosen and a synthesis setting is assigned to it. The network then interpolates between these examples.
From speech to phonemes
The first approach analysed spoken text according to vocal effort (x) and breathiness (y). The training material was a paragraph from Camus’ The Plague, read in six vocal styles ranging from clear and relaxed to breathy and strained. The MFCC analysis, however, captured the nuanced variations of continuous prose inconsistently, and it was hard to establish reliable correlations between vocal character and perceptual coordinates.
I therefore switched to training data built from individual phonemes (s, sh, h, v, u), placed at distinct positions in the perceptual space. Sustained phonemes have clearer, more stable spectral signatures, which the MFCC analysis captures more consistently and the networks learn from more effectively. This indicates that the system is particularly suited to tracking variations in the timbral quality of the input material.
Synthesis engine
The synthesis engine is built around a DemandEnvGen driven by a Brownian sequence, wave-folded against a pair of sine oscillators and passed through mid/side filtering and a final effects stage. The eight parameters map as follows:
An additional control uses the incoming voice as a ring-modulation source for the synthesis.
The engine produces interesting material on its own, but it is not ideally suited to vocal control. Its output changes quickly and dramatically, which can feel disconnected from the voice, an input that moves between subtle, gradual change and rapid modulation of the vocal folds.
Technical overview
Two directions: low-loss and high-loss
The system suggests two diverging artistic applications. They differ in two respects: the complexity of the training data, and the precision of the training stage, i.e. how carefully perceptual coordinates are assigned within the parametrised space.
Low-loss: audio input as controller. A robust set of training data that the MFCC analysis interprets easily, such as the phoneme set, is mapped meticulously first into the perceptual space of Network 1 and then into that of Network 2. The outcome is predictable, and the voice works as a control mechanism for the synthesis. This approach suits a live setting.
High-loss: audio input as material generator. A broader, more diverse training set is combined with a looser mapping of perceptual coordinates in Network 2. The voice acts less as a controller and more as a seed for novel and varied sonic material. The result is less predictable and needs further curation before it can be used in composition, which makes this approach better suited to use as a material generator.
Further exploration
The next step may simply be further experimentation with training data and parameter mappings, to build more intuitive relationships between vocal gestures and sonic results.
SCD Code
(
s.waitForBoot{
var column1 = 420, column2 = 600, buttonWidth = 150, buttonHeight = 30, vSpacing = 40;
// Counters for training points
~counter = 0;
~counter2 = 0;
// Buffers for Network 1 (audio → perceptual)
~dsInput = FluidDataSet(s);
~dsOutput = FluidDataSet(s);
~mfccBuf = Buffer.alloc(s, 13);
~perceptualBuf = Buffer.alloc(s, 2);
// Buffers for Network 2 (perceptual → synthesis)
~dsInput2 = FluidDataSet(s);
~dsOutput2 = FluidDataSet(s);
~synthParamBuf = Buffer.alloc(s, 8);
// Neural Networks
~nn = FluidMLPRegressor(s, [10, 8], FluidMLPRegressor.tanh, FluidMLPRegressor.sigmoid, learnRate:0.05, batchSize:1, validation:0);
~nn2 = FluidMLPRegressor(s, [7], FluidMLPRegressor.sigmoid, FluidMLPRegressor.sigmoid, learnRate:0.1, batchSize:1, validation:0);
// Load voice file
~voiceFile = Buffer.read(s, "/Users/heu/code/sc/sounds/clear_relaxed.wav");
Window.closeAll;
~win = Window("Audio → Perceptual → Synthesis", Rect(0,0,800,700)).background = "black";
// 2D slider for vocal effort (x) and breathiness (y)
~slider2d = Slider2D(~win, Rect(10,20,400,400))
.action_({
arg s2d;
~perceptualBuf.setn(0, [s2d.x, s2d.y]);
//("Vocal Effort: % Breathiness: %").format(s2d.x.round(0.01), s2d.y.round(0.01)).postln;
});
// Labels for 2D slider
StaticText(~win, Rect(20, 30, 200, 20)).string_("ss").font_(Font.default.size_(12)).stringColor_(Color.white);
StaticText(~win, Rect(20, 395, 200, 20)).string_("sh").font_(Font.default.size_(12)).stringColor_(Color.white);
StaticText(~win, Rect(205, 150, 200, 20)).string_("hh").font_(Font.default.size_(12)).stringColor_(Color.white);
StaticText(~win, Rect(205, 290, 200, 20)).string_("ah").font_(Font.default.size_(12)).stringColor_(Color.white);
StaticText(~win, Rect(375, 30, 200, 20)).string_("vv").font_(Font.default.size_(12)).stringColor_(Color.white);
StaticText(~win, Rect(375, 395, 100, 20)).string_("uu").font_(Font.default.size_(12)).stringColor_(Color.white);
// Calibration buttons - consistent spacing
Button(~win, Rect(column1, 20, buttonWidth, buttonHeight))
.states_([["[0,1] Relaxed+Clear"]])
.action_({ ~slider2d.x_(0).y_(1); });
Button(~win, Rect(column1, 60, buttonWidth, buttonHeight))
.states_([["[0,0] Relaxed+Breathy"]])
.action_({ ~slider2d.x_(0).y_(0); });
Button(~win, Rect(column2, 20, buttonWidth, buttonHeight))
.states_([["[1,1] Strained+Clear"]])
.action_({ ~slider2d.x_(1).y_(1); });
Button(~win, Rect(column2, 60, buttonWidth, buttonHeight))
.states_([["[1,0] Strained+Breathy"]])
.action_({ ~slider2d.x_(1).y_(0); });
// Audio source selection
~audioSourceButton = Button(~win, Rect(column1, 120, buttonWidth, buttonHeight))
.states_([["Use Live Input", Color.red, Color.white], ["Use Voice File", Color.green, Color.white]])
.action_({
arg button;
Ndef(\audioAnalyzer).set(\useFile, button.value);
if(button.value == 0) {
"Switched to live input".postln;
} {
"Switched to voice file".postln;
};
})
.value_(1);
// Monitor On/Off toggle
Button(~win, Rect(column1, 160, buttonWidth, buttonHeight))
.states_([["Monitor Off", Color.gray, Color.white], ["Monitor On", Color.blue, Color.white]])
.action_({
arg button;
Ndef(\audioAnalyzer).set(\monitor, button.value);
if(button.value == 1) {
"Audio monitoring ON".postln;
} {
"Audio monitoring OFF".postln;
};
})
.value_(1);
// Load new voice file
Button(~win, Rect(column2, 120, buttonWidth, buttonHeight))
.states_([["Load New Voice File"]])
.action_({
Dialog.openPanel({ |path|
~voiceFile.free;
~voiceFile = Buffer.read(s, path);
("Loaded new voice file: " ++ PathName(path).fileName).postln;
});
});
//////////////////////////////////////////////////////////
// Network 2: MFCC Analysis → (mapping to) Perceptual Space
//////////////////////////////////////////////////////////
// Training controls - Network 1
Button(~win, Rect(column1, 200, buttonWidth, buttonHeight))
.states_([["Add Point"]])
.action_({
var id = "mfccPoint%".format(~counter);
~dsInput.addPoint(id, ~mfccBuf);
~dsOutput.addPoint(id, ~perceptualBuf);
~counter = ~counter + 1;
// Print MFCC values
~mfccBuf.getn(0, 13, { |mfccValues|
("MFCC values: %").format(mfccValues.round(0.001)).postln;
});
// Print Perceptual values
~perceptualBuf.getn(0, 2, { |values|
("Added point %: effort=%, breathiness=%").format(
~counter,
values[0].round(0.01),
values[1].round(0.01)
).postln;
});
});
Button(~win, Rect(column1, 240, buttonWidth, buttonHeight))
.states_([["Train Network 1"]])
.action_({
~nn.fit(~dsInput, ~dsOutput, {
arg loss;
"Network 1 loss: %".format(loss).postln;
});
});
Button(~win, Rect(column1, 280, buttonWidth, buttonHeight))
.states_([["Not Predicting", Color.yellow, Color.black], ["Is Predicting", Color.green, Color.black]])
.action_({
arg but;
Ndef(\audioAnalyzer).set(\predicting, but.value);
});
// Save/Load Data buttons
Button(~win, Rect(column2, 160, buttonWidth, buttonHeight))
.states_([["Save Data"]])
.action_{
FileDialog({
arg folder;
~dsInput.write(folder+/+"mfccData.json");
~dsOutput.write(folder+/+"perceptualData.json");
"Training data saved".postln;
},{},2,0,true);
};
Button(~win, Rect(column2, 200, buttonWidth, buttonHeight))
.states_([["Load Data"]])
.action_{
FileDialog({
arg folder;
~dsInput.read(folder+/+"mfccData.json");
~dsOutput.read(folder+/+"perceptualData.json");
~dsInput.size({ |size|
~counter = size;
("Loaded % training points").format(size).postln;
});
},fileMode:2,acceptMode:0,stripResult:true);
};
// Save/Load Neural Network buttons
Button(~win, Rect(column2, 240, buttonWidth, buttonHeight))
.states_([["Save Network 1"]])
.action_{
Dialog.savePanel({
arg path;
if(PathName(path).extension != "json"){
path = "%.json".format(path);
};
~nn.write(path);
("Network 1 saved to: " ++ path).postln;
});
};
Button(~win, Rect(column2, 280, buttonWidth, buttonHeight))
.states_([["Load Network 1"]])
.action_{
Dialog.openPanel({
arg path;
~nn.read(path);
("Network 1 loaded from: " ++ path).postln;
});
};
//////////////////////////////////////////////////////////
// Network 2: Perceptual Coordinates → Synthesis
//////////////////////////////////////////////////////////
StaticText(~win, Rect(column1, 410, 300, 20))
.string_("NETWORK 2: Perceptual → Synthesis")
.font_(Font.default.size_(14))
.stringColor_(Color.white);
// MultiSliderView for synthesis parameters
~synthMultiSlider = MultiSliderView(~win, Rect(10, 440, 400, 200))
.elasticMode_(1)
.isFilled_(1)
.indexThumbSize_(40)
.gap_(2)
.action_({
arg ms;
~synthParamBuf.setn(0, ms.value);
("Synth params: %").format(ms.value.collect(_.round(0.01))).postln;
})
.value_(0.5.dup(8)); // 8 synthesis parameters, initial values at 0.5
// Parameter labels
StaticText(~win, Rect(32, 650, 400, 15))
.string_("d1 d2 d3 d4 d5 d6 d7 d8")
.font_(Font.default.size_(10))
.stringColor_(Color.white);
// Network 2 training buttons
Button(~win, Rect(column1, 440, buttonWidth, buttonHeight))
.states_([["Randomize Synth"]])
.action_({
var randomValues = 8.collect({ 1.0.rand });
~synthMultiSlider.value_(randomValues);
~synthParamBuf.setn(0, randomValues);
"Randomized synthesis parameters".postln;
});
Button(~win, Rect(column1, 480, buttonWidth, buttonHeight))
.states_([["Add Synth Point"]])
.action_({
var id = "synthPoint%".format(~counter2);
~dsInput2.addPoint(id, ~perceptualBuf);
~dsOutput2.addPoint(id, ~synthParamBuf);
~counter2 = ~counter2 + 1;
~perceptualBuf.getn(0, 2, { |coords|
("Added synth point %: effort=%, breathiness=%").format(
~counter2, coords[0].round(0.01), coords[1].round(0.01)
).postln;
});
});
Button(~win, Rect(column1, 520, buttonWidth, buttonHeight))
.states_([["Train Network 2"]])
.action_({
~nn2.fit(~dsInput2, ~dsOutput2, {
arg loss;
"Network 2 loss: %".format(loss).postln;
});
});
// Save/Load Network 2 buttons
Button(~win, Rect(column2, 440, buttonWidth, buttonHeight))
.states_([["Save Network 2 Data"]])
.action_{
FileDialog({
arg folder;
~dsInput2.write(folder+/+"perceptualData2.json");
~dsOutput2.write(folder+/+"synthParamData.json");
"Network 2 data saved".postln;
},{},2,0,true);
};
Button(~win, Rect(column2, 480, buttonWidth, buttonHeight))
.states_([["Load Network 2 Data"]])
.action_{
FileDialog({
arg folder;
~dsInput2.read(folder+/+"perceptualData2.json");
~dsOutput2.read(folder+/+"synthParamData.json");
~dsInput2.size({ |size|
~counter2 = size;
("Loaded % Network 2 points").format(size).postln;
});
},fileMode:2,acceptMode:0,stripResult:true);
};
Button(~win, Rect(column2, 520, buttonWidth, buttonHeight))
.states_([["Save Network 2"]])
.action_{
Dialog.savePanel({
arg path;
if(PathName(path).extension != "json"){
path = "%.json".format(path);
};
~nn2.write(path);
("Network 2 saved to: " ++ path).postln;
});
};
Button(~win, Rect(column2, 560, buttonWidth, buttonHeight))
.states_([["Load Network 2"]])
.action_{
Dialog.openPanel({
arg path;
~nn2.read(path);
("Network 2 loaded from: " ++ path).postln;
});
};
Button(~win, Rect(column1, 320, buttonWidth, buttonHeight))
.states_([["Check MFCC Now"]])
.action_({
~mfccBuf.getn(0, 13, { |mfccValues|
("Current MFCC: %").format(mfccValues.round(0.001)).postln;
if(mfccValues.sum == 0) {
"MFCC buffer is empty!".postln;
"1. Load real audio file".postln;
"2. Start Audio Analyzer".postln;
"3. Turn Monitor On".postln;
};
});
});
Slider(~win, Rect(column1, 650, buttonWidth, buttonHeight))
.value_(0.0) // Start with dry signal (0.0)
.action_({ |sl|
Ndef(\synthEngine).set(\inWet, sl.value);
//("Wet/Dry Mix: %").format(sl.value.round(0.01)).postln;
});
// Add a label for the slider
StaticText(~win, Rect(column2, 650, 120, buttonHeight))
.string_("RM Signal In Dry/Wet")
.font_(Font.default.size_(12))
.stringColor_(Color.white);
//////////////////////////////////////////////////////////
// System Control Buttons
//////////////////////////////////////////////////////////
StaticText(~win, Rect(column1, 590, 200, 20))
.string_("SYSTEM CONTROL")
.font_(Font.default.size_(14))
.stringColor_(Color.white);
// Start/Stop Audio Analyzer
Button(~win, Rect(column1, 610, buttonWidth, buttonHeight))
.states_([["Start Audio Analyzer", Color.green, Color.white], ["Stop Audio Analyzer", Color.red, Color.white]])
.action_({
arg but;
if(but.value == 1) {
Ndef(\audioAnalyzer).play;
"Audio Analyzer started".postln;
} {
Ndef(\audioAnalyzer).stop;
"Audio Analyzer stopped".postln;
};
});
// Start/Stop Synthesis Engine
Button(~win, Rect(column2, 610, buttonWidth, buttonHeight))
.states_([["Start Synthesis", Color.green, Color.white], ["Stop Synthesis", Color.red, Color.white]])
.action_({
arg but;
if(but.value == 1) {
Ndef(\synthEngine).play;
"Synthesis Engine started".postln;
} {
Ndef(\synthEngine).stop;
"Synthesis Engine stopped".postln;
};
});
~win.front;
//////////////////////////////////////////////////////////
// Ndef Definitions
//////////////////////////////////////////////////////////
// Audio Analysis and Neural Network Processing
Ndef(\audioAnalyzer, {
arg predicting = 0, useFile = 1, monitor = 1;
var input, mfccs, trig, predictedCoords, output;
// Audio source selection
input = Select.ar(useFile, [
SoundIn.ar(0)*0.5, // Live input
PlayBuf.ar(1, ~voiceFile, loop: 1) // File input
]);
// MFCC analysis
mfccs = FluidMFCC.kr(input,
numCoeffs: 13, startCoeff: 1,
maxFFTSize: 1024, hopSize: 256
);
// Store MFCC values in buffer
FluidKrToBuf.kr(mfccs, ~mfccBuf);
// Prediction trigger
trig = Impulse.kr(10) * predicting; // 10 Hz prediction rate
// Network 1: MFCC → perceptual coordinates
~nn.kr(trig, ~mfccBuf, ~perceptualBuf);
// Network 2: Perceptual coordinates → synthesis parameters
~nn2.kr(trig, ~perceptualBuf, ~synthParamBuf);
// Read predicted perceptual coordinates
predictedCoords = FluidBufToKr.kr(~perceptualBuf);
// Send predictions back to update the 2D slider
SendReply.kr(trig, "/predictions", predictedCoords);
// Also send synthesis parameters to update MultiSlider
SendReply.kr(trig, "/synthParams", FluidBufToKr.kr(~synthParamBuf));
// Conditional monitoring output
output = Select.ar(monitor, [
Silent.ar(2), // Monitor off - no audio output
Splay.ar((input*0.5)!2) // Monitor on - stereo output
]);
output;
});
// Synthesis Engine
// An explanation of the parameter mapping:
// d1 → Affects the duration argument of the DemandEnvGen by changing the "hi" arg of the Dbrown
// d2 → Frequency of the LEFT channel of the SinOsc used with the binary operator .fold2 on the DemandEnvGen
// d3 → Frequency of the RIGHT channel of the SinOsc used with the binary operator .fold2 on the DemandEnvGen
// d4 → Frequency of the notch filter used in the mid-filtering. Also determines selection of BPF/HPF on the side-filtering
// d5 → Frequency of the LPF used in the mid-filtering
// d6 → Frequency of the LEFT BPF/HPF
// d7 → Frequency of the RIGHT BPF/HPF
// d8 → Selector value for the last stage of signal processing, i.e. dry / reverb / echo.
(Ndef(\synthEngine, {|inWet=0|
var params = FluidBufToKr.kr(~synthParamBuf);
var d1, d2, d3, d4, d5, d6, d7, d8;
var sig;
var sd;
var loMul = 1, hiMul=1000, min=0.01, max=0.8;
var foldFreqA, foldFreqB;
var bandStopFreq, hpFreqA, hpFreqB, lpFreq, whichFilt;
var reverbs, whichVerb;
//#d1, d2, d3, d4, d5, d6, d7, d8 = params;
d1 = Lag.kr(params[0], 0.4);
d2 = Lag.kr(params[1], 0.4);
d3 = Lag.kr(params[2], 0.4);
d4 = Lag.kr(params[3], 0.6);
d5 = Lag.kr(params[4], 0.6);
d6 = Lag.kr(params[5], 0.6);
d7 = Lag.kr(params[6], 0.6);
d8 = Lag.kr(params[7], 1.0);
hiMul = d1.linexp(0.0,1.0,1,20);
sd = SampleDur.ir;
# foldFreqA, foldFreqB = [d2, d3].linexp(0,1,9,40);
sig = DemandEnvGen.ar(
Dseq([min, min.neg, [max.neg, max]], inf),
Dbrown(sd * loMul, sd * hiMul, sd*2);
).fold2(SinOsc.ar([foldFreqA, foldFreqB])) * 0.5;
bandStopFreq=d4.linexp(-1.0,1.0,10,10000);
lpFreq = Lag.kr(d5.linexp(0,1,600,15000));
# hpFreqA, hpFreqB = Lag.kr([d6,d7].linexp(0,1,10,3000));
whichFilt = d4;
// Some Mid/Side Filtering
sig = Splay.ar([
// Left
Select.ar(whichFilt, [
BPF.ar(sig[0], hpFreqA,0.25).distort,
HPF.ar(sig[0], hpFreqA).distort
]),
// Mid
LPF.ar(BBandStop.ar(sig.sum, bandStopFreq,SinOsc.kr(0.1).range(0.5,5)).tanh, lpFreq),
// Right
Select.ar(whichFilt, [
BPF.ar(sig[1], hpFreqB,0.3).round(0.04),
HPF.ar(sig[1], hpFreqB).round(0.04)
])
]);
sig = LPF.ar(sig, 15000);
sig = sig*0.5;
// Some reverbs
reverbs = [
sig,
sig,
sig,
//Splay.ar(Resonz.ar(Greyhole.ar(sig,0.1,modDepth: 0.5), [60,63,67,74].midicps,0.2)),
Greyhole.ar(sig,0.1,modDepth: 0.5)*0.15,
CombC.ar(sig, 0.4,LFNoise1.kr(0.2).range(0.1,0.4),2)
];
whichVerb = d8.linlin(0,1,0, (reverbs.size-1)).postln;
sig = SelectX.ar(whichVerb, reverbs);
// An attempt to use the synth for vocal processing (doesn't work terribly well)
sig = XFade2.ar(sig, (SoundIn.ar(0) * sig)+sig, inWet.linlin(0, 1, -1, 1));
sig;
});
);
// Start the audio analyzer by default
Ndef(\audioAnalyzer).play(numChannels: 2);
// Update the 2D slider with predictions
OSCdef(\predictions, {
arg msg;
var effort = msg[3];
var breathiness = msg[4];
{
~slider2d.x_(effort).y_(breathiness);
("Predicted - Effort: % Breathiness: %").format(effort.round(0.01), breathiness.round(0.01)).postln;
}.defer;
}, "/predictions");
// Update the MultiSlider with synthesis parameters
OSCdef(\synthParams, {
arg msg;
var params = msg[3..10]; // Extract the 8 synthesis parameters
{
~synthMultiSlider.value_(params);
}.defer;
}, "/synthParams");
"Neural Network Audio System loaded!".postln;
"Audio Analyzer: Ndef(\\audioAnalyzer) - handles playback and analysis".postln;
"Synthesis Engine: Ndef(\\synthEngine)".postln;
"Use MultiSlider to control 8 synthesis parameters".postln;
}
)
About
My work spans performance, acousmatic and interactive sound works, and installations. Working through a conceptual lens, I am interested in the moments of in-betweenness where meaning must be continuously renegotiated. Central to this are questions of control and its absence; how agency and perception shift, dissolve, or are redistributed.
Recent projects range from acousmatic spatial compositions to explorations of light-sound transduction, digital feedback systems, and algorithmic interpretations of graphic scores.
I hold an M.Sc. in Architecture and Urban Planning from the Karlsruhe Institute of Technology and a Diplom (MFA) in Media Arts from HfG Karlsruhe, and studied Sonology at the Royal Conservatory of The Hague. My works and performances have been presented at venues such as Museum West (NL), Garage Rotterdam (NL), ZKM Kubus (DE), Rewire Festival (NL), Lapinlahden Lähde Studios (FI), Národní dům (CZ), and Nova (BE).
Get in touch
contact (at) lythgoe (dot) studio
Links
About
My work spans performance, acousmatic and interactive sound works, and installations. Working through a conceptual lens, I am interested in the moments of in-betweenness where meaning must be continuously renegotiated. Central to this are questions of control and its absence; how agency and perception shift, dissolve, or are redistributed.
Recent projects range from acousmatic spatial compositions to explorations of light-sound transduction, digital feedback systems, and algorithmic interpretations of graphic scores.
I hold an M.Sc. in Architecture and Urban Planning from the Karlsruhe Institute of Technology and a Diplom (MFA) in Media Arts from HfG Karlsruhe, and studied Sonology at the Royal Conservatory of The Hague. My works and performances have been presented at venues such as Museum West (NL), Garage Rotterdam (NL), ZKM Kubus (DE), Rewire Festival (NL), Lapinlahden Lähde Studios (FI), Národní dům (CZ), and Nova (BE).
Get in touch
contact (at) lythgoe (dot) studio
Links