Hand Controlled Synth - Kinect + TouchDesigner + Ableton Live 11

Pointcloud me controlling some parameters of a REPRO-5 in Ableton Live with my right hand’s X and Y axis through the use of TouchDesigner’s TDAbleton component and Kinect CHOP.

Thought some of you guys might dig into this kind of experiments.

I've been using my first-gen Kinect almost exclusively for deliberately low quality 3d scanning (swapping in a different object for one or two images in the stack can be interesting), but this made me think that it should be fairly simple to use TouchDesigner or PD to make a small patch that divides the image into sectors and sends OSC or MIDI to OBS according to which sector has motion  in it, for automated camera switching.  

I recommend everyone to first watch this video muted and try to imagine what sort of music is going on.

I think the concept is cool but it may have unintended consequences if the artist does any kind of unplanned movement on stage.

I imagined some kind of cross between the music from the Myanmar coup workout clip and early Nitzer Ebb.

Honestly, even just playing those two tracks at the same time without any attempt to match the beats or tempo makes me think I might be on to something.

  On 6/28/2021 at 3:07 PM, TubularCorporation said:

I've been using my first-gen Kinect almost exclusively for deliberately low quality 3d scanning (swapping in a different object for one or two images in the stack can be interesting), but this made me think that it should be fairly simple to use TouchDesigner or PD to make a small patch that divides the image into sectors and sends OSC or MIDI to OBS according to which sector has motion  in it, for automated camera switching.  

That's a great idea! I've been experimenting with producing MIDI, but no greatness yet achieved. Need to improve it a lot. Let me know if you get any results!

I won't be able to test that because the computer I run OBS on doesn't have enough CPU to run any more video capture than it already does.

Actually what might be even more useful (but also a lot more work and more expensive) would be to use a Raspberry Pi 3 or 4 with four cheap HDMI capture dongles and some short range proximity sensors (maybe 12") connected via Bluetooth.  Make a PD patch to monitor the proximity sensors and when something moved in range of one, switch to the associated HDMI input.  Set it all up to run headless and display the currently active input fullscreen so you could patch the DMI output to a capture device on a separate computer for composition and endcoding.   Standalone, 1040p, lossless, automatic video switcher for probably around $50-$200 depending on which Pi version you needed (a 3b+ might be enough).   Good-enough HDMI capture dongles are about $10 each, proximity sensors are 4 or 5, but I don't know yet what and how expensive the best option for wirelessly connecting the sensors to the Pi would be.

Maybe some day I'll try putting something together, I can't afford to do it any time soon.

  On 6/28/2021 at 8:23 PM, TubularCorporation said:

I won't be able to test that because the computer I run OBS on doesn't have enough CPU to run any more video capture than it already does.

 

Actually what might be even more useful (but also a lot more work and more expensive) would be to use a Raspberry Pi 3 or 4 with four cheap HDMI capture dongles and some short range proximity sensors (maybe 12") connected via Bluetooth.  Make a PD patch to monitor the proximity sensors and when something moved in range of one, switch to the associated HDMI input.  Set it all up to run headless and display the currently active input fullscreen so you could patch the DMI output to a capture device on a separate computer for composition and endcoding.   Standalone, 1040p, lossless, automatic video switcher for probably around $50-$200 depending on which Pi version you needed (a 3b+ might be enough).   Good-enough HDMI capture dongles are about $10 each, proximity sensors are 4 or 5, but I don't know yet what and how expensive the best option for wirelessly connecting the sensors to the Pi would be.

 

Maybe some day I'll try putting something together, I can't afford to do it any time soon.

Expand  

I would love to have the knowledge to make something like that! Doesn't seem so complicated, but haven't used Raspberries ever. 

  On 8/13/2021 at 1:46 AM, justanotheruser said:

I would love to have the knowledge to make something like that! Doesn't seem so complicated, but haven't used Raspberries ever. 

I could probably manage the hardware OK - it's all of the shelf stuff - but I'm a hack with PD.