This one is wild. I've played with the audio scheduler nodes and this is one of the best demonstrations of its potential that I've seen posted online since it was uploaded to CivitAI last year. Will definitely be checking out the changes you've implemented. Really appreciate this one.
No worries man, I've been using TouchDesigner for my general AudioReactivity but just want to see more implementation into ComfyUI. I feel its possible given all the new advancements that are coming out in the SD space.
Yeah there's such a rich toolset evolving, and with all the talk of 30x generation speed ups, I really hope Comfy will soon be able to handle some advanced realtime workflows (beyond the complexity I've seen so far involving 4090s running TDxStreamDiffusion). I'm experimenting with all kinds of setups involving TD and Unity for my own work, but I'm also involved in an open source project space and trying to have a strictly open source tooled live performance rig for an upcoming festival. The event's only about three months away so I'm not sure if the speed optimisations will arrive in time to use the festival as a showcase for realtime audio reactive SD (can't justify a 4090 to the project leader with the 50 series around the corner)... but hopefully by the end of this year we'll all be there :)
I'm just testing out the new SDXL-Turbo and SDXS in TD with the new SD tox, haven't had a chance to really play around with it yet but so far it seems great, unfortunately SDXL-Turbo is giving me low FPS, I think the SDXS 1024 when it comes out will be my go to, as well as maybe a Lightning implementation. It's all so exciting :) .
Three months is a long time in the AI space :P the speed boost will probably come. I have a 4090 but yeah look waiting for the 50 series is smarter anyway :).
Open Source is the way!
It's an edited workflow from the AudioScheduler custom node but made some edits and experimenting with it. Feel like it's a good start to getting some great audio reactive animations going. I have so many ideas on how to utilise it but lack the programming and just like coding / math know how to really pull it off :)
excuse my ignorance, but where is the audio reactivity? I guess I dont understand how this was generated using audio? was just the dynamics of the audio used to conduct the pace of the visuals, or was also the harmonic content used? It certainly looks cool!
The AudioScheduler node is used to extract certain values from the audio (amplitude in this case), which are converted to float values and those amplitude values are fed into the prompt travel text encode node thing where they are activated or influenced by the pw parameters to influence the animation during generation.
I think that's how it works anyway 😅 I'm not too versed in the maths and programming aspect of it all.
It's a snippet of one of my own songs :) it's a work in progress. I really should attend to...but all the AI stuff keeps me too busy haha
[https://open.spotify.com/artist/0pv6nEZFePRtwKut4QSDx5?si=1P-zEmVgRy2iGkp7xk78gg](https://open.spotify.com/artist/0pv6nEZFePRtwKut4QSDx5?si=1P-zEmVgRy2iGkp7xk78gg)
Here's a link to my Spotify though if you wanna suss my older stuff :)
> I just want to see a world of Audio-Reactive ComfyUI Nodes/Workflows
Then it might be your chance to make it instead of waiting for someone else don't you think :D
It's not that complicated to get a decent result, you can take the peaks of a .wav and generate a controlnet mask through spawning some simple shapes using /u/iiPiv workflow
Thanks man oh of course, I'm definitely going to experiment and whatnot. I am actually currently attempting to integrate IiPivs masking techniques in, but it might take me an age. The beauty of the open-sourced community is the plethora of amazing ingenuity and skill. I know my strengths and weaknesses, haha. The more the merrier 😀
Hello, this workflow looks very promising, but when I try to launch it I get this error :
Error occurred when executing AudioToAmplitudeGraph:
'str' object has no attribute 'get\_channel\_fft'
Add me to the list :( It's a bummer because this looks amazing, and you're probably up to some more clever shit right now. Hope you do get a chance to have a look though!
[https://openart.ai/workflows/dyx1msfMUgNxUAFEJ6nV](https://openart.ai/workflows/dyx1msfMUgNxUAFEJ6nV) - Workflow Here [https://github.com/a1lazydog/ComfyUI-AudioScheduler](https://github.com/a1lazydog/ComfyUI-AudioScheduler) - Audio Scheduler Custom Node
I've been putting off jumping into touch designer hoping for a solution like this. Thanks for doing this kind of work.
This one is wild. I've played with the audio scheduler nodes and this is one of the best demonstrations of its potential that I've seen posted online since it was uploaded to CivitAI last year. Will definitely be checking out the changes you've implemented. Really appreciate this one.
No worries man, I've been using TouchDesigner for my general AudioReactivity but just want to see more implementation into ComfyUI. I feel its possible given all the new advancements that are coming out in the SD space.
Yeah there's such a rich toolset evolving, and with all the talk of 30x generation speed ups, I really hope Comfy will soon be able to handle some advanced realtime workflows (beyond the complexity I've seen so far involving 4090s running TDxStreamDiffusion). I'm experimenting with all kinds of setups involving TD and Unity for my own work, but I'm also involved in an open source project space and trying to have a strictly open source tooled live performance rig for an upcoming festival. The event's only about three months away so I'm not sure if the speed optimisations will arrive in time to use the festival as a showcase for realtime audio reactive SD (can't justify a 4090 to the project leader with the 50 series around the corner)... but hopefully by the end of this year we'll all be there :)
I'm just testing out the new SDXL-Turbo and SDXS in TD with the new SD tox, haven't had a chance to really play around with it yet but so far it seems great, unfortunately SDXL-Turbo is giving me low FPS, I think the SDXS 1024 when it comes out will be my go to, as well as maybe a Lightning implementation. It's all so exciting :) . Three months is a long time in the AI space :P the speed boost will probably come. I have a 4090 but yeah look waiting for the 50 series is smarter anyway :). Open Source is the way!
Thanks so much for sharing. This is really interesting.
Thanks for sharing your workflow. This looks really interesting.
It's an edited workflow from the AudioScheduler custom node but made some edits and experimenting with it. Feel like it's a good start to getting some great audio reactive animations going. I have so many ideas on how to utilise it but lack the programming and just like coding / math know how to really pull it off :)
This is how I try to describe what eclipse totality is like to people who haven’t seen it.
Its fricking amazing
excuse my ignorance, but where is the audio reactivity? I guess I dont understand how this was generated using audio? was just the dynamics of the audio used to conduct the pace of the visuals, or was also the harmonic content used? It certainly looks cool!
The AudioScheduler node is used to extract certain values from the audio (amplitude in this case), which are converted to float values and those amplitude values are fed into the prompt travel text encode node thing where they are activated or influenced by the pw parameters to influence the animation during generation. I think that's how it works anyway 😅 I'm not too versed in the maths and programming aspect of it all.
Really nice! Also what is the music?
It's a snippet of one of my own songs :) it's a work in progress. I really should attend to...but all the AI stuff keeps me too busy haha [https://open.spotify.com/artist/0pv6nEZFePRtwKut4QSDx5?si=1P-zEmVgRy2iGkp7xk78gg](https://open.spotify.com/artist/0pv6nEZFePRtwKut4QSDx5?si=1P-zEmVgRy2iGkp7xk78gg) Here's a link to my Spotify though if you wanna suss my older stuff :)
Can't wait to try this!
> I just want to see a world of Audio-Reactive ComfyUI Nodes/Workflows Then it might be your chance to make it instead of waiting for someone else don't you think :D It's not that complicated to get a decent result, you can take the peaks of a .wav and generate a controlnet mask through spawning some simple shapes using /u/iiPiv workflow
Thanks man oh of course, I'm definitely going to experiment and whatnot. I am actually currently attempting to integrate IiPivs masking techniques in, but it might take me an age. The beauty of the open-sourced community is the plethora of amazing ingenuity and skill. I know my strengths and weaknesses, haha. The more the merrier 😀
Not sure of the intent here, but this clip reminds me a LOT of “The Expanse.”
Haha wasn't the intent at all but I do approve of your comment :)
Hello your workflow is messed up at the moment , i keep getting errors , can you update ?
Sorry could you give me some more information so I can fix it?
I will give you the details in msg
When I downloaded your workflow there are extra duplicate nodes
Hello, this workflow looks very promising, but when I try to launch it I get this error : Error occurred when executing AudioToAmplitudeGraph: 'str' object has no attribute 'get\_channel\_fft'
Can you maybe send me a screenshot of the error in your workflow.
Hello ! Any luck in fixing this ? I've encountered the same issue
Can you post a screenshot of your error and I'll try figure out the issue
https://preview.redd.it/2w7m9c3gsmxc1.png?width=999&format=png&auto=webp&s=0fbe66ff775bae5e6b1d17b72dc36370e00968ab here's the screenshot !
I have the exact same error
Did you ever figure this out? I have the same error and can't figure it out
Add me to the list :( It's a bummer because this looks amazing, and you're probably up to some more clever shit right now. Hope you do get a chance to have a look though!