What?
For my graduation work I made a research paper (and project) on audio analysis communicating with DMX, MIDI and OSC protocols. I found this topic fairly interesting since it combines my love for audio, programming and going to concerts/festivals.
To reflect this, I made a project that simulates a festival stage filled with static and moving lights, pyros and fireworks. I used Unreal Engine 4 and had to enable some extra plugins in order to work with the added protocols such as DMX, MIDI and OSC.
What were my biggest challenges?
The most difficult thing in this whole project was trying to grasp all the different audio analysis theory. When I started this project, I had some basic understanding of sound; frequencies, amplitude, waves etc. But in order to understand, and be able to implement these topics myself, I had to dive into the world of audio analysis. I learned a lot about FFTs, spectrograms, audio analysis libraries, audio feature extraction, sound maths etc.
Once I familiarized myself with audio analysis, I started looking at what Unreal Engine 4 could provide. I found out that they have some simple non-real time analysers which was just what I needed. (Back when I made this project these analysers were non-real time, which meant the music/audio file was analysed before playing it back. With the advent of Unreal Engine 5, they are now introducing real-time audio analysis, which probably means that I could update this project to support real-time audio analysis as well.)
Of course I also had to learn about the different protocols I was going to send information to. I educated myself on DMX, MIDI and OSC and for each protocol I made a special implementation:
• The lights, pyros and fireworks of the festival stage are all controlled by DMX.
• I created a MIDI converter, which extracts MIDI notes from the audio that is currently playing.
• I created an OSC communication system that communicates to the TouchOSC app on a smartphone, with which you can actually control the DMX environment inside the project. It also receives the data from the audio analysers.
I really loved working on this project and I believe this project was the turning point where I decided I wanted a future in audio programming/sound design
