• Intro to ffmpeg: audio-to-video filters

    by meii

    August 3rd, 2019

    Thanks to the interest of Oddlogic and BFNK in creating an automated way to create visual content based on our music, I took their advice on looking up to ffmpeg. Ffmpeg is "A complete, cross-platform solution to record, convert and stream audio and video", free & open-source. Ffmpeg makes easy and fast to do all kind of manipulations to video and audio.

    Installation steps are here.

    ffmpeg is the audio/video converter. It requires always an input and an output. ffplay is the player included. It only requires an input.

    For ffmpeg the syntax is

    ffmpeg -f lavfi -i INPUT -vf filtername OUTPUT

    For ffplay the syntax is:

    ffplay -f lavfi movie=input.mp4 //to play a video
    ffplay -f lavfi amovie=input.mp4 //to play only audio
    ffplay -f lavfi movie=input.mp4 -vf filter //to add a filter to the video

    We will focus on the audio-to-video capabilities.

    When creating video from audio, we need to duplicate the input so we can add it to the resulting video. A nice example from the dev site is:

    input --> split ---------------------> overlay --> output
                |                             ^
                |[tmp]                  [flip]|
                +-----> crop --> vflip -------+

    ffmpeg -i INPUT -vf "split [main][tmp]; [tmp] crop=iw:ih/2:0:0, vflip [flip]; [main][flip] overlay=0:H/2" OUTPUT

    So basically you have to split your media and then put them together after processing them. That's all, but it's important to remember when applying more than 1 filter.

    A list of audio-to-video filters:









    The one that have the weirdest results imo is avectorscope. But the magic is in the combination of several filters in serie or parallel. For example,

    ffplay -f lavfi 'amovie=input1.wav, asplit [a][out1];[a]avectorscope=m=polar:mirror=x:draw=line:s=600x400:r=60:zoom=5:rc=165:gc=82:bc=0:rf=5:gf=10:bf=15'

    Adding now a filter at the end (mirror effect),

    ffplay -f lavfi 'amovie=input.wav, asplit [a][out1];
    [a]avectorscope=m=polar:mirror=x:draw=line:s=600x400:r=60:zoom=10:rc=165:gc=82:bc=0:rf=5:gf=10:bf=15' -vf "crop=iw/2:ih:0:0,split[left][tmp];[tmp]hflip[right];[left][right] hstack"

    And that is most of it! now it is only a matter of testing new values, filters, etc. Some of the results I've been getting are:

    My main goal here is not to become some serious programmer/databending wizard. I'm not a visual artist and my knowledge and experience in this topic is almost null. But with tools like this I found very exciting and easy to have a visual representation of your own music that you can keep processing and processing for much better results.

    And I think you can do that too!

    Next step would be to start creating some shell scripts to go further in manipulating the parameters. If you are curious and want to do more, check "frei0r" plugins for cool video effects.