Software To Match Midi To Audio

[BigBubbaBrown]

In Blender (3D modeling/animation software) there's a script to import a MIDI file and bind it to various animation channels. It works pretty well when it's strictly MIDI, but when it comes to lip synching or audio (drums recorded live, for instance) you have to load a file into the video editor and try to "read" the waveform. For a drum track it becomes a hassle and when lip synching the results are almost always inaccurate, even if it's just to a barely noticeable degree.

Recently I got the idea to use notes in an unused channel to drive phonemes in Blender but there are timing issues. When you record a singer the results won't be snapped to a grid. If someone's singing a series of 8th notes at 120 bpm and you enter a series of 8th notes on a piano roll they're not going to match up. In best case scenarios most notes will be off by a 64th note to a 32nd note in either direction and worst case you just have to give up if rerecording isn't an option.

Is there any software that facilitates matching MIDI to audio? Perhaps one where you can load an audio file and the waveform will render in the background of a piano roll so instead of constantly clicking out of the piano roll in your DAW to see if the notes line up you can check as you add the notes.

In theory it would be easy to write something like that but since I'm also trying to port the MIDI import script to the current version of Blender, modeling the characters, writing the music, and trying to juggle non music and animation related things I'm hoping someone knows of existing software.

[john]

Hey

I would try using SMPTE code and / or one of the midi sync options (probably Midi Time Code - MTC)

You might find this article a useful introduction:

http://midi.songstuff.com/article/MIDI_sync/

Give me a shout if you have a specific question you think I can help with, though much of it is experimenting with the options.

Do you use Blender a lot? I found it very flexible but the interface was really cumbersome to get into because of just that. Time being a limited factor my little side investigation of Blender didn't go very far.

What sort of things have you used Blender for? What is your main vid editor software?

Cheers

John

[BigBubbaBrown]

I don't really have much in the way of sync options. I can use the MIDI out of my drum kit to control it (in fact, that's the only way I can get the patterns in, as otherwise there's so much lag that it just sounds like noise). CWMC3 lists audio as a sync option but I've never been able to get it to work. I can't afford to upgrade past version 3, either so I'm limited to what I already have.

Your suggestion did give me an idea, though. I'll still have to "read" the waveform, but by typing the H:M:S:F values (SMPTE isn't an option in my software) into a spreadsheet and printing it out I'll have a sort of dope sheet that will help me place the notes, and even if the MIDI controller idea doesn't work I can use the info to jump to the proper frame by converting it to seconds (should be a simple formula in a spreadsheet) and using the formula frame_rate*time_in_seconds.

I've noticed Blender is either one of those "love it for life" or "run away screaming" propositions. I probably have an advantage over a lot of people because when I started using it (don't remember the year, but the version number was 1.8. They're up to 2.6 now for perspective) I had a lot of time on my hands, the industry standards weren't an option, and it was my second experience with 3D after POV-Ray, where you had to type everything in and you didn't know if everything was right until you rendered and animation was a series of mathematical formulas.

I've used Blender for just about everything. Could have made a living using it, in conjunction with Ogre (a game engine) to model buildings for architects to show off to clients if I didn't keep getting cheated.

As far as my main video editor, that would also be Blender. Render out a series of PNG (or EXR if I need the depth information for compositing) files, use the compositor for post processing (color correction, cropping, effects that can't be achieved during the rendering process such as anaglyph 3D, etc.), load the resulting image sequence into the sequence editor along with the relevant audio files, and then render that out to an Ogg theora file.

With the latest version I don't even have to do much work with the audio to get positional effects. Maybe some EQ if I want it to sound like it's coming out of a radio, but if I want to animate a drum line playing a cadence while walking past the camera or record a "rehearsal" where they wouldn't have any mixing equipment I no longer have to mess with automating panning and volume. Just set up a "speaker" object for each instrument, load the audio files, and I can go from a mastered track that would otherwise play over the top of the animation to having the drums sound like they're coming from the drum set, the guitar sounding like it's coming out of an amp, etc. then mix them down for the final render.

If you want to give Blender another try send me a message and I can help get you started. The mistake a lot of people make is they try to learn everything about it. When you only learn the things that pertain to what you want to use it for (for instance, if you only use it for character animations you wouldn't have to learn about particle effectors, no need to learn how to use the cloth simulator if you only want it for static backgrounds, no need to learn modeling if you just want to use it to process and mix video, etc.) it's actually a pretty good alternative to spending a few grand on the industry standards. Plus everything's self contained so once the music's recorded you won't have to switch from Blender to post processing software to a video editor.

[john]

Blender doesn't support SMPTE? Wow.

There must be a SMPTE time code converter you can use. In terms of generating that time code just about any DAW should be able to generate it (set to master and to actually produce the code).

I did a wuick search and found this:

http://blendit.xaa.pl/index.php?p=middrv&l=eng

Hopefully that is of help. If not, if there is one, perhaps there are others.

Many thanks for your offer of help with Blender. Very much appreciated. My initial interest was for 3d modelling and animation. The idea of combining greenscreen with those 3d models is quite appealing.

Available time is an issue right at the moment. I'm in the middle of a major re-work of Songstuff. It should be completed in a month or two and at that point I'll be able to able to get back to my other creative interests including completing an album and video creation. Will it be okay to contact you at that point?

Thanks again

John

[BigBubbaBrown]

After consulting Cakewalk's help files it turns out it does support SMPTE. My mistake was assuming the acronym was something along the lines of Seconds, Minutes, ?, ?, ? instead of being named for the standard. Blender has support for it, though it's a bit cumbersome. It's only available on a few screens (the dope sheet and possibly the NLA (nonlinear action) editor) and in order to get it to display the frame portion you have to really zoom in. When editing on a large scale you pretty much only have access to the H:M:S portion of it and you only get the frames when fine tuning.

So in short both programs support it, though my ignorance of the subject of SMPTE kept me from seeing it.

I tried out the script you linked to when I was evaluating MIDI import scripts. It seems powerful but it throws errors at random points. The one I use is MIDI Import X. It doesn't have all of the whistles and bells but it's stable and gets the job done.

I'll PM you my contact info for when you're ready to give Blender another try. To use it with MIDI you'll have to download two versions unless I get the MIDI import script ported (recently the program, including the python API, received a complete overhaul so it won't work with the current version). Fortunately they keep links to the older versions so I can show you how to obtain it.