I made some major progress on Zack today. Firstly, I fixed the vertical jitter. It was due to the fact that the camera runs asynchronously and I made the mistake of zeroing out the height array on each video frame before computing new height values. Naturally the game rendering thread sometimes saw a zeroed out array, depending on where the video thread got interrupted.
I also made a first cut on some physics. Notice the new background setup, I can now pull the video camera further back.