The issue ended up being commutation. With the commutator being freshly cut, I needed to build a new film on the surface of the comm. Even after sanding the Crap out of all of the brushes yesterday, it still looked like a compounding issue.

Today I noticed that I was only building an output of 3.5-4 volts. After running the car for a while on inspection and then on high speed, my brushes got nice and shiny and the comm got a nice chocolate color; didn't have to move any taps. Up and down speeds were within 10fpm.

Thank you everybody for your responses.