Well I finally fixed it.

It wasn't the timers.

I designed a bitmasking input system, which in hindsight I have no idea what the hell I was thinking because there is already a global bitmask just for that purpose. But basically it was getting cleared every tick instead of me properly handling released key presses.

This was unnoticeable at higher framerates because the renderer would update inputs faster than the world could update and have key events ready long before the next game tick.

However at lower framerates this would cause anything reading the bitmask to constantly start and stop moving because the bitmask was completely empty whenever multiple ticks occurred within one frame.

This also explains why visual effects worked fine.

So yeah I wasted 3 days reimplementing various timer systems over and over before realizing the system to prevent dropped inputs was dropping inputs and I could have avoided all of this