Speedometer pointer falls below zero

Hello, I am working on one Lexus hybrid vehicle, it has problem with its speedometer. The speedometer falls below zero, I opened the combination meter so that I can access the speedometer pointer (without removing any wire) with the engine running, I used my finger to push it back to zero as I removed my finger it was still at zero. Test drove the vehicle it responded properly, I thought it has worked. After parking the vehicle for some hours, the speedometer falls below zero again. The vehicle was test drive without adjusting it with my finger, it was observed that the pointer was moving gradually from where it was (below zero) when the vehicle is accelerating. This will give a false reading to the driver because it is not starting from zero. What could be the cause and how can it be fixed?
I drew a sketch of what I mean below.