If a player runs out of lives, the player will get a game over and he/she has to start the cup all over.
However, lives are unlimited in Mario Kart 64.
As of Mario Kart: Double Dash!!, lives are no longer a thing and instead you just go onto the next race regardless of your place.
No more Mario Kart Lives! Yay!