Q: If one day is not exactly 24 hours and is in fact 23 hours, 56 minutes, shouldn't the error add up, and shouldn't we see 12 a.m. becoming noon?

You're right that a "sidereal" day is about 23 hours, 56 minutes, 4 seconds. But this is not a day in the everyday sense.

A sidereal day is how long it takes the earth (on average) to make one rotation relative to the faraway stars and other galaxies in the sky.

If you find a star that is directly above you at midnight one night, the same star will be directly above you again at 11:56:04 p.m. the next evening.

Similarly, if you were sitting on the star Proxima Centauri looking through a powerful telescope at earth, you would see Toledo, Ohio, go by every 23 hours, 56 minutes, and 4 seconds.

However, we don't keep time by the faraway stars -- we measure time by a much closer star, the sun! And we are actually in orbit around the sun, orbiting in the same direction that the earth is spinning on its own axis. From our perspective, the sun goes a little slower in the sky because we are also orbiting around it.

How fast are we orbiting around the sun? We make one full orbit every year, or roughly 366.25 sidereal days.

So after a year, the faraway stars will have done 366.25 rotations around the earth, but the sun will only have done 365.25 rotations. We "lose" a sunset because of the complete orbit. (The extra quarter day is why we need a leap year every four years.)

So there are 365.25 "mean solar days" in 366.25 "sidereal" days. How long is a "mean solar day"? Let's do the math: One sidereal day is 23 hours, 56 minutes, 4 seconds, or 86164 seconds. Multiply this by 366.25 sidereal days in a year, and you get 31557565 seconds. Divide by 365.25 solar days, and we get that a solar day is.... 86,400 seconds. That's 24 hours exactly!

It's this "mean solar day" (24 hours) that is the normal definition of day.

If you want to do the math more exactly, a sidereal day is 86164.09054 seconds, and a tropical year is 366.242198781 sidereal days. That works out very closely.

(P.S. Unfortunately, the earth's spin has been slowing down because the moon is sucking away the earth's energy. Every time the high tide of the Atlantic Ocean slams into the east coast of North America, the earth slows its spin a little bit. The definition of the second is based on the speed the earth was spinning back in 1820, and we have slowed down since then. As a result, we occasionally have to add in a "leap" second to the world's clocks. See http://online.wsj.com/article_email/SB112258962467199210-lMyQjAxMTEyMjIyNTUyODU5Wj.html?mod=wsj_valetleft_email)