Another possibility is a Low Dropout Regulator IC (LDO), such as the:
* MCP1702-5002E/TO [Digi-Key part#: MCP1702-5002E/TO-ND]
* LP2950CZ-5.0 [Digi-Key part#: LP2950CZ-5.0/NOPB-ND]
Make sure to include the required capacitor on the output [and in some cases, input] -- check the datasheet!
This is less efficient than a DC-to-DC converter [Stephan's suggestion], though, but it's a simpler/less expensive solution.
And, also, as Stephan pointed out, you neglected to divulge the battery type/voltage, which can make a world of difference. Also, if the battery will drain down to below 5 Volts + the dropout voltage of the regulator, while the battery still has power to give, then this is a poor solution -- the DC-to-DC converter would be better in that case.