This power supply really isn't suited to audio applications. Not enough ripple rejection. I would replace the transistor with a 78L12 Voltage Regulator, remove the Pot and short the wiper line to ground (i.e. you want the ground pin of the 78L12 to go to ground). This will make the supply no longer variable, but it sounds like you don't need a variable supply for your application. If you do, then I would use an LM317L and rewire the pot and limit resistor (i.e. the variable voltage divider) as required for the LM317L (check the LM317L datasheet if you're not sure how to do that). With the LM317L, though, it won't adjust to 0 (more like 1.25V).
Another alternative is to try something like a 3300uf cap from the base of Q1 to ground. The size of this cap depends on the Beta of Q1 and how much ripple noise can be tolerated. Basically this turns Q1 into a capacitance amplifier. You might be able to go as low as 600uf.
Here's my reasoning: IT=CV so C=IT/V if I=Ic/Beta and the Beta of Q1 is, say, 70, then I=50ma/70=0.71ma T is approx 6ms and V is whatever maximum voltage will not produce perceptible noise in your Train Whistle Synthesizer. Lets say 1mV. So, C=0.71ma*6ms/1mv=4260uf. Round down to the next standard value and you get 3300uf. Best case might be a Beta of 100, and a ripple voltage tolerance of 5mV: C=(50ma/100)*6ms/5mV=600uf.
This is all linear approximation math, but it should get you in the ballpark.
