Here is what the switches do. Ground all and the input goes to zero, unground all and connect to Vref, and Vout should approach 2x Vref. So as to what you want Vref to be that depends, if you want the output to represent the original sampled voltage, then Vref would be Vmax (sampled)/2.
Say you sampled some voltage and the max sampled voltage was 5V, the Vref would be 5V/2. Therefore when you set all the switched to Vref, the output would approach 5V (never actually reached 5V because of resolution).
Have you actually written out a table that shows what the output voltage is as a function of the switch settings? I hope so. I don't know where this particular algorithm came from but that doesn't matter, it has a purpose.
Have you sorted out any of this, I mean how it works as designed? If so is that how you want it to work?
