That's odd. I use this all the time, admittedly with the D2sroba frontend. Try it maybe.
My bitrate settings are Max:8000, Min:2000, Min avg:2000 (unchecked), Max Avg:7200, CD size 4464.
Under Misc tab I've got the DVD2VCD level set to advanced (sure it makes no difference).
On the encoder tab you haven't got the Calc bitrate as max box checked have you?
My encoder is Cinema Craft.
Can't see anything that would make a difference.
If all else fails you can always author the DVD & then shrink with
DVD Shrink, although this is far from ideal & might affect the image quality.