Hi Eve,
It's definitely possible (but not probable) that the VFD is only using two-phases of the supply during the braking operation. If it were, the DC it generates could be flowing through either two or three of the phase connections to the motor, depending on how the VFD is designed. Typically though, a VFD is generating DC using all three phases during normal operation (and it chops up this DC to run the motor). Then when it wants to brake, it stops chopping (or chops differently) and just provides DC to make the motor stop. All this to say that it would be slightly odd for it to switch to only two phases for braking.
You may want to also check the phase voltages at the supply. It's possible another machine nearby (or something else on the line) is causing a slight dip in voltage on one of the phases and that could easily result in a reduction of current on that one phase.
Also - are you measuring with a "true-rms" current meter? The current waveform is bound to be funky (non-sinusoidal) which will create erroneous readings on a 'normal' current meter (since normal meters make assumptions about waveform shape). Also - make sure you're measuring AC currents upstream of the VFD. I would take current measurements during normal motor operation and again during braking and see if the current (or voltage!) imbalance is present in both conditions.
Most VFDs are very light with their braking and provide very little current (when compared to motor startup currents or dedicated DC-Injection brakes) so it's interesting that this would be enough to trip the breaker.
If this system has been working fine for a while and just recently started having this malfunction, then I would look more closely at the VFD or the breaker itself as something internal probably failed. Or, less probably, the motor itself could be failing or having intermittent faults between phases. The latter would only be likely if the motor was very old or has been thermally abused.
We design and manufacture DC-Injection brakes so this is right in my wheelhouse and I'm happy to answer any other questions.