Yeah, even thinking about it more - the dual build thing is almost more annoying than it'd be worth for that anyway. With the characters that I use a second build for, it's usually for a specific purpose. I'd hate to have to waffle between those on a TF just to "shortcut" the recharge on that.
You'd have to make your builds tailored for that (which I guess for the diehards that's probably doable/appealing), but for the average player I can't imagine it being used much for that purpose.