In most situations the restricted lane (regardless of how you pick who gets to use it), does in fact benefit everyone else.
Under high congestion traffic throughput plummets. Restricted access to one or more lanes lets you keep them flowing at near the peak, increasing the overall throughput of the system by much more than one of the congested lanes.
Most of the Bay Area HOV lanes are not limited access. They let you enter/exit wherever, creating congestion. They also slow down traffic at the points where people have to cross lots of lanes to enter/exit.
When before/after studies have been done, the HOV lanes around here generally make everything worse.
Under high congestion traffic throughput plummets. Restricted access to one or more lanes lets you keep them flowing at near the peak, increasing the overall throughput of the system by much more than one of the congested lanes.