I've been playing with the camera track_visual element. I've setup to position the camera at a fixed position when follow is toggled:

    <track_visual>
      <static>1</static>
      <use_model_frame>1</use_model_frame>
      <xyz>8 0 5</xyz>
      <inherit_yaw>1</inherit_yaw>
    </track_visual>

However, once the follow mode exited, the camera orbit mode (middle mouse button) behaviour changes. "Normally" the middle mouse button let you ajust the camera yaw and pitch angle but once I enter and exit follow mode orbit also affects the camera roll angle.