During a manually flown SAR mission, in mountainous terrain, the aircraft entered IMC. While attempting to exit these conditions on a pre-briefed escape heading with the autopilot coupled, the autopilot entered an unexpected mode that resulted in the aircraft not responding as intended. The autopilot was disconnected and the flight continued manually without further incident.
