The gunmen snatched Abdallah al-Khalidi as he was about to get into his car and they sped off with him in another vehicle, police in the city's Mansoura district said.
Security in the country's second city, and in southern Yemen generally, has deteriorated during the political turmoil that began with mass protests against then-president Ali Abdullah Saleh in early 2011, and saw fighting among pro- and anti-Saleh factions of the military as well as tribal militias.
A Saudi foreign ministry spokesman confirmed the kidnapping, saying: "It happened this morning. The embassy already contacted the highest security authority in Yemen. They are investigating and trying to find out the reason."
A string of security officials have been assassinated in recent months in southern Yemen, where an Islamist group linked to al Qaeda has seized territory and claimed responsibility for attacks on Yemeni troops and a U.S. security team last month.
Kidnapping is common in Yemen, frequently in the context of regional or tribal disputes with Yemeni authorities. The victims are sometimes held for ransom, particularly if they are foreigners.