No details were immediately available on the identity of the attackers.
The three were killed at the site known to Muslims as the Haram Al-Sharif and to Jews as the Temple Mount, the location of regular clashes between Palestinians and Israeli police, but gunfire rarely occurs there.
The site includes the Al-Aqsa mosque and the Dome of the Rock.
A number of attacks have occurred around Jerusalem’s Old City in recent months, but they have often involved knives.
Police locked down the area and the Al-Aqsa compound at the holy site was closed for Friday prayers. Gates leading to the site were sealed off.
Video being shared on social media appeared to show gunshots ringing out at the holy site.
“I was standing here and then I heard the shooting. I thought it was fireworks,” Basem Badawi, a 60-year-old water seller in the Old City, told AFP.
“But then I saw the police coming from everywhere.”
The Haram Al-Sharif/Temple Mount is considered the third-holiest site in Islam and the most sacred in Judaism.
It is central to the Israeli-Palestinian conflict, with Palestinians fearing Israel may one day seek to assert further control over it.
It is located in east Jerusalem, occupied by Israel in the 1967 Six Day War and later annexed in a move never recognized by the international community.
Jews are allowed to visit, but not pray there to avoid provoking tensions. The site is administered by the Islamic Waqf organization.
Waqf officials said its guards at the site had been detained by Israeli police following the attack.
A wave of unrest that broke out in October 2015 has claimed the lives of at least 277 Palestinians, 42 Israelis, two Americans, two Jordanians, an Eritrean, a Sudanese and a Briton, according to an AFP toll.
Israeli authorities say most of the Palestinians killed were carrying out knife, gun or car-ramming attacks.
Others were shot dead in protests and clashes, while some were killed in Israeli air strikes on the Gaza Strip.
The violence has greatly subsided in recent months.
