On the south side of the entrance of Mahón harbour lie the ruins of the castle of San Felipe. The original castle was built during the mid 18th century, by the engineer Juan Baurista Calvi. The castle was built in the shape of a four pointed star, with four bastions and four curtain walls and was surrounded by a dry moat excavated from the underlying rock. In the 17th century, the castle was enlarged by covering the curtain walls with ravelins on the far side of the dry moat covering the whole of the building with covered walk ways, palisades and glacis.

Although the British first occupied Menorca in 1708, it was not until 1713, with the signing of the Treaty of Utrecht, that Britain obtained sovereignty of the island. Even before the signing of the treaty the British began expanding the castle and strengthening its defence systems. The work consisted of adding counterguards covering the bastions, enlarging the ravelins opposite the curtain walls, communicating the moats from the counterguards and ravelins at three different levels, to the main moat. The original four pointed star building was surrounded by lunettes stretching from Mahón harbour to the bay of Cala San Estaban. Next to the castle was the Arrabal, or town, which had grown up near the castle walls, the proximity of the town enabled the French to mount batteries in the town during the siege of 1756. It was later re-built and is the present day village of Es Castell.

In 1782, the island was taken by the Spanish; the castle surrendered and was handed back to Spain, after more than half a century of British rule. On the orders of Carlos III, the castle was demolished. It was to be rebuilt and demolished for the second time in 1805 by the Spanish. Ariel views of the area still show the distinct outline of the castle.

Many years after the demolition of the castle of San Felipe, when the artillery were on La Mola, two batteries were placed on the ruins of the site of the old castle, at the points of San Carlos, and Prince´s battery overlooking the harbour interior.

The Ministry of Defence restored the castle walls after it suffered five landslides caused by easterly storms.