There has been a church on the site for over 900 years. Its name is usually taken to mean that it is the oldest of the City churches dedicated to the Virgin Mary. The patronage of the rectory of St Mary Aldermary belonged to the prior and chapter of Canterbury, but was transferred to the Archbishop of Canterbury in 1400.

In 1510, Sir Henry Keeble financed the building of a new church. The tower was still unfinished when he died in 1518. In 1629, two legacies enabled it to be completed, and the work, begun 120 years before, was finished within three years. Keble was buried in a vault beneath the floor of church, but his grave was not allowed to remain for long. Richard Newcourt recorded that

Sir William Laxton, who died in 1556, and Sir Tho. Lodge, who died in 1583 (both which were Grocers and had been Mayors of this City), were buried in the Vault of this Sir Henry Keeble, his bones unkindly cast out, and his Monument pull’d down, in place whereof, Monuments were set up of the others.

John Stow mentions various dignitaries buried in the early church in his 1598 Survey of London. They include Richard Chaucer, vintner, said by Stow to be the father of the poet Geoffrey Chaucer. John Milton married his third wife, Elizabeth Minshull, in the church in 1663. The parish registers date from 1558, and are now deposited in the Guildhall Library.

St Mary Aldermary was badly damaged in the Great Fire of London of 1666, although parts of its walls and tower survived. It was mostly rebuilt by Sir Christopher Wren in a Gothic style. A legacy of £5,000 had been left by one Henry Rogers for the rebuilding of a church, and his widow agreed to use it to fund the reconstruction of St Mary’s. According to some sources, she stipulated that the new church should be an exact imitation of the one largely destroyed.

The church as rebuilt has an aisled nave, six bays long, with a clerestory. There is a short chancel. The tower is attached to the south west corner of the building, and is entered through a western lobby. It is divided into storeys by string courses; the corners have octagonal turrets, terminating in what George Godwin called "carved finials of impure design". The nave and aisles are separated by arcades of clustered columns, supporting somewhat flattened Gothic arches. The ceilings are decorated with elaborate plaster fan vaulting. The east wall of the chancel is set askew in relation to the axis of the church.

According to Nikolaus Pevsner, St Mary Aldemary is "the chief surviving monument of the 17th-century Gothic revival in the City and – with Warwick – the most important late 17th-century Gothic church in England".

The parish of the church of St Thomas the Apostle, destroyed in the Great Fire and not rebuilt, was united with that of St Mary’s.

In 1781 a new organ was installed, built by George England and Hugh Russell.

St Mary Aldermary was damaged by German bombs in the London Blitz during the Second World War. All the windows were shattered and some plaster fell from the vaulting but the building itself remained intact.

The church has been repaired and restored many times over the years. In 1876-7 there were major changes to the interior: a new oak screen was inserted dividing the church from the lobby; the pews and stalls were replaced, the organ was moved from the western gallery to the chancel; the floor was repaved, new stained glass put into the windows, and a new reredos installed.

The latest interior restoration was finished in April 2005, with special attention paid to the plaster ceilings and the memorials on the north wall. A service was held on April 21, 2005 to celebrate the restoration, presided over by Richard Chartres, the Bishop of London.

The church was designated a Grade I listed building on 4 January 1950.