The Hermitage

Linlithgo, Columbia County, New York

Built from 1774, for Colonel Peter R. Livingston (1737-1794) and his wife, Margaret Livingston (1738-1809). Peter was the eldest son and principal heir of the 3rd and last Lord of Livingston Manor but having become encumbered with debt during his father's lifetime, as part of the action taken to curtail his inheritance, his father started building him this home. He hoped his son would in time add another story but that didn't happen. It wasn't until 1939 that the house was finished, by Ida H. Ogilvie, founder and former head of Columbia University's Geology department. The Dutchess County Historical Society applauded her and the architect for their careful attention to detail, but it was razed in 1983 by Mrs Peggy Rockefeller, wife of the Chairman of Chase Manhattan Bank. She didn't consider the locally-loved property "historic," defending her decision by saying that it would have been too expensive for her to restore....

This house is best associated with...

Peter Robert Livingston

Colonel Peter R. Livingston, of "The Hermitage" Linlithgo, New York


Robert Thong Livingston

Robert Thong Livingston, of "Linlithgo" Manor of Livingston, New York


Henry Walter Livingston

H.W. Livingston, Congressman from New York & Member of the State Assembly


Peter Schuyler Livingston

P. Schuyler Livingston, U.S. Congressman from New York


Robert L. Livingston

Robert L. Livingston, of New York


Ida Helen Ogilvie

Professor Ida H. Ogilvie Ph.D., Geologist, of "The Hermitage" New York


Peter's lack of business acumen and soaring debts left his family fearing for the future of the vast lands that comprised the Manor of Livingston that were Peter's birthright. His father sought the help of his son-in-law, James Duane, and they successfully broke the entail in 1771 but nonetheless ensured that Peter was well-provided for by building him this home on 600-acres and giving him an ample trust fund with which to maintain his family and lifestyle. To make sure the home could never be taken by his creditors, they put the house in the name of Peter's eldest son, Robert Thong Livingston (1759-1813).

In Hope of Redemption...

The house in which Peter lived was 3.5-miles south of Hudson and stood one story over a raised basement measuring 50-feet square which gave it a footprint of 2,500 square feet. His initials "P.R.L." were seen on the iron fireback in the Drawing Room. It is still a matter of some conjecture as to whether construction was halted half way through due to the Revolution - and a roof hastily added - or whether this was indeed the finished article that his father intended. Judged by the position of the stairs in what was then the back hall - built in the same way as Peter's brother's house, Teviotdale, which adjoined The Hermitage to the south - historians concluded that his father had hoped that by building him only half a house this would act as an incentive for Peter to bring his finances into order and complete the rest himself. 

The Revolving Door at Hermitage Farm (1794 to 1930)

Peter died in 1794 and three years later his son (whose initials "R.T.L" and "Serena" were scratched onto one of the windows in the southeast room on the main floor) sold the estate to his first cousin, Henry W. Livingston, to cover their debts. Perhaps Henry had intended to finish the house for his family but instead he built a significantly grander home, The Hill, and seemingly sold The Hermitage to his younger brother, P. Schuyler Livingston, who was the owner when it was publicly advertised for sale in 1799.

The new owner was Schuyler's brother, Robert L. Livingston, but having married the younger daughter and co-heiress of Chancellor Robert R. Livingston they moved into Arryl House and The Hermitage once again became superfluous. In 1804, he sold it to John Richmond (1746-1829) who lived here with his wife, Mary, until his death during which time he turned the property into a successful farm, known as the "Hermitage Farm".

The Richmonds had four "rather eccentric" sons (none of whom married) and one daughter, Mary (1796-1843), who married John Taylor (1790-1863), Mayor of Albany. Their eldest son, Timothy Richmond, died in 1863 and three years later his executors sold The Hermitage and its 658-acres for a considerable $35,000 to Alexander Fisher.

During Fisher's tenure, he maintained extensive orchards here but the woollen mills built earlier were abandoned. After his death, his son, Frank, could not keep up with the mortgage repayments and the bank foreclosed on the property. In 1899 it was bought at auction for $19,000 by the heirs of the late James B. Jermain of Albany, who included among them Catherine B.J. McClure. On the partition of the Jermain estate in 1906, the Hermitage was conveyed to Catherine though she never lived there, presumably leasing it to various tenants. In 1928, she sold it to Albert M. Reed, of Manchester, Vermont.

Ida H. Ogilvie, Finishing the Lord's Work

In 1930, The Hermitage was purchased by Professor Ida H. Ogilvie (1874-1963), of Barnard College, Columbia University, the first woman to teach Geology at graduate level in a major university. During the war she became involved with the Women's Land Army that helped out with farmwork while the men were fighting overseas and she transformed the farm at the Hermitage into an "excellent" dairy farm, stocking it with Jersey Cattle from her partner's (Delia West Marble's) farm, Airlie, in Westchester. It was Ida who in 1939 finally finished the house as the 3rd Lord of Livingston Manor had originally intended just short of 150-years before, flawlessly integrating a second story, porch, and wing.

The result was met with overwhelming approval and on visiting in 1939 Brigadier-General John Ross Delafield of Montgomery Place wrote on behalf of the Dutchess County Historical Society: "The restoration and enlargement of the mansion have been well done indeed. Everything that could be saved has been restored and kept and the new work has been carefully done to match the old. Miss Ogilvie and her architect, Harold R. Sleeper, are to be congratulated". It was at her "beloved" home where Ida died in 1963.


Whereas Teviotdale was fortunate to have been restored by Harrison Cultra and Richard Barker after having been left to rot for over 50 years, The Hermitage was not so lucky: In 1982, the house and farm was bought by Mrs Margaret "Peggy" Rockefeller, the wife of David Rockefeller, Chairman of the Chase Manhattan Bank. They lived on the Kykuit estate in Westchester but in 1980 bought their first farm in Columbia County to raise Simmental cattle. In 1982, they paid $1.5 million for three farms to its south - including The Hermitage - totalling 1,137-acres on which to grow hay, corn, and oats.

To the anger of locals and preservationists, Mrs Rockefeller had demolished The Hermitage by April the following year. She was quoted in The New York Times as saying: "When we first bought it, we felt we would keep the house, but we found it was in terrible shape... We tried to sell it, but nobody wanted to buy it. We tried to give it away, but no one would accept it. It was in terrible shape and very expensive to repair and it was useless for us". 

Despite being what James Ivory of Clavarack called, "the only building on the property that people cared about," it had never been added to the National Historic Register. Mrs Rockfeller once more defended her decision by saying, "if it had been a totally historic house, not three-quarters Ida Ogilvie, I would have felt differently". Several of the original features such as lintels, doors, windows etc., were salvaged and later dispersed and sold. 

You May Also Like...

Contributed by Mark Meredith on 13/06/2020 and last updated on 05/01/2021.
Images Courtesy of the Library of Congress, 1937; Year Book of the Dutchess County Historical Society, Volumes 23-30; Razing of 1774 Home Causes Dispute Upstate, The New York Times, April 10, 1983.


Be the first to connect to this house. Connect to record your link to this house. or just to show you love it! Connect to The Hermitage →