In 1903, the library commission announced plans to build a branch library in the vicinity of Field School at Field and Agnes Streets. While previous branch libraries were located in rented buildings, Branch No. 4 was the first owned by the city when it opened on November 22nd, 1906, built at a cost of $12,000. "Detroit is no longer a one library town," librarian Henry M. Utley told the Detroit Free Press on the occasion.
In March of 1913, the library was remodeled and expanded. Malcomson and Higginbotham designed the addition, which more than doubled its size and moved the entrance from Agnes Street to Field.
Facing a severe budgetary shortfall, the public library commission announced that three of the city's 29 library branches would be closing, including Campbell, Monnier, and Gray. It was the first time that branches were permanently closed, with 140 of the system's 700 employees losing their jobs. Gray would eventually reopen in a new facility inside the Butzel Family Center, though the center would not open until 1975.
While initially busy, Butzel would prove to be a difficult location to manage. In 2004, the center began to require identification for entry due to security problems, resulting in a sharp decrease in use. Coupled with declining maintenance and vandalism, the library commission decided to close the Gray Branch in July of 2005.
Gray would eventually become home to St. Christopher House, an outreach ministry for the homeless founded in 1971 by Sister Hilda Manson. St. Christopher House served the east side community for almost 30 years, operating a summer camp for children, a soup kitchen, and mental health services. But by 1998, St. Christopher was experiencing serious financial difficulties, and closed permanently in 1999 or 2000.
Starting in 2022 the library was gutted in preperation for conversion into 12 apartments.