Following the death of his brother Charles Joseph M. Spencer then took charge of the woolen mill pirchased by his father in 1832. In a few years he established an immense business, people coming with wool to be carded or spun from all sections of the country. Sometimes, for weeks, the mill would run night and day. In latter years, this mill passed into other hands, and was finally destroyed by fire.

The Spencer brothers also owned, and operated a saw mill at the same place. Later on, Joseph M. Spencer built a large grist mill which is still [1897] in operation.