In 1660 he published his Optica Promota, in which the compact reflecting telescope known by his name, the Gregorian telescope, is described. The telescope design attracted the attention of several people in the scientific establishment: Robert Hooke, the Oxford physicist who eventually built the telescope, Sir Robert Moray, polymath and founding member of the Royal Society and Isaac Newton, who was at work on a similar project of his own. Later, Gregory, who was and enthusiastic supporter of Newton, carried on much friendly correspondence with him and incorporated his ideas into his own teaching, ideas which at that time were controversial and considered quite revolutionary.
In 1667 he issued his Vera Circuli et Hyperbolae Quadratura, in which he showed how the areas of the circle and hyperbola could be obtained in the form of infinite convergent series. This work contains a remarkable geometrical proposition to the effect that the ratio of the area of any arbitrary sector of a circle to that of the inscribed or circumscribed regular polygons is not expressible by a finite number of terms. Hence he inferred that the quadrature of the circle was impossible; this was accepted by Montucla, but it is not conclusive, for it is conceivable that some particular sector might be squared, and this particular sector might be the whole circle. This book contains also the earliest enunciation of the expansions in series of sin x, cos x, sin^(-1) x or arc sin x, and cos^(-1) x or arc cos x. It was reprinted in 1668 with an appendix, Geometriae Pars, in which Gregory explained how the volumes of solids of revolution could be determined.
In 1671, or perhaps earlier, he established the theorem that , the result being true only if θ lie between -(1/4)π and (1/4)π. This is the theorem on which many of the subsequent calculations of approximations to the numerical value of &pi have been based.