Abstract The structural dynamics of a machine tool at the tool center point characterizes its vibration response and machining stability which affects productivity. The dynamics are mostly dominated by the spindle-holder-tool assembly whose main vibration mode can change during machining due to centrifugal forces, thermal expansion, and gyroscopic moments generated at high spindle speeds. This paper proposes the identification of the spindle's in-process modal parameters: natural frequency, damping ratio and modal constant, by using a limited number of vibration transmissibility and critical chatter stability measurements. The classical inverse stability solution, which tunes the modal parameters to minimize prediction errors in chatter stability limits, is augmented with vibration transmissibility under two methods: (1) transmissibility-enhanced inverse stability solution: the modal parameters are updated to minimize prediction errors in chatter stability, and vibration transmissibility; (2) artificial neural network (ANN)-integrated inverse stability solution: the ANN uses vibration transmissibility to estimate the natural frequency and damping ratio, such that the inverse stability solution only needs to identify the modal constant. While both methods are experimentally validated, it is shown that the transmissibility-enhanced inverse stability solution is a more effective approach than the time-consuming ANN alternative for the estimation of in-process spindle dynamics.