In this paper, we study the linear quadratic (LQ) optimal control of time-varying difference system with terminal state constraints. The main contribution is to provide the Q-learning algorithm for the optimal controller under the case that the time-varying system matrices and input matrices are both unknown, which consists of learning the solution of the Riccati equation and calculating the specific Lagrange multiplier from the data-driven matrix equation. Different from the existing Q-learning algorithms that mainly focus on unconstrained optimal control problems, the novelty of the proposed algorithm can be applied to handle situations with terminal state constraints. The effectiveness of the proposed Q-learning algorithm is demonstrated through a numerical example.