This paper presents a novel framework integrating Stackelberg game theory and reinforcement learning for cyber-physical system (CPS) security. A hierarchical game model is developed, in which defenders and attackers interact through sequential decision-making. The defender-attacker dynamics are formulated as an optimization problem combining H₂ and H_∞ control objectives. Key innovations include a unified game-theoretic approach for modeling hybrid attack-defense mechanisms, online reinforcement learning algorithms for real-time strategy adaptation, and rigorous stability analysis using the Lyapunov theory. Theoretical guarantees of convergence are established for the proposed learning scheme. Comprehensive experiments on a robotic platform validate the effectiveness of the framework in maintaining control performance under diverse attack scenarios.