Match summary
The dominant Edinburgh side powered eight points clear of Celtic at the summit of the Scottish Premiership to strengthen their early title credentials and deepen the struggles around the Celtic team.
A superb goal from Kyziridis and Lawrence Shankland's penalty gave Derek McInnes' side a 3-1 lead that the title holders could not overcome.
Dane Murray - among three youth players in the Celtic starting lineup - scored an own goal early on to put Hearts ahead at a bouncing Tynecastle, and it was the 22-year-old's foul that led to the spot-kick.
A stylish finish from McGregor of the season had quickly cancelled out Hearts' opener, and the Swedish attacker almost put Celtic in front, choosing placement over power and allowing Alexander Schwolow to make a crucial stop.
But the unstoppable Hearts side seized control again to hand Celtic to back-to-back losses for the initial occasion since last year.
It also gives the Hearts a commanding advantage at the summit.
The previous occasion they were this far ahead was in 2005-06 under their former manager, when they ended as runners-up - a rare instance that a team has not secured the title after building such an advantage in the last three decades.
McInnes, whose Aberdeen side were second to Celtic on four occasions, has been at pains to play down title talk, as the club seek their maiden league title since the 1960s.
Rodgers has experienced this previously at the club - both perfect and shaky beginnings to a season. He's never ended as runner-up.
But on Sunday's evidence, this Edinburgh outfit showed they need to be taken seriously.
The Edinburgh side started with the self-assurance of a side unbeaten in the league this season - the popular attacker delivered a ball for the striker to fire off target.
Celtic didn't aid their cause with some unusual sloppiness in possession and they buckled under their first real examination.
Claudio Braga's header was touched on to a post by Kasper Schmeichel and the follow-up from Shankland was diverted goalwards by the unfortunate Murray, in for the injured Cameron Carter-Vickers.
Celtic found their rhythm and started building moves. From a flowing attack, Sebastian Tounekti released the full-back to square for McGregor, who slotted home.
Apart from the Nygren chance, Celtic struggled to create much for striker Kenny or his replacement Shin Yamada amid the unavailability of stricken attackers their key strikers.
Hearts rediscovered their energy and adaptability in creating chances in the opposition half. The Greek forward, one of the finds of the club's scouting network, struck low past Schmeichel from the just outside the area.
Almost straight away, the defender tripped Braga in the right side of the box and Shankland converted. And it could've been an more comprehensive victory, had replacement Ageu not been stopped by Schmeichel.
Celtic's second half was somewhat summed up in two incidents involving big-money recruit Engles.
The midfielder's wayward free-kick delivery was in stark contrast to his midweek exploits in the continental victory over Sturm Graz and the Belgium international was then booked for a cynical foul.
The Hearts manager: "You see how hard the players work, the team spirit. That's not sufficient on its own to win games week in, week out.
"What we have got, is skillful players. You're witnessing it from multiple individuals who have been here previously. Recent additions are eager to show that.
"We're in a positive moment at the minute, but it's just one win. It's crucial for us to keep winning. We've got much work ahead, loads to do. After the first two rounds is typically a benchmark of your standing."
Celtic manager Brendan Rodgers: "The second goal was disappointing from our perspective. We didn't move when the ball went back as a line so when they pass the ball, they ought to be caught offside but didn't. We have got to do better with the attempt.
"Plenty of matches to play for. It's only the beginning. It's only nine games in. Clearly, being eight adrift is not what we would want. The positive is there's numerous fixtures to play.
"We now have to focus on the here and now and look to find more consistency and better performance in our play."
The Glasgow club are back at home on Wednesday against promoted side Falkirk (19:45 GMT). At the identical hour in the Scottish Premiership, the Edinburgh outfit are at St Mirren.