Chasing 91 after the match was reduced to eight overs a side due to a wet outfield, Rohit finished with a 20-ball unbeaten 46 while Dinesh Karthik scored the winning runs with four balls to spare after Axar Patel (2-13) did the damage with the ball for the hosts.
India were cruising at 51-1 at the halfway stage, with Adam Zampa briefly halting their momentum by taking out opener KL Rahul (10) before turning the game on its head by dismissing Virat Kohli (11) and Suryakumar Yadav (0) in successive balls.
The hosts never lost control of the chase, despite Zampa's outstanding 3-16 spell, thanks to captain Rohit leading the charge before Karthik hit a four and a six off the only two balls he faced to wrap up the victory.
"I didn't expect it to go quite as well as that," Rohit, who smashed four fours and as many sixes to see India through, said after collecting the Player of the Match award.
"Glad (Karthik) could finish well. It's been a while since he had some time in the middle," he added of the veteran middle-order batsman.
Overnight rain had rendered the outfield waterlogged and although the weather had been clear on Friday, the start of play was postponed by two-and-a-half hours after several pitch inspections.
India elected to bowl first and reduced Australia to 31-3 in 3.1 overs after Cameron Green was run out and Glenn Maxwell fell for a golden duck to Axar, who bowled Tim David in his next over.
Jasprit Bumrah, playing his first match in over two months, marked his return with a searing yorker to dismiss Aaron Finch (31) before Matthew Wade's unbeaten 43 off 20 carried Australia to 90-5 as Harshal Patel conceded 19 runs in the final over.
Australia clinched a four-wicket victory in the series opener in Mohali on Tuesday after chasing down 208 runs with four balls to spare.
The teams will meet in Hyderabad on Sunday for the series decider before South Africa head to India.