The former Italy international was a hero to the Biancocelesti faithful but the fans turned against him after he joined the Rossoneri in 2002.
The Lazio supporters felt betrayed by one of their own but Nesta maintains that he agreed to a transfer for the good of the club, who were experiencing serious financial problems at the time.
"I would never have wanted to leave but the club had to raise cash and it all happened in such a hurry that I could not make a decision," the centre-back told Sky Sport Italia. "I deserved more respect.
"Passing me off as a traitor hurt because the truth came out that Lazio was in debt and had to sell.
"I remember that day like it was yesterday. We were training in Formello and Massimo Cragnotti [son of former president Sergio] called us in because someone had to be sold.
"He handed me the phone and told me that I should agree [to a transfer] because Lazio needed money.
"He gave me no time to get my things and I ran to catch the plane to Milan.
"There was sort of presentation at San Siro before a friendly, and at one point I turned around to see [Hernan] Crespo: they had also sold him, to Inter!"
Nesta, who was capped 78 times by his country before retiring from international football after the 2006 World Cup triumph, has won two Champions League crowns and two Serie A titles during his time with Milan.