Lindsay has been on house arrest, if Lindsay journeyed out of state, or even out of county without her probation officer's permission, that could be a violation. It would not be hard for the District Attorney's office to subpoena the photographer to determine exactly where the shoot took place.
She could also get into trouble if the pictorial shows alcohol. Again, the district attorney's office needs to prove that there is a bottle of wine in the pic and it is not just a prop. If memory serves, Britney Spears got in trouble for appearing next to guns in a photo shoot while she was on probation.
As for the nudity itself, Lindsay has a first amendment right to "express" herself since the case of Schad v. Mt. Ephraim, a famed Supreme court case about nude dancing. Lindsay is supposedly not going to do full nudity in the pictorial, but there would be nothing preventing her from displaying her wares.
Her payout is presumably in the hundreds of thousands, if not millions. The other way she can get in trouble is by not declaring it on her income taxes. One group that will certainly be checking her out...the IRS!