The rule says to put them back after you finish moving, not at the end of any one specific Movement phase or player or game turn.
We play that the tree is moved out of the way. Once the unit is no longer occupying the spot that the tree came from, it is moved back.
Of course, having magnets on the trees and on the forest base helps with this quite a bit!

See it just your rubbish terrain! I told you that.

@ LH, we never fight, no idea where you got that from

I have always played it that way. Infact we still do. But I was always concerned at what point the trees went back. It made sense the way we were doing it. The debate only came about out of a general conversation.
But I still think my take on the assault into the ruin is right. You can't just have a model floating or balancing unrealistically in mid air.

Your right, you can't. That's why you place it were you can on another level and make it clear who is in assault with who.

I hope I haven't stumbled into a private argument here!
Take a look at the following diagram;

Both units "B" and "C" are outside of a ruin.
Both units have to make difficult terrain tests to move into the ruins.
If there is an enemy unit on the first level of the ruin closest to the edge where "B" and "C" are, and both "B" and "C" rolled a '6' on their diffidult terrrain test;
"B" could not assault because moving 3" to the ruin and 3" vertically would still leave it outside the ruin.
"C" could assault because it can move into the ruin and up 3" to the first level.
But "C" can't be physically placed there, so it would be okay to place it on the ground level noting that it is in base to base with the enemy.

No argument TW, we are just trying to get it right.

I agree with your description.