The following code will reproduce this problem. Note that we are saying NO to useformat. That appears to be what is triggering the problem. Just setting the format and setting the editmask.mask is not a problem.
My code did all of this via a single modify, i split it out so you could see the issue.
ls_mod = "amt.Format='#,##0.00' "
ls_result = dw_1.modify(ls_mod )
ls_mod = "amt.editmask.Mask='###,###,###,###,###.00'"
ls_result = dw_1.modify(ls_mod )
ls_mod = "amt.editmask.UseFormat='no'"
ls_result = dw_1.modify(ls_mod )
at this point the editmask limits user entry to 9,999.99
The work around is to set the useformat FIRST, and then afterwards set the editmask.mask. Then it seems to work ok.
ls_mod = "amt.editmask.Mask='###,###,###,###,###.00'"
ls_result = dw_1.modify(ls_mod )
Occurs in appeon 2015 WEB. Did not occur in 6.6 web. No idea if it occurs anywhere else.