The easiest way that I have found is to use an iframe in the modal body. It is possible to create a form in the modal without an iframe, but that does present a whole set of problems with form tokens, upload tokens etc. You don’t get these problems when using an iframe as the tokens and upload js are already in the iframe document head.
You need to alter your form function to output a form suitable for this - no header or footer for example. Jamroom provides some easy functions to help with that, which makes it incredibly simple to alter your forms to work within an iframe (and you don't even need a modal for that, your iframe could be within the main body of your page).
If you are using a normal Jamroom modal the basics for a modal are summarised here:
https://www.jamroom.net/ultrajam/documentation/code/1120/how-to-use-a-modal-in-a-template