I just had to set some fields in a form opened via GlideModalFormV3, all answers found seems to be outdated or at least not working in scoped application on the madrid release. So I thought I share my findings if somebody else stumbles across that problem.

It seems this was the old way to do it, but apparently in madrid the window object is null in the onLoad Callback Function.

var d = new GlideModalForm("Create Problem", "problem");
d.setOnloadCallback(onLoadCallbackFn);
d.setSysID(-1);
d.render();

function onLoadCallbackFn(oGlideModal) {
  //window is null
  var oModalGForm = window.frames["dialog_frame"].g_form;
}

After some trial and error I found out that the parameter of the callback function has a $window variable, this can be used to find the g_form of the opened ModalForm.

var d = new GlideModalForm('Problem', 'problem');
d.setOnloadCallback(onLoadCallbackFn);
d.setSysID(-1);
d.render();

function onLoadCallbackFn(modalForm) {
  //This works in madrid (at least in a scoped app, didn't test it in global scope)
  var iframe = modalForm.$window.find("#dialog_frame")[0];
  var iframeWindow = iframe.contentWindow ? 
      iframe.contentWindow :
      iframe.contentDocument.defaultView;
  var d_form = iframeWindow.g_form;
}

EDIT:

Just found out if you just want to set field values you can also use "sysparm_query", didn't try it out yet. But I thought this also is worth sharing.

var d = new GlideModalForm("Problem", "problem");
d.addParm("sysparm_query", "short_description=TestSetField"); // populate fields
d.setSysID(-1);
d.render();