You can already achieve entering a new instance of a reference attribute with its form in a new tab instead of the default popup dialog which the 'default implementation' resolves to. I don't think you can do it using a modeless window though.
- create a process
..its Input is the parent (owner) object
..its Action is ENTER NEW child WITH child.parent=parent AND VIEW IN TAB
- in the Panel Operations dialog where you select 'Add New Reference', select 'Use custom implementation' and select the process you created at 1.
this inherently passes the instance of 'parent' in context to the process without it having to be specified like it would if you instead started a process (which you could also do to achieve this)
The hurdle you have to overcome is when the instance of parent (owner) is still new/unsaved.
Adding a child in this way (tab) when the parent does not yet exist will obviously result in an orphan record (which does not happen if you use the default popup dialog or inline editing to add new children while the parent is still new/unsaved. in these scenarios the child.parent attribute is updated for each child instance when the parent is saved).
Even if you set the child.parent attribute to be mandatory, Aware erroneously allows saving of the first orphan record (when parent is still new/unsaved), though does give an error message if you try and save any more children via the same form.
BUT, if users open multiple tabs to enter new children, then Aware will erroneously allow the first child entered in the form in each tab to be saved without a child.parent value even if the child.parent attribute is mandatory/required.
Perhaps when the parent form is opened for a new instance of parent, a process using the CREATE action first creates an instance but the object has an attribute which is used as a Flag for whether it is 'live' or not.
VIEW would then be used to load the form of the parent instance in Context from the CREATE action.
New functionality suggestion: as well as 'AND VIEW' and 'AND VIEW IN TAB', it seems logical for 'AND VIEW IN MODELESS' to be possible.