I was able to figure out a solution. I will post it here so others may benefit from it as I saw similar questions asked in the forum.
In my dialog view model, I defined two properties for the commands:
public UICommand OKCommand
public UICommand CancelCommand
Within my view I am using the EventToCommand behavior to execute a command in the view model when a record is double-clicked:
[C#]<dxmvvm:Interaction.Behaviors>
<dxmvvm:EventToCommand EventName="MouseDoubleClick" Command="{Binding SelectNodeCommand}" PassEventArgsToCommand="True" MarkRoutedEventsAsHandled="True">
<dxmvvm:EventToCommand.EventArgsConverter>
<dx:EventArgsToDataRowConverter />
</dxmvvm:EventToCommand.EventArgsConverter>
</dxmvvm:EventToCommand>
</dxmvvm:Interaction.Behaviors>
The command that gets executed is defined as a POCO command and it calls the Close method on the CurrentDialogService and passes the UICommand so the ShowDialog method gets the correct response like this:
[C#]public void SelectNode(StrategyNode node)
{
if (node != null)
CurrentDialogService.Close(OKCommand);
}