Silverlight Toolbox GestureListener not working great when developing using Remote Desktop

The Silverlight for Windows Phone Toolkit is working fine in normal environments, great toolkit. However when I started developing WP7 apps in the living room and used a Remote Desktop connection to the development machine I encountered very bad gesture responses in the Windows 7 Emulator. The Flick gesture on the emulator screen didn't do anything (well about 1 out of 100). Doing it again 1 minute later on de development machine without RDS and it worked all the time. Other gestures worked fine. Maybe it's something with the mouse events?

The XAML code I used:

<toolkit:GestureService.GestureListener>
    <toolkit:GestureListener Flick="GestureListener_Flick">
    </toolkit:GestureListener>
</toolkit:GestureService.GestureListener>

Code behind:

   1: if (ViewModel.ViewModelLocator.MainStatic.TossCmd.CanExecute(e))

   2: {

   3:     ViewModel.ViewModelLocator.MainStatic.TossCmd.Execute(e);

   4: }

Setting a breakpoint on line 1 never kicked in.