Wrong switch for wiring in matrix...

Maybe using relay, see this "pulse" circuit, when the switch is closed (and in your case light the LED) the relay send a momentary press for controller.

http://desktopaviator.tripod.com/Articles/USB_Switches/index3_files/image013.gif
http://desktopaviator.tripod.com/Articles/sch.gif

http://desktopaviator.tripod.com/Articles/USB_Switches/index3.htm