HI,
I think the problem is in the way the encoder works. When you turn the shaft one way you get the equivalent of a switch being pressed. Turn it the other way and you get a different switch pressed. Push down on the shaft and you get the third switch pressed.
Your problem is that all three functions are separate.
Your encoder gets a clock pulse from somewhere in to the CLK pin. When this is supplied the DT (data) pin is read. If it's logic high then the shaft was turned one way, if logic low then the shaft was turned the other way.
SW is the output when the switch is pushed and may be either logic high or low.
In something like an arduino you'll have to monitor the SW output. When it is activated, you then send a CLK pulse. Then read the DT output to see if it was turned left or right.
Based on that information you would then increase or decrease the lighting level.
I haven't had a lot of experience with encoders, just the basic functions so someone else may have a better idea of how to make it do what you want.
Good luck anyway.
Andy
PS In my pit I just used an analogue circuit with a pot to alter the voltage to the instrument lighting led bus.