This past Sunday I had a short flight planned in my M20J. My run-up was normal with all gauges indicating normal and no sign of engine roughness. I took off and climbed to 3500 MSL and leveled off. During the climb everything indicated normally with no sign of engine roughness and all gauges were within normal tolerances. When I leveled off at 3500 MSL I pulled the power back to 23" with no sign of roughness. As soon as I began to pull the prop back the engine got very rough. I immediately went full forward with the prop and the roughness went away. I informed the tower I needed to turn back to the airport due to engine roughness. During the few minutes I had returning to the airport I left the power and prop full forward due to the roughness that occurred. I came in high incase of an engine failure due to the roughness we encountered. On short final, I began to pull the power back and lost the engine completely. I immediately went full forward with the power and the engine started after a few seconds. I landed and taxied off the runway where I stopped and tried to determine what the problem was. Once again, I did an extensive run-up and the plane had no indications of roughness and all gauges indicated normally just moments after the engine quit on final. Two weeks prior to this incedent, I had the plane at the avionics shop getting a new Garmin GTN 650 installed. During that time I received a S/B from AVStar that the fuel servo that had been rebuilt at the last annual needed to be sent in due to numerous planes with the same servo losing complete power during flight. The servo was rebuilt and reinstalled before I picked up the plane. I flew the plane back at 8000 MSL from the avionics shop with no issues. My mechanic is stumped and has spoke with AVStar and they tell him that if a servo is going to fail it will be a complete failure with no possibility of working after the failure. Does anyone have any suggestions on where we should start looking for the problem?