Yup. This is why getting an A&P/IA that will work with you is important if you want to do more stuff on your own. The IA's signature at annual basically blesses things that came before that, but it still won't stop somebody else from demanding that something be undone or removed before they'll sign it off. Even, "my FSDO said it was okay" may not work in a different district, even if you have something in writing from the previous FSDO. If you find a good A&P/IA/shop that you like working with, keep them as long as you can.
When I was a teenage lineboy at a US flying club in Europe, our DPE was a German lady whose husband was a PanAm 747 Captain and ex-AF fighter pilot. They had a 1960 Cessna 210, and the airspeed indicator was out of an F-104 that her husband had scrounged when he was in the AF. It had a piece of red tape where the redline should be, which was all it needed to be legal. They always thought it was cool to have that in there, and I always thought it was cool, too. They had a place here in AZ that they had moved back to and I used to see them once in a while, as she was still a DPE operating out of Goodyear airport. The last time I talked to her several years ago she was an angry hen because her new IA had made her take out that ASI and replace it with one that had all of the color arcs, etc. Nobody had bothered her about that ASI for over fifty years, but she finally ran across somebody who wouldn't sign it off unless it had all of the markings on it. I wish I'd had my IA at the time, but I didn't, so couldn't help her.
So, yeah, you're always at the mercy of the next guy. It may not matter at all who said what or when or how at any time in the past. The entire VARMA program is an effort to try to help mitigate that at least from a parts substitution perspective, but it won't help if somebody has a problem with an owner-performed repair, like wiring, or something like that.