Lyon makes a good point about being more critical with refs in which you have a vested interest in the game. Refs can certainly have an impact in deciding the game. Especially down the stretch or if they call fouls overly aggressively early on.
However, I ref in another sport and I can tell you that it is certainly a thankless job. And, that all sports struggle to bring in new refs. The pay is barely worth the time invested and certainly not worth getting yelled at by moms, dads, grandmas and grandpas. You need a thick skin and you do it for the love of the sport.
That isn't to say that criticism of refs isn't warranted at some times. Basketball has to be one of the toughest sports to ref due to the speed, level of subjectivity in some fouls and the proximity of the spectators and coaches. People would have a much different perspective if they ever gave it a try.