I think when it comes to RPG books, it’s good to have an eclectic mix of art styles in the illustrations. The early D&D books were aces at this, ranging from the actual cartoons of Tom Wham, to the grounded illustrations of Dave Trampier, to the trippy works of Errol Otus. I think this kind of diversity is important because you never know what will strike a chord and provide inspiration for a game. Books that have a very clear “house style”, like some of the later editions of D&D, have a harder time of this, imo. If it hits, it hits, but if it doesn’t there may not be anything in the whole book that really sparks your imagination.
Now some of the time, like maybe a setting guide or something based on some specific IP like StarWars, setting tight boundaries on the tone and imagery might be exactly what you want. But I think in the general case, you want to have the art run the gamut.
Tales of the Rambling Bumblers 