Goku, Vegeta, and Broly are anamolies amongst the Saiyans though when you really think about it. I keep revisiting this thought that there are truly no limitations in the Dragon Ball universe only advantages and disadvantages. At base level, among all the races in the universe Saiyans are actually mid-tier -- I'm talking about universe 7 specifically.
Vegeta and Nappa, elites of their race, were looked at like pipsqueaks by other members of Freeza's army which was composed of various alien races. Even Raditz, a veteran member of Frieza's army paled in comparison to an average Namekian warrior. Despite living on a peaceful planet with gravity comparable to Earth's those warrior Namekians managed to be twice as strong as Raditz, someone who was apparently very active in combat. It's possible that those warriors had already engaged in fusion but also very doubtful seeing as how Nail described it as a last resort that was basically taboo in the culture.
We also know that not all Saiyans have or can tap into their S-Cells. Only a rare special breed can. The only transformation every Saiyan has access to is Oozaru a x10 power boost that introduces other vulnerabilities. And zenkais, while greatly beneficial for a combat race, don't seem to typically be as wildly imbalanced as they were on Namek. Normally, they seem to yield much smaller percentages. The zenkais we see from Namek and via Zamasu seem to be dramatic mutations.
We've also seen that each race has many outliers when it comes to overall strength levels. The main cast are composed of the strongest members their respective races, same with the antagonists and villains. So essentially we are seeing mutants fight other mutants. With that in mind it's difficult to say which race possesses what potential. For fighting specifically though, Saiyans seem to have the most advantages from the base level onward but I'm not including potential in that.
I think that those who push past their "limitations" genetic or otherwise, which the entire cast does several times over, will be able to gather large reserves of power no matter what. Humans, for instance, may only have low reserves of power compared to Saiyans because they are a race that aren't accustomed to wielding ki. Maybe some years down the line they will gain that proficiency. Our elite human fighters are all stronger than the classic elites of Saiyan society by a manner of several magnitudes. Tien would mop the floor with King Vegeta for instance despite the king likely having much more combat experience. Though to be fair to King Vegeta, he may have access to S-Cells as well.
There's also racial mysticism to consider as well. When you add that into the mix, Namekians and Saiyans would have the highest potential due to Namekian Fusion and the Super Saiyan God ritual. With a sizeable population and the right conditions, warriors from these two races would naturally be the strongest, though we don't know much about the average capabilities of Frieza's clan. Humans with their sealing abilities and methods of attaining immortality without Dragon Balls could be up there too if they honed in on those methods properly. Then there's human tech which is pretty scary in the grand scheme of things.
I think other than the infinite energy, which may or may not be generated by a mechanical device of some sort, that you could possibly see the androids as a genetic mutation of humans. It's the same base after all and they are altered on a molecular level. They are kind of like the X-Men or even the Captain America-esque super-soliders of Dragon World. It's not natural power but it could be possible with human genetics. I'm not saying that it IS possible, but that it may be a possibility locked inside the human gene.