I mean, yes - but I feel like you missed the biggest thing here. He had to fight stronger enemies because that was the only way to grow his haki. He was outmatched, but his haki grew throughout the fight. Without the growth from the battle he wouldn't have even had a small chance against Kaido or Big Mom. This wasn't plot armor it was character development. Rayleigh says the only way to get stronger is to fight strong enemies. He needed a strong stepping stone enemy to grow against.