And what topology do you have? The 3 esp are at what distance from each other? Because maybe you only need one that handles all the slaves over I2C and the whole thing is overkill? Because you probably have enough resources in a single esp.
Added after 2 [minutes]:
PS. if you have some longer distances, then pulling a bare UART out into the world is not a very good idea. And if short, you don't need 3 esp
.