Will the panels have a negative impact on cooling performance?

Without question there will be less airflow available because of the lack of ventilation from the side panels but with the correct choice of hardware and placement of upper/lower panel fans, this can be mitigated.