Hardware impairments (HWI) in transceivers have been found to cause severe performance degradation in wireless relay networks. However, literature investigating their impact on networks with imperfect channel estimation (ICE) is limited. In this paper, the impact of HWI on channel estimation error (CEE) and their combined impact on the performance of amplify-and-forward variable-gain (AF-VG) two-way relay network (TWRN) and one-way relay network (OWRN) have been investigated. The expressions for equivalent signal-to-noise plus distortion-and-error ratio (SNDER) and a closed-form expression for the exact analysis of outage probabilities (OP) are derived by taking a Rayleigh block fading channel. In addition, the combined impacts of HWI and CEE on outage performance of TWRN and OWRN systems for low and high transmission rates are evaluated. For further insights, we perform asymptotic analysis of OP and observe irreducible outage floors which act as fundamental performance limits. This SNDER ceiling acts as an upper limit for practical transmission rates. From this we infer that at high transmit powers, HWI and CEE are largely responsible in elevating the outage floors. Finally, the derived results are validated using Monte Carlo simulations.