ReLU networks have the nice property of being piecewise linear, but also during training they optimise their own non-linear transformation over time.