What is wrong with my brunfelsia?

I planted 5 brunfelsia plants in autumn last year and they appeared to be growing well. They get decent sun in summer and good lunch/early afternoon sun in winter. In summer they were watered 1-2x a week and winter 1x a fortnight.

2 of them are doing really well. 1 of them hasn’t really grown and I’m not sure why, but hopefully it takes off this spring. 2 others were growing nicely, but now their leaves are browning and falling off. What is wrong with my brunfelsia plants?

Additionally info: I am in Brisbane. It has been a wet autumn and early winter, and I reduced watering during these stages, but we’ve dried off now. I’ve checked the soil and it’s damp, so they are definitely not being under watered. We had a spider mite issue at the beginning of the year and I used white oil to manage it, probably needs another spray soon but it’s pretty much gone. I’ve uploaded a pic of a sad looking one and a happy one. Please help