Tree in a pot,currently fruiting ill knock all those off Tree not strong enough to bear fruit. Just some browning on some leaves,can't see any insects
The most common causes of brown leaves in Kaffir Lime Trees are related to water, either too much or too little, the soil needs to be moist but well drained. In pots it is worth allowing the soil to dry out between watering and making the watering good and deep. Citrus are gross feeders they require a balanced mix of nutrients. The easiest solution is to use a sustained release fertilizer that is specifically for citrus plants.
You are quite correct to knock off the small fruits for the first 2-3 years whilst the tree is growing and the trunk becoming stronger.