I would have thought you shoudl probably write ( recorded delivery) to the landlord expressing you unhappiness with this behavior and copy in your local council approved landlord scheme (if there is one).

Specify in the letter that you expect him to give you written 24 hour notice

If the you do that and he carries on making unannounced visits you will have a stronger case for termination.

Also if there is more than one tenant who can witness the "unannounced visits" that will also strengthen your case.

Reality is that it would be expensive/ difficult/ time consuming  for landlord to persue you for this through small claims court if you have valid case but you may struggle to get your deposit returned unless he is registered with a tenancy deposit protection scheme?