The main ethical issue is potential conflict of interest. Your employer (if they knew about it) might well want to avoid their employees having independent business relationships with the company's clients, because there might come a time when what's best for your day job is different from what's best for your other business. They even more want to avoid employees having undisclosed business relationships.
If there's no conceivable conflict then your employer might agree to you going ahead, but then again they might be paranoid and if you asked them they'd say no even if a conflict is astonishingly unlikely. Deciding for yourself that there's no conceivable conflict that your employer might be worried about is slightly dodgy even if you're 100% correct.
You say you've never worked directly with these clients on behalf of your employer. Fair enough, that reduces the risk of conflict. However, the fact that you consider it significant means you know that if you were working directly with that client then there would be more of a problem. So by working with that client you're creating an imposition on your employer, that if they re-assign you to different clients then they have a problem.
I won't say it's inherently unethical to create potential problems for your employer, or even that it's inherently unethical to create potential problems and keep them secret. If your company has no contractual terms and no policy about additional employment then they've made their own bed -- really they ought to have written down somewhere whether you're allowed to do side-work for their clients, and if they had they'd almost certainly have written "not without permission". So you might think it's fair game to take advantage of their failure. But it's not overtly ethical either, especially if you think that your employer might have a different idea of "reasonable behaviour" from you, and therefore is (however unwisely) assuming you would never do this thing you're about to do.