An example to make this clear:
Let say you specify folder c:/test in the File delete Task with option "delete empty sub folders".
c:/test has two sub folders:
c:/test/folder1 <- this is empty
c:/test/folder2 <- this contains a file that matches the file filter
Now, the file filter will only find the file in folder2. And after deletion of that file it will notice that folder2 is empty. Now, it deletes folder two.
Folder1 will never be found and excluded because it did not match a file with that file filter.
What you might request is a "Delete folder" Task which you can just specify a start folder on and a _folder mask_ for. * for delete all.