The logs (with debug on) showed the task would successfully download multiple files in one go, but the Result.DownloadedFiles variable used by the next task would only contain one file name, or sometimes none. It kept causing the job to fail on me, I would rather not start looking at that route again.
What about dependencies? Could they be used to control when the FTP Remote File Trigger is looking at the FTP.