Ah, figured it out.
Any tasks that run while trying to connect to the server are throwing this error message.
I have a job that has about 30 tasks in it, takes a minute to run.
By starting the job, then running my program to connect to the server, I see the task ID's are missing, and looking at them, I see they are sequential tasks in my job.
(And manually looking at the times, they look like they were running right at the same time I was trying to connect.)
(I added output of all jobs and task names/IDs on the server in the program once I connect to verify that these tasks are in fact there.)