Passive-aggressive or what? If it was a good friend or family member, the response that you received would be maybe a bit justified. Since he was a co-worker that you saw rarely, you owe him nothing!