I had the exact same problems with sunflowers this year. I did finally get 4 full size flowers in September.
I think you are correct and your soil needs serious amending. A good place to start is a soil test. I'd suggest using Logan Labs because with those results, you can read and make the correct amendment mix for your soil, instead of a sort of broad, might work, no micro-nutrients fix that state universities do. I've made posts about how to do this:
https://steemit.com/gardening/@goldenoakfarm/reading-a-soil-test-part-1
https://steemit.com/gardening/@goldenoakfarm/how-to-read-a-soil-test-part-ii
Hope these help your garden be much improved next year!