I like Bury St. Edmund's, but I have to admit that part of its attraction to me is things to see. There are, however, a number of nice tea rooms there; or were when last I went. And Britain's smallest pub.
It's a pretty market town. It's fairly prosperous, so there's much less of filling high street shops with betting & charity shops than other places. However, Clare, Lavenham, and Hadleigh are prettier, if smaller and embody the image of rural Suffolk, while Bury St. Edmunds has big employers Greene King and British Sugar, and consequently a ring of modern housing estates.
The coastal road might be a little chilly yet in April, but otherwise, Norfolk is very nice, too. If you re as late as the May bank holiday weekend, I would avoid the coastal road then. It can be very busy bank holidays and it's too narrow an windy to be nice in heavy traffic.
If it's the stops you're after, have look at http://www.norfolkctc.org.uk/?q=caf-s
and plan around them?
There used to be a list for Suffolk somewhere, to, but I can't find it at the moment. But most of the pretty little villages and towns of Suffolk have nice tea rooms. I don't recall ever being disappointed. If you decide to go that way, say so, and I will either find a list, or post my own favourites.
edited to add: you might be able to park in Thetford Forest on a pass card? I haven't checked, but it used to be possible to buy a week's pass that included parking. IIRC the fees for a week's pass were reasonable.
“In some ways, it is easier to be a dissident, for then one is without responsibility.”
― Nelson Mandela, Long Walk to Freedom