Hi,
I hope you've gotten this issue resolved already, but in case you haven't: I had a similar problem at first, but resizing the images in my image editor (I use SnagIt) fixed the problem. (I use PNG format, by the way.) I haven't tried resizing images in HelpStudio, but maybe that's the problem.
I also started using the Replace with Thumbnail widget, so people using the .chm version of the help file can still access the larger versions of the images, and that works nicely. To use this widget, in Full Page Edit mode, right-click the image and select Replace with Thumbnail... Then in the compiled help file, there's a little magnifying glass with text that reads Click to enlarge, or if it's already enlarged, it says Click to shrink.
Hope this helps!
Kim Nathans