Try hotmelt glueing them into place first then building up the filler onto that or just hotmelt them into place, cover with speaker cloth and resin/hardener, thats what i'm doing as per this guide:
http://www.talkaudio.co.uk/vbb/showthread.php?t=159311
I like!