I used to do something similar way back in the days of Soundfonts - the specs of the PC I was using then were not powerful enough to work with lots of audio loops directly in my DAW, but soundfont samples were loaded into some separate ram in my soundcard. So I used to make vast banks of drum loops and other samples, and I loved how easy it was to just cycle through a bunch of different samples on the fly. So yeah, I can definitely see the appeal of doing that with the EXS24, although these days I'm just too lazy
The only pitfall really that I remember was that all the samples effectively became "1-shot", in other words I could trigger a sample with a key, but it had to play from the beginning each time. If I wanted to chop up a loop, I would have to first use an audio editor to physically split up the file, then load each piece onto a separate note in the Soundfount patch and then play in the various note combinations as a sequence. But it was quite a cool creative tool though, because I could mess around with the various pieces in real time as the track played, until I liked the result.