Peter,
XSLT mapping consumes lots of Memory and I think you are getting the error due to memory issue at run time (As you mentioned you were able to reprocess the message without any issue). Check your System Logs you will find some evidence of same.
There are few Notes to be considered to get rid of issue:Note 1484825 - Mapping Pool for XSLT messages consumes lots of memory
Thanks,
Sunil Singh
