Matlab provides its own version of some libraries and redefines the environment variables. When you run Metafor with the system command, the Matlab PATH is used and Metafor crashes because it uses incompatible libraries, at least in Linux.
One way to overcome the problem is to temporarily redefine the Matlab path before running Metafor :
matlabPath = getenv( 'LD_LIBRARY_PATH' );
setenv( 'LD_LIBRARY_PATH', getenv('PATH') )
system( '/home/user/Metafor/release/bin/Metafor -nogui -run cont2' );
setenv( 'LD_LIBRARY_PATH', matlabPath )
Feel free to give a less dirty solution if you have one.