You need to make sure that you are feeding enough data to the IP and the output buffer is big enough to receive the result.
Please verify the IP documentation,
https://www.xilinx.com/support/documentation/ip_documentation/rs_encoder/v9_0/pg025_rs_encoder.pdf
I also suggest to run the IP test bench to understand better how it works.
Mario