Update return value from EVP_Encode_Update in one error case

Update return value from EVP_Encode_Update in one error case #1660