Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support lz4 files when running UI #67

Open
pjfanning opened this issue Sep 20, 2017 · 2 comments
Open

support lz4 files when running UI #67

pjfanning opened this issue Sep 20, 2017 · 2 comments

Comments

@pjfanning
Copy link

pjfanning commented Sep 20, 2017

I have tried a few combinations when starting the UI

docker run -v /path/to/logs/dir:/logs -p 23763:23763 roboxue/sparklint -d /logs && open localhost:23763

But if I have .lz4 files in the directory, they are not picked up as sources.
I notice there is some code in sparklint that uses the lz4 codec.

@roboxue
Copy link
Contributor

roboxue commented Sep 20, 2017

I've been noticing this for a while. The spark's lz4 format cannot easily unpacked by lz4 unpacker. I'm using this line in my testing new LZ4BlockInputStream(getClass.getClassLoader.getResourceAsStream("application_xxxxxxxx.lz4")), which works fine with standalone log files. I'll try to update the public branch this weekend. Thanks for reporting

@pjfanning
Copy link
Author

Thanks @roboxue - I'm doing something similar with my lz4 logs. It's a nice-to-have feature to let the UI read lz4 files directly but Sparklint is very useful regardless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants