Support for Jupyter notebooks in excess of 1 MB

[hansec]

At present it appears as though linguist on GitHub handles large files (> 1MB) by reporting their byte count (or something similar) en lieu of their actual line count. For my repo https://github.com/hansec/OpenFUSIONToolkit, this manifests as the primary language being reported as "Jupyter Notebook", when the number of lines of code of this type is significantly less than the primary language (Fortran).

When running linguist on the repo in a container I find only two files with non-zero line counts. Counting individually, these total totaling 1,504 lines of code (as reported by linguist). However, when linguist is run on the full repo it reports 6677629 lines of Jupyter code. This is comparable too, but greater than, the total size of all Jupyter files at 6260766 bytes.

Jupyter Notebook:
src/examples/Marklin/cylinder/Marklin_ex1.ipynb
src/examples/TokaMaker/ITER/ITER_baseline_ex.ipynb
src/examples/TokaMaker/ITER/ITER_mesh_ex.ipynb
src/examples/TokaMaker/fixed_boundary/fixed_boundary_ex1.ipynb
src/examples/TokaMaker/fixed_boundary/fixed_boundary_ex2.ipynb

57.75%  6677629    Jupyter Notebook
36.04%  4167016    Fortran
4.46%   515698     Python
0.62%   72091      CSS
0.38%   44332      CMake
0.29%   33881      C
0.19%   21918      Makefile
0.14%   16676      C++
0.11%   13267      HTML
0.00%   520        Shell

.../src/examples/Marklin/cylinder/Marklin_ex1.ipynb: 594 lines (594 sloc)
  type:      Text
  mime type: text/plain
  language:  Jupyter Notebook

.../src/examples/TokaMaker/ITER/ITER_baseline_ex.ipynb: 460 lines (460 sloc)
  type:      Text
  mime type: text/plain
  language:  Jupyter Notebook

.../src/examples/TokaMaker/ITER/ITER_mesh_ex.ipynb: 0 lines (0 sloc)
  type:      Text
  mime type: text/plain
  language:  Jupyter Notebook
  blob is too large to be shown

.../src/examples/TokaMaker/fixed_boundary/fixed_boundary_ex1.ipynb: 0 lines (0 sloc)
  type:      Text
  mime type: text/plain
  language:  Jupyter Notebook
  blob is too large to be shown

.../src/examples/TokaMaker/fixed_boundary/fixed_boundary_ex2.ipynb: 0 lines (0 sloc)
  type:      Text
  mime type: text/plain
  language:  Jupyter Notebook
  blob is too large to be shown

It appears as though there is a 1 MB limit for notebooks and other files being handled in a standard way. While I certainly understand a file size limit, 1 MB seems particularly low for Jupyter notebooks that may include one or more images.

[lildude]

At present it appears as though linguist on GitHub handles large files (> 1MB) by reporting their byte count (or something similar) en lieu of their actual line count.

Correct. This is documented:

The percentages are calculated based on the bytes of code for each language as reported by the List Languages API.

As for:

However, when linguist is run on the full repo it reports 6677629 lines of Jupyter code.

The output is showing bytes in the second column, not lines.

As for linguist not supporting larger Jupyter files, it certainly does. If it didn't, you wouldn't get a language here and the figure in the sidebar and analysis would be significantly lower:

.../src/examples/TokaMaker/fixed_boundary/fixed_boundary_ex2.ipynb: 0 lines (0 sloc)
  type:      Text
  mime type: text/plain
  language:  Jupyter Notebook <=== ‼️ HERE
  blob is too large to be shown

The line you see below it isn't an indication of a lack of support and also doesn't apply for things like Jupyter notebooks which are rendered by a completely independent service. This information only applies to files rendered directly in GitHub's UI without any processing, like plain text python or Fortran files.

So what you are seeing is expected and correct behaviour. If you want to change this to better suit your desires, you can implement an override.

[hansec]

@lildude thanks for the quick response!

I certainly do not think that large Jupyter notebooks are not supported. My issue is that 3,712 lines of code are reported as 6,677,629 lines (an 179,893% increase). Assuming 1 byte per line (or something similar) seems pretty low.

Perhaps an improvement to only count \n characters (or linux/windows equivalent) can be provided? Separately perhaps the language count can be updated via a GitHub action? If not, perhaps an increase from a 1 MB limit to a few MB would mitigate for most cases? I realize these may be significant efforts, so just want to surface this issue.

[lildude]

I certainly do not think that large Jupyter notebooks are not supported. My issue is that 3,712 lines of code are reported as 6,677,629 lines (an 179,893% increase). Assuming 1 byte per line (or something similar) seems pretty low.

You're not interpretting the output correctly. As I've mentioned before, it's not reporting 3,712 lines as 6,677,629 lines, it's reporting the total bytes as 6,677,629 bytes. The lines of code have nothing to do with this. You can confirm this independently of Linguist using du:

➜ github-linguist OpenFUSIONToolkit/
58.37%  6677629    Jupyter Notebook
35.29%  4038050    Fortran
4.55%   520080     Python
0.63%   72091      CSS
0.40%   45203      CMake
0.30%   33881      C
0.19%   21918      Makefile
0.15%   16676      C++
0.13%   15067      HTML
0.00%   520        Shell
➜ du -cb OpenFUSIONToolkit/**/*.ipynb | tail -1
6677629	total

Note that the figures match.

Perhaps an improvement to only count \n characters (or linux/windows equivalent) can be provided?

I don't see the need for this. Lines of code has nothing to do with how Linguist counts and reports the language usage. The lines and sloc are only provided as a convenience for the GitHub UI. Linguist doesn't use it at all for language analysis or percentage determination.

Separately perhaps the language count can be updated via a GitHub action?

There's no need. The language stats are currently updated and cached whenever a push is made to the repo (documented). Anything more frequent than this would be pointless and a waste of resources as there would be no changes.

If not, perhaps an increase from a 1 MB limit to a few MB would mitigate for most cases?

Mitigate what? This is not a Linguist enforced limit. GitHub's UI will not render a file over 1MB in size and instead presents the message "Sorry about that, but we can’t show files that are this big right now." As the UI isn't going to render the file, there's no need to show the lines and sloc values so Linguist doesn't even attempt to determine this, once again, because it's an unnecessary waste of resources.

[hansec]

Gotcha, I was under the impression that the GitHub language breakdown, and thus linguist, was based on lines of code not file size. Sorry I didn't catch that on the first reply.

In my mind it would be more useful for the language breakdown to indicate percentages based on lines of code vs file size, but that is a whole different discussion for probably a different setting.

Thank you again for your thorough responses to my dense question!