CLI Options Reference for v3.1.1 #27

AyanSinhaMahapatra · 2019-10-15T04:00:19Z

This PR adds to Scancode-Toolkit documentation the Command Line Interface Options reference pages.

Documentation Style is enforced by using Doc8 and Doc Contribution Guide is updated accordingly. 300+ Style errors fixed. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

This commit adds to the Scancode Version 3.1.1 doc branch -> 2 main sample tutorials, How to Run a Scan, and How to Visualize Scan Results. It also makes changes to How to Format Scan output and How to Set what will be Detected in a Scan to be compatible with Scancode version 3.1.1, and adds images/gifs (May be updated later). Also adds minor links/referances. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

This commit Adds minor changes to Doc Version 3.1.1 which doesn't belong to CLI options or the Basic Tutorials. CLI options are to be added next. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

This commit Adds to the Scancode Doc 3.1.1 rst-snippets, which contains major sections in CLI options, to be included in list-options.rst(also added in this commit) and other major CLI Options pages. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

This commit adds to Scancode Doc version 3.1.1 a Synopsis page, Getting Help From Command Line Page, Includes the How to Perform a scan Tutorial as an example, and adds Gifs/Images (some outdated ones will be replaced later). Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

This commit adds to the Scancode Version 3.1.1 docs, A page detailing all the output formats. Also modifies rst_snippets related to the custom_output format case. (An Error present, marked [ToDo], will be solved) Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

This commit adds to Scancode Doc Version 3.1.1, pages for Core Options, Output Filters and OutputControl, Post Scan Options, and Pre Scan Options Pages. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

This commit adds minor changes to contrib_doc.rst, and at cli-referance/, adds changes to synopsis, index and help-text-options. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

pombredanne

Thank you! I added a first set of review comments for your consideration.

pombredanne · 2019-10-15T09:02:43Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+`Core` Options
+==============
+
+.. include::  /scancode-toolkit/rst_snippets/basic_options.rst


Is it normal to have absolute slash-prefixed paths there?

It is not. Even doc8 gives an error when checking for doc standards here. I thought it'd be better if all the snippets are in one location. If you prefer otherwise I'll change that.
[Note]
Some snippets are used cross-section but a majority of them are not. For example,
rst_snippets/custom_output_format.rst is included both in cli-referance/output-format.rst and tutorials/how_to_format_scan_output.rst. What should I do in these cases?

I don't think there is a problem with the location of the snippets, per-say. I think @pombredanne just means we should use relative paths if possible? Not sure how sphinx is configured in this case, but changing to ./scancode-toolkit/rst_snippets/basic_options.rst might be enough here?

That'd give an error InputError: [Errno 2] No such file or directory: 'source/scancode-toolkit/cli-reference/scancode-toolkit/rst_snippets/post_scan_options.rst'. as in sphinx using ./ relates to the rst_snippets being located in the same directory as the file it's being included in. Right?

pombredanne · 2019-10-15T09:04:43Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+
+.. include::  /scancode-toolkit/rst_snippets/core_options.rst
+
+``--generated`` Options


That's great doc, but as I am reading this I wonder if this should not be included in the --info option rather than be a separate option. (this is just a remark... but your opinion is welcomed as a future evolution of the scancode CLI user experience)

This is an interesting point. Most other options are in some way dependent on other options, (You've also asked -> Should not --max-email be side-by-side with the -e option?) like the --summary option is dependent on --info and has some dependent options of its own. Rather than "including" them in each other, I think they should be kept in separate sections as it is presently. What can be done to better show these dependencies is I could add Notes on every option who they are dependent on and what other options are dependent.

@pombredanne it probably should be included in the --info, but this is a scancode-toolkit specific issue I think.

Another question, looks like --generated is independent of --info, and even without -i it works normally without errors. Should this dependency be pointed out?
In other cases, I'm going with notes like :

The option --max-email is a sub-option of and requires the option(s) --email

pombredanne · 2019-10-15T09:05:33Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+    ./scancode -clpieu --json-pp sample_generated.json samples --generated
+
+In the results, for each file the following attribute is added with it's corresponding True/False
+value ::


May be True/False should be lowercase since the JSON output is true/false?

Yes. Corrected this. Should I push a single commit with all requested changes?

@AyanSinhaMahapatra Yes

This is updated.

pombredanne · 2019-10-15T09:06:26Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+
+    "is_generated": true
+
+For the samples folder, the following files have a true value for their is_generated attribute::


Do you think it is a good idea to keep samples in the tool itself? I was willing to take the sample files out entirely and have a separate download for this.

It's is a good question. I don't think it's done normally that this kind of sample folders are included with the software itself. Separating the sample folder and having a separate download will definitely be a better practice I think. The sample folder size is about ~1.2 MB, so it doesn't make a significant difference in the download size though. Even if you keep the folder inside scancode, having a separate download link will be great (perhaps even with multiple bigger samples) !

I agree with separating out the samples - they are very old and not particularly interesting.

Yes, If you could put together better and more interesting samples that'd be great.

pombredanne · 2019-10-15T09:07:13Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+..
+    [ToDo] Research and Write Better
+
+``--max-email`` Options


Should not this be side-by-side with the --email option? (same for the --url option below?)

I've followed the groupings and order of the options in the help-text as users can easily switch between both without having to search. The present grouping (and in the help-text) is, in my opinion, the most logical one. What can be done is adding Notes to --max-email at -e and the same for the email. I'll do that.

This issue has been fixed.

pombredanne · 2019-10-15T09:12:03Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+Some important INTEGER values of the ``--license-score INTEGER`` option:
+
+- **0**     - Default and Lowest Value, All matches are reported.
+- **100**    - Highest Value, Only licences with a much better match are reported


We probably need a section on scoring elements for license matches.

Yeah, I too feel this needs elaboration, and so I've left a ToDo here. But I'd need your/other developers guidance on that. That being said, I still have some unpushed work, and some improvements to make. After that, I was thinking we should mark sections for improvement, and then you/others could guide me/discuss together what can be done.

Yes, this can be added later and expanded by other developers/users. We can track this with a ticket for now: #28

@MaJuRG There are some more issues I need help on, along with this one. They all are marked as [ToDo] in the .rst files in their respective locations (Not all [ToDo]s, some of them I need help on). Should I open more tickets for them?

pombredanne · 2019-10-15T09:15:20Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+``--license-text`` Options
+--------------------------
+
+The ``--license-text`` option includes in the scan result, the matched text for the detected


This is a general comment on the way to craft some sentences:
Here

The --license-text option includes in the scan result, the matched text for the detected
license, under the attribute "matched text".

may be easier to read as:

With the --license-text option, the scan results attribute "matched text" includes the matched text for the detected
license.

@DennisClark what do you think? I am not a native speaker of English!

This is really an important issue. I'll look into this and correct stuff all across the docs. Suggestions and help on this is much appreciated.

Or something like "--license-text: Include the matched licensed text in the scan results for each detected license."

After a join chat today, no major change needed, though it is better to use direct vs. indirect style as general guideline.

pombredanne · 2019-10-15T09:18:14Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+
+A scan example using the ``--license-url-template TEXT`` option ::
+
+    ./scancode -clpieu --json-pp sample_lic_url_template.json samples --license-url-template https://opensource.org/licenses/{}


That's a great example but the likeliness that a ScanCode TK license key is the same as a the opensource.org key is super low IMHO.
Could you instead use an example using this base URL https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses ? and that would point to the .LICENSE file?

You are right. I'll update this.

This one is updated.

pombredanne · 2019-10-15T09:19:25Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+
+    "reference_url": "https://opensource.org/licenses/zlib",
+
+The referance URL changes for all detected licenses in the scan, across the scan result file.


nit: referance -> reference ? see also this PR title

Thanks for pointing out, I'll resolve all spelling/sentence structure stuff ASAP.

pombredanne · 2019-10-15T09:20:59Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+causes the following differance in scan result of the file
+``samples/JGroups/licenses/bouncycastle.txt``.
+
+Without Diagnostics::


There is another important thing too is that whole line of text are included (including possibly unmatched parts) by default unless you request license-text-diagnostics

Updated this part.

pombredanne · 2019-10-15T09:25:30Z

I also see that you added a lot of screen shots. When this is for JSON, it might be better to use plain text instead (and we likely could configure Sphinx to provide some coloring later?) ?

AyanSinhaMahapatra · 2019-10-15T19:13:26Z

I was using screenshots because in some cases it is easier to hide parts of the text and show the overall structure, but yeah using plain-text and then configuring sounds better! I'll update accordingly.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

steven-esser · 2019-10-17T00:19:27Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+`Core` Options
+==============
+
+.. include::  /scancode-toolkit/rst_snippets/basic_options.rst


I don't think there is a problem with the location of the snippets, per-say. I think @pombredanne just means we should use relative paths if possible? Not sure how sphinx is configured in this case, but changing to ./scancode-toolkit/rst_snippets/basic_options.rst might be enough here?

steven-esser · 2019-10-17T00:21:17Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+
+.. include::  /scancode-toolkit/rst_snippets/core_options.rst
+
+``--generated`` Options


@pombredanne it probably should be included in the --info, but this is a scancode-toolkit specific issue I think.

steven-esser · 2019-10-17T00:22:26Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+    ./scancode -clpieu --json-pp sample_generated.json samples --generated
+
+In the results, for each file the following attribute is added with it's corresponding True/False
+value ::


@AyanSinhaMahapatra Yes

steven-esser · 2019-10-17T00:31:54Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+Some important INTEGER values of the ``--license-score INTEGER`` option:
+
+- **0**     - Default and Lowest Value, All matches are reported.
+- **100**    - Highest Value, Only licences with a much better match are reported


Yes, this can be added later and expanded by other developers/users. We can track this with a ticket for now: #28

steven-esser · 2019-10-17T00:34:05Z

docs/source/scancode-toolkit/cli-reference/core-options.rst

+``--license-text`` Options
+--------------------------
+
+The ``--license-text`` option includes in the scan result, the matched text for the detected


Or something like "--license-text: Include the matched licensed text in the scan results for each detected license."

steven-esser · 2019-10-17T00:36:09Z

docs/source/scancode-toolkit/tutorials/how_to_visualize_scan_results.rst

+
+.. image:: data/filter_sample.gif
+
+Componets


Spelling error, this should be Components

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra · 2019-10-18T00:11:53Z

Pushed some major changes on Options dependency. I'm still working on resolving some issues (Some are solved, I've pointed them out above in individual threads). I'll ping on Gitter channels and re-request reviews when I'm done, possibly today.

AyanSinhaMahapatra · 2019-10-18T00:28:06Z

I also think that these Issues:

And these PRs:

are related to Scancode-Toolkit docs and we can close all of them as they are inactive and redundant at this point.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra · 2019-10-20T03:45:49Z

@pombredanne @mjherzog @MaJuRG @DennisClark I've pushed fixes on changes requested and resolved all the issues discussed above. Please review.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra · 2019-10-20T04:02:44Z

Another Minor thing, you'd have to go to https://readthedocs.org/dashboard/aboutcode/advanced/, then Default settings -> Requirements file and enter "docs/requirements.txt" for proper ReadTheDocs builds. There is some sphinx version-specific behavior that changes builds, thus this addition. Also, you can head to https://aboutcode-test.readthedocs.io/en/v3.1.1/ to check out the docs.

steven-esser

There is a lot here, as there should be when migrating docs from another source. From a glance, it looks like everything made it over correctly. I would think that this is good enough to merge as-is and more specific corrections and improvements can be handled in individual smaller tickets if needed.

@DennisClark and @pombredanne, thoughts?

DennisClark · 2019-10-21T17:28:48Z

@MaJuRG I agree completely with your assessment, thanks.

AyanSinhaMahapatra added 16 commits September 17, 2019 01:00

Documentation Style Enforced for Scancode-Toolkit

b049637

Documentation Style is enforced by using Doc8 and Doc Contribution Guide is updated accordingly. 300+ Style errors fixed. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds Minor Changes

311b7be

This commit Adds minor changes to Doc Version 3.1.1 which doesn't belong to CLI options or the Basic Tutorials. CLI options are to be added next. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds Core Options, Output Filters/Control, Post/Pre Scan Options Pages

a342a43

This commit adds to Scancode Doc Version 3.1.1, pages for Core Options, Output Filters and OutputControl, Post Scan Options, and Pre Scan Options Pages. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Sets Version 3.1.1 to Doc8 standards Compliant

7821ccd

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds RST Snippets and Image Data For Scancode-Toolkit CLI

ecc8f83

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds Minor Changes to Scancode CLI Options

2b1b00b

This commit adds minor changes to contrib_doc.rst, and at cli-referance/, adds changes to synopsis, index and help-text-options. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds Core Options Page to CLI referance

70bda0e

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds Pre-Scan CLI Options Referance

1001b58

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds Post-Scan CLI Options Referance

395debb

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds Output Filters and Control page to CLI referance

af0e85b

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds Output Formats in CLI referance

084f8a9

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Merge branch 'master' into v3.1.1

0c91358

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

pombredanne requested changes Oct 15, 2019

View reviewed changes

mjherzog changed the title ~~CLI Options Referance for v3.1.1~~ CLI Options Reference for v3.1.1 Oct 15, 2019

Fix minor Style Problems and Update License Template Option

948eace

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

steven-esser reviewed Oct 17, 2019

View reviewed changes

AyanSinhaMahapatra added 2 commits October 18, 2019 05:35

Add modifications to rst_snippets

11ac684

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add Options dependency support and other Modifications/Fixes

012fc0f

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra added 4 commits October 20, 2019 03:31

Add JSON Model Snippets and Delete Screenshots

4524ed2

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add minor fixes

aa74646

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Fix Spelling and Grammer Errors

50da75b

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Enforce doc standards

3942d68

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra requested review from mjherzog, pombredanne and steven-esser October 20, 2019 03:42

Add requirements.txt file for RTD build

810dd0b

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

steven-esser reviewed Oct 21, 2019

View reviewed changes

pombredanne approved these changes Oct 22, 2019

View reviewed changes

DennisClark merged commit f2b25dc into aboutcode-org:master Oct 22, 2019

This was referenced Oct 24, 2019

Scan command shows error #23

Closed

modify outdated command of how_to_scan #24

Closed

updated How_to_scan in this Doc #21

Closed

AyanSinhaMahapatra deleted the v3.1.1 branch November 1, 2019 16:50

AyanSinhaMahapatra mentioned this pull request Nov 2, 2019

Improve examples aboutcode-org/scancode-toolkit#1031

Open


		.. include:: /scancode-toolkit/rst_snippets/core_options.rst

		``--generated`` Options


		"is_generated": true

		For the samples folder, the following files have a true value for their is_generated attribute::


		A scan example using the ``--license-url-template TEXT`` option ::

		./scancode -clpieu --json-pp sample_lic_url_template.json samples --license-url-template https://opensource.org/licenses/{}


		"reference_url": "https://opensource.org/licenses/zlib",

		The referance URL changes for all detected licenses in the scan, across the scan result file.

CLI Options Reference for v3.1.1 #27

CLI Options Reference for v3.1.1 #27

Conversation

AyanSinhaMahapatra commented Oct 15, 2019

pombredanne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AyanSinhaMahapatra Oct 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pombredanne commented Oct 15, 2019

AyanSinhaMahapatra commented Oct 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AyanSinhaMahapatra commented Oct 18, 2019 • edited Loading

AyanSinhaMahapatra commented Oct 18, 2019

AyanSinhaMahapatra commented Oct 20, 2019

AyanSinhaMahapatra commented Oct 20, 2019 • edited Loading

steven-esser left a comment

Choose a reason for hiding this comment

DennisClark commented Oct 21, 2019

AyanSinhaMahapatra Oct 18, 2019 •

edited

Loading

AyanSinhaMahapatra commented Oct 18, 2019 •

edited

Loading

AyanSinhaMahapatra commented Oct 20, 2019 •

edited

Loading