ATOMS - Automatic Trivial Oxidized Model Spec (#3361)

This commit introduces ATOMS (Automatic Trivial Oxidized Model Spec), which greatly simplifies model unit tests. Most of the code is from @ytti, @robertcheramy documented it and updated device2yaml.rb. --------- Co-authored-by: Saku Ytti <[email protected]>
ytti · Jan 9, 2025 · 0e19aed · 0e19aed
1 parent aa89265
commit 0e19aed
Show file tree

Hide file tree

Showing 98 changed files with 6,008 additions and 6,424 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,7 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
+
 ### Added
 - junos: add unit test (@systeembeheerder)
 - apc_aos: support for scp (@robertcheramy)
@@ -13,6 +14,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - sonicos: accept policy message. Fixes #3339 (@Steve-M-C, @robertcheramy)
 - input/ssh: change input.debug to dump all characters and include sent commands. (@robertcheramy)
 - cumulus: remove ANSI Escape codes and fix prompt issues. The prompt is more specific now (@alchemyx, @robertcheramy)
+- model unit tests: the tests are automated and simpler to use (@ytti, @robertcheramy)
+- device2yaml.rb: moved to extra/, commands can be specified from the command line or from a file (no cmdsets provided anymore) (@robertcheramy)
 
 ### Fixed
 - tplink: send 'enable' before the enable password. Fixes #3271 (@robertcheramy)

diff --git a/Rakefile b/Rakefile
@@ -74,6 +74,7 @@ task :chmod do
     extra/oxidized.runit
     extra/syslog.rb
     extra/update-ca-certificates.runit
+    extra/device2yaml.rb
   ]
   dirs = []
   %x(git ls-files -z).split("\x0").reject { |f| f.match(/^(test|spec|features)\//) }.each do |file|

diff --git a/docs/Creating-Models.md b/docs/Creating-Models.md
@@ -132,32 +132,16 @@ Intuitively, it is also possible to:
 * Create a completely new model, with a new name, for a new operating system type.
 * Testing/validation of an updated model from the [Oxidized GitHub repo models](https://github.com/ytti/oxidized/tree/master/lib/oxidized/model) by placing an updated model in the proper location without disrupting the gem-supplied model files.
 
-## Create unit tests for the model
-> :warning: model unit tests are still a work in progress and need some polishing.
-
-If you want the model to be integrated into oxidized, you can
-[submit a pull request on github](https://github.com/ytti/oxidized/pulls).
+## Create Unit Tests for the Model
+If you want the model to be integrated into Oxidized, you can
+[submit a pull request on GitHub](https://github.com/ytti/oxidized/pulls).
 This is a greatly appreciated submission, as there are probably other users
 using the same network device as you are.
 
 A good (and optional) practice for submissions is to provide a
-[unit test for your model](/spec/model). This reduces the risk that further
-developments could break it, and facilitates debugging issues without having
-access to a physical network device for the model.
-
-In order to simulate the device in the unit test, you need a
-[YAML simulation file](/examples/device-simulation/), have a look at the
-link for an explanation on how to create one.
-
-Creating the unit test itself is explained in
-[README.md in the model unit test directory](/spec/model/README.md).
-
-Remember - producing a YAML simulation file and/or writing a unit test is
-optional.
-The most value comes from the YAML simulation file. The unit
-test can be written by someone else, but you need access to the device for the
-YAML simulation file. If you encounter problems, open an issue or ask for help
-in your pull request.
+[unit test for your model](/docs/ModelUnitTests.md). This reduces the risk that
+further developments could break it, and facilitates debugging issues without
+having access to a physical network device for the model.
 
 ## Advanced features
 

diff --git a/docs/DeviceSimulation.md b/docs/DeviceSimulation.md
@@ -0,0 +1,184 @@
+# Device Simulation
+Oxidized supports [150+ devices](/docs/Supported-OS-Types.md).
+
+No developer has access to all of these devices, which makes the task of
+maintaining Oxidized difficult:
+
+- Issues can't be resolved because the developer has no access to the device.
+- Further developments can produce regressions.
+
+In order to address this, we can simulate the devices. An example of a
+simulation is the [model unit tests](/spec/model), but one could also simulate a
+device within an SSH server.
+
+The simulation of devices is currently focused on SSH-based devices. This may be
+extended to other inputs like Telnet or FTP in the future.
+
+## YAML Simulation Data
+The underlying data for the simulation is a [YAML](https://yaml.org/) file in
+which we store all relevant information about the device. The most important
+information is the responses to the commands used in the Oxidized models.
+
+The YAML simulation files are stored under
+[/spec/model/data/](/spec/model/data/), with the naming convention
+`<model>:<description>:simulation.yaml`, where `<model>` is the lowercase name
+of the Oxidized model and `<description>` is the name of the test case.
+`<description>` is generally formatted as `<hardware>_<software>` or
+`<hardware>_<software>_<information>`.
+
+### Creating a YAML Simulation File with device2yaml.rb
+A device does not only output the ASCII text we can see in the console.
+It adds ANSI escape codes for nice colors, bold and underline, \r, and so on.
+These are key factors in prompt issues, so they must be represented in the YAML
+file. We use the Ruby string format with interpolations like \r, \e, and so on.
+Another important point is trailing spaces at the end of lines. Some text
+editors automatically remove trailing spaces, so we code them with \x20.
+
+Although a YAML file could be written by hand, this is quite a tedious task to
+catch all the extra codes and code them into YAML. This can be automated with
+the Ruby script [extra/device2yaml.rb](/extra/device2yaml.rb).
+
+`device2yaml.rb` needs Ruby and the gem
+[net-ssh](https://rubygems.org/gems/net-ssh/) to run. On Debian, you can install
+them with `sudo apt install ruby-net-ssh`.
+
+Run `extra/device2yaml.rb`, the online help tells you the options.
+```
+oxidized$ extra/device2yaml.rb
+Missing a host to connect to...
+
+Usages:
+- device2yaml.rb [user@]host -i file [options]
+- device2yaml.rb [user@]host -c "command1
+  command2
+  command3" [options]
+
+-i and -c are mutualy exclusive, one must be specified
+
+[options]:
+    -c, --commands "command list"    specify the commands to be run
+    -i, --input file                 Specify an input file for commands to be run
+    -o, --output file                Specify an output YAML-file
+    -t, --timeout value              Specify the idle timeout beween commands (default: 5 seconds)
+    -e, --exec-mode                  Run ssh in exec mode (without tty)
+    -h, --help                       Print this help
+```
+
+- `[user@]host` specifies the user and host to connect to the device. The
+password will be prompted interactively by the script. If you do not specify a
+user, it will use the user executing the script.
+- The commands that will be run on the device must be defined in
+`deviceyaml.rb`. You can give the commands online with `-c` or read them from a
+file (one line per command) with `-i`. The commands should match exactly the
+ones of the model (no abbreviations) and include the commands of the
+`post_login` and `pre_logout` sections. When using `-c` and editing the shell
+command line, `CTRL-V CTRL-J` is very useful to add a line break.
+- `device2yaml.rb` waits an idle timeout after the last received data
+before sending the next command. The default is 5 seconds. If your device makes
+a longer pause than 5 seconds before or within a command, you will see that the
+output of the command is shortened or slips into the next command in the YAML
+file. You will have to change the idle timeout to a greater value to address
+this.
+- When run without the output argument, `device2yaml.rb` will only print the SSH
+output to the standard output. You must use `-o <model:HW_SW:simulation.yaml>`
+to store the collected data in a YAML file.
+- If your Oxidized model uses SSH exec mode (look for `exec true` in the model),
+you will have to use the option `-e` to run `device2yaml.rb` in SSH exec mode.
+
+Note that `device2yaml.rb` takes some time to run because of the idle timeout of
+(default) 5 seconds between each command. You can press the "Escape" key if you
+know there is no more data to come for the current command (when you see the
+prompt for the next command), and the script will stop waiting and directly
+process the next command.
+
+
+Running the script against an ios device would look like:
+```shell
+extra/device2yaml.rb oxidized@r61 -c "terminal length 0
+terminal width 0
+show version
+show vtp status
+show inventory
+show running-config
+exit" -o spec/model/data/ios:C8200L_16.12.1:simulation.yaml
+```
+### Publishing the YAML Simulation File to Oxidized
+Publishing the YAML simulation file of your device helps maintain Oxidized. This
+task may take some time, and we are very grateful that you take this time for
+the community!
+
+You should pay attention to removing or replacing anything you don't want to
+share with the rest of the world, for example:
+
+- Passwords
+- IP Addresses
+- Serial numbers
+
+You can also shorten the configuration if you want - we don't need 48 times the
+same configuration for each interface, but it doesn't hurt either.
+
+Take your time, this is an important task: after you have uploaded your file on
+GitHub, it may be impossible to remove it.
+You can use search/replace to make consistent and faster changes, for example
+change the hostname everywhere.
+
+The YAML simulation files are stored under
+[/spec/model/data/](/spec/model/data/), with the naming convention
+`<model>:<description>:simulation.yaml`, where `<model>` is the lowercase name
+of the Oxidized model and `<description>` is the name of the test case.
+`<description>` is generally formatted as `<hardware>_<software>` or
+`<hardware>_<software>_<information>`.
+
+Using a correct name for the file is important to ensure it is included in
+automatic model unit tests.
+
+Examples:
+
+- spec/model/data/aoscx:R0X25A-6410_FL.10.10.1100:simulation.yaml
+- spec/model/data/asa:5512_9.12-4-67_single-context:simulation.yaml
+- spec/model/data/ios:C9200L-24P-4G_17.09.04a:simulation.yaml
+
+When you are finished, commit and push to your forked repository on GitHub, and
+submit a Pull Request. Thank you for your help!
+
+### Interactive Mode
+The `device2yaml.rb` script is basic and sometimes needs some help, especially
+when dealing with a device that sends its output page by page and requires you
+to press space for the next page. `device2yaml.rb` does not know how to handle
+this.
+
+While `device2yaml.rb` is running, you can type anything on the keyboard, and it
+will be sent to the remote device. So you can press space or 'n' to get the next
+page.
+
+You can also use this to enter an enable password.
+
+If you press the "Esc" key, `device2yaml.rb` will not wait for the idle timeout
+and will process the next command right away.
+
+### YAML Format
+The YAML file has two sections:
+- init_prompt: describing the lines sent by the device before we can send a
+command. It usually includes MOTD banners and must include the first prompt.
+- commands: the commands the Oxidized model sends to the network device and
+their outputs.
+
+The outputs are multiline and use YAML block scalars (`|`), with the trailing \n
+removed (`-` after `|`). The outputs include the echo of the given command and
+the next prompt. Escape characters are coded in Ruby style (\n, \r...).
+
+Here is a shortened example of a YAML file:
+```yaml
+---
+init_prompt: |-
+  \e[4m\rLAB-R1234_Garderos#\e[m\x20
+commands:
+  show system version: |-
+    show system version
+    grs-gwuz-armel/003_005_068 (Garderos; 2021-04-30 16:19:35)
+    \e[4m\rLAB-R1234_Garderos#\e[m\x20
+# ...
+  exit: ""
+```
+
+
diff --git a/docs/Issues.md b/docs/Issues.md
@@ -49,7 +49,7 @@ contributing code via a pull request (PR) or hiring a developer.
 
 ## Sumbit a YAML Simulation File
 To help developers troubleshoot device-specific issues, you may be asked to submit a
-[YAML simulation file](https://github.com/ytti/oxidized/blob/master/examples/device-simulation/README.md#creating-a-yaml-file-with-device2yamlrb) for your device.
+[YAML simulation file](/docs/DeviceSimulation.md#creating-a-yaml-file-with-device2yamlrb) for your device.
 
 Here's a brief overview how to do it, you can find more details in the link
 above.
@@ -63,22 +63,28 @@ sudo apt install git ruby-net-ssh
 ```
 git clone [email protected]:<your github user>/oxidized.git
 ```
-- run the device2yaml.rb script (you’ll be provided with the command set and
-  output filename to use)
+- run the `extra/device2yaml.rb` script (you’ll be provided with the command to
+run) from the repository root:
+
 ```
-cd oxidized/examples/device-simulation
-# Replace user and devicename to appropriate values
-./device2yaml.rb user@devicename -c cmdsets/ios -o yaml/asr900_26.8.1b.yaml
+extra/device2yaml.rb oxidized@r61 -c "terminal length 0
+terminal width 0
+show version
+show vtp status
+show inventory
+show running-config
+exit" -o spec/model/data/ios:C8200L_16.12.1:simulation.yaml
 ```
+
 - The script waits 5 seconds between commands, and outputs the response of the
   device. You can press "ESC" if you see the prompt and want to pass to next
   command without waiting for the timeout.
-- The result will be stored in `oxidized/examples/device-simulation/yaml/`.
+- The result will be stored in `spec/model/data/`.
 - Replace any sensitive information with placeholder values in the output file.
 - Commit & push the file to github
 ```
-git add yaml/asr900_26.8.1b.yaml
-git commit -m "Device simulation for ASR900"
+git add spec/model/data/ios:C8200L_16.12.1:simulation.yaml
+git commit -m "Device simulation for C8200L"
 git push
 ```
 - Create a pull request (PR) in GitHub, referencing the issue number (e.g.,