NetCov is an open-source tool that can be used with Batfish to analyze test coverage for network configurations. Given a set of Batfish queries, it analyzes which lines of configurations has/has not been covered. The result can be used to assess the rigorousness of the test suite and help discover the blind spots. Please refer to our paper for technical details.
NetCov is written in Python and can be used in concert with pybatfish, Batfish's Python API.
NetCov supports coverage tracking for the following Batfish questions:
- Reachability test via traceroute question
- Routing policy evaluation via testRoutePolicies question
- Direct inspection of routing tables via routes and bgpRib questions
- Querying interfaces via interfaceProperties question
NetCov supports configuration syntax of the following vendors:
| Interface | BGP neighbor | Routing Policy | Prefix list | Community list | As-path list | |
|---|---|---|---|---|---|---|
| Cisco | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| Juniper | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| Arista | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| Palo Alto Networks | ✔️ | |||||
| SONiC | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
If you’d like support for additional vendors (which are supported by Batfish but unsupported by NetCov) or currently-unsupported configuration features, let us know via GitHub issue. We'll try to add support, and it would help if you share a configuration example. You may want to use an anonymizer before sharing.
NetCov reports configuration coverage as the percentage of configuration lines that are covered, such as:
NetCov can also report fine-grained coverage results as colored annotations on source configurations (lines in blue are covered, lines in red are not covered):
You can find an example of the coverage report here.
Install NetCov using pip. A virtual environment and Python version 3.7 is recommended.
$ pip install netcovNetCov leverages LCOV to generate HTML report. If you would like to use this feature, install LCOV using Homebrew (MacOS) or apt (Ubuntu):
Install lcov on MacOS using Homebrew:
$ brew install lcovInstall lcov on Ubuntu using apt:
$ sudo apt install lcovNetCov can be used seamlessly with pybatfish. It provides a hooked version of pybatfish APIs that automatically tracks coverage during the execution of supported pybatfish questions.
It takes only two simple steps to measure coverage for your existing pybatfish scripts/notebooks.
- For import, replace pybatfish client session with the one provided by NetCov:
#from pybatfish.client.session import Session
from netcov import NetCovSession as Session- Generate coverage results at the end of your script. To print coverage metrics to the console, use
bf.cov.result(). To generate HTML report, usebf.cov.html_report().
We provide a demo video and an example to help you get started. If you are new to pybatfish, we recommend reading the pybatfish doc first.
Sometimes not all information retrieved by Batfish questions are meant to be tested, for example, when you retrieve all RIB entries but only assert on a subset of them. To help NetCov model coverage more accurately, you can pause coverage tracking and add tested information use a NetCov API:
# pause coverage tracking to avoid over-estimation
bf.cov.pause()
routes = bf.q.routes(nodes="edge-0000").answer().frame()
bf.cov.resume()
# filter RIB entries to test
tested = routes[routes["Network"] == '0.0.0.0/0'].head(1)
# add tested route to coverage trace
bf.cov.add_tested_routes(tested)
bf.cov.result() prints coverage metrics using logging module and writes to stderr by default. To save the coverage report to file, you can customize logger by:
import logging
fh = logging.FileHandler('cov.log')
logging.getLogger('netcov').addHandler(fh)@inproceedings {netcov-nsdi-2023,
author = {Xieyang Xu and Weixin Deng and Ryan Beckett and Ratul Mahajan and David Walker},
title = {Test Coverage for Network Configurations},
booktitle = {20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)},
year = {2023},
isbn = {978-1-939133-33-5},
address = {Boston, MA},
pages = {1717--1732},
url = {https://www.usenix.org/conference/nsdi23/presentation/xu},
publisher = {USENIX Association},
month = apr,
}```