This tool helps to generate and layout cloud client libraries. Synthtool runs the GAPIC (Generated API Client) Generator via Google API Artifact Manager (artman).
-
Linux This tool runs on Linux only. No other platforms are supported.
-
Python 3.6 Either install it from python.org or use pyenv to get 3.6.
-
Bazel can be downloaded from bazel.build.
-
Docker Some synth.py files require Docker to generate code.
-
Clone this repository and install this library with pip:
cd synthtool python3 -m pip install -e .
To start the process of generation, clone the destination repository.
git clone [email protected]:googleapis/python-tasks.git
cd python-tasks/
If a synth.py
script is not present, create a new one.
You can create one from scratch or copy one from another library.
Run synthtool
:
python3 -m synthtool
After synthtool
runs successfully:
- Investigate the changes it made
- Run the library tests
- Commit and push the changes to a branch and open a Pull Request
Find examples below in different programming languages (Cloud Tasks API used as an example).
- Clone the destination repository:
git clone [email protected]:googleapis/python-tasks.git cd python-tasks/
- Run
synthtool
to generate using the existingsynth.py
file for the Python Client for Cloud Tasks API:python3 -m synthtool
- See the Python Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.
- Clone the destination repository:
git clone [email protected]:googleapis/java-tasks.git cd java-tasks/
- Run
synthtool
to generate using the existingsynth.py
file for the Google Cloud Java Client for Cloud Tasks:python3 -m synthtool
- See the Java Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.
- Clone the destination repository:
git clone [email protected]:googleapis/nodejs-tasks.git cd nodejs-tasks/
- Run
synthtool
to generate using the existingsynth.py
file for the Google Cloud Tasks Node.js Client:python3 -m synthtool
- See the Node.js Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.
- Clone the destination repository:
git clone [email protected]:googleapis/google-cloud-php.git cd google-cloud-php/
- Navigate to the destination directory:
cd Tasks/
- Run
synthtool
to generate using the existingsynth.py
file for the Google Cloud Tasks client for PHP:python3 -m synthtool
- See the PHP Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.
- Clone the destination repository:
git clone [email protected]:googleapis/google-cloud-ruby.git cd google-cloud-ruby/
- Navigate to the destination directory:
cd google-cloud-tasks/
- Run
synthtool
to generate using the existingsynth.py
file for the Ruby Client for Cloud Tasks API:python3 -m synthtool
- See the Ruby Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.
Functions in synthtool make it easier to copy files, merge files, etc. See the pydocs for more details.
SynthTool supports template files using Jinja.
Templates are found in subdirectories of synthtool/gcp/templates/
for each language,
You can generate and copy templates using gcp.CommonTemplates
in your synth.py
:
common_templates = gcp.CommonTemplates()
templates = common_templates.node_library()
s.copy(templates)
You can provide variables to templates as keyword arguments to the library generation method:
common_templates = gcp.CommonTemplates()
templates = common_templates.node_library(version=5, show_version=True, previous_versions=[1,2,3,4])
s.copy(templates)
Template files can access any values provided, e.g.
README.md.j2
{% if show_version %} The version is {{ version }} {% if previous versions is defined %} Previous versions: {% for ver in previous_versions %} - {{ ver }} {% endfor %} {% endif %} {% endif %}
For more information on how to use Synthtool templating for Python Samples, view /py_templating_instructions
You can learn more about Jinga templating in the Template Designer Documentation.
SynthTool supports generation from googleapis/googleapis-private.
gapic = gcp.GAPICGenerator()
library = gapic.node_library('speech', 'v1', private=True)
2FA is required to clone a private repo.
- Using SSH: Before running Synthtool, set the environment variable
AUTOSYNTH_USE_SSH
totrue
.
The repo is cloned using SSH.
- Using HTTPS: Generate a GitHub Personal Access Token with scope
repo
. Runsynthtool
.
When GitHub prompts for your GitHub password, provide the access token instead.
synthtool > Cloning googleapis-private.
Username for 'https://github.com': busunkim96
Password for 'https://[email protected]':
SynthTool uses the latest version of the Artman Docker image.
You can change this by setting the environment variable SYNTHTOOL_ARTMAN_VERSION
to the desired version tag.
export SYNTHTOOL_ARTMAN_VERSION=0.16.2
SynthTool uses the latest version of gcr.io/gapic-images/gapic-generator-python. You can change this by
setting the environment variable SYNTHTOOL_GAPIC_GENERATOR_PYTHON_VERSION
to the desired version tag.
export SYNTHTOOL_GAPIC_GENERATOR_PYTHON_VERSION=0.22.0
Alternatively you can set the generator version by passing it to gapic.py_library
.
import synthtool as s
import synthtool.gcp as gcp
gapic = gcp.GAPICMicrogenerator()
library = gapic.py_library(
"bigquery/connection", "v1beta1", generator_version="0.22.0"
)
SynthTool supports generation from a local copy of googleapis.
Specify the path to googleapis
in the environment variable SYNTHTOOL_GOOGLEAPIS
.
export SYNTHTOOL_GOOGLEAPIS=path/to/local/googleapis
SynthTool supports generation from a local copy of gapic-generator.
Specify the path to gapic-generator
in the environment variable SYNTHTOOL_GENERATOR
.
export SYNTHTOOL_GENERATOR=path/to/local/gapic-generator
Don't forget to compile gapic-generator
before running SynthTool.
cd path/to/local/gapic-generator
./gradlew fatJar
SynthTool supports specifying a local directory of templates. Specify the path to the root
template directory (not a SynthTool clone) in the environment variable SYNTHTOOL_TEMPLATES
.
export SYNTHTOOL_TEMPLATES=path/to/local/templates
SynthTool supports copying .proto API definition files from googleapis.
gapic = gcp.GAPICGenerator()
library = gapic.node_library('speech', 'v1', include_protos=True)
Autosynth runs synthtool on your synth.py
nightly or more frequently.
By default, it runs synthtool once, and if the generated code differs,
creates a PR with the differences.
Autosynth can also find which changes in upstream repositories triggered changes in the generated code. To enable this behavior (context-aware commits), set one or both of the following flags in you synth.py file:
AUTOSYNTH_MULTIPLE_COMMITS
AUTOSYNTH_MULTIPLE_PRS
Assume that since the library source code was last generated, A, B and X, Y were committed to googleapis and synthtool respectively, and they all triggered changes in the generated library code.
googleapis | synthtool (templates) |
---|---|
A | X |
B | Y |
Here's what autosynth generates for each flag setting.
AUTOSYNTH_MULTIPLE_COMMITS = True
Autosynth creates one PR, with a single commit for each original commit:
PR |
---|
A |
B |
X |
Y |
AUTOSYNTH_MULTIPLE_COMMITS = True
AUTOSYNTH_MULTIPLE_PRS = True
Autosynth creates two PRs, with a single commit for each original commit:
PR1 |
---|
A |
B |
PR2 |
---|
X |
Y |
AUTOSYNTH_MULTIPLE_PRS = True
Autosynth creates two PRs, with a single commit combining all the original commits.
PR1 |
---|
AB |
PR2 |
---|
XY |
SynthTool runs Artman which creates generated code that
can be found at ~/.cache/synthtool/googleapis<-private>/artman_genfiles
. This is useful for figuring out
what it is you need to copy for your specific library.
Ben had the misfortune to discover a corner case where autosynth deleted a file that Ben never intended or expected it to delete.
Here is what happened:
-
Autosynth cannot directly observe which files your synth.py generates, because synth.py could literally do anything, including launch the space shuttle. So, Autosynth figures out which files were generated by examining all the file system reads and writes that happened while synth.py was executing. Any file that is written to or copied to is deemed to have been generated by synth.py and recorded as a generatedFiles in synth.metadata.
-
NodeJS's synth.py ran the linter on manually-written sample files in the repo. A new version of the linter was pulled in, which modified a manually-written sample file. Autosynth observed the write and concluded that the manually-written file was a generated file, and listed it in generatedFiles in synth.metadata. The next time Autosynth ran, the linter made no changes, and the manually-written sample file was not written to. Autosynth concluded the manually-written sample file was no longer being generated, and deleted it.
Make sure your synth.py does not touch any manually-written files in the repo.