The main goal of this service is to allow the users to import a certain project (be it software or hardware) from an import_service and exporting it to a target export_service.
By service we refer to some storage or collaborative editing platforms. Currently, we provide support to the following services:
- Wikifactory
- Git / Github
- Google Drive
Once the import-export process has finished, the user will have a copy of the project from the import_service inside the export_service.
This project has been thought as a FastAPI based service. The structure of the code follows the specification defined here. The management of the processing tasks is handled by means of Celery. PostgreSQL has been used as the database provider.
Next, an overview of the process can be found.
First, the user must select the source project that will be imported. In order to abstract the process as much as possible, the ImportExport service requires the URL of the source project. This URL will have different meanings given the associated service and will have an asssociated import_service (e.g. the web address of a github repository or the addres of a project in Wikifactory). It may be required to perform some authentication or authorization steps, to provide the ImportExport service access to the original files or to the target platform. As the result of this step, a new piece of information can be integrated into the request: the import_token and export_token .
Below, we provide a schema of the request as must be sent to the ImportExport service, in JSON format:
{
"import_url":"https://github.com/nasa- jpl/open-source-rover",
"import_service":"git",
"import_token":"TOKEN_HERE",
"export_url":"http://wikifactory.com/@dummyuser/imported_rover_project",
"export_service":"wikifactory",
"export_token":"TOKEN_HERE"
}
At this point of the implementation we assume that the export_url directs to an already created project, be it a github repository, a Google Drive folder or a Wikifactory project.
All the source code associated to the IEService can be found inside the app
folder
- The
api
folder holds the information of the API exposed by the service. In particular, the folderapi_v1
contents the current exposed api. Later, more versions of the API could be added inside this folder. In any case, theapi.py
file define the paths exposed by the service - The
core
folder is in charge of two main functionalities. First, it configures the details of the Celery tasks with thecelery_app
file. Second, with theconfig.py
file it defines the most basic information of the service (e.g. where the downloaded files will be stored or the credentials to access the data base). - The logic of importing and exporting for each service is defined inside the
importers
andexporters
folder. Each individual importer inherits from BaseImporter, which defines its basic functionallity. Exporters have an equivalent process, but descending from BaseExporter.
The following folders inside the app
one are strongly related with the data model of the service:
- The
db
folder holds the required information of the db associated to the IEService. More in detail, theinit_db.py
file is in charge of initializing the db (i.e. creating the tables), and this is done by reading all the classes defined inbase.py
. Finally, the db session is created insidesession.py
, from where the rest of the service will have access. - The data model of the service is defined inside the
schemas
folder. Each entity in the service is defined by a class that inherits from Pydantics' BaseModel (more info here). Pydantics is seamless integrated inside FastAPI and allows automatic validation. - The
models
folder on its behalf, defines how those schemas will be represented inside the database. This definition is based on the sqlalchemy toolkit. - Finally, iside the
crud
folder we defined the main methods used to create, read update and delete (CRUD) the elements inside the database. Thiscrud
module is used across the IEService to access the data inside the db as well as updating it.
In order to implement a new importer, the following steps are required:
- Create the import class inside
app/importers
. The newly created class must extend from BaseImporter. At this point, you can define your own initialization steps, or you can copy the basic schema from any of the already implemented ones. - Next, you have to implement the
process
method inside your importe,r taking into account the details of the import_service. As the result of this process, a Manifest instance will be created in the db and the files will be downloaded in the computer. - Create the service entry in the
app/importers/__init__.py
file. Inside this file, first you must set the string identifier of your service (e.g. "git" or "google_drive"). After that, you must add an entry to the validator map, which ties that same service identifier to an associated URL validator.
The steps required to create a new exporter are similar to those ones, with some minor changes:
- The exporter must be created inside the
app/exporters
folder and it should extend from the BaseExporter class - You have to implement the
process
method, which is in charge of reading all the files associated to the project and upload them to the export_service. - Finally, you have to integrate the exporter into the process by adding an entry inside the validator map defined in
app/exporters/__init__.py
as well as the service identifier.
You can run the service by doing: docker-compose up -d
and stop it by using docker-compose down
. This will start the service in the background.
By default, the access to the database has been disabled (i.e. not exposed). If you want to access directly to that data, you must change the docker-compose.override.yaml
file. In particular, you must expose the db port inside your computer. For example, you could do this:
networks:
- default
ports: # Add this line
- "8004:5432" # Add this line
You can run the tests inside the docker environment by using the following command:
docker-compose run backend pytest