Apertium on Github

Source Browser

Proposal

See PMC_proposals/Move_Apertium_to_Github.

Structure

• Each package will have its own repository.
• "Meta repositories" will contain only submodules.
  • apertium-all will be the root repo, containing repos with topics apertium-core and apertium-all (the latter are the repos below).
  • apertium-tools will contain tools (repos with topic apertium-tools).
  • apertium-staging will contain staging language pairs (repos with topic apertium-staging).
  • apertium-nursery will contain nursery language pairs (repos with topic apertium-nursery).
  • apertium-incubator will contain staging language modules/pairs (repos with topic apertium-incubator).
  • apertium-trunk will contain trunk language modules/pairs (repos with topic apertium-trunk).
• Meta repositories will be kept up-to-date by sync.py (described below).

N.B. The topics (a.k.a. tags) are integral.

Transition

Organization

Members

Initially,

• All package repositories (pairs/languages/tools) will belong to the apertium GitHub organization.
• Members of the PMC will be owners of this organization.
• All other Apertium contributors will be members. A member of the PMC can make someone a member of the organization after they commit code as an outside collaborator or via pull request and are then sponsored by two existing committers. This is not analogous to giving commit access in SVN since non-members of the organization can easily contribute code with pull requests or as an outside collaborator.

The permissions associated with these roles are described in detail.

invite-members.sh will invite all the non-SourceForge emails in svn-authors.txt to the organization as members. For technical reasons, surrounding rate limits and caching, this script should be run a few days prior to repository migration when add-contributors.sh will give members of the organization write access to repositories they have committed to in the past.

Teams

Members of the apertium organization can organize themselves into teams. A team and its membership can be private or public, but for Apertium, it is preferable to have all teams public. A team can:

• have a team homepage
• plan projects using GitHub's projects interface
• be mentioned as '@xyz-team' in discussions

A member of the PMC can grant write permissions for specific repositories to an entire team. Current and future members of the team will have those permissions as long as they are not revoked by a member of the PMC.

Upon request, a member of the PMC can make members of a team into team maintainers. The privileges of a team maintainer are:

• adding or removing members of the team
• giving or revoking the team access to repositories

Repositories

Permissions

Each repository will have the following permission levels:

• owner: (same for all repositories) PMC members (same as the organization owners). Can do anything.
• admin: (repository-specific) Organization members that serve as 'package maintainers'. A PMC member can designate a member of the organization as an admin. Of particular relevance to Apertium are the following permissions they have in addition to write permissions:
  • manage repository settings
  • delete the repository
  • add/delete outside collaborators (people with write access who are not Apertium members)
  • give other contributors admin access
  • manage topics (including moving from e.g. staging to trunk)
• write: (repository-specific) Organization members that can commit to the repository. Any PMC member can give a member of the organization write permission to a repository.
• read: (same for all repositories) any GitHub user since our repositories are all public

Meta Repositories

The meta-repositories apertium-all, apertium-tools, apertium-staging, apertium-nursery, apertium-incubator, and apertium-trunk should be 'pinned repositories' for the organization so that they are at the top of the list of repositories when a user lands on the organization page.

Note that nobody should have write permissions for these meta-repositories, except owners of course. Their contents will, for the most part, be updated automatically via sync.py.

Scripts

These scripts rely on SubGit being present in the current (top) directory and use the svn-authors.txt file to convert SVN users to GitHub emails to establish connections between commits and GitHub accounts. Utility functions are located in util.sh.

• import-repo.sh "svn-path" "github-repo-name" "github-topic" will import the SVN folder at svn-path to GitHub with the name github-repo-name and the given topic.
• import-modules.sh imports all language pairs, modules, and the Apertium core from SVN to Github.
• create-metarepos.sh creates all the meta repos and syncs their submodules
• invite-members.sh invites non-SourceForge emails from svn-authors.txt to the GitHub organization
• add-contributors.sh grants push permissions for each repository to those GitHub users who have previously contributed to the repo
• add-gitmeta.sh adds/updates existing .gitignore and .gitattributes
• lock-files.sh locks SVN files recursively (using SF shell service to add a pre-commit hook would be optimal but it is down)

Migration

For the actual migration, an owner of the apertium GitHub organization needs to:

1. Create a GitHub OAuth token with 'repo' permissions.
2. Run export GITHUB_OAUTH_TOKEN=<token from above> to set the environment variable.
3. Download and unzip SubGit in the current directory.
4. Edit line 5 of util.sh with their own GitHub username.
5. Run ./invite-members.sh. Wait for some time to allow people to accept invitations.
6. Run ./lock-files.sh.
7. Run ./import-modules.sh.
8. Run ./create-metarepos.sh. Wait for ~two hours to allow GitHub to sync contributors.
9. Run ./add-contributors.sh.
10. Pin meta repositories.

Maintenance

• sync.py recieves events from GitHub web hooks.
• Any updates to repositories with the appropriate tags will be pushed to the appropriate meta-repository.
• New repositories with a valid topic will be added to the appropriate meta-repository.
• Deleted repositories with a valid topic will be deleted from the appropriate meta-repository.

Usage:

usage: sync.py [-h] [--verbose] [--dir DIR]
              [--repo {apertium-nursery,apertium-incubator,apertium-tools,apertium-trunk,apertium-staging,apertium-languages,apertium-all}]
              [--port PORT] --token TOKEN [--sync-interval SYNC_INTERVAL]
              [--author AUTHOR]
              {startserver,sync}

Sync Apertium meta repositories.

positional arguments:
  {startserver,sync}    use "startserver" to start the server and "sync --repo
                        [name]" to force a meta-repo sync

optional arguments:
  -h, --help            show this help message and exit
  --verbose, -v         add verbosity (maximum -vv)
  --dir DIR, -d DIR     directory to clone meta repos
  --repo {apertium-nursery,apertium-incubator,apertium-tools,apertium-trunk,apertium-staging,apertium-languages,apertium-all}, -r {apertium-nursery,apertium-incubator,apertium-tools,apertium-trunk,apertium-staging,apertium-languages,apertium-all}
                        meta-repo to sync (default: all)
  --port PORT, -p PORT  server port (default: 9712)
  --token TOKEN, -t TOKEN
                        GitHub OAuth token
  --sync-interval SYNC_INTERVAL, -i SYNC_INTERVAL
                        min interval between syncs (default: 3s)
  --author AUTHOR, -a AUTHOR
                        commit author (default: Apertium Bot
                        <[email protected]>)

The GitHub OAuth token is described in the 'Scripts' section above. For sync.py, it can also be set through the environment variable GITHUB_OAUTH_TOKEN.

Interface

We provide a wrapper on top of GitHub's organization view since it only supports pinning up to six repositories, searching is cumbersome and there are no custom layout options.

The source for this interface is source-browser.html. For the sake of simplicity, only modern browsers are supported. It is made available via GH pages.

Git Tips

• Remember, GitHub has a Subversion bridge that makes it possible to use SVN to work with any of the non meta repositories.
• Kernel.org's Git for SVN users cheatsheet.
• Meta repositories
  • Don't push updated submodules to a meta repository, let sync.py handle it.
  • To checkout a meta repository, use git clone --recursive -j8 [url].
  • To pull (update) a meta repository, use git pull --recurse-submodules. Never use git submodule update, you will get conflicts with the sync script's pushes.
  • To commit changes to all submodules within a meta repository, use git submodule foreach 'git commit -m "my message"'. Add the -a flag to add unstaged files. If some submodules are dirty, use git commit -m "my message" || true.
  • To push changes to submodules within a meta repository, use git submodule foreach git push.
• Use Git alias for any oft used commands.