Skip to content

Commit

Permalink
Resolve conflicts
Browse files Browse the repository at this point in the history
  • Loading branch information
Dmitry committed Mar 7, 2018
2 parents bd191ad + 8c3fb0d commit 2a88588
Show file tree
Hide file tree
Showing 19 changed files with 268 additions and 759 deletions.
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1 @@
*.swp
*~
/setup.yml
36 changes: 13 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# postgrest-translation-proxy
PostgreSQL/PostgREST proxy to Google, Bing and Prompt Translate APIs, with caching and ability to combine multiple text segments in one single request. It allows to work with those Translate APIs right from Postgres or via REST API requests.
# postgrest-google-translate
PostgreSQL/PostgREST proxy to Google Translate API, with caching and ability to combine multiple text segments in one single request. It allows to work with Google Translate API right from Postgres or via REST API requests.

[![Build Status](https://circleci.com/gh/NikolayS/postgrest-google-translate.png?style=shield&circle-token=fb58aee6e9f98cf85d08c4d382d5ba3f0f548e08)](https://circleci.com/gh/NikolayS/postgrest-google-translate/tree/master)

This tiny project consists of 2 parts:

1. SQL objects to enable calling Translation APIs right from SQL environment (uses [plsh](https://github.com/petere/plsh) extension)
1. SQL objects to enable calling Google API right from SQL environment (uses [plsh](https://github.com/petere/plsh) extension)
2. API method (uses [PostgREST](http://postgrest.com))

Part (1) can be used without part (2).

Table `translation_proxy.cache` is used to cache API responses to speedup work and reduce costs.
Table `google_translate.cache` is used to cache Google API responses to speedup work and reduce costs.
Also, it is possible to combine multiple phrases in one API request, which provides great advantage (e.g.: for 10 uncached phrases, it will be ~150-200ms for single aggregated request versus 1.5-2 seconds for 10 consequent requests). Currently, Google Translate API accepts up to 128 text segments in a single request.

:warning: Limitations
Expand All @@ -29,32 +29,22 @@ Dependencies
---
1. cURL
2. [PostgREST](http://postgrest.com) – download the latest version. See `circle.yml` for example of starting/using it.
3. `plsh` – PostgreSQL contrib module, it is NOT included to standard contribs package. To install it on Ubuntu/Debian run: `apt-get install postgresql-X.X-plsh` (where X.X could be 9.5, depending on your Postgres version). For Archlinux use AUR package 'postgresql-plsh'.
4. Ruby for easy installer (optional)
2. `plsh` – PostgreSQL contrib module, it is NOT included to standard contribs package. To install it on Ubuntu/Debian run: `apt-get install postgresql-X.X-plsh` (where X.X could be 9.5, depending on your Postgres version)

Installation and Configuration
---
Simple method
----
Edit `setup.yml` then execute `setup.rb`. You need to have the ruby interpreter been installed.

Step-by-step method
----
For your database (here we assume that it's called `DBNAME`), install [plsh](https://github.com/petere/plsh) extension and then execute `_core` SQL scripts, after what configure your database settings:
`translation_proxy.promt_api_key`, `translation_proxy.bing_api_key` and
`translation_proxy.google_api_key` (take it from Google Could Platform Console):
For your database (here we assume that it's called `DBNAME`), install [plsh](https://github.com/petere/plsh) extension and then execute two SQL scripts, after what configure your database setting `google_translate.api_key` (take it from Google Could Platform Console):
```sh
psql DBNAME -c 'create extension if not exists plsh;'
psql DBNAME -f install_core.sql
psql -c "alter database DBNAME set translation_proxy.google_api_key = 'YOUR_GOOGLE_API_KEY';"
psql -c "alter database DBNAME set translation_proxy.google_begin_at = '2000-01-01';"
psql -c "alter database DBNAME set translation_proxy.google_end_at = '2100-01-01';"
psql -c "alter database DBNAME set google_translate.api_key = 'YOU_GOOGLE_API_KEY';"
psql -c "alter database DBNAME set google_translate.begin_at = '2000-01-01';"
psql -c "alter database DBNAME set google_translate.end_at = '2100-01-01';"
```

Alternatively, you can use `ALTER ROLE ... SET google_translate.api_key = 'YOU_GOOGLE_API_KEY';` or put this setting to `postgresql.conf` or do `ALTER SYSTEM SET google_translate.api_key = 'YOU_GOOGLE_API_KEY';` (in these cases, it will be available cluster-wide).

Alternatively, you can use `ALTER ROLE ... SET translation_proxy.google_api_key = 'YOUR_GOOGLE_API_KEY';` or put this setting to `postgresql.conf` or do `ALTER SYSTEM SET translation_proxy.google_api_key = 'YOUR_GOOGLE_API_KEY';` (in these cases, it will be available cluster-wide).

Parameters `translation_proxy.google_begin_at` and `translation_proxy.google_end_at` are responsible for the period of time, when Google Translate API is allowed to be called. If current time is beyond this timeframe, only the cache table will be used.
Parameters `google_translate.begin_at` and `google_translate.end_at` are responsible for the period of time, when Google Translate API is allowed to be called. If current time is beyond this timeframe, onlic cache will be used.

To enable REST API proxy, install [PostgREST](http://postgrest.com), launch it (see `cirle.yml` as an example), and initialize API methods with the additional SQL script:
```sh
Expand All @@ -74,10 +64,10 @@ Usage
In SQL environment:
```sql
-- Translate from English to Russian
select translation_proxy.google_translate('en', 'ru', 'Hello world');
select google_translate.translate('en', 'ru', 'Hello world');

-- Combine multiple text segments in single query
select * from translation_proxy.google_translate('en', 'ru', array['ok computer', 'show me more','hello world!']);
select * from google_translate.translate('en', 'ru', array['ok computer', 'show me more','hello world!']);
```

REST API:
Expand Down
25 changes: 12 additions & 13 deletions circle.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,24 @@ dependencies:
- cd ~ && wget https://github.com/begriffs/postgrest/releases/download/v0.3.2.0/postgrest-0.3.2.0-ubuntu.tar.xz
- cd ~ && tar xf postgrest-0.3.2.0-ubuntu.tar.xz
- echo "~/postgrest postgres://apiuser:SOMEPASSWORD@localhost:5432/test --pool=200 --anonymous=apiuser --port=3000 --jwt-secret notverysecret --max-rows=500 --schema=v1" > ~/postgrest-run.sh && chmod a+x ~/postgrest-run.sh
- npm install -g npm newman ava
- npm install -g npm newman
database:
override:
- sudo -u postgres psql -c "create role apiuser password 'SOMEPASSWORD' login;"
- sudo -u postgres psql -c "create database test;"
- sudo -u postgres psql -c "alter database test set translation_proxy.google_api_key = 'AIzaSyCauv2HRjprFX3DcGhorJFYGyeVmzvunuc';"
- sudo -u postgres psql -c "alter database test set translation_proxy.google_begin_at = '2000-01-01';"
- sudo -u postgres psql -c "alter database test set translation_proxy.google_end_at = '2100-01-01'"
- sudo -u postgres psql -c "alter database test set google_translate.api_key = 'AIzaSyCauv2HRjprFX3DcGhorJFYGyeVmzvunuc';"
- sudo -u postgres psql -c "alter database test set google_translate.begin_at = '2000-01-01';"
- sudo -u postgres psql -c "alter database test set google_translate.end_at = '2100-01-01'"
- sudo -u postgres psql test -c "create extension if not exists plsh;"
- sudo -u postgres psql test -f ~/postgrest-translation-proxy/install_google_core.sql
- sudo -u postgres psql test -f ~/postgrest-translation-proxy/install_promt_core.sql
- sudo -u postgres psql test -f ~/postgrest-translation-proxy/install_bing_core.sql
- sudo -u postgres psql test -f ~/postgrest-translation-proxy/install_api.sql
- sudo -u postgres psql test -f ~/postgrest-google-translate/install_core.sql
- sudo -u postgres psql test -f ~/postgrest-google-translate/install_api.sql
- ~/postgrest-run.sh:
background: true
background: true
test:
override:
- ~/postgrest-translation-proxy/test/run.sh -f junit >$CIRCLE_TEST_REPORTS/junit.xml
- ~/"$CIRCLE_PROJECT_REPONAME"/test/run.sh -f junit >$CIRCLE_TEST_REPORTS/junit.xml
- nc -z -v -w5 localhost 3000
- newman run ~/postgrest-translation-proxy/test/postman/postgrest-translation-proxy.postman_collection --bail -e ~/postgrest-translation-proxy/test/postman/local.postman_environment --reporter-junit-export $CIRCLE_TEST_REPORTS/newman.xml
- sudo -u postgres psql test -v ON_ERROR_STOP=1 -f ~/postgrest-translation-proxy/uninstall_api.sql
- sudo -u postgres psql test -v ON_ERROR_STOP=1 -f ~/postgrest-translation-proxy/uninstall_core.sql
- newman run ~/"$CIRCLE_PROJECT_REPONAME"/test/postman/postgrest-google-translate.postman_collection --bail -e ~/postgrest-google-translate/test/postman/local.postman_environment --reporter-junit-export $CIRCLE_TEST_REPORTS/newman.xml
- sudo -u postgres psql test -v ON_ERROR_STOP=1 -f ~/"$CIRCLE_PROJECT_REPONAME"/uninstall_api.sql
- sudo -u postgres psql test -v ON_ERROR_STOP=1 -f ~/"$CIRCLE_PROJECT_REPONAME"/uninstall_core.sql

41 changes: 25 additions & 16 deletions install_api.sql
Original file line number Diff line number Diff line change
@@ -1,17 +1,26 @@
CREATE OR REPLACE FUNCTION v2.translate_array(source CHAR(2), target CHAR(2), q JSON)
RETURNS TEXT[] AS $BODY$
DECLARE
rez TEXT[];
BEGIN
SELECT
CASE current_setting('translation_proxy.api.current_engine')
WHEN 'google' THEN
translation_proxy.google_translate_array( source, target, q )
WHEN 'promt' THEN
translation_proxy.promt_translate_array( source, target, array_agg( json_array_elements_text(q) ) )
END INTO rez;
RETURN rez;
END;
$BODY$ LANGUAGE PLPGSQL SECURITY DEFINER;
create schema if not exists v1;
do
$$
begin
if not exists (
select *
from pg_catalog.pg_user
where usename = 'apiuser'
) then
create role my_user password 'SOMEPASSWORD' login;
end if;
end
$$;

GRANT EXECUTE ON FUNCTION v2.translate_array(CHAR(2), CHAR(2), JSON) TO apiuser;
grant usage on schema v1 to apiuser;

create or replace function v1.google_translate_array(source char(2), target char(2), q json) returns text[] as $$
select * from google_translate.translate_array(source, target, q);
$$ language sql security definer;

create or replace function v1.google_translate(source char(2), target char(2), q text) returns text as $$
select * from google_translate.translate(source, target, q);
$$ language sql security definer;

grant execute on function v1.google_translate_array(char, char, json) to apiuser;
grant execute on function v1.google_translate(char, char, text) to apiuser;
17 changes: 0 additions & 17 deletions install_api_vars.sql

This file was deleted.

48 changes: 0 additions & 48 deletions install_bing_core.sql

This file was deleted.

3 changes: 0 additions & 3 deletions install_bing_vars.sql

This file was deleted.

Loading

0 comments on commit 2a88588

Please sign in to comment.