Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(#6289): Bumps CouchDb version to 3.3.2 #8107

Merged
merged 93 commits into from
Sep 12, 2023
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
2cb5ff5
update couchdb container
dianabarsan Feb 21, 2023
244ff4a
update e2e tests
dianabarsan Feb 22, 2023
64c49e7
update e2e tests
dianabarsan Feb 22, 2023
7d166b7
update unit tests
dianabarsan Feb 22, 2023
30edc75
add missed code.
dianabarsan Feb 22, 2023
5034915
the story behind the messy commits is ...
dianabarsan Feb 22, 2023
4df0da5
add e2e tests
dianabarsan Feb 23, 2023
5b2c045
add e2e tests
dianabarsan Feb 23, 2023
9c5c5a6
remove old file
dianabarsan Feb 23, 2023
7ed10b5
adds unit test
dianabarsan Feb 23, 2023
f7ea09d
remove test only
dianabarsan Feb 24, 2023
5c1bd75
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan Feb 24, 2023
b1a600c
remove test echo
dianabarsan Feb 27, 2023
afda1a6
reload page before loading analytics
dianabarsan Feb 27, 2023
5e94b8b
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan Mar 29, 2023
57fe903
use lua-load-per-thread
dianabarsan Mar 22, 2023
0e77a02
upgrade to haproxy 2.7
dianabarsan Apr 5, 2023
bfd5cb4
add option abortonclose
dianabarsan Apr 5, 2023
10df1b1
remove tune options from haproxy config
dianabarsan Apr 6, 2023
f7f158c
remove old comment about tune.bufsize
dianabarsan Apr 6, 2023
3e98060
Merge branch '8166-new-haproxy' into 7964-couch3
dianabarsan Apr 13, 2023
9786dda
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan Apr 13, 2023
ade5bb1
disable repeatedly failing test
dianabarsan Apr 17, 2023
0120c35
"patch" node-fetch in api and sentinel.
dianabarsan Apr 28, 2023
a66ae9d
rm in grunt only.
dianabarsan Apr 28, 2023
709e11c
revert unwanted change
dianabarsan May 1, 2023
e2773db
add comment.
dianabarsan May 1, 2023
2a5e0b7
Merge remote-tracking branch 'origin/master' into 8173-pouchdb-upgrade
dianabarsan May 3, 2023
a6ed30b
remove code that is no longer necessary
dianabarsan May 3, 2023
fce25bd
partial cherrypick of haproxy edits
dianabarsan May 4, 2023
c6d1527
don't wait for API when haproxy is down, it'll never start.
dianabarsan May 4, 2023
bf8f239
don't wait for API when haproxy is down, it'll never start.
dianabarsan May 4, 2023
144c273
always retry writing the infodoc.
dianabarsan May 5, 2023
98e42e6
fix eslint
dianabarsan May 5, 2023
f4c7971
add e2e test
dianabarsan May 9, 2023
9fb8065
update test
dianabarsan May 9, 2023
cddd945
fix eslint and start server before restarting sentinel.
dianabarsan May 9, 2023
35c8452
Merge branch '8160-retry-infodoc-write' into 7964-couch3
dianabarsan May 9, 2023
3551baa
Merge branch '8173-pouchdb-upgrade' into 7964-couch3
dianabarsan May 9, 2023
ca1a654
merge things.
dianabarsan May 10, 2023
aa15d4a
wait longer to sync passwords in cluster.
dianabarsan May 10, 2023
2f9f918
reorder refresh in protractor target test
dianabarsan May 10, 2023
2284dd2
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan May 11, 2023
8ed0fde
reload session before asserting languages.
dianabarsan May 16, 2023
797f7c3
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan May 16, 2023
0256ed6
call _sync_shards to sync password updates across the cluster
dianabarsan May 16, 2023
b7c7b6d
wait after starting haproxy
dianabarsan May 17, 2023
f19f32f
wait for 10s for logs (in case many forms are pushed)
dianabarsan May 17, 2023
db20407
wait for 10s for logs (in case many forms are pushed)
dianabarsan May 17, 2023
4dc5f0f
wait for 10s instead of 5.
dianabarsan May 17, 2023
eb5deff
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan Jun 8, 2023
e714c3e
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan Jul 4, 2023
2789484
more queue time
dianabarsan Jul 5, 2023
8fc3e42
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan Jul 6, 2023
c953ee3
fix bad merge
dianabarsan Jul 6, 2023
5c05230
add delay before pushing a new doc
dianabarsan Jul 6, 2023
5a1c00b
revert unwanted changes
dianabarsan Jul 9, 2023
13e4eaf
revert unwanted changes
dianabarsan Jul 9, 2023
16dcc3d
revert unwanted changes
dianabarsan Jul 9, 2023
2cbf56b
pretty paths in new e2e test
dianabarsan Jul 9, 2023
0fc1a6b
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan Jul 9, 2023
3e02a4b
we rolled back time, telemetry date is supposed to be yesterday
dianabarsan Jul 9, 2023
a9542e5
Merge branch 'master' into 7964-couch3
dianabarsan Jul 10, 2023
ad51a7b
couchdb is supposed to be faster ffs
dianabarsan Jul 11, 2023
76afdf3
it's actually slower
dianabarsan Jul 12, 2023
6dfbb37
it's actually slower
dianabarsan Jul 13, 2023
8504760
review feedback
dianabarsan Jul 18, 2023
bf8ab8e
fix eslint
dianabarsan Jul 18, 2023
f78fb3e
fix bad function reference
dianabarsan Jul 19, 2023
f4f719d
fix e2e test
dianabarsan Jul 26, 2023
3a19675
add compressible types config
dianabarsan Jul 26, 2023
5074224
Merge branch 'master' into 7964-couch3
dianabarsan Aug 1, 2023
2255758
Merge branch 'master' into 7964-couch3
dianabarsan Aug 1, 2023
b7897bb
Merge branch 'master' into 7964-couch3
dianabarsan Aug 10, 2023
efee2eb
remove national-admin
dianabarsan Sep 8, 2023
1a1d3af
fix unit tests
dianabarsan Sep 8, 2023
6243617
fix integration tests
dianabarsan Sep 8, 2023
747ddd4
fix validate doc update test
dianabarsan Sep 8, 2023
16d1769
revert haproxy / nginx symlink removal
dianabarsan Sep 8, 2023
c445b81
Merge branch '8243-remove-national-admin' into 7964-couch3
dianabarsan Sep 8, 2023
d55eb9e
just add all files haproxy / nginx symlink removal
dianabarsan Sep 9, 2023
bc7c65f
omg symlink hell
dianabarsan Sep 9, 2023
403fc5a
omg symlink hell
dianabarsan Sep 9, 2023
7a82d7f
Merge branch '8243-remove-national-admin' into 7964-couch3
dianabarsan Sep 9, 2023
b7777a4
omg symlink hell
dianabarsan Sep 9, 2023
a1f35fa
Merge branch '8243-remove-national-admin' into 7964-couch3
dianabarsan Sep 9, 2023
8c0c4f8
fix linting
dianabarsan Sep 9, 2023
ad14c34
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan Sep 11, 2023
bb5b951
fix all permissions e2e test error.
dianabarsan Sep 11, 2023
ffc7eaf
remove unnecessary env set
dianabarsan Sep 11, 2023
de879d2
fix bad merge
dianabarsan Sep 11, 2023
e8f02fa
wtb compiler plis
dianabarsan Sep 11, 2023
e4c3fbf
Merge remote-tracking branch 'origin/master' into 7964-couch3
dianabarsan Sep 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion api/src/db.js
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,8 @@ if (UNIT_TEST_ENV) {
'allDbs',
'activeTasks',
'saveDocs',
'createVault'
'createVault',
'addRoleToSecurity',
];

const notStubbed = (first, second) => {
Expand Down Expand Up @@ -162,4 +163,37 @@ if (UNIT_TEST_ENV) {

throw new Error(`Error while saving docs: ${errors.join(', ')}`);
};

const DEFAULT_SECURITY_STRUCTURE = {
names: [],
roles: [],
};
dianabarsan marked this conversation as resolved.
Show resolved Hide resolved

module.exports.addRoleToSecurity = async (dbname, role, addAsAdmin) => {
dianabarsan marked this conversation as resolved.
Show resolved Hide resolved
if (!dbname || !role) {
return;
dianabarsan marked this conversation as resolved.
Show resolved Hide resolved
}

const securityUrl = new URL(environment.serverUrl);
securityUrl.pathname = `${dbname}/_security`;

const securityObject = await rpn.get({ url: securityUrl.toString(), json: true });
const property = addAsAdmin ? 'admins' : 'members';
if (!securityObject[property]) {
securityObject[property] = DEFAULT_SECURITY_STRUCTURE;
}

if (!securityObject[property].roles) {
dianabarsan marked this conversation as resolved.
Show resolved Hide resolved
throw new Error('Invalid database security %o', securityObject);
dianabarsan marked this conversation as resolved.
Show resolved Hide resolved
}

if (securityObject[property].roles.includes(role)) {
return;
}

logger.info(`Adding "${role}" role to ${dbname} ${property}`);
securityObject[property].roles.push(role);
await rpn.put({ url: securityUrl.toString(), json: true, body: securityObject });
};

}
54 changes: 3 additions & 51 deletions api/src/migrations/add-national_admin-role.js
Original file line number Diff line number Diff line change
@@ -1,61 +1,13 @@
const request = require('request-promise-native');
const url = require('url');
const environment = require('../environment');
const logger = require('../logger');
const db = require('../db');

const DEFAULT_STRUCTURE = {
names: [],
roles: [],
};

const addRole = (dbname, role) => {
return request.get({
url: url.format({
protocol: environment.protocol,
hostname: environment.host,
port: environment.port,
pathname: `${dbname}/_security`,
}),
auth: {
user: environment.username,
pass: environment.password
},
json: true
})
.then(body => {
// In CouchDB 1.x, if you have not written to the _security object before
// it is empty.
if (!body.admins) {
body.admins = DEFAULT_STRUCTURE;
}

if (!body.admins.roles.includes(role)) {
logger.info(`Adding ${role} role to ${dbname} admins`);
body.admins.roles.push(role);
}
return request.put({
url: url.format({
protocol: environment.protocol,
hostname: environment.host,
port: environment.port,
pathname: `${dbname}/_security`,
}),
auth: {
user: environment.username,
pass: environment.password
},
json: true,
body: body
});
});
};

module.exports = {
name: 'add-national_admin-role',
created: new Date(2017, 3, 30),
run: () => {
return Promise.resolve()
.then(() => addRole('_users', 'national_admin'))
.then(() => addRole(environment.db, 'national_admin'));
.then(() => db.addRoleToSecurity('_users', 'national_admin'))
.then(() => db.addRoleToSecurity(environment.db, 'national_admin'));
}
};
18 changes: 16 additions & 2 deletions api/src/services/config-watcher.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ const generateXform = require('./generate-xform');
const generateServiceWorker = require('../generate-service-worker');
const manifest = require('./manifest');
const config = require('../config');
const environment = require('../environment');

const MEDIC_DDOC_ID = '_design/medic';

Expand Down Expand Up @@ -87,6 +88,7 @@ const handleSettingsChange = () => {
logger.error('Failed to reload settings: %o', err);
process.exit(1);
})
.then(() => addUserRolesToDb())
.then(() => initTransitionLib())
.then(() => logger.debug('Settings updated'));
};
Expand All @@ -99,7 +101,7 @@ const handleTranslationsChange = () => {
};

const handleFormChange = (change) => {
logger.info('Detected form change for', change.id);
logger.info('Detected form change for: %s', change.id);
if (change.deleted) {
return Promise.resolve();
}
Expand Down Expand Up @@ -131,6 +133,7 @@ const load = () => {
loadViewMaps();
return loadTranslations()
.then(() => loadSettings())
.then(() => addUserRolesToDb())
.then(() => initTransitionLib())
.then(() => db.createVault());
};
Expand All @@ -139,7 +142,6 @@ const listen = () => {
db.medic
.changes({ live: true, since: 'now', return_docs: false })
.on('change', change => {

if (tombstoneUtils.isTombstoneId(change.id)) {
return Promise.resolve();
}
Expand Down Expand Up @@ -170,9 +172,21 @@ const listen = () => {
});
};

const addUserRolesToDb = async () => {
const roles = config.get('roles');
if (!roles || typeof roles !== 'object') {
return;
}

for (const role of Object.keys(roles)) {
await db.addRoleToSecurity(environment.db, role, false);
}
};

module.exports = {
load,
listen,
updateServiceWorker,
loadTranslations,
addUserRolesToDb,
};
131 changes: 131 additions & 0 deletions api/tests/mocha/db.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -202,4 +202,135 @@ describe('db', () => {
expect(db.close.args).to.deep.equal([[dbObject]]);
});
});

describe('addRoleToSecurity', () => {
it('should add role as member to db security', async () => {
sinon.stub(env, 'serverUrl').get(() => 'http://admin:pass@couchdb:5984');
sinon.stub(rpn, 'get').resolves({ admins: { roles: ['role1'] }, members: { roles: ['role2'] } });
sinon.stub(rpn, 'put').resolves();

await db.addRoleToSecurity('dbname', 'rolename');

expect(rpn.get.args).to.deep.equal([[ { url: 'http://admin:pass@couchdb:5984/dbname/_security', json: true } ]]);
expect(rpn.put.args).to.deep.equal([[
{
url: 'http://admin:pass@couchdb:5984/dbname/_security',
json: true,
body: {
admins: { roles: ['role1'] },
members: { roles: ['role2', 'rolename'] },
}
}
]]);
});

it('should add role as admin to db security', async () => {
sinon.stub(env, 'serverUrl').get(() => 'http://admin:pass@couchdb:5984');
sinon.stub(rpn, 'get').resolves({ admins: { roles: ['role1'] }, members: { roles: ['role2'] } });
sinon.stub(rpn, 'put').resolves();

await db.addRoleToSecurity('dbname', 'rolename', true);

expect(rpn.get.args).to.deep.equal([[ { url: 'http://admin:pass@couchdb:5984/dbname/_security', json: true } ]]);
expect(rpn.put.args).to.deep.equal([[
{
url: 'http://admin:pass@couchdb:5984/dbname/_security',
json: true,
body: {
admins: { roles: ['role1', 'rolename'], },
members: { roles: ['role2'] },
}
}
]]);
});

it('should skip member roles that already exist', async () => {
sinon.stub(env, 'serverUrl').get(() => 'http://admin:pass@couchdb:5984');
sinon.stub(rpn, 'get').resolves({ admins: { roles: ['role1'] }, members: { roles: ['role2'] } });
sinon.stub(rpn, 'put').resolves();

await db.addRoleToSecurity('dbname', 'role2');

expect(rpn.get.args).to.deep.equal([[ { url: 'http://admin:pass@couchdb:5984/dbname/_security', json: true } ]]);
expect(rpn.put.called).to.equal(false);
});

it('should skip admin roles that already exist', async () => {
sinon.stub(env, 'serverUrl').get(() => 'http://admin:pass@couchdb:5984');
sinon.stub(rpn, 'get').resolves({ admins: { roles: ['role1'] }, members: { roles: ['role2'] } });
sinon.stub(rpn, 'put').resolves();

await db.addRoleToSecurity('dbname', 'role1', true);

expect(rpn.get.args).to.deep.equal([[ { url: 'http://admin:pass@couchdb:5984/dbname/_security', json: true } ]]);
expect(rpn.put.called).to.equal(false);
});

it('should set members security property if not existing', async () => {
sinon.stub(env, 'serverUrl').get(() => 'http://admin:pass@couchdb:5984');
sinon.stub(rpn, 'get').resolves({ admins: { roles: ['role1'] } });
sinon.stub(rpn, 'put').resolves();

await db.addRoleToSecurity('dbname', 'rolename');

expect(rpn.get.args).to.deep.equal([[ { url: 'http://admin:pass@couchdb:5984/dbname/_security', json: true } ]]);
expect(rpn.put.args).to.deep.equal([[
{
url: 'http://admin:pass@couchdb:5984/dbname/_security',
json: true,
body: {
admins: { roles: ['role1'] },
members: { roles: ['rolename'], names: [], },
}
}
]]);
});

it('should set admins security property if not existing', async () => {
sinon.stub(env, 'serverUrl').get(() => 'http://admin:pwd@host:6984');
sinon.stub(rpn, 'get').resolves({ members: { roles: ['role2'] } });
sinon.stub(rpn, 'put').resolves();

await db.addRoleToSecurity('name', 'arole', true);

expect(rpn.get.args).to.deep.equal([[ { url: 'http://admin:pwd@host:6984/name/_security', json: true } ]]);
expect(rpn.put.args).to.deep.equal([[
{
url: 'http://admin:pwd@host:6984/name/_security',
json: true,
body: {
admins: { roles: ['arole'], names: [] },
members: { roles: ['role2'] },
}
}
]]);
});

it('should throw get security errors', async () => {
sinon.stub(env, 'serverUrl').get(() => 'http://admin:pass@couchdb:5984');
sinon.stub(rpn, 'get').rejects(new Error('not_found'));

await expect(db.addRoleToSecurity('data', 'attr')).to.be.rejectedWith(Error, 'not_found');
});

it('should throw put security errors', async () => {
sinon.stub(env, 'serverUrl').get(() => 'http://admin:pass@couchdb:5984');
sinon.stub(rpn, 'get').resolves({ members: { roles: ['role2'] } });
sinon.stub(rpn, 'put').rejects(new Error('forbidden or something'));

await expect(db.addRoleToSecurity('data', 'attr')).to.be.rejectedWith(Error, 'forbidden or something');
});

it('should throw error when security is invalid', async () => {
sinon.stub(env, 'serverUrl').get(() => 'http://admin:pass@couchdb:5984');
sinon.stub(rpn, 'get').resolves({ members: 'this is actually a string' });

await expect(db.addRoleToSecurity('data', 'attr')).to.be.rejectedWith(Error, 'Invalid database security');
});

it('should skip when missing dbname or role', async () => {
await db.addRoleToSecurity('', 'arole');
await db.addRoleToSecurity('dbanme', );
});
});
});
35 changes: 35 additions & 0 deletions api/tests/mocha/services/config-watcher.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ const generateXform = require('../../../src/services/generate-xform');
const config = require('../../../src/config');
const bootstrap = require('../../../src/services/config-watcher');
const manifest = require('../../../src/services/manifest');
const environment = require('../../../src/environment');

let on;
const emitChange = (change) => {
Expand Down Expand Up @@ -226,12 +227,46 @@ describe('Configuration', () => {
it('reloads settings settings doc is updated', () => {
settingsService.update.resolves();
settingsService.get.resolves({ settings: 'yes' });
sinon.stub(db, 'addRoleToSecurity');
sinon.stub(config, 'get').withArgs('roles').returns({ chw: {} });
sinon.stub(environment, 'db').get(() => 'medicdb');

return emitChange({ id: 'settings' }).then(() => {
chai.expect(settingsService.update.callCount).to.equal(1);
chai.expect(settingsService.get.callCount).to.equal(1);
chai.expect(config.set.callCount).to.equal(1);
chai.expect(config.set.args[0]).to.deep.equal([{ settings: 'yes' }]);
chai.expect(config.get.withArgs('roles').callCount).to.equal(1);
chai.expect(db.addRoleToSecurity.args).to.deep.equal([['medicdb', 'chw', false]]);
});
});

it('should add all configured user roles to the main database', () => {
settingsService.update.resolves();
settingsService.get.resolves({ settings: 'yes' });
sinon.stub(db, 'addRoleToSecurity');
sinon.stub(config, 'get')
.withArgs('roles')
.returns({
chw1: {},
chw2: {},
chw3: {},
chw4: {},
});
sinon.stub(environment, 'db').get(() => 'medicdb');

return emitChange({ id: 'settings' }).then(() => {
chai.expect(settingsService.update.callCount).to.equal(1);
chai.expect(settingsService.get.callCount).to.equal(1);
chai.expect(config.set.callCount).to.equal(1);
chai.expect(config.set.args[0]).to.deep.equal([{ settings: 'yes' }]);
chai.expect(config.get.withArgs('roles').callCount).to.equal(1);
chai.expect(db.addRoleToSecurity.args).to.deep.equal([
['medicdb', 'chw1', false],
['medicdb', 'chw2', false],
['medicdb', 'chw3', false],
['medicdb', 'chw4', false],
]);
});
});

Expand Down
Loading