Skip to content

Commit

Permalink
TestNet 04-18-20 (#33)
Browse files Browse the repository at this point in the history
* Roll up 4-14-20 (#1)

* only generate enclave pem when needed, use absolute path (#1)

* switch local network to not use ssl (#6)

* This commit moves mobilecoin to the mobilecoinofficial fork of prost (#8)

* This commit moves mobilecoin to the mobilecoinofficial fork of prost

Removing cbeck88 permissions controls
I have ensured that there are branch protection rules covering the
commits

* Fix view enclave cargo toml to have the same revision as others

* Fix cargo.lock files

* Remove selfsigned from README (#9)

* Adds Java namespaces to protocol buffers (#12)

* fix ecies MC-1216 (#11)

* fix ecies MC-1216

was mobilecoinofficial/mobilecoin-internal#321

changes since then:
- Removed alloc feature
- Added *_in_place_detached api (like aead crate)
  This reduces the amount of noise in the actual crypto part,
  the noise being "which bytes go where in the buffer"
- Marked the `encrypt_into` and `decrypt_into` apis as not public,
  because those APIs suck, it should really be as much like aead
  crate as possible, which is a better thought-out API

* add comments about fixing part of API

* those APIs have to be public for now, sigh. maybe they aren't so bad

* Add comments about API

* [MC-1172] rm tranasction::encoders

* Reorganize SCP to Cargo standards (#18)

* Make /opt/intel/sgxsdk/lib64 part of LD_LIBRARY_PATH in dockerfile (#21)

* Make /opt/intel/sgxsdk/lib64 part of LD_LIBRARY_PATH in dockerfile

and uprev the dockerfile.

This intended to fix ci in PR 14

This fixes issues like `...epid_sim.so` not being found by the test
targets. It is not getting installed in `/opt/intel/sgxsdk/sdk_libs`,
it is getting installed in the path mentioned

```
     Running target/debug/deps/tx_recovery-3449d1ea71010602
/tmp/mobilenode/target/debug/deps/tx_recovery-3449d1ea71010602: error while loading shared libraries: libsgx_epid_sim.so: cannot open shared object file: No such file or directory
error: test failed, to rerun pass '-p fog_ingest_server --test tx_recovery'
root@cb7f949bccb2:/tmp/mobilenode# ldd /tmp/mobilenode/target/debug/deps/tx_recovery-3449d1ea71010602
	linux-vdso.so.1 (0x00007ffc62df8000)
	libsgx_epid_sim.so => not found
	libsgx_urts_sim.so => /opt/intel/sgxsdk/sdk_libs/libsgx_urts_sim.so (0x00007fc5c8061000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc5c5fd9000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc5c5dd5000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc5c5bcd000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc5c59ae000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc5c5796000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc5c53a5000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc5c7e6c000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc5c5007000)
	libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fc5c4b3c000)
	libsgx_uae_service_sim.so => /opt/intel/sgxsdk/sdk_libs/libsgx_uae_service_sim.so (0x00007fc5c8059000)
root@cb7f949bccb2:/tmp/mobilenode# ls /opt/intel/sgxsdk/sdk_libs/libsgx_u
libsgx_uae_service_sim.so  libsgx_urts_sim.so
root@cb7f949bccb2:/tmp/mobilenode# ls /opt/intel/sgxsdk/sdk_libs/
libsgx_uae_service_sim.so  libsgx_urts_sim.so
root@cb7f949bccb2:/tmp/mobilenode#
```

* Make circle ci not source the intel sgx environment

As much as possible, the setting up of the build enviornment should
be done in the Dockerfile. Not duplicating these lines throughout
jenkins, k8s, mob tool, and README is a good thing. (they can be
in readme for people who don't want to use the container.)

* mobilecoind: b58 endpoints (#22)

* mobilecoind b58 rpc endpoints

* mobilecoind2: implement generate transfer code tx

* test_generate_transfer_code_tx

* Basic framework for Java mobilecoind client based on Gradle (#23)

* Fixes class name in build (#24)

* Replaces "Mobilenode" with "MobileCoin" in several READMEs (#4)

* Replaces Mobilenode with MobileCoin in several READMEs

* Adds src/README.md

* avoid "consensus node" language

* Update cloudbuild/README.md

Co-Authored-By: Robb Walters <[email protected]>

Co-authored-by: Robb Walters <[email protected]>

* Move ledger_enclave_server.proto from mobilecoin_api to fog_api (#25)

* Move ledger_enclave_server.proto from mobilecoin_api to fog_api

* fix build

* Actually fix build

* Check outbuf_used for null in mobileenclave_call (#17)

Merging this.

* Tx uses SignatureRctBulletproofs (#2)

* Applies patch from mobilecoin-internal

* unit tests encodings

* Removes unwrap in validate_transaction_signature

* Re-enables test_validate_key_images_are_unique_rejects_duplicate

* Removes unwraps in TransactionBuilder::build

* fmt

* Increases proptest cases, reorders imports

* Adds CL params to Java code along with monitors and balance check (#33)

* Adds CL params to Java code along with monitors and balance check

* Fix help message

* Build comments

* Prettify

* Unused import

* Added a README, changed the language and flags around entropy

* parameter fix

* README format

* Update proto to reflect current mobilecoind API

* First round of suggested fixes

* Adds ssl flag

* Fix the 'no-vars-given' case. (#38)

* Upgrade sentry to 0.18 (#36)

* upgrade sentry to 0.18

* lock files

* Implement mob client python session (#30)

* Implement mob client python session

* Reference sign up rather than create account

* Start testnet client script

* Rename and introduce exchange

* start-testnet-mobilecoind

* Add public_address

* Transfer to public address

* Versioned ecies (#28)

* Do another pass on ecies API, `encrypt_into` -> `encrypt_in_place`

This also allows `encrypt` to return an error, previously I didn't
allow that, but I looked in tarcieri's actual aes-gcm crate, and
it does return an error if the plaintext is larger than a huge number.

I think it might be more sensible to panic there, but anyways,
I'd like to make the `ecies` API close to `aead` and make it generic.

In order to make that change, I needed to fix the places that were
doing `encrypt_into` and `decrypt_into`, which were in the fog hints
in transaction crate. So this is a good case study of how the API
wokrs out.

I also added a special wrapper over `&mut [u8]` called `FixedBuffer`
to try to make using it nicer and close to how it worked without
the `encrypt_into` functions.

If we like `FixedBuffer` then I think we should try to open a PR
to `aead` crate and see if Tony wants it.

LMK what you think -- if we're happy with this, then in the next
PR I'm going to turn this into a trait, then make a version of it
that includes two "version tag" bytes so that we can have a nice
forwards-and-backwards-compatible wire format for the ECIES
ciphertexts. Once they get into the blockchain and into the recovery
db we cannot easliy change the algorithm if we don't have that.

* fix bug

* fix tests

* fix clippy

* Create versioned Ecies wire format, integrated with Ecies trait

* Add and use encrypt_fixed_length APIs for ECIES

This is much cleaner than the FixedBuffer thing

* Add better docu, references, naming, per code review comments

trait ECIES -> RistrettoEcies

* fixup previous

* Rename `ecies` to `ristretto-box`, and better README / docu

* Rename again per @jcape

* Move crate `public/crypto/mc-crypto-box` to `public/crypto/box` per discussion

* Additional functions in Java client for request codes and transfers (#42)

* Additional functions in Java client for request codes and transfers

* Document transfer function

* Change target to recipient

* 'host' = 'server'

* Client subaddress (#40)

* Use account/subaddress syntax

* Flesh out new account

* introduce mc-grpc-build and use it mobilecoind-api (#41)

* introduce mc-grpc-build and use it mobilecoind-api

* delete old autogenerated code

* build issue fix and comments

* grpc-build -> build-grpc

* comment and lock file

* readme and lock file

* use mcbuild-utils

Co-authored-by: Eran Rundstein <[email protected]>
Co-authored-by: Chris Beck <[email protected]>
Co-authored-by: tsegaran <[email protected]>
Co-authored-by: Robb Walters <[email protected]>
Co-authored-by: m a t t f a u l k n e r <[email protected]>
Co-authored-by: Brian Anderson <[email protected]>
Co-authored-by: James Cape <[email protected]>

* Roll up 4-15-20 (#2)

* introduce mc-grpc-build and use it mobilecoind-api (#41)

* introduce mc-grpc-build and use it mobilecoind-api

* delete old autogenerated code

* build issue fix and comments

* grpc-build -> build-grpc

* comment and lock file

* readme and lock file

* use mcbuild-utils

* Propose Values in Slot (#16)

* Adds a status call to the Java client (#44)

* Adds transaction receipt and status checks to Java client

* Document parameters for status function

* Balance fix (#46)

* Client fixes

* Better error display

* Pep8

* Small fixups in mc-crypto-box README, per joey feedback (#45)

I tried to make the commentary about the user-provided nonce
more accurate as well

* Rewrite the README for digestible crate (#47)

* Rewrite the README for digestible crate

* Add another sentence

* Add another sentence about framing, after reading again

* Fix typo

* Update public/crypto/digestible/README.md

Co-Authored-By: sugargoat <[email protected]>

* Update public/crypto/digestible/README.md

Co-Authored-By: sugargoat <[email protected]>

* Update public/crypto/digestible/README.md

Co-Authored-By: sugargoat <[email protected]>

* Try to fix the sentences sarah commented on, fixup conclusion

Co-authored-by: sugargoat <[email protected]>

* Updates README for java client and a couple of minor fixes (#54)

* Fix missing check (#51)

* Fix missing check

* Speed up grpc install

* Better wording

* Update path for test network (#55)

* Update READMEs with sigstruct (#7)

* Update READMEs with sigstruct

* Add css info

* Add signed enclave info

* Remove aws

* Fix typo

* Remove privkey option and provide signed and css to consensus

* Add IAS_MODE to mobilecoind

Co-authored-by: Eran Rundstein <[email protected]>
Co-authored-by: Robb Walters <[email protected]>
Co-authored-by: tsegaran <[email protected]>
Co-authored-by: garbageslam <[email protected]>

* Switch to build-grpc (#7)

* switch mobilecoin-api to using the new build-grpc crate

* switch attest-api to using the new build-grpc crate

* switch grpc-util to using the new build-grpc crate

* CircleCI build improvements (#8)

* add circleci task for running "cargo build", dedupe pem file generation

* fix spacing

* call check-dirty-git

* lint and save caches in the faster job

* rename job

* fix typo in check-dirty-git

* fix typo

* Changes from internal repo (#5)

* Removes MAX_TINY_MOB (#9)

* Fix crypto directory README (#3)

* MC-1283: Export protos directories as cargo depvars (#10)

* moves tombstone_block from Tx to TxPrefix (#13)

* Nicer mob behavior (check for docker being installed) (#12)

This might help the user experience in issues like
#6

* Improvements to build instructions in README.md (#16)

* Improvements to build instructions in README.md

* Fixup SGX_MODE=SIM vs. SGX_MODE=SW, and give explanation about env vars

* small tweak

* Expand upon the enclave build part

* Spelling and grammar

* Simplify python example (#14)

* Simplify python example

* Update README

* Reference java example on top level readme

* Fixup markdown rendering in various readme's (#19)

This is what I get for not rendering them locally before making PR

* Update PROD endpoint (#21)

* Unit tests that TransactionBuilder returns error if value is not conserved (#22)

* Amount and signatures use CompressedRistretto (#17)

* Amount and signatures use CompressedRstretto

* Comment cleanup

* Unit test for Commitment

* Unit test for CompressedCommitment

* Comment fixes

* Extracts ring decompression into a function

* Comment fix

* RingMLSAG derives Message (#23)

* MC-1292 Replace MIN_RING_SIZE and MAX_RING_SIZE with RING_SIZE (#20)

* Uses single RING_Size constant

* Changes MIN_RING_SIZE to RING_SIZE

* Restores test_validate_ring_sizes

* Update consensus/api/proto/consensus_common.proto

Co-Authored-By: James Cape <[email protected]>

* Restores error variant

Co-authored-by: James Cape <[email protected]>

* Improve readme (#27)

* Improve readme

Thanks to @joekottke for suggestion to use markdown footnotes in
the earlier PR comments

* Remove footnote extension, it's not supported in githug-flavored-markdown

* SignatureRctBulletproofs derives Message (#25)

* SignatureRctBulletproofs derives Message

* Removes dead code

* Update command with missing params (#26)

* Update command

* peer-responder-id does not have public key

* Typo fix

* Example testnet client (#29)

* wip

* wip

* mob and such

* begin rpc integration

* iterate on the flow

* flow works

* command line args, comments

* check that mobilecoind is running

* clippy

Co-authored-by: Eran Rundstein <[email protected]>
Co-authored-by: Chris Beck <[email protected]>
Co-authored-by: tsegaran <[email protected]>
Co-authored-by: Robb Walters <[email protected]>
Co-authored-by: m a t t f a u l k n e r <[email protected]>
Co-authored-by: Brian Anderson <[email protected]>
Co-authored-by: James Cape <[email protected]>
  • Loading branch information
8 people committed Apr 19, 2020
1 parent e57b690 commit bceca62
Show file tree
Hide file tree
Showing 118 changed files with 6,744 additions and 4,699 deletions.
67 changes: 39 additions & 28 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
version: 2.1

defaults:
builder-install: &builder-install gcr.io/mobilenode-211420/builder-install:1_8
builder-install: &builder-install gcr.io/mobilenode-211420/builder-install:1_9

executors:
build-executor:
Expand Down Expand Up @@ -103,11 +103,26 @@ commands:
paths:
- "~/.cargo"

build_setup:
generate-pem-file:
steps:
- run:
name: Generating Enclave_private.pem
command: |
openssl genrsa -out $(pwd)/Enclave_private.pem -3 3072
export CONSENSUS_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
export INGEST_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
export LEDGER_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
export VIEW_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
prepare-for-build:
steps:
- checkout
- print_versions
- env_setup
- enable_sccache
- restore-cargo-cache
- restore-sccache-cache
- generate-pem-file

check-dirty-git:
steps:
Expand All @@ -121,6 +136,7 @@ commands:
echo "repo is dirty"
git status
exit 1
fi
jobs:
# A job that builds all the tests in the workspace, and stores them in a test-bins/ directory.
Expand All @@ -129,11 +145,7 @@ jobs:
build-parallel-tests:
executor: build-executor
steps:
- build_setup
- enable_sccache
- restore-cargo-cache
- restore-sccache-cache

- prepare-for-build
- run:
name: Build/prepare unit tests
command: |
Expand Down Expand Up @@ -202,7 +214,6 @@ jobs:
# Crates that define macros (e.g. `digestible_derive`) link dynamically against libtest*.so, which sits here.
export LD_LIBRARY_PATH="$HOME/.rustup/toolchains/$(rustup show active-toolchain | awk '{print $1}')/lib"
source /opt/intel/sgxsdk/environment
echo "LD_LIBRARY_PATH = $LD_LIBRARY_PATH"
# Run the test binaries for all the packages we're assigned by CircleCI's test splitting mechanism,
Expand Down Expand Up @@ -235,44 +246,44 @@ jobs:
executor: build-executor
parallelism: 1
steps:
- build_setup
- enable_sccache
- restore-cargo-cache
- restore-sccache-cache
- prepare-for-build
- run:
name: Run all unit tests
command: |
openssl genrsa -out $(pwd)/Enclave_private.pem -3 3072
export CONSENSUS_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
export INGEST_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
export LEDGER_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
export VIEW_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
cargo test
- check-dirty-git

rm $(pwd)/Enclave_private.pem
# Build and lint in debug mode
build-all-and-lint-debug:
executor: build-executor
steps:
- prepare-for-build
- run:
name: Lint/fmt
name: Cargo build
command: |
openssl genrsa -out $(pwd)/Enclave_private.pem -3 3072
export CONSENSUS_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
export INGEST_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
export LEDGER_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
export VIEW_ENCLAVE_PRIVKEY=$(pwd)/Enclave_private.pem
cargo build
- check-dirty-git

# The lint and saving of caches happens here since this job is faster than the run-all-tests job.
# This results in shorter CI times.
- run:
name: Lint/fmt
command: |
./tools/lint.sh
rm $(pwd)/Enclave_private.pem
- save-cargo-cache
- save-sccache-cache

workflows:
version: 2
# Build and run tests on a single container
run-all-tests:
build-and-run-all-tests:
jobs:
# Run tests on a single container
- run-all-tests

# Build everything in debug
- build-all-and-lint-debug

# Build and run tests in parallel - not needed at the moment since the test suite is fast enough.
# build-and-run-tests:
# jobs:
Expand Down
20 changes: 0 additions & 20 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -33,26 +33,6 @@ logs
*.so
*.a

# grpc
attest_api/src/attest.rs
attest_api/src/attest_grpc.rs
grpc_util/src/health_api.rs
grpc_util/src/health_api_grpc.rs
consensus/api/src/blockchain.rs
consensus/api/src/blockchain_grpc.rs
consensus/api/src/consensus_client.rs
consensus/api/src/consensus_client_grpc.rs
consensus/api/src/consensus_common.rs
consensus/api/src/consensus_peer.rs
consensus/api/src/consensus_peer_grpc.rs
consensus/api/src/external.rs
consensus/api/src/ledger_enclave_server.rs
consensus/api/src/ledger_enclave_server_grpc.rs
consensus/api/src/ledger_server.rs
consensus/api/src/ledger_server_grpc.rs
mobilecoind/api/src/mobilecoind_api.rs
mobilecoind/api/src/mobilecoind_api_grpc.rs

# Random junk
.DS_Store
.factorypath
Expand Down
105 changes: 105 additions & 0 deletions BUILD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
Build
=====

## Build environment

Services that create SGX enclaves depend on the Intel SGX SDK. This must be installed
in the build environment, as well as the runtime environment.

#### Dockerized build

An easy way to get this environment is to build in the docker image that we use for CI.
The dockerfile for this image lives in `docker/Dockerfile`.

You can use `./mob prompt` to pull this image, (or to build it locally), and get a prompt
in this environment.

```
# From the root of the repo
./mob prompt
# At the container prompt
cargo build
```

If you have SGX-enabled hardware (activated in BIOS, and with SGX kernel module installed),
you can use `./mob prompt --hw` to get SGX in the container. Then you can both build and
run the tests in `SGX_MODE=HW`. (See below for an explanation.)

#### No-docker build

A docker-less build also works fine for development:
- Follow instructions [consensus/service/BUILD.md](consensus/service/BUILD.md)
- Set up your environment like the [Dockerfile](docker/Dockerfile)

## Build configuration

There are two project-wide SGX-related configuration variables `SGX_MODE` and `IAS_MODE`.

These are set by environment variables, and they must be the same for all artifacts,
even those that don't depend directly on SGX. E.g. `mobilecoind` must have the same configuration
as `consensus_service` for Intel Remote Attestation to work, otherwise an error will occur at runtime.

For testing, you should usually use `SGX_MODE=SW` and `IAS_MODE=DEV`.

#### SGX_MODE

`SGX_MODE=SW` means that the enclaves won't be "real" enclaves -- consensus service will link
to Intel-provided "_sim" versions of the Intel SGX SDK, and the enclave will be loaded approximately
like a shared library being `dlopen`'ed. This means that you will be able to use `gdb` and get
backtraces normally through the enclave code. In this mode, the CPU does not securely compute
measurements of the enclave, and attestation doesn't prove the integrity of the enclave.

`SGX_MODE=HW` means that the real Intel libraries are used, and the enclave is loaded securely.
This mode is required for Intel Remote Attestation to work and provide security.

The clients and servers must all agree about this setting, or attestation will fail.

#### IAS_MODE

`IAS_MODE=DEV` means that we will hit the Intel provided "dev endpoints" during remote attestation.
These won't require the real production signing key in connection to the MRENCLAVE measurements.

`IAS_MODE=PROD` means that we will hit the real Intel provided endpoints for remote attestation.

In code, this discrepancy is largely handled by the `attest-net` crate.

The clients and servers must all agree about this setting, or attestation will fail.

#### Why are these environment variables?

`cargo` supports crate-level features, and feature unification across the build plan.
`cargo` does not support any notion of "global project-wide configuration".

In practice, it's too hard invoke cargo to get all the features enabled exactly correctly on
all the right crates, if every crate has an `sgx_mode` and `ias_mode` feature.

Even if cargo had workspace-level features, which it doesn't, that wouldn't be good enough for us
because our build requires using multiple workspaces. We must keep the cargo features on some
targets separated and not unified.
Unifying cargo features across enclave targets and server targets will break the enclave builds.
This is because the enclave builds in a special `no_std` environment.

Making `SGX_MODE` and `IAS_MODE` environment variables, and making `build.rs` scripts that read
them and set features on these crates as needed, is the simplest way to make sure that there is
one source of truth for these values for all of the artifacts in the whole build.

The `SGX_MODE` environment variable configuration is also used throughout Intel SGX SDK examples.

## Building the enclave

For technical reasons, the `consensus_enclave` must be in a separate workspace.
It is also built using `cargo build`.

The enclave build is invoked *automatically* if needed from the `consensus_service` `build.rs`.

To reproducibly build the enclave, (get exactly the right MRENCLAVE value), you must build
in the container.

For local testing, you don't need to get exactly the right MRENCLAVE value. You can set up
test networks with whatever MRENCLAVE your build produces, and clients that check this value
using the Remote Attestation process.

If you want to download a prebuilt enclave, signed using the production signing key, in order use `IAS_MODE=PROD`
and participate in a production-environment network, check out the `enclave-signing-material` instructions:
https://github.com/mobilecoinofficial/mobilecoin/blob/master/consensus/service/BUILD.md#enclave-signing-material
Loading

0 comments on commit bceca62

Please sign in to comment.