Skip to content

Commit

Permalink
Upstream sync (#3)
Browse files Browse the repository at this point in the history
* Updating docs to reflect new CLI syntax (fnproject#138)

* Updating docs to reflect new CLI syntax

* Missed a few

* fn-java-fdk: post-1.0.60 version bump [skip ci]

* remove printenv

* manual version bump

* fn-java-fdk: post-1.0.62 version bump [skip ci]

* Update FN container name (fnproject#139) [skip ci]

Update FN container name from functions to fnserver at grabbing internal docker network IP

* Updated dependency versions to reduce security problems (fnproject#143)

Using snyk.io tools we found several dependencies had security vulnerabilities
and could be updated. This does not remove all of them, but the remaining CVEs
are in transitive dependencies brought in by 3rd party code.

CVEs remain in:

async-thumbnails example (low priority to fix) through
  - minio
  - wiremock

fn-spring-cloud-function through
  - spring-cloud-function-context

Additional changes were needed to adjust for API changes in updated dependencies:

async-thumbnails example:
  - new minio client uses chunked file uploads so assertions needed changing

fn-spring-cloud-function:
  - Several breaking API changes, and it seems now that Consumer behaviour is broken.

* fn-java-fdk: post-1.0.63 version bump [skip ci]

* preparations for the junit5 module

- separation of the junit4 and junit5 code into different modules.
- core module to share code between different TDD frameworks

* cleaning of unused folders/modules

* fixed deps in build-image module

* fixed integration-tests examples

* fixed deps in build-image module

* deactivated integration-test 3 and 7 - cycle dependency to cli

* fn-java-fdk: post-1.0.64 version bump [skip ci]

* pom.xml files - refactoring

* switched all versions to the latest available in maven central

* switched all plugin-versions to the latest available in maven central

* removed comment

* will make the google compiler happy...

* fn-java-fdk: post-1.0.65 version bump [skip ci]

* Add Unix Stream codec, Split flow from FDK, update integration tests (fnproject#150)

* http input codec

* add jetty-core and UDS deps

* PoC jetty core UDS event codec

* revise FDK contract for initial relase, split flow from fn, support new unix domain socket contract

* First pass of refactor

* added annotations for for features to support flow

* moving flow feature to runtime package

* rename flow testing and add docs, fix race in Stream testing

* fixing failing tests

* Update to reflect FN_ID

* more integration test fixing

* fix up format name

* first stab at home-grown UDS implementation

*  docker build fix

* UDS rework, remove old UDS dep and test new one

* adjust permissions on file on start

* new integration tests

* simlify build and integration tests, add tests for http-stream

* fix ci

* typo

* make igntests standalone

* itests run 1.8

* fix itest result

* env for itests

* add createDir

* test failures in circle

* fix test interference

* Use standard properties in fdk tests, fix versions in itests

* hmm no entropy on circle boxes :)

* use consistent properties for versions everywhere

* add waits for start

* this time with feeling

* diagnosing timeout

* close input, try more aggressive testing

* C errors

* fixing up accept and twiddling with integration test start

* set DOCKER_LOCALHOST

* integration test fixing/finessing

* cludge docker localhost

* no_proxy messing

* set fnserver IP correctly

* typo

* nits in C code

* review nits

* more C fix nits

* Make mandatory headers mandatory again

* minus comment

* remove Content-Length from stripped input

* fix call test

* bump version to fix broken release [skip ci]

* Initial cut at HTTP gateway prtotocol support

* regen sig file

* unit tests on HTTPGw Stuff

* ign tests code

* integration tests

* update QR code demo to be HTTP Gateway friendly

* tests and docs, organize inputs

* Readme typos

* fn-java-fdk: post-1.0.67 version bump [skip ci]

* Fix typos, minor lints, and a small race condition in stream codec cleanup (fnproject#155)

* Fix typos, minor lints, and a small race condition in stream codec cleanup

* review feedback

* debug log

* add more logging

* mask binary stuff in c debug code

* Fix issue with uninitialised addres in accept

* default REPO location in ign tests, fix race condition in Flow test tests

* fix typo from fnproject#154

* bump jetty client to latest

* fn-java-fdk: post-1.0.68 version bump [skip ci]

* trying new delpoyment setting

* fn-java-fdk: post-1.0.69 version bump [skip ci]

* Try another new deployment setting (fnproject#158)

* fn-java-fdk: post-1.0.70 version bump [skip ci]

* make FnFeature multi-valued remove use of getAnnotationsByType (fnproject#160)

* fn-java-fdk: post-1.0.71 version bump [skip ci]

* Build and Init Images for substrate VM 

This adds a a build and init image that builds functions using substrateVM as consolidated binaries.

* fn-java-fdk: post-1.0.72 version bump [skip ci]

* CI/Release of  substrate related images (fnproject#161)

* first cut at substrate images

* fix docker build

* bump fdk for release

* fn-java-fdk: post-1.0.74 version bump [skip ci]

* Rev  jackson-databind to deal with latest vulnerabilities.  (fnproject#163)

* Fix Jackson databind to deal with latest CVE
* rev minio version used in async thumbnails test to pick up newer jetty version
* fix up async thumbnails to test new minio version

* fn-java-fdk: post-1.0.75 version bump [skip ci]

* [164] Support http-stream protocol in substrate/native java.

* This gets the local build.sh working again on the latest JDK8

* Remove comment

* Include some CI fixes from @tteggel

* Update file permissions on surefire CI fix script

* Remove old default and http formats from boilerplate test

* Add some logging to boilerplate test

* Update FunctionsTest.java

* Update FunctionsTest.java

* Update FunctionsTest.java

* Update FunctionsTest.java

* Update FunctionsTest.java

* Added some output to help if fn operation fails

* Update IntegrationTestRule.java

* Update IntegrationTestRule.java

* Update IntegrationTestRule.java

* Update pom.xml

* tweak to surefire version and config

* Add specific 8-friendly test and pom for boilerplate test.

* Update FunctionsTest.java

* Another attempt at logging a failed result...sigh

* Rename func.yaml to prevent overwrite

* Let's go again...

* fn-java-fdk: post-1.0.76 version bump [skip ci]

* fn-java-fdk: post-1.0.77 version bump [skip ci]

* Add jdk11 build/runtime images (fnproject#169)

* Add build for J11 images
* Remove JDK9 VM memory options

* fn-java-fdk: post-1.0.78 version bump [skip ci]

* Use JRE instead of JDK for 11 runtime image, and add push of 11 images into CI release.sh

* Reinstate SNAPSHOT version in init-native pom

* fn-java-fdk: post-1.0.79 version bump [skip ci]

* Replace java9 tests with java11

* Ooops...

* Add JDK9 test back whilst CLI still supports it.

* fn-java-fdk: post-1.0.80 version bump [skip ci]

* Bump jackson version

* fn-java-fdk: post-1.0.81 version bump [skip ci]

* removed the (now) useless Docker & Format entries (fnproject#178)

* fn-java-fdk: post-1.0.82 version bump [skip ci]

* Native java fixes (fnproject#179)

* Fixed maven surefire classloader.

* Fixed SVM problem with max heap size in container.

* fn-java-fdk: post-1.0.83 version bump [skip ci]

* Removed old format codecs and tests

* Remove Java 9 dockerfiles, update builds and tests

* Update tests to java11

* Fix broken release

* Bump version number past previous broken release

* fn-java-fdk: post-1.0.85 version bump [skip ci]

* Update HTTPGatewayFunctions.md

* removed some legacy stuff (fn call & route)

* A first cut at updating and simplifying the Java FDK Readme. Removed user tutorial already included in tutorials. Reorganized and simplified. More work to do reviewing each of the docs linked here.

* simplifying the fdk repo readme along with mikes update

* Update README.md

* add generic link back to official docs

* WIP fn user added to runtime image, working on adding user to the init-image

* added fn user to runtime java

* changes to RUN cmd syntax

* trying to run fn user with correct cli flags

* added user name

* added gid

* create app before deploy app

* remove --local from create app

* create app before deploy in all integ tests

* deploy after context created

* --create-app change to integ tests

* removal of comments

* fn-java-fdk: post-1.0.86 version bump [skip ci]

* Update Java FDK string-reverse example

* post-1.0.87 bump (manual)

* fn-java-fdk: post-1.0.88 version bump [skip ci]

* WIP adding logframer and tests to fdk-java, before each function invocation

* WIP debugging test

* Quick fix..?

* checking if env variables are logged out to stdout and stderr, stdout gets redirected to stderr hence the duplication in the assertion

* removing comments of test in wrong file

* fn-java-fdk: post-1.0.89 version bump [skip ci]

* adding explicit newline characters in print statements

* editing tests

* correcting \n

* fn-java-fdk: post-1.0.90 version bump [skip ci]

* Update pom.xml so circleci test will pass. JDK 11 not allowed currently.

* fnproject/functions -> fnproject/fnserver

* add fn user to jre11 runtime image

* fn-java-fdk: post-1.0.91 version bump [skip ci]

* fn-java-fdk: post-1.0.92 version bump [skip ci]

* Re-order release artifact publishing

By publishing tag to github first we ensure that no artifacts are pushed
for a version which isn't tagged.
Publishing to bintray last ensures that the CLI won't pick up a version
which is not pushed everywhere.

* Post-1.0.93 version bump [skip ci]

* fn-java-fdk: post-1.0.94 version bump [skip ci]

* Skip deployment of example code

Do not attempt to publish artifacts for the string-reverse exapmle
function

* fn-java-fdk: post-1.0.95 version bump [skip ci]

* Bump Jackson version

* fn-java-fdk: post-1.0.96 version bump [skip ci]

* Fix path of java binary in Dockerfile (fnproject#210)

* fn-java-fdk: post-1.0.97 version bump [skip ci]

* Updated to GraalVM 19. (fnproject#209)

* fn-java-fdk: post-1.0.98 version bump [skip ci]

* Add env to test server to allow setting fn limits (fnproject#214)

work around changed paths in new j11 image

* fn-java-fdk: post-1.0.99 version bump [skip ci]

* Updated native-image to GraalVM 19.2.0. (fnproject#213)

* Updated to GraalVM 19.2.0.

* Fixed build-native Dockerfile.

* fn-java-fdk: post-1.0.100 version bump [skip ci]

* Bump Jackson version to 2.9.10 (fnproject#216)

* fn-java-fdk: post-1.0.101 version bump [skip ci]

* Update all names to Function Development Kit for Java per Shaun, per Legal, and for consistency.

* Removed hyphen as best practice seems to be no hyphen

* Updating abbrev to FDK for Java

* fn-java-fdk: post-1.0.102 version bump [skip ci]
  • Loading branch information
denis256 authored Oct 13, 2019
1 parent 3130cc2 commit 88b76f5
Show file tree
Hide file tree
Showing 324 changed files with 8,685 additions and 5,447 deletions.
101 changes: 29 additions & 72 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,27 @@ version: 2
jobs:
build:
working_directory: ~/fn-java-fdk
machine:
java:
version: oraclejdk8
machine: true
environment:
# store_artifacts doesn't shell substitute so the variable
# definitions are duplicated in those steps too.
FDK_ARTIFACT_DIR: /tmp/artifacts/fdk
TEST_ARTIFACT_DIR: /tmp/artifacts/tests
STAGING_DIR: /tmp/staging-repository
REPOSITORY_LOCATION: /tmp/staging_repo
steps:
- checkout
- run:
name: Update Docker to latest
command: ./.circleci/install-docker.sh
- run:
name: Install fn binary (as it is needed for the integration tests)
command: ./.circleci/install-fn.sh
- run:
name: Workaround for https://issues.apache.org/jira/browse/SUREFIRE-1588
command: ./.circleci/fix-java-for-surefire.sh
- run:
name: Install junit-merge
command: npm install -g junit-merge
- run:
name: Set release to latest branch version
command: |
Expand All @@ -22,83 +32,22 @@ jobs:
name: Determine the release version
command: ./.circleci/update-versions.sh
- run:
name: Build and deploy locally
name: Build and Test FDK
command: |
rm -rf $STAGING_DIR
mkdir -p $STAGING_DIR
mvn deploy -DaltDeploymentRepository=localStagingDir::default::file://"$STAGING_DIR"
- store_test_results:
path: runtime/target/surefire-reports
- store_test_results:
path: testing/target/surefire-reports

export FN_FDK_VERSION=$(cat ./release.version)
./build.sh
- run:
name: Copy FDK artifacts to upload folder
name: Run integration tests
command: |
mkdir -p "$FDK_ARTIFACT_DIR"
cp -a api/target/*.jar "$FDK_ARTIFACT_DIR"
cp -a runtime/target/*.jar "$FDK_ARTIFACT_DIR"
- store_artifacts:
name: Upload FDK artifacts
path: /tmp/artifacts/fdk

- run:
name: Update Docker to latest
command: ./.circleci/install-docker.sh

export FN_JAVA_FDK_VERSION=$(cat release.version)
./integration-tests/run_tests_ci.sh
timeout: 1200
- run:
name: Login to Docker
command: |
if [[ "${CIRCLE_BRANCH}" == "master" && -z "${CIRCLE_PR_REPONAME}" ]]; then
docker login -u $DOCKER_USER -p $DOCKER_PASS
fi
- run:
name: Build fn-java-fdk Docker build image
command: |
cd build-image
./docker-build.sh -t fnproject/fn-java-fdk-build:$(cat ../release.version) .
- run:
name: Build fn-java-fdk-jdk9 Docker build image
command: |
cd build-image
./docker-build.sh -f Dockerfile-jdk9 -t fnproject/fn-java-fdk-build:jdk9-$(cat ../release.version) .
- run:
name: Build fn-java-fdk Docker runtime image
command: |
cd runtime
docker build -t fnproject/fn-java-fdk:$(cat ../release.version) .
- run:
name: Build fn-java-fdk-jdk9 Docker runtime image
command: |
cd runtime
docker build -f Dockerfile-jdk9 -t fnproject/fn-java-fdk:jdk9-$(cat ../release.version) .
- run:
name: Install fn binary (as it is needed for the integration tests)
command: ./.circleci/install-fn.sh

- run:
name: Run integration tests
command: REPOSITORY_LOCATION="$STAGING_DIR" FN_JAVA_FDK_VERSION=$(cat release.version) ./integration-tests/run-local.sh
timeout: 1200

- run:
name: Copy integration test results to test artifact dir
command: |
mkdir -p "$TEST_ARTIFACT_DIR"
for test_dir in ./integration-tests/main/test-*; do
test_name="$(basename "$test_dir")"
mkdir "${TEST_ARTIFACT_DIR}/${test_name}"
cp -a $(find "$test_dir" -maxdepth 1 -type f) "${TEST_ARTIFACT_DIR}/${test_name}"
done
find "${TEST_ARTIFACT_DIR}"
when: always
- store_artifacts:
name: Upload integration test results to artifacts
path: /tmp/artifacts/tests
- deploy:
name: Release new version
command: |
Expand All @@ -108,4 +57,12 @@ jobs:
git branch --set-upstream-to=origin/${CIRCLE_BRANCH} ${CIRCLE_BRANCH}
./.circleci/release.sh
fi
- run:
name: Gather test results
when: always
command: |
junit-merge $(find . -wholename "*/target/surefire-reports/*.xml") --createDir -o test_results/merged_results.xml
- store_test_results:
when: always
path: test_results

19 changes: 19 additions & 0 deletions .circleci/fix-java-for-surefire.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/usr/bin/env bash

set -ex

cat << 'EOF' > $HOME/.m2/settings.xml
<settings>
<profiles>
<profile>
<id>SUREFIRE-1588</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<argLine>-Djdk.net.URLClassPath.disableClassPathURLCheck=true</argLine>
</properties>
</profile>
</profiles>
</settings>
EOF
32 changes: 0 additions & 32 deletions .circleci/install-go.sh

This file was deleted.

101 changes: 59 additions & 42 deletions .circleci/release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@ USER=fnproject
SERVICE=fn-java-fdk
RUNTIME_IMAGE=${SERVICE}
BUILD_IMAGE=${SERVICE}-build
NATIVE_INIT_IMAGE=fn-java-native-init
NATIVE_BUILD_IMAGE=fn-java-native

release_version=$(cat release.version)
if [[ $release_version =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] ; then
if [[ ${release_version} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] ; then
echo "Deploying version $release_version"
else
echo Invalid version $release_version
Expand All @@ -21,63 +23,78 @@ version_parts=(${release_version//./ })
new_minor=$((${version_parts[2]}+1))
new_version="${version_parts[0]}.${version_parts[1]}.$new_minor"

if [[ $new_version =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] ; then
if [[ ${new_version} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] ; then
echo "Next version $new_version"
else
echo Invalid new version $new_version
echo Invalid new version ${new_version}
exit 1
fi

# Push result to git

# Deploy to bintray
mvn -s ./settings-deploy.xml \
-DskipTests \
-DaltDeploymentRepository="fnproject-release-repo::default::$MVN_RELEASE_REPO" \
-Dfnproject-release-repo.username="$MVN_RELEASE_USER" \
-Dfnproject-release-repo.password="$MVN_RELEASE_PASSWORD" \
-DdeployAtEnd=true \
clean deploy
echo ${new_version} > release.version
git tag -a "$release_version" -m "version $release_version"
git add release.version
git commit -m "$SERVICE: post-$release_version version bump [skip ci]"
git push
git push origin "$release_version"


# Regenerate runtime and build images and push them
(
moving_version=${release_version%.*}-latest

## jdk8 runtime
docker tag $USER/$RUNTIME_IMAGE:${release_version} $USER/$RUNTIME_IMAGE:latest
docker tag $USER/$RUNTIME_IMAGE:${release_version} $USER/$RUNTIME_IMAGE:${moving_version}
docker push $USER/$RUNTIME_IMAGE:latest
docker push $USER/$RUNTIME_IMAGE:${release_version}
docker push $USER/$RUNTIME_IMAGE:${moving_version}
docker tag ${USER}/${RUNTIME_IMAGE}:${release_version} ${USER}/${RUNTIME_IMAGE}:latest
docker tag ${USER}/${RUNTIME_IMAGE}:${release_version} ${USER}/${RUNTIME_IMAGE}:${moving_version}
docker push ${USER}/${RUNTIME_IMAGE}:latest
docker push ${USER}/${RUNTIME_IMAGE}:${release_version}
docker push ${USER}/${RUNTIME_IMAGE}:${moving_version}

## jdk8 build
docker tag $USER/$BUILD_IMAGE:${release_version} $USER/$BUILD_IMAGE:latest
docker tag $USER/$BUILD_IMAGE:${release_version} $USER/$BUILD_IMAGE:${moving_version}
docker push $USER/$BUILD_IMAGE:latest
docker push $USER/$BUILD_IMAGE:${release_version}
docker push $USER/$BUILD_IMAGE:${moving_version}

## jdk9 runtime
docker tag $USER/$RUNTIME_IMAGE:jdk9-${release_version} $USER/$RUNTIME_IMAGE:jdk9-latest
docker tag $USER/$RUNTIME_IMAGE:jdk9-${release_version} $USER/$RUNTIME_IMAGE:jdk9-${moving_version}
docker push $USER/$RUNTIME_IMAGE:jdk9-latest
docker push $USER/$RUNTIME_IMAGE:jdk9-${release_version}
docker push $USER/$RUNTIME_IMAGE:jdk9-${moving_version}

## jdk9 build
docker tag $USER/$BUILD_IMAGE:jdk9-${release_version} $USER/$BUILD_IMAGE:jdk9-latest
docker tag $USER/$BUILD_IMAGE:jdk9-${release_version} $USER/$BUILD_IMAGE:jdk9-${moving_version}
docker push $USER/$BUILD_IMAGE:jdk9-latest
docker push $USER/$BUILD_IMAGE:jdk9-${release_version}
docker push $USER/$BUILD_IMAGE:jdk9-${moving_version}
docker tag ${USER}/${BUILD_IMAGE}:${release_version} ${USER}/${BUILD_IMAGE}:latest
docker tag ${USER}/${BUILD_IMAGE}:${release_version} ${USER}/${BUILD_IMAGE}:${moving_version}
docker push ${USER}/${BUILD_IMAGE}:latest
docker push ${USER}/${BUILD_IMAGE}:${release_version}
docker push ${USER}/${BUILD_IMAGE}:${moving_version}

## jre11 runtime
docker tag ${USER}/${RUNTIME_IMAGE}:jre11-${release_version} ${USER}/${RUNTIME_IMAGE}:jre11-latest
docker tag ${USER}/${RUNTIME_IMAGE}:jre11-${release_version} ${USER}/${RUNTIME_IMAGE}:jre11-${moving_version}
docker push ${USER}/${RUNTIME_IMAGE}:jre11-latest
docker push ${USER}/${RUNTIME_IMAGE}:jre11-${release_version}
docker push ${USER}/${RUNTIME_IMAGE}:jre11-${moving_version}

## jdk11 build
docker tag ${USER}/${BUILD_IMAGE}:jdk11-${release_version} ${USER}/${BUILD_IMAGE}:jdk11-latest
docker tag ${USER}/${BUILD_IMAGE}:jdk11-${release_version} ${USER}/${BUILD_IMAGE}:jdk11-${moving_version}
docker push ${USER}/${BUILD_IMAGE}:jdk11-latest
docker push ${USER}/${BUILD_IMAGE}:jdk11-${release_version}
docker push ${USER}/${BUILD_IMAGE}:jdk11-${moving_version}

## native init image
docker tag ${USER}/${NATIVE_INIT_IMAGE}:${release_version} ${USER}/${NATIVE_INIT_IMAGE}:latest
docker tag ${USER}/${NATIVE_INIT_IMAGE}:${release_version} ${USER}/${NATIVE_INIT_IMAGE}:${moving_version}
docker push ${USER}/${NATIVE_INIT_IMAGE}:latest
docker push ${USER}/${NATIVE_INIT_IMAGE}:${release_version}
docker push ${USER}/${NATIVE_INIT_IMAGE}:${moving_version}

)

(
if [ -f images/build-native/native_build.image ] ; then
native_build_image=$(cat images/build-native/native_build.image)
docker tag ${native_build_image} ${USER}/${NATIVE_BUILD_IMAGE}:latest
docker push ${USER}/${NATIVE_BUILD_IMAGE}:latest
docker push ${native_build_image}
fi
)

# Push result to git

echo $new_version > release.version
git tag -a "$release_version" -m "version $release_version"
git add release.version
git commit -m "$SERVICE: post-$release_version version bump [skip ci]"
git push
git push origin "$release_version"
# Deploy to bintray
mvn -s ./settings-deploy.xml \
-DskipTests \
-DaltDeploymentRepository="fnproject-release-repo::default::${MVN_RELEASE_REPO}" \
-Dfnproject-release-repo.username="${MVN_RELEASE_USER}" \
-Dfnproject-release-repo.password="${MVN_RELEASE_PASSWORD}" \
clean deploy
7 changes: 6 additions & 1 deletion .circleci/update-versions.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
#!/bin/bash
#
# this is instead for the maven release plugin (which sucks) - this sets the versions across the project before a build
# for branch builds these are ignored (nothing is deployed)
# For master releases this sets the latest version that this branch would be released as
#

release_version=$(cat release.version)
if [[ $release_version =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] ; then
Expand All @@ -14,6 +19,6 @@ mvn versions:set -D newVersion=${release_version} versions:update-child-modules
# We need to replace the example dependency versions also
# (sed syntax for portability between MacOS and gnu)
find . -name pom.xml |
xargs -n 1 sed -i.bak -e "s|<fnproject\\.version>.*</fnproject\\.version>|<fnproject.version>${release_version}</fnproject.version>|"
xargs -n 1 sed -i.bak -e "s|<fdk\\.version>.*</fdk\\.version>|<fdk.version>${release_version}</fdk.version>|"
find . -name pom.xml.bak -delete

4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,7 @@ logs/
*.versionsBackup
.gradle
examples/gradle-build/build
**/*.classpath
**/*.project
**/*.settings

Loading

0 comments on commit 88b76f5

Please sign in to comment.