.github/workflows/zxc-verify-docker-build-determinism.yaml

fixing how files are copied and debugging #1

Workflow file for this run

.github/workflows/zxc-verify-docker-build-determinism.yaml at f23ea44

	##
	# Copyright (C) 2023-2025 Hedera Hashgraph, LLC
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software
	# distributed under the License is distributed on an "AS IS" BASIS,
	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	# See the License for the specific language governing permissions and
	# limitations under the License.
	##
	name: "ZXC: Verify Docker Build Determinism"
	# Here, the ZXC prefix:
	# Z - Ensures sort order such that this script appears at the bottom of the UI
	# X - Indicates it's not for direct user consumption
	# C - Indicates this is a 'workflow_call' based reusable workflow
	on:
	workflow_call:
	inputs:
	ref:
	description: "The branch, tag, or commit to checkout:"
	type: string
	required: false
	default: ""
	java-distribution:
	description: "Java JDK Distribution:"
	type: string
	required: false
	default: "temurin"
	java-version:
	description: "Java JDK Version:"
	type: string
	required: false
	default: "21.0.4"

	defaults:
	run:
	shell: bash

	permissions:
	id-token: write
	contents: read
	packages: write

	env:
	DOCKER_MANIFEST_GENERATOR: .github/workflows/support/scripts/generate-docker-artifact-baseline.sh
	DOCKER_MANIFEST_PATH: ${{ github.workspace }}/.manifests/docker
	DOCKER_REGISTRY: ghcr.io
	DOCKER_IMAGE_NAME: hashgraph/hedera-block-node
	DOCKER_CONTEXT_PATH: server/docker
	SKOPEO_VERSION: v1.14.0
	OWNER: hashgraph

	jobs:
	generate-baseline:
	name: Generate Baseline
	runs-on: block-node-linux-medium
	outputs:
	sha: ${{ steps.commit.outputs.sha }}
	sha-abbrev: ${{ steps.commit.outputs.sha-abbrev }}
	source-date: ${{ steps.commit.outputs.source-date }}
	path: ${{ steps.baseline.outputs.path }}
	file: ${{ steps.baseline.outputs.file }}
	name: ${{ steps.baseline.outputs.name }}

	steps:
	- name: Harden Runner
	uses: step-security/harden-runner@0080882f6c36860b6ba35c610c98ce87d4e2f26f # v2.10.2
	with:
	egress-policy: audit

	- name: Checkout Code
	uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
	with:
	ref: ${{ inputs.ref }}

	- name: Authenticate to Google Cloud
	id: google-auth
	uses: google-github-actions/auth@6fc4af4b145ae7821d527454aa9bd537d1f2dc5f # v2.1.7
	with:
	workload_identity_provider: "projects/235822363393/locations/global/workloadIdentityPools/hedera-builds-pool/providers/hedera-builds-gh-actions"
	service_account: "swirlds-automation@hedera-registry.iam.gserviceaccount.com"

	- name: Setup Google Cloud SDK
	uses: google-github-actions/setup-gcloud@6189d56e4096ee891640bb02ac264be376592d6a # v2.1.2

	- name: Retrieve Commit Hash
	id: commit
	run: \|
	echo "sha=$(git rev-parse HEAD)" >> "${GITHUB_OUTPUT}"
	echo "sha-abbrev=$(git rev-parse HEAD \| tr -d '[:space:]' \| cut -c1-8)" >> "${GITHUB_OUTPUT}"
	echo "source-date=$(git log -1 --pretty=%ct)" >> "${GITHUB_OUTPUT}"

	- name: Baseline Existence Check
	id: baseline
	run: \|
	BASELINE_NAME="${{ steps.commit.outputs.sha }}.tar.gz"
	BASELINE_PATH="gs://hedera-ci-ephemeral-artifacts/${{ github.repository }}/docker/baselines"
	BASELINE_FILE="${BASELINE_PATH}/${BASELINE_NAME}"
	BASELINE_EXISTS="false"

	if gsutil ls "${BASELINE_FILE}" >/dev/null 2>&1; then
	BASELINE_EXISTS="true"
	fi

	echo "exists=${BASELINE_EXISTS}" >> "${GITHUB_OUTPUT}"
	echo "path=${BASELINE_PATH}" >> "${GITHUB_OUTPUT}"
	echo "name=${BASELINE_NAME}" >> "${GITHUB_OUTPUT}"
	echo "file=${BASELINE_FILE}" >> "${GITHUB_OUTPUT}"

	- name: Setup Java
	uses: actions/setup-java@8df1039502a15bceb9433410b1a100fbe190c53b # v4.5.0
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	with:
	distribution: ${{ inputs.java-distribution }}
	java-version: ${{ inputs.java-version }}

	- name: Setup Gradle
	uses: gradle/actions/setup-gradle@cc4fc85e6b35bafd578d5ffbc76a5518407e1af0 # v4.2.1
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	with:
	cache-disabled: true

	- name: Install Skopeo and JQ
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	run: \|
	sudo apt-get update
	sudo apt-get install --yes --no-install-recommends skopeo jq

	- name: Setup QEmu Support
	uses: docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf # v3.2.0
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}

	- name: Setup Docker Buildx Support
	uses: docker/setup-buildx-action@c47758b77c9736f4b2ef4073d4d51994fabfe349 # v3.7.1
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	with:
	version: v0.16.2
	driver-opts: network=host

	- name: Setup Local Docker Registry
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	run: docker run -d -p 5000:5000 --restart=always --name registry registry:latest

	- name: Show Docker Version
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	run: docker version

	- name: Show Docker Info
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	run: docker info

	- name: Build Gradle Artifacts
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	run: ./gradlew assemble --scan

	- name: Login to GitHub Container Registry
	uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446 # v3.2.0
	with:
	registry: ${{ env.DOCKER_REGISTRY }}
	username: ${{ github.actor }}
	password: ${{ secrets.GITHUB_TOKEN }}

	# - name: Prepare for Docker Build
	# if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	# run: \|
	# mkdir -p "${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}/sdk/data"
	#
	# echo "::group::Copying Application Artifacts"
	# cp -Rvf "${{ github.workspace }}/server/build/libs" "${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}/sdk/data/"
	# echo "::endgroup::"

	# - name: Write Artifact Version Descriptor
	# run: \|
	# printf "VERSION=%s\nCOMMIT=%s\nDATE=%s" "$(./gradlew -q showVersion)" "$(git log -1 --format='%H' \| cut -c1-8)" "$(date -u)" \| tee "${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}/sdk/VERSION"

	- name: Extract version
	id: extract_version
	run: \|
	VERSION=$(grep 'version=' gradle.properties \| cut -d '=' -f2)
	echo "VERSION=${VERSION}" >> $GITHUB_ENV

	- name: Build Docker Image
	uses: docker/build-push-action@48aba3b46d1b1fec4febb7c5d0c644b249a11355 # v6.10.0
	env:
	SOURCE_DATE_EPOCH: ${{ steps.commit.outputs.source-date }}
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	with:
	push: true
	no-cache: true
	platforms: linux/amd64,linux/arm64
	build-args: \|
	SOURCE_DATE_EPOCH=${{ steps.commit.outputs.source-date }}
	VERSION=${{ env.VERSION }}
	context: ./${{ env.DOCKER_CONTEXT_PATH }}
	file: ./${{ env.DOCKER_CONTEXT_PATH }}/Dockerfile
	tags: ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:${{ env.VERSION }}
	build-contexts: \|
	distributions=./server/build/distributions

	- name: Generate Manifest
	id: manifest
	env:
	MANIFEST_PATH: ${{ env.DOCKER_MANIFEST_PATH }}
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	run: GITHUB_SHA="${{ needs.generate-baseline.outputs.sha-abbrev }}" ${{ env.DOCKER_MANIFEST_GENERATOR }}

	- name: Amend Manifest with Gradle Artifacts
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	working-directory: ${{ env.DOCKER_MANIFEST_PATH }}
	run: \|
	EXTRACTED_FILE_NAME="${{ steps.commit.outputs.sha }}.tar"
	echo "Before manifest operations ----------"
	pwd
	ls -la
	gunzip "${{ steps.manifest.outputs.name }}"
	echo "After gunzip ----------"
	ls -la
	tar -rvf "${EXTRACTED_FILE_NAME}" -C "${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}" ../../server/build/
	gzip "${EXTRACTED_FILE_NAME}"
	echo "After augmenting and gzipping ----------"
	ls -la

	- name: Upload Baseline
	if: ${{ steps.baseline.outputs.exists == 'false' && !failure() && !cancelled() }}
	run: gsutil cp "${{ steps.manifest.outputs.file }}" "${{ steps.baseline.outputs.file }}"

	verify-artifacts:
	name: "Verify Artifacts (${{ join(matrix.os, ', ') }})"
	runs-on: ${{ matrix.os }}
	needs:
	- generate-baseline
	strategy:
	fail-fast: false
	matrix:
	# Windows is not supported due to GitHub not supporting Docker Desktop/Podman Desktop and Docker CE on Windows
	# not supporting BuildKit and the Buildx plugin.
	# GitHub hosted MacOS and Ubuntu runners are temporarily disabled.
	os:
	#- ubuntu-22.04
	#- ubuntu-20.04
	#- macos-12
	#- macos-11
	- block-node-linux-medium
	steps:
	- name: Harden Runner
	uses: step-security/harden-runner@0080882f6c36860b6ba35c610c98ce87d4e2f26f # v2.10.2
	with:
	egress-policy: audit

	- name: Standardize Git Line Endings
	run: \|
	git config --global core.autocrlf false
	git config --global core.eol lf

	- name: Checkout Code
	uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
	with:
	ref: ${{ inputs.ref }}

	- name: Setup Python
	uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
	with:
	python-version: 3.9

	- name: Install JQ (Linux)
	if: ${{ runner.os == 'Linux' }}
	run: \|
	sudo apt-get update
	sudo apt-get install --yes --no-install-recommends jq

	- name: Install Skopeo (Linux)
	if: ${{ runner.os == 'Linux' }}
	run: \|
	source /etc/os-release
	if [[ "${VERSION_ID}" != "20.04" ]]; then
	sudo apt-get install --yes --no-install-recommends skopeo
	fi

	- name: Authenticate to Google Cloud
	id: google-auth
	uses: google-github-actions/auth@6fc4af4b145ae7821d527454aa9bd537d1f2dc5f # v2.1.7
	with:
	workload_identity_provider: "projects/235822363393/locations/global/workloadIdentityPools/hedera-builds-pool/providers/hedera-builds-gh-actions"
	service_account: "swirlds-automation@hedera-registry.iam.gserviceaccount.com"

	- name: Setup Google Cloud SDK
	uses: google-github-actions/setup-gcloud@6189d56e4096ee891640bb02ac264be376592d6a # v2.1.2
	env:
	CLOUDSDK_PYTHON: ${{ format('{0}{1}', env.pythonLocation, runner.os == 'Windows' && '\python.exe' \|\| '/bin/python3') }}

	- name: Download Baseline
	env:
	CLOUDSDK_PYTHON: ${{ format('{0}{1}', env.pythonLocation, runner.os == 'Windows' && '\python.exe' \|\| '/bin/python3') }}
	run: \|
	mkdir -p "${DOCKER_MANIFEST_PATH}"
	cd "${DOCKER_MANIFEST_PATH}"
	gsutil cp "${{ needs.generate-baseline.outputs.file }}" .
	tar -xzf "${{ needs.generate-baseline.outputs.name }}"
	pwd
	ls -la
	mv *.json "${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}/"
	mv ./server/build/distributions/*.tar.gz "${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}/"
	echo "Moved files to: ${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}/"

	- name: Determine Home Directory
	id: home
	run: echo "directory=$(tr -d '[:space:]' < <(cd ~ && pwd))" >> "${GITHUB_OUTPUT}"

	- name: Setup QEmu Support
	uses: docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf # v3.2.0

	- name: Setup Docker Buildx Support
	uses: docker/setup-buildx-action@c47758b77c9736f4b2ef4073d4d51994fabfe349 # v3.7.1
	with:
	version: v0.16.2
	driver-opts: network=host
	# buildkitd-config-inline: \|
	# [registry."docker.io"]
	# mirrors = ["https://hub.mirror.docker.lat.ope.eng.hashgraph.io"]

	- name: Setup Local Docker Registry
	run: docker run -d -p 5000:5000 --restart=always --name registry registry:latest

	- name: Show Docker Version
	run: docker version

	- name: Show Docker Info
	run: docker info


	- name: Login to GitHub Container Registry
	uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446 # v3.2.0
	with:
	registry: ${{ env.DOCKER_REGISTRY }}
	username: ${{ github.actor }}
	password: ${{ secrets.GITHUB_TOKEN }}

	- name: Extract version
	id: extract_version
	run: \|
	VERSION=$(grep 'version=' gradle.properties \| cut -d '=' -f2)
	echo "VERSION=${VERSION}" >> $GITHUB_ENV

Check failure on line 348 in .github/workflows/zxc-verify-docker-build-determinism.yaml View workflow run for this annotation GitHub Actions / .github/workflows/zxc-verify-docker-build-determinism.yaml Invalid workflow file `You have an error in your yaml syntax on line 348`
	- name: Docker Image Debugging
	run: \|
	echo "zero ---------------------"
	echo "${{ env.VERSION }}\n"
	cd ${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}/
	# echo "${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}/"
	# ls -la "${{ github.workspace }}/${{ env.DOCKER_CONTEXT_PATH }}/"
	# echo "first ---------------------"
	pwd
	ls -la
	# echo "second ---------------------"
	# ls -la ./server/
	# echo "third ---------------------"
	# ls -la ../
	# echo "fourth ---------------------"
	# ls -la ./server/build/distributions/
	# echo "fifth ---------------------"

	- name: Build Docker Image
	uses: docker/build-push-action@48aba3b46d1b1fec4febb7c5d0c644b249a11355 # v6.10.0
	env:
	SOURCE_DATE_EPOCH: ${{ needs.generate-baseline.outputs.source-date }}
	with:
	push: true
	no-cache: true
	platforms: linux/amd64,linux/arm64
	build-args: \|
	SOURCE_DATE_EPOCH=${{ needs.generate-baseline.outputs.source-date }}
	VERSION=${{ env.VERSION }}
	context: ./${{ env.DOCKER_CONTEXT_PATH }}
	file: ./${{ env.DOCKER_CONTEXT_PATH }}/Dockerfile
	tags: ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:${{ needs.generate-baseline.outputs.sha-abbrev }}
	build-contexts: \|
	distributions=./

	- name: Regenerate Manifest
	id: regen-manifest
	env:
	MANIFEST_PATH: ${{ env.DOCKER_MANIFEST_PATH }}/regenerated
	run: GITHUB_SHA="${{ needs.generate-baseline.outputs.sha-abbrev }}" ${{ env.DOCKER_MANIFEST_GENERATOR }}

	- name: Validate Layers (linux/amd64)
	run: \|
	if ! diff -u "${DOCKER_MANIFEST_PATH}/linux-amd64.layers.json" "${{ steps.regen-manifest.outputs.path }}/linux-amd64.layers.json" >/dev/null 2>&1; then
	echo "::group::Layer Differences"
	diff -u "${DOCKER_MANIFEST_PATH}/linux-amd64.layers.json" "${{ steps.regen-manifest.outputs.path }}/linux-amd64.layers.json"
	echo "::endgroup::"
	exit 1
	fi

	- name: Validate Layers (linux/arm64)
	run: \|
	if ! diff -u "${DOCKER_MANIFEST_PATH}/linux-arm64.layers.json" "${{ steps.regen-manifest.outputs.path }}/linux-arm64.layers.json" >/dev/null 2>&1; then
	echo "::group::Layer Differences"
	diff -u "${DOCKER_MANIFEST_PATH}/linux-arm64.layers.json" "${{ steps.regen-manifest.outputs.path }}/linux-arm64.layers.json"
	echo "::endgroup::"
	exit 1
	fi

	- name: Validate Full Manifest (linux/amd64)
	run: \|
	if ! diff -u "${DOCKER_MANIFEST_PATH}/linux-amd64.comparable.json" "${{ steps.regen-manifest.outputs.path }}/linux-amd64.comparable.json" >/dev/null 2>&1; then
	echo "::group::Layer Differences"
	diff -u "${DOCKER_MANIFEST_PATH}/linux-amd64.comparable.json" "${{ steps.regen-manifest.outputs.path }}/linux-amd64.comparable.json"
	echo "::endgroup::"
	exit 1
	fi

	- name: Validate Full Manifest (linux/arm64)
	run: \|
	if ! diff -u "${DOCKER_MANIFEST_PATH}/linux-arm64.comparable.json" "${{ steps.regen-manifest.outputs.path }}/linux-arm64.comparable.json" >/dev/null 2>&1; then
	echo "::group::Layer Differences"
	diff -u "${DOCKER_MANIFEST_PATH}/linux-arm64.comparable.json" "${{ steps.regen-manifest.outputs.path }}/linux-arm64.comparable.json"
	echo "::endgroup::"
	exit 1
	fi

	- name: Publish Manifests
	uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
	if: ${{ steps.regen-manifest.conclusion == 'success' && failure() && !cancelled() }}
	with:
	name: Docker Manifests [${{ join(matrix.os, ', ') }}]
	path: ${{ env.DOCKER_MANIFEST_PATH }}/**

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fixing how files are copied and debugging #1

Workflow file

fixing how files are copied and debugging #1

Jobs

Run details

Workflow file for this run

GitHub Actions / .github/workflows/zxc-verify-docker-build-determinism.yaml