In highly-regulated environments such as healthcare, pharmaceuticals, or finance, protecting model weights and code artifacts is essential—not just for data privacy and security, but also for safeguarding proprietary intellectual property. This article introduces a reproducible method for encrypting models and training logic within Rhino’s Federated Computing Platform (FCP). The code examples throughout the article are taken directly from the Secrets Manager Integration and Encryption with Rhino FCP to emphasize how to use it with the secret manager. It also provides NVIDIA FLARE Example - Encrypted Model Code and Weights, which demonstrates an alternate approach to encrypting without relying on a secret manager.
Prerequisites
Access to the Rhino FCP platform
Imported relevant dataset available in FCP
What You'll Learn
Encrypt code and model weights
Configure encrypted Federated Learning (FL) jobs in FCP
Use secret manager or manual key injection
Overview
To learn more about why you might need to use encryption and about the Rhino FCP encryption workflow, read the following subsections.
Why Encrypt in Federated Computing?
FL enables data to remain distributed across collaborators—but model artifacts can still leak sensitive information. FL systems are vulnerable to several types of attacks that can compromise privacy, security, and model integrity, including:
Intellectual property protection: Model weights and code artifacts are proprietary assets that encode valuable architectures and data patterns.
Poisoning attacks: Malicious clients inject harmful or biased updates into the model to degrade performance or introduce backdoors.
Inference attacks: Adversaries analyze shared model updates to recover sensitive patterns about training data.
Communication attacks: Attackers intercept, modify, or replay messages between clients and the server to compromise training integrity.
Free-riding attacks: Participants try to obtain the trained global model without actually contributing real local training.
If model parameters are exposed, they can leak sensitive data patterns, reduce competitive advantage, violate contracts, damage the resulting model’s integrity and commercial value, and expose the organization to regulatory penalties, legal liability, and reputational harm.
In addition, malicious participants may attempt to corrupt model updates, extract private data patterns from shared parameters, intercept communications, or contribute fake updates to gain access to the final model without legitimate participation. A detailed analysis of these attack types can be found in this paper.
How Does Encryption Happen?
This workflow applies when a collaborator has already trained a machine learning model outside of Rhino FCP and now needs to upload that model into Rhino for secure inference, fine-tuning, or federated training. In addition, encryption applies to storing final model weights after the training / fine-tuning process. This protects the artifact from leakage during upload or storage and satisfies compliance requirements in regulated environments like healthcare or finance.
Secure Key Handling Modes
Rhino supports two secure key handling modes for decrypting encrypted models at runtime. You may adapt these approaches to fit your own container build and encryption workflows. We generally recommend using a Cloud Secrets Manager for production workloads, as it offers automated, secure, and scalable key management. Alternatively, local JSON keyfiles can be used in development or non-cloud environments but require careful handling to maintain security.
- Scenario #1: AWS Secrets Manager (automatic key management)
Rhino automatically retrieves the encryption key stored in AWS Secrets Manager and injects it into the container environment as a JSON file at runtime. This approach is ideal for production workloads due to its automation, security, and scalability. This article explains how to implement the workflow, in detail, using the Secrets Manager Integration and Encryption with Rhino FCP.
As of July 2025, Rhino supports AWS Secrets Manager as a productized integration, but also supports other secret managers through custom integrations.
- Scenario #2: Manual JSON Keyfile:
The encryption key is stored in a JSON file that you manage yourself. You can add this file directly to your container image or provide it manually as a secret parameter at runtime through the Rhino UI. This article describes how to implement this alternative approach using the NVIDIA FLARE Example - Encrypted Model Code and Weights.
Note: Be sure to coordinate with your organization's IT or Cyber Security team to follow internal security protocols when managing encryption keys.
Scenario 1: Secrets Manager
The following directory structure shows the complete example used for all code samples in this scenario.
secrets-manager-encryption/ ├── README.md ├── data/ # Sample molecular datasets │ ├── cyp3a4_A.csv # Training dataset A │ ├── cyp3a4_B.csv # Training dataset B │ ├── cyp3a4_C.csv # Training dataset C │ └── cyp3a4_test.csv # Test dataset └── code/ # Application code ├── chemprop_fl_classification.py # Main federated learning model ├── requirements.txt # Python dependencies ├── Dockerfile # Container configuration ├── entrypoint.sh # Container entrypoint for decryption ├── infer.py # Inference script ├── meta.conf # Model metadata ├── model_parameters.pt # Pre-trained model weights ├── encrypt_code/ # Encryption utilities │ └── encrypt_code.py # File encryption with AWS Secrets Manager integration └── app/ # Application files ├── custom/ # Custom application code │ ├── chemprop_fl_classification.py.enc # Encrypted main model │ ├── model_parameters.pt.enc # Encrypted model weights │ ├── decrypt_code.py # Runtime decryption utility │ └── encrypted_persistor.py # Encrypted model persistence └── config/ # NVFlare configuration ├── config_fed_client.conf # Federated client configuration └── config_fed_server.conf # Federated server configuration
This scenario can be understood in two main groups of steps:
- Code and Initial Model Provisioning Procedure - where the developers prepare and encrypt their model artifacts before upload.
- Code Execution Procedure - inside Rhino Orchestrator, where the encrypted model is securely decrypted using the secret managed in the Secret Manager and used at runtime for inference or fine-tuning.
Code and Initial Model Provisioning Procedure
-
Add decryption logic in model code.
Example of the relevant part from infer.py :
# Orchestrate decryption and model loading from decrypt_code import decrypt_weights decrypted_data = decrypt_weights(model_params_file_path) model.load_state_dict(torch.load(decrypted_data)["model"]) model.to(device)
- Get key from key generation package or KMS.
-
Encrypt code and model parameter files (e.g.,
.pt→.pt.enc).
Note: While encryption is performed outside Rhino (by the client or their CI/CD system), we include this step here to illustrate the process clearly for users. Customers should use their preferred encryption tools or KMS solutions (e.g., AWS KMS, Google Cloud KMS) to ensure artifacts are encrypted before upload.
Example command (adapted from a customer implementation):
python ../encrypt_code.py -i ./model_parameters.pt -o ./model_parameters.pt.enc -k <model_name> -dParameters:
-i - input file to be encrypted
-o - encrypted output file, for NVFLARE, must end in .enc
-k - key id of the secret in secrets manager (must match container name)
-d - boolean flag to delete the original file to prevent it from being Packaged in the container
- Store keys and Set secret access policies and in the external secret manager.
- Build the container with encrypted code and model.
Example Dockerfile line to include the encrypted model:
COPY --chown=$UID:$GID ./model_parameters.pt.enc ./-
Push the container to the workgroup private container
registry.
Include encrypted initial model parameters in the container or import the encrypted model from the in workgroup artifact bucket.
Code Execution Procedure
Cloud Orchestration
- Trigger Code Run using the Rhino GUI or SDK with dataset IDs.
- Secret Retrieval Using Secret Manager (Recommended) – The FL Server obtains the key from the secret manager and securely provides it as a parameter to the Code Run.
- Dispatch encrypted container and secret to Rhino Clients over TLS.
- Decrypt the files using the secret and execute the code in the Trusted Execution Environment.
This is easily done by using the Rhino-provided
decrypt_weights()
function from
decrypt_code.py,
which is imported and called in
infer.py.
The relevant part from infer.py :
# Orchestrate decryption and model loading
decrypted_data = decrypt_weights(model_params_file_path)
model.load_state_dict(torch.load(decrypted_data)["model"])
model.to(device)
The relevant part from decrypt_code.py:
import json
from pathlib import Path
from Crypto.PublicKey import RSA
from Crypto.Cipher import AES, PKCS1_OAEP
from io import BytesIO
def decrypt_weights(model_parameters_path):
secret_run_params_file_path = Path("/input/secret_run_params.json")
with secret_run_params_file_path.open("r") as secret_file:
secret_run_params = json.load(secret_file)
private_key = RSA.import_key(secret_run_params["decrypt_key"])
with open(model_parameters_path, 'rb') as f:
enc_session_key = f.read(private_key.size_in_bytes())
nonce = f.read(16)
tag = f.read(16)
ciphertext = f.read()
cipher_rsa = PKCS1_OAEP.new(private_key)
session_key = cipher_rsa.decrypt(enc_session_key)
cipher_aes = AES.new(session_key, AES.MODE_EAX, nonce)
decrypted_data = cipher_aes.decrypt_and_verify(ciphertext, tag)
return BytesIO(decrypted_data)- Produce results (e.g., model weights, aggregate statistics).
-
Apply differential privacy (optional). Send parameters
over
TLS.
If FHE is enabled, parameters are also encrypted.
Result Aggregation, Encryption, and Storage in the Rhino Orchestrator
Steps 7–9 repeat for every federated round, processing contributions from each participating client in the workgroup.
- Aggregation and Encryption - Each federated round, the server aggregates client updates and produces a new global model. Before storing or distributing this model, Rhino Orchestrator encrypts it using hybrid RSA–AES encryption with the public key from the Secrets Manager.
- Store the encrypted global model in a workgroup storage bucket. This is a specific and isolated bucket that is private to the workgroup. Access to the stored model is controlled through Rhino Orchestrator, which manages secure communication with the Secrets Manager service.
encrypted_persistor.py shows how the server loads the RSA public encryption key to encrypt the aggregated model artifact before saving it back to the workgroup storage bucket:
import json
import os
import torch
from typing import Any
from pathlib import Path
from Crypto.PublicKey import RSA
from Crypto.Random import get_random_bytes
from Crypto.Cipher import AES, PKCS1_OAEP
import nvflare.app_opt.pt.file_model_persistor as mp
class EncryptedPersistor(mp.PTFileModelPersistor):
def _load_encryption_key(self) -> RSA.RsaKey:
"""Load RSA public key from secret params file."""
secret_path = Path("/server-credentials/secret_run_params.json")
if not secret_path.exists():
raise ValueError(f"Secret params file not found at {secret_path}")
with open(secret_path, 'r') as f:
params = json.load(f)
try:
if 'encrypt_key' not in params:
raise ValueError("encrypt_key not found in secret params")
return RSA.import_key(params['encrypt_key'])
except Exception as e:
raise ValueError(f"Failed to load encryption key: {str(e)}")
def save_model_file(self, save_path: str):
"""Override save_model_file to encrypt the model using RSA/AES hybrid encryption."""
# First save the model state dict to temp file
temp_location = f"{save_path}.temp"
save_dict = self.persistence_manager.to_persistence_dict()
torch.save(save_dict, temp_location)
# Read the saved model file
with open(temp_location, 'rb') as f:
data = f.read()
# Generate random AES session key
session_key = get_random_bytes(16)
# Encrypt session key with RSA
public_key = self._load_encryption_key()
cipher_rsa = PKCS1_OAEP.new(public_key)
enc_session_key = cipher_rsa.encrypt(session_key)
# Encrypt file data with AES
cipher_aes = AES.new(session_key, AES.MODE_EAX)
ciphertext, tag = cipher_aes.encrypt_and_digest(data)
# Save encrypted data
with open(save_path, 'wb') as f:
[f.write(x) for x in (enc_session_key, cipher_aes.nonce, tag, ciphertext)]
# Clean up temp file
os.remove(temp_location)- Repeat process – For each federated round, Rhino Orchestrator securely retrieves the private key from the configured Secrets Manager and injects it into the container environment for use by the decryption logic. This process repeats each round, with clients decrypting (as elaborated in step 4) and training locally while the server aggregates and re-encrypts the global model.
This repetitive workflow ensures that pre-trained models can be brought into Rhino safely, without ever exposing sensitive model internals or private training patterns. The encryption and decryption steps are automated and isolated, providing strong protection with minimal friction to the developer.
Scenario 2: Manual Secret
This scenario describes how teams can manually manage and supply their encryption key without relying on an integrated Secrets Manager. NVIDIA FLARE Example - Encrypted Model Code and Weights demonstrates this approach in detail.
Unlike Scenario #1, there is no automated key retrieval. Instead, you generate and store the key yourself, then provide it manually at run time easily through the UI.
The following directory structure shows the complete example used for all code samples in this scenario:
(Highlighted files are the specific ones provided as examples throughout the workflow)
.
├── Dockerfile # Container configuration for building the image
├── README.md
├── config
│ ├── config_fed_client.json # Federated client configuration
│ └── config_fed_server.json # Federated server configuration
├── custom
│ ├── decrypt_code.py # Runtime decryption utility called by entrypoint.sh
│ ├── network.py.enc # Encrypted main model source code
│ ├── pneumonia_trainer.py # Model training logic
│ ├── pt_constants.py # Training constants and settings
│ └── pt_secured_model_persistor.py # Secure persistor to handle encrypted model weights
├── encrypt_code
│ ├── encrypt_code.py # Utility to encrypt code or model weights before upload
│ └── generate_key.py # Script to generate local encryption keys
├── entrypoint.sh # Container entrypoint script that runs decryption
├── infer.py # Inference script for serving the model
├── network.py # Decrypted model code (produced at runtime)
└── requirements.txt # Python dependenciesSimilar to Scenario #1, this Scenario will be Understood in Two Main Groups of Steps
- Code and Initial Model Provisioning Procedure - where the developers prepare and encrypt their model artifacts before upload.
- Code Execution Procedure - inside Rhino Orchestrator, where the encrypted model is securely decrypted at runtime using the secret manually injected by the user and used for inference or fine-tuning.
Code and Initial Model Provisioning Procedure
These steps are performed by the developer before uploading to Rhino.
1. Generate Encryption Key - Unlike Scenario #1 (which retrieves a key from AWS Secrets Manager), you generate your own encryption key locally.
Example command using generate_key.py file from the full NVFlare example:
python ./encrypt_code/generate_key.py ~/my_model_key- This produces a local key file that only you hold. (in this example — my_model_key)
2. Encrypt Model Code and Weights - Encrypt both your source code and trained model weights ahead of time so the raw files are never included in the container.
Example command:
python ./encrypt_code/encrypt_code.py ./network.py ~/my_model_key ./custom/network.py.enc
python ./encrypt_code/encrypt_code.py ./model_parameters.pt ~/my_model_key ./model_parameters.pt.encNote: This step is conceptually identical to Scenario #1’s encryption step, but here you use your own locally stored key instead of a cloud KMS/Secrets Manager.
3. Delete or Move Unencrypted Files to ensure it's excluded from the container.
4. Build the Container Image - before you doing that, make sure again that it includes only the encrypted code and weights in the image:
Relevant line from the Dockerfile :
COPY --chown=$UID:$GID ./custom ./custom- The raw unencrypted files are never included in the container image.
5. Push the Container - Push the container to the workgroup’s private container registry on Rhino FCP.
Code Execution Procedure
These steps occur inside Rhino Orchestrator when the federated run is launched.
Cloud Orchestration
- Provide the Key Manually at Runtime
- Instead of Rhino Orchestrator retrieving the key from Secrets Manager, you manually supply the key at runtime by easily pasting via the Rhino UI
-
Rhino Orchestrator mounts your key as
/input/secret_run_params.json.
Example JSON file:
{
"decrypt_key": "YOUR_PRIVATE_KEY_HERE"
"encrypt_key": "YOUR_PUBLIC_KEY_HERE"
}
- Trigger Code Run using the Rhino GUI or SDK with dataset IDs.
- Dispatch encrypted container and secret to Rhino Clients over TLS.
Edge Computation at Each Client
- Decrypt – The encrypted code and model weights are decrypted before training or inference using the secret that is manually provided at runtime.
Unlike Scenario #1, this decryption does not happen inside your Python model code (there’s no decrypt_weights() function). Instead, the container’s entrypoint.sh script automatically decrypts all .enc files at startup, before launching the training or inference process.
#!/bin/bash
set -eux -o pipefail
# Decrypt all files with a .enc extension.
find . -type f -name '*.enc' -exec bash -c 'python ./custom/decrypt_code.py "$1" "${1%.enc}"' bash {} ';'
exec "$@"This command:
-
Reads the secret key from
/input/secret_run_params.json -
Decrypts the
.encfile using Fernet symmetric encryption from the decrypt_code.py file. -
Writes the decrypted
.pyfile to the specified location inside the container
#!/usr/bin/env python
import argparse
import json
import logging
import sys
from pathlib import Path
from cryptography.fernet import Fernet
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
stream=sys.stdout,
)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Decrypt a file using a key from the GC run parameters")
parser.add_argument("input_filename", help="input filename")
parser.add_argument("output_filename", help="output filename")
args = parser.parse_args()
if not Path(args.input_filename).exists():
print(f"File not found: {args.input_filename}", file=sys.stderr)
sys.exit(1)
if Path("/server-credentials").exists():
secret_run_params_file_path = Path("/server-credentials/secret_run_params.json")
elif Path("/input").exists():
secret_run_params_file_path = Path("/input/secret_run_params.json")
else:
print("Could not find directory for secret_run_params.json!", file=sys.stderr)
sys.exit(1)
if not secret_run_params_file_path.is_file():
print("secret_run_params.json file is missing.", file=sys.stderr)
sys.exit(1)
with secret_run_params_file_path.open("rb") as secret_run_params_file:
try:
secret_run_params = json.load(secret_run_params_file)
except json.JSONDecodeError as exc:
print(f"Error decoding JSON in secret_run_params.json: {str(exc)}", file=sys.stderr)
sys.exit(1)
try:
key = secret_run_params["key"]
except KeyError:
print('Missing key in secret_run_params.json: "key"', file=sys.stderr)
sys.exit(1)
logging.info(f"Decrypting input file '{args.input_filename}'")
encrypted = Path(args.input_filename).read_bytes()
logging.info(f"Read {len(encrypted)} bytes of encrypted content")
fernet = Fernet(key)
decrypted = fernet.decrypt(encrypted)
logging.info(f"Decrypted contents resulted in {len(decrypted)} bytes")
logging.info(f"Writing decrypted contents to output file '{args.output_filename}'")
Path(args.output_filename).write_bytes(decrypted)
logging.info("Done.")- Produce results (e.g., model weights, aggregate statistics).
-
Apply differential privacy (optional). Send parameters
over
TLS.
If FHE is enabled, parameters are also encrypted.
Result Aggregation, Encryption, and Storage in the Rhino Orchestrator
- Aggregation and Encryption - After each federated training round, Rhino Orchestrator aggregates the updates received from all participating clients to produce a new global model. Before storing or sharing this model, the server encrypts it using a hybrid RSA–AES scheme with the public key manually provided at the beginning through the UI before hitting RUN.
- Store the encrypted global model in the workgroup’s private bucket within Rhino Orchestrator. This is a specific and isolated bucket that is private to the workgroup. Access to this storage is governed by workgroup-level permissions, ensuring that only approved users and processes can retrieve or manage the encrypted artifacts.
- Repeat process – For each federated training round, this workflow repeats in the same way: the server distributes the encrypted global model to participating clients. Each client decrypts the model locally using the manually provided key (supplied at runtime), trains on its private data, re-encrypts the updated weights with the same key, and sends them securely back to Rhino Orchestrator. This cyclical process ensures robust security and consistent protection of model artifacts across all rounds of federated learning.
This repetitive workflow ensures that pre-trained models can be brought into Rhino safely, without ever exposing sensitive model internals or private training patterns. The encryption and decryption steps are automated and isolated, providing strong protection with minimal friction to the developer. However, the secret handling is done manually by the user which can reduce the overall security of this process.
Thanks again for investing your time in learning how to use the FCP in this use case, we can't wait to see what you will do with it! If you need support at any time, reach out to us at Rhino Support .
Additional Resources
This is an additional manual-encryption example that is provided in the Rhino user-resources repository:
Generalized Compute Example – Run Encrypted Code
This example shows a simpler GC approach to encrypt and decrypt only a specific
file you define, rather than the entire container. It explains how to encrypt
the file, decrypt it at runtime with a secret, and run the code securely.