Problem: Conveting weights from local to global

[LeandroDiL]

Hello,
I'm trying to modify the NVFlare 2.3.0 version of the advanced example cifar10 (simulation one). I use a classification problem starting from a csv file containing numerical data, using a modified NN model that fit the size of the data, CrossEntropy as criterion and SGD as optimizer. All is working fine until during validation and training I encounter the part of code
"# Before loading weights, tensors might need to be reshaped to support HE for secure aggregation.
local_var_dict = self.model.state_dict()
model_keys = global_weights.keys()
for var_name in local_var_dict:
if var_name in model_keys:
weights = global_weights[var_name]
try:
# reshape global weights to compute difference later on
global_weights[var_name] = np.reshape(weights, local_var_dict[var_name].shape)
# update the local dict
local_var_dict[var_name] = torch.as_tensor(global_weights[var_name])
except BaseException as e:
raise ValueError(f"Convert weight from {var_name} failed") from e
self.model.load_state_dict(local_var_dict)"
that is similar between validation and training and bringing to the same error:
"File "/my-workspace/my-workspace/provaaa/simulate_job/app_site-2/custom/cifar10_learner.py", line 533, in validate
local_var_dict[var_name] = torch.as_tensor(torch.reshape(weights[8192:], local_var_dict[var_name].shape))
RuntimeError: shape '[64, 128]' is invalid for input of size 0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/nvflare/app_common/executors/learner_executor.py", line 77, in execute
return self.train(shareable, fl_ctx, abort_signal)
File "/usr/local/lib/python3.10/dist-packages/nvflare/app_common/executors/learner_executor.py", line 94, in train
validate_result: Shareable = self.learner.validate(shareable, fl_ctx, abort_signal)
File "/my-workspace/my-workspace/provaaa/simulate_job/app_site-2/custom/cifar10_learner.py", line 536, in validate
raise ValueError(f"Convert weight from {var_name} failed") from e
ValueError: Convert weight from fc1.weight failed"
In order to have a better understanding of the issue, I will post the code used to manage data during inizialization and the architecture used. Furthermore, the csv file used contains 4967 rows and 129 columns of which the first row contains the data labels while last columns contains the 3 class classification output. The remaining 4966 rows and 128 columns contains the numeric data.
Data initialization and Architecture:

data = pd.read_csv(self.train_idx_root, header=None, low_memory=False)

        features = data.iloc[1:, :-1]  
        labels_temp = data.iloc[1:, -1]     

        label_encoder = LabelEncoder()
        labels = label_encoder.fit_transform(labels_temp)

        X_train, X_test, Y_train, Y_test = train_test_split(features, labels, test_size=0.2, stratify=labels)
        X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.25, stratify=Y_train)

        X_train = X_train.apply(pd.to_numeric)
        X_test = X_test.apply(pd.to_numeric)
        X_val = X_val.apply(pd.to_numeric)

        X_train_tensor = torch.tensor(X_train.values, dtype=torch.float)
        X_val_tensor = torch.tensor(X_val.values, dtype=torch.float)
        X_test_tensor = torch.tensor(X_test.values, dtype=torch.float)

        self.train_dataset = CustomDataset(X_train_tensor, Y_train)
        self.val_dataset = CustomDataset(X_val_tensor, Y_val)
        self.test_dataset = CustomDataset(X_test_tensor, Y_test)

        self.train_loader = DataLoader(self.train_dataset, batch_size=self.batch_size, shuffle=True)
        self.valid_loader = DataLoader(self.val_dataset, batch_size=self.batch_size, shuffle=False)  
        self.test_loader = DataLoader(self.test_dataset, batch_size=self.batch_size, shuffle=False)  


class CustomModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(128, 64)
        self.fc2 = nn.Linear(64, 3)  # 3 classi di output

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

To run the experiment I use the code: nvflare simulator -w /my-workspace/my-workspace/test -n 2 -t 2 NVFlare/jobs/cifar10_fedavg
Could someone help me? Ty in advance!

[YuanTingHsieh]

@LeandroDiL this is a bit hard to follow without your codes.
Can you share your codes or jobs?
And have you tried running our examples first to see if they works?

[LeandroDiL]

@YuanTingHsieh ty for your response. I have already tried to run the example and all is working fine locally running the ./run_experiment bash file or with the nvflare simulator. All worked fine even when I tried to run it in 2 embedded devices using the NVFlare provisiong tool.

About the code the only thing that is different from the example is the initialize method in the cifar10_learner.py file. Infact I modified it into:

def initialize(self, parts: dict, fl_ctx: FLContext):
        """
        Note: this code assumes a FL simulation setting
        Datasets will be initialized in train() and validate() when calling self._create_datasets()
        as we need to make sure that the server has already downloaded and split the data.
        """

        # when the run starts, this is where the actual settings get initialized for trainer

        # Set the paths according to fl_ctx
        self.app_root = fl_ctx.get_prop(FLContextKey.APP_ROOT)
        fl_args = fl_ctx.get_prop(FLContextKey.ARGS)
        self.client_id = fl_ctx.get_identity_name()
        self.log_info(
            fl_ctx,
            f"Client {self.client_id} initialized at \n {self.app_root} \n with args: {fl_args}",
        )

        self.local_model_file = os.path.join(self.app_root, "local_model.pt")
        self.best_local_model_file = os.path.join(self.app_root, "best_local_model.pt")

        # Select local TensorBoard writer or event-based writer for streaming
        #self.writer = SummaryWriter(self.app_root)
        self.writer = parts.get(self.analytic_sender_id)  # user configured config_fed_client.json for streaming
        if not self.writer:  # use local TensorBoard writer only
            self.writer = SummaryWriter(self.app_root)

        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.model = CustomModel().to(self.device)
        self.optimizer = optim.SGD(self.model.parameters(), lr=self.lr, momentum=0.9)
        self.criterion = torch.nn.CrossEntropyLoss()

        data = pd.read_csv(self.train_idx_root, header=None, low_memory=False)

        features = data.iloc[1:, :-1]  
        labels_temp = data.iloc[1:, -1]     


        label_encoder = LabelEncoder()
        labels = label_encoder.fit_transform(labels_temp)

        X_train, X_test, Y_train, Y_test = train_test_split(features, labels, test_size=0.2, stratify=labels)

        X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.25, stratify=Y_train)

        X_train = X_train.apply(pd.to_numeric)
        X_test = X_test.apply(pd.to_numeric)
        X_val = X_val.apply(pd.to_numeric)
        X_train_tensor = torch.tensor(X_train.values, dtype=torch.float)
        X_val_tensor = torch.tensor(X_val.values, dtype=torch.float)
        X_test_tensor = torch.tensor(X_test.values, dtype=torch.float)
       

        self.train_dataset = CustomDataset(X_train_tensor, Y_train)
        self.val_dataset = CustomDataset(X_val_tensor, Y_val)
        self.test_dataset = CustomDataset(X_test_tensor, Y_test)

        self.train_loader = DataLoader(self.train_dataset, batch_size=self.batch_size, shuffle=True)
        self.valid_loader = DataLoader(self.val_dataset, batch_size=self.batch_size, shuffle=False)  
        self.test_loader = DataLoader(self.test_dataset, batch_size=self.batch_size, shuffle=False)  

In order to manage numeric data and the transformation before the DataLoader I used a CustomDataset like below:

`class CustomDataset(Dataset):
def init(self, features, labels):
self.features = features
self.labels = torch.tensor(labels, dtype=torch.long)

def __len__(self):
    return len(self.features)

def __getitem__(self, idx):
    return self.features[idx], self.labels[idx]`

Obviously in order to adapt the model to the new numeric data different from a CIFAR-10 dataset based on image I used the following model:

`class CustomModel(nn.Module):
def init(self):
super().init()
self.fc1 = nn.Linear(128, 64)
self.fc2 = nn.Linear(64, 3) # 3 classi di output

def forward(self, x):
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x`

This code lead to a problem switching from global to local weights like described in the previous post. I'm sure the model and the managing of the data is fine since I already managed to make it work on other FL frameworks. Ty in advance!

[YuanTingHsieh]

@LeandroDiL thanks for providing the detail information!

So from your posts I assume your ONLY modify the "cifar10_learner.py" file.
This is not enough, you need to also modify this line: https://github.com/NVIDIA/NVFlare/blob/2.3/examples/advanced/cifar10/cifar10-sim/jobs/cifar10_fedavg/cifar10_fedavg/config/config_fed_server.json#L27

in the config_fed_server.json to use your own "CustomModel"

Let me explain the reason below.
The FedAvg workflow works like this:

1. Server initializes an initial model
2. For each round:
   2.1. Client receives a model from server
   2.2. Client does local training
   2.3. Client sends local trained model back
   2.4. Server aggregates all the model from all clients

So in your case, you change the "cifar10_learner.py" file, which change the steps 2.1, 2.2, 2.3.
You need to also modify config_fed_server.json so step 1 and step 2.4 use your new custom model.

[LeandroDiL]

Ye sorry @YuanTingHsieh my bad. I didn't remember to mention that I had modified the json files appropriately. Let's say that the error lies in the part of the code reported in the first question asked then in the line of code

# reshape global weights to compute difference later on
global_weights[var_name] = np.reshape(weights, local_var_dict[var_name].shape)

In my mind, the fact that it only crashes here and there is no other error makes me think of a problem with the implementation of the code when dealing perhaps with datasets not related to image classification but of another type.

[YuanTingHsieh]

@holgerroth ^ can you provide some insights to this reshaping?
You were trying to reshape to flatten tensors for the HE filters?

[holgerroth]

Looks like there is a mismatch between the global model and local model architectures. Did you update the persistor in config_fed_server.json to initialize/load your custom model architecture?

[LeandroDiL]

Yes, sorry, I am stupid. After testing several models I forgot to change the configurations for the model architecture even within the json files. Thank you @YuanTingHsieh and @holgerroth , very kind and helpful as always! Everything works perfectly now!

[YuanTingHsieh]

@LeandroDiL thanks for the update!

We understand it might be a bit tedious to go through each json file one by one.
We will be introducing the new job create cli that hope will make things easier for people!