Unexpected Visualization of Segmented Masks

[halwai]

Hey @liutinglt ,

I was trying to generate the segmentation masks for the provided snapshot(LIP_epoch_149.pth) while using evaluate.py where I added the following changes after Line 87:

temp = np.asarray(np.argmax(parsing, axis=3), dtype=np.uint8)
bs = temp.shape[0]
for i in range(0,bs):
    im_name = './temp/' + meta['name'][i] + '.png'
    org_name = os.path.join(args.data_dir,args.dataset+'_images',meta['name'][i] + '.jpg')
    savePng(temp[i], im_name, org_name)

where savePng saves the segmented image along-with corresponding original resized image.

Weirdly I get the following image (corresponding val image is also shown):

Could you please clarify this? Am I doing something wrong with inference.
Please ignore the colour coding.

Plus I get the following result:

Pixel accuracy: 33.734847 
Mean accuracy: 41.195624 
Mean IU: 16.228184 

OrderedDict([('Background', 21.21032650802787), ('Hat', 8.353540984716123), ('Hair', 28.628801550075778), ('Glove', 1.6443718260508893), ('Sunglasses', 0.4208788907181844), ('Upper-clothes', 43.008366280529145), ('Dress', 21.814854264377896), ('Coat', 25.210757775797134), ('Socks', 1.1099624827991603), ('Pants', 42.92961786592803), ('Jumpsuits', 10.25370439733039), ('Scarf', 1.104068135592208), ('Skirt', 11.54700741191835), ('Face', 28.341544521307533), ('Left-arm', 13.436388969552615), ('Right-arm', 21.208934571047447), ('Left-leg', 18.449373720229296), ('Right-leg', 15.446476235285603), ('Left-shoe', 1.144701502828927), ('Right-shoe', 9.29999506950886), ('Pixel accuracy', 33.73484655858868), ('Mean accuracy', 41.19562429552275), ('Mean IU', 16.22818364818107)])

Thanks,
Abhijeet

[liutinglt]

@halwai Can you reproduce the mIoU without any change? You can add the code for saving the mask in utils/miou.py after Line 72,
output_im = PIL.Image.fromarray(np.asarray(pred, dtype=np.uint8))
output_im.putpalette(palette)
output_im.save(your_save_dir+im_name+'.png')

[MoAbd]

@halwai are you using the same crop size in the original code? [380,380].

[halwai]

@MoAbd I haven't changed anything else, but the crop size is [384,384] instead of [380,380]

I get this by adding print(input_size) at Line 102 in evaluate.py

[halwai]

@liutinglt I have retried without any change and the results are still the same (reposting the results):

Pixel accuracy: 33.734848 
Mean accuracy: 41.195624 
Mean IU: 16.228182 

OrderedDict([('Background', 21.210327763730174), ('Hat', 8.353542295326081), ('Hair', 28.628796705634524), ('Glove', 1.6443744176306494), ('Sunglasses', 0.420878820477885), ('Upper-clothes', 43.008373660290054), ('Dress', 21.81482459075151), ('Coat', 25.210751193798952), ('Socks', 1.1099617273149016), ('Pants', 42.929615973315656), ('Jumpsuits', 10.25370690180522), ('Scarf', 1.1040681692383612), ('Skirt', 11.547003820568868), ('Face', 28.341545252759033), ('Left-arm', 13.436355812506454), ('Right-arm', 21.208949800840106), ('Left-leg', 18.449391121512875), ('Right-leg', 15.446470245549879), ('Left-shoe', 1.1447247808724323), ('Right-shoe', 9.299973249663141), ('Pixel accuracy', 33.73484796686051), ('Mean accuracy', 41.195623645121266), ('Mean IU', 16.22818181517934)])

[liutinglt]

@halwai Is your environment same with me? conda env create -f environment.yaml

[halwai]

@liutinglt Some minor differences exists for following
PILLOW =5.4.1
libgcc=7.2.0
protobuf=3.7

Complete List of packages in the conda environment

blas                      1.0                         mkl  
bzip2                     1.0.6                h14c3975_5  
ca-certificates           2018.12.5                     0  
cairo                     1.14.12              h8948797_3  
certifi                   2018.11.29               py36_0    anaconda
cffi                      1.11.5           py36he75722e_1  
cudatoolkit               9.0                  h13b8566_0  
cudnn                     7.1.2                 cuda9.0_0  
ffmpeg                    4.0                  hcdf2ecd_0  
fontconfig                2.13.0               h9420a91_0  
freeglut                  3.0.0                hf484d3e_5  
freetype                  2.9.1                h8a8886c_1  
glib                      2.56.2               hd408876_0  
graphite2                 1.3.13               h23475e2_0  
harfbuzz                  1.8.8                hffaf4a1_0  
hdf5                      1.10.2               hba1933b_1  
icu                       58.2                 h9c2bf20_1  
intel-openmp              2019.1                      144  
jasper                    2.0.14               h07fcdf6_1  
jpeg                      9b                   h024ee3a_2  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0  
libglu                    9.0.0                hf484d3e_1  
libopencv                 3.4.2                hb342d67_1  
libopus                   1.3                  h7b6447c_0  
libpng                    1.6.36               hbc83047_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.10               h2733197_2  
libuuid                   1.0.3                h1bed415_2  
libvpx                    1.7.0                h439df22_0  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.9                he19cac6_0  
mkl                       2018.0.3                      1  
mkl_fft                   1.0.6            py36h7dd41cf_0  
mkl_random                1.0.1            py36h4414c95_1  
nccl                      1.3.5                 cuda9.0_0  
ncurses                   6.1                  he6710b0_1  
ninja                     1.8.2            py36h6bb024c_1  
numpy                     1.15.4           py36h1d66e8a_0  
numpy-base                1.15.4           py36h81de0dd_0  
opencv                    3.4.2            py36h6fd60c2_1  
openssl                   1.1.1a               h7b6447c_0  
pcre                      8.42                 h439df22_0  
pip                       18.1                     py36_0  
pixman                    0.36.0               h7b6447c_0  
py-opencv                 3.4.2            py36hb342d67_1  
pyparsing                 2.3.1                      py_0    conda-forge
python                    3.6.8                h0371630_0  
pytorch                   0.4.1            py36ha74772b_0  
readline                  7.0                  h7b6447c_5  
scipy                     1.1.0            py36hfa4b5c9_1  
setuptools                40.6.3                   py36_0  
sqlite                    3.26.0               h7b6447c_0  
tk                        8.6.8                hbc83047_0  
wheel                     0.32.3                   py36_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               h7b6447c_3  
cuda90                    1.0                  h6433d27_0    pytorch
opencv                    3.4.2            py36h6fd60c2_1
pillow                    5.4.1            py36h34e0f95_0  
protobuf                  3.7.0            py36hf484d3e_0    conda-forge
pycocotools               2.0.0           py36h14c3975_1000    conda-forge
six                       1.12.0                   py36_0  
tensorboardx              1.6                        py_0    conda-forge
pytorch                   0.4.1            py36ha74772b_0  
torchvision               0.2.1                      py_2    pytorch

[halwai]

@liutinglt any updates on this thread ?

[lkj1114889770]

Did you solve the problem? I have the same problem。

[DerrickWang005]

I got this problem too. Is there any solution?

[Serhiy-Shekhovtsov]

Hm... I've just downloaded the code and model. I can't reproduce this problem. Here is what I am doing:

1) In evaluate.py I break the for loop in valid method after the first iteration

So I can analyze the input and results for the first batch.

2) Checked the input data:

from PIL import Image
from utils.utils import inv_preprocess

image_to_check = 2

# un preprocess image - denormalize
images_inv = inv_preprocess(image, num_images)

# tensor to numpy
img_array = images_inv[image_to_check].cpu().numpy()

# CHW -> HWC
img_array = np.transpose(img_array, [1, 2, 0])

# convert back to RGB
img_array = img_array[...,::-1]

Image.fromarray((img_array * 255).astype(np.uint8))

Gives me this image:

3) Checked the mask:

from utils.miou import get_palette

mask_img = Image.fromarray(parsing_preds[image_to_check].astype(np.uint8))
mask_img.putpalette(get_palette(args.num_classes))
mask_img