Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

About local frame! #1

Closed
haolyuan opened this issue Jul 9, 2022 · 12 comments
Closed

About local frame! #1

haolyuan opened this issue Jul 9, 2022 · 12 comments

Comments

@haolyuan
Copy link

haolyuan commented Jul 9, 2022

where can I find the specific defination of each local frame in this MANO code?

@lixiny
Copy link
Owner

lixiny commented Jul 9, 2022

each rotation (local frame) is represented as axis-angle w.r.t its parent frame.

The order is:

# **** axis order right hand
#         14-13-12-\
#                   \
#    2-- 1 -- 0 -----*
#   5 -- 4 -- 3 ----/
#   11 - 10 - 9 ---/
#    8-- 7 -- 6 --/

Lixin

@haolyuan
Copy link
Author

Thanks a lot! I have another question, Why the render result of app.py is not consistent to random_pose? When random_pose is all zero, the render result is as shown below. Is T-pose of MANO not consistent to T-pose of Smplx?

hand

@lixiny
Copy link
Owner

lixiny commented Jul 13, 2022

@Yamato-01

Sorry for the confusion.

The all zeros T-pose of MANO is consistent with T-pose of SMPL-X only when the flat_hand_mean = True.
You can type --flat_hand_mean in your command line when running app.py
see code here

@haolyuan
Copy link
Author

Thanks a lot! One more question. How to avoid abnormal hand pose after designing local frame in your project? Such as setZero?

@haolyuan haolyuan reopened this Jul 13, 2022
@lixiny
Copy link
Owner

lixiny commented Jul 14, 2022

If I understand your question correct, the abnormal poses can not be penalized without knowing its parent, or child.

We use a coordinate adaptation axis layer to convert the original rotational axes to twist-bend-splay (in our code: back-left-up), and then penalize the abnormal rotations w.r.t twist-bend-splay.

Thanks a lot! One more question. How to avoid abnormal hand pose after designing local frame in your project? Such as setZero?

@haolyuan
Copy link
Author

Thanks again! I have found three loss in geo_loss.py: joint_u_axis_loss, joint_l_limit_loss and joint_b_axis_loss. For example, You calculate cos value using b_axis and axis. Do the b_axis and axis mean the twist axis direction of the same local frame with this joint's pose (include fathers' joints along kintree) and with only its father joints' pose?

@haolyuan
Copy link
Author

If I understand your question correct, the abnormal poses can not be penalized without knowing its parent, or child.

We use a coordinate adaptation axis layer to convert the original rotational axes to twist-bend-splay (in our code: back-left-up), and then penalize the abnormal rotations w.r.t twist-bend-splay.

Thanks a lot! One more question. How to avoid abnormal hand pose after designing local frame in your project? Such as setZero?

One more question! I am confusing about abnormal pose of thumb recently. It seems that the three joints of thumb have more DOFs than other fingers. I would like to know how to penalize the abnormal rotation of thumb specifically. Could you please help me?

@lixiny
Copy link
Owner

lixiny commented Sep 19, 2022

Thanks again! I have found three loss in geo_loss.py: joint_u_axis_loss, joint_l_limit_loss and joint_b_axis_loss. For example, You calculate cos value using b_axis and axis. Do the b_axis and axis mean the twist axis direction of the same local frame with this joint's pose (include fathers' joints along kintree) and with only its father joints' pose?

Yes, the b_axis represent the back direction (aka twist in our paper), and it is calculated through:
current joint's child's xyz minus current joint's xyz.

axis-angle:
Since rotation can be represented as an direction (unit 3D vector, axis) and a scalar value (angle), calculating the cosine value between axis and the b_axis (unit vector along the back direction) give us the magnitude of the rotation vector along the b_axis. Intuitively, we want this magnitude to be close to zero.

@lixiny
Copy link
Owner

lixiny commented Sep 19, 2022

If I understand your question correct, the abnormal poses can not be penalized without knowing its parent, or child.
We use a coordinate adaptation axis layer to convert the original rotational axes to twist-bend-splay (in our code: back-left-up), and then penalize the abnormal rotations w.r.t twist-bend-splay.

Thanks a lot! One more question. How to avoid abnormal hand pose after designing local frame in your project? Such as setZero?

One more question! I am confusing about abnormal pose of thumb recently. It seems that the three joints of thumb have more DOFs than other fingers. I would like to know how to penalize the abnormal rotation of thumb specifically. Could you please help me?

We approximate the rotation DoFs on the first joints (closest to the wrist) of the thumb as a combination of motions in two orthogonal directions: bend and splay

Hope these figures illustrate our implementation:

bend

splay

@haolyuan
Copy link
Author

Thanks again! I have found three loss in geo_loss.py: joint_u_axis_loss, joint_l_limit_loss and joint_b_axis_loss. For example, You calculate cos value using b_axis and axis. Do the b_axis and axis mean the twist axis direction of the same local frame with this joint's pose (include fathers' joints along kintree) and with only its father joints' pose?

Yes, the b_axis represent the back direction (aka twist in our paper), and it is calculated through: current joint's child's xyz minus current joint's xyz.

axis-angle: Since rotation can be represented as an direction (unit 3D vector, axis) and a scalar value (angle), calculating the cosine value between axis and the b_axis (unit vector along the back direction) give us the magnitude of the rotation vector along the b_axis. Intuitively, we want this magnitude to be close to zero.

Thank you! I get it! It seems that you split the axis angle direction into three vertical direction. You restrict one direction to avoid abnormal pose so the final direction is got by suming the other two direction. But in my opinion, the axis angle is not in euclidean space. That is, "exp(aa^)exp(bb^)" is not equal to "exp((aa+bb)^)". So is it available to restrict the rotation using axis angle directly? Sorry for so many questions~I just started learing pose estimation.

@lixiny
Copy link
Owner

lixiny commented Sep 19, 2022

Thanks again! I have found three loss in geo_loss.py: joint_u_axis_loss, joint_l_limit_loss and joint_b_axis_loss. For example, You calculate cos value using b_axis and axis. Do the b_axis and axis mean the twist axis direction of the same local frame with this joint's pose (include fathers' joints along kintree) and with only its father joints' pose?

Yes, the b_axis represent the back direction (aka twist in our paper), and it is calculated through: current joint's child's xyz minus current joint's xyz.
axis-angle: Since rotation can be represented as an direction (unit 3D vector, axis) and a scalar value (angle), calculating the cosine value between axis and the b_axis (unit vector along the back direction) give us the magnitude of the rotation vector along the b_axis. Intuitively, we want this magnitude to be close to zero.

Thank you! I get it! It seems that you split the axis angle direction into three vertical direction. You restrict one direction to avoid abnormal pose so the final direction is got by suming the other two direction. But in my opinion, the axis angle is not in euclidean space. That is, "exp(aa^)exp(bb^)" is not equal to "exp((aa+bb)^)". So is it available to restrict the rotation using axis angle directly? Sorry for so many questions~I just started learing pose estimation.

Wow, good question! you are almost there.
Indeed the sum operation upon so3 algebra should be conducted as the multiplication in SO(3) space.

However,
here we use neural network to produce rotation R , but to penalize the R with respect to those abnormal direction.
It is feasible to represent the rotation as the Euler angles, and the angles can only be applied to the valid directions, in our case, bend and splay.
However, we find that axis-angles (recently we have moved to the ortho6D representation) is more suitable for neural networks.

@haolyuan
Copy link
Author

Thank a lot!

@lixiny lixiny pinned this issue Sep 19, 2022
@lixiny lixiny unpinned this issue May 19, 2023
@lixiny lixiny pinned this issue Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants