Is it possible to change the type of the robot tool

[madinwei]

Hello,
Thank you for your awesome code and repository.
I am interested in your work and I have implemented it, and now I want to use a different tool instead of the gripper tool.
do you have diffrent trianed model for diffrint tool. or how can I train a different robot tool, according to my needs.

am looking forward to your replay. any help and advice will be valuable to me.

[yiyeChen]

Hi,

Thanks for your interest in this project!

Can I ask what you mean by "a different tool"? Do you want to apply it to a specific two-finger gripper, or want to change to a different type of end-effector (e.g. suction cup)? Either way, our method should be applicable. I think you can simply swap our dataset with your specific dataset, with the keypoints generated by the GT pose of your specific end-effector, and then re-train the model.

Feel free to reach out if further problems arise! I am glad to help.

[madinwei]

@yiyeChen
Am very grateful for your reply and help, and I apologize for disturbing you during your summer vacation.

Can I ask what you mean by "a different tool"? Do you want to apply it to a specific two-finger gripper, or want to change to a different type of end-effector (e.g. suction cup)?

Yes, I want to use a different end-effector, like a welding torch.

Either way, our method should be applicable. I think you can simply swap our dataset with your specific dataset, with the keypoints generated by the GT pose of your specific end-effector, and then re-train the model.

So only I need to build my own data with my end-effector GT pose.
I have to check your generated data function, I would like to modify it for my welding torch tool, do you have any suggestions for me?

Feel free to reach out if further problems arise! I am glad to help.

again I am very grateful for your reply and help, I am Looking forward to your response.

[yiyeChen]

@madinwei

I am happy to assist you to test on different end-effector, and I am looking forward to the results!

I think the general process to test on your own dataset with novel end-effector is:

1. Define the end-effector frame keypoint for your own tool.
2. Generate 2D keypoint projections and scales from the annotated GT end-effector pose for training

For step 1, you can take a look at the grasp_kpts.py file. For step 2, you will need to take a look at the datasets/dataset/ps_grasp.py and datasets/sample/grasp_pose.py. Specifically, self._get_gt_grasps from the former class load the ground truth grasp pose and generate the 2D annotations such as keypoints and open width (which might not be necessary for you depending on your application) for a single data, indexed by scene id and camera id. I believe you will need to re-write this function to accommodate to your own dataset format and application requirement. After that, the latter (sampler) should be able to use the this function (these lines) to obtain the materials for training.

Those are on the top of my head. But if you encounter any further problem, feel free to let me know!

[madinwei]

@yiyeChen ,
that's very helpful and valuable for me.

Looking forward to contributing my end-effector to your KGN 👍 ^__^

[madinwei]

Hello, @yiyeChen , I hope you are doing well ^__^

I am following the instruction you have given me and haven't figured out how to do it.
I want to generate an object like this :

or like this:

I started with the last object, but had these issues:

1. I modified the cuboid class, which is easy to change the dimensions, however, I couldn't figure out how to make the class produce two object side by side with a random or specific distance.
2. I tried modifying the current tools grasp_kpts, since my weld tool just needs the orientation and position and I can make the open_width=0. but I also had an issue with how to specify where the tools are generated in the object, as in the screenshot:

I haven't figured out in def generate_grasp_family(self) how to make the tools on the side, or with the two pieces in between them.

first, I want to learn how to make the two-piece and generate tools accordingly and then I will be able to do other kinds of welding parts like the first screenshots.
I hope you offer me some intuitions or advice, looking forward to your response.

[yiyeChen]

@madinwei

It seems like you are trying to generate two boards and want to label the welding pose that can touch the connected part to join them. Then I would suggest you align the connected line with one of the axis, and shift both boards and weld torch accordingly. For example you can treat the connection line as the z axis. Then for the second image, you just need to shift one board along +y direction, the other board along -y direction for the same amount. I would recommend creating another class that encompasses two Box instances.

Then for the weld tool pose, by virtue of how we define the end-effector frame (pointing towards +x & the head aligns with the origin), you should already get a feasible weld pose following above steps without even applying any transformation, where the torch will be perpendicular to the two boards. You can also apply any rotation along the z axis, which should also be a feasible pose as the head is still touching the connected part (z axis). This will be the new generate_grasp_family() function for you.

You can also rotate the boards along the z axis or shift them along y direction to generate more arrangement of the two boards, including the one in the first image.

Besides, feel free to overwrite the generate_gripper_mesh function to change the appearance for your tool. Seems like your tool resembles a simple stick more.