Student assistant (m/f/d)-Multi-Modal Sensor Fusion, Object Detection in Clutter and Object Tracking

The Fraunhofer-Gesellschaft (www.fraunhofer.com) currently operates 76 institutes and research units throughout Germany and is a leading applied research organization. Around 32 000 employees work with an annual research budget of 3.4 billion euros.

Advertisement for the field of study such as: Automation technology, cybernetics, aerospace engineering, mechanical engineering, mechatronics, physics, control engineering, software engineering, computer science, robotics or comparable.

Reliable and precise perception is essential for robot-based bin picking and assembly, especially for bulk goods and object manipulation with robotic hands. The focus here is robust multi-camera fusion, accurate object detection in cluttered scenes (bulk goods), and multi-modal tracking that combines camera data with tactile feedback. Using deep learning, multi-view geometry, and simulation via digital twins, the goal is a real-time, deployable perception stack for an industrial robot arm using a robotic hand as an end effector.

What you will do

Literature Review: Survey multi-camera sensor fusion, object detection under heavy clutter/occlusion, and multi-modal (vision + tactile) tracking in robotics and manufacturing.
Framework Development: Set up the digital twin in Isaac Sim; build scene generators for clutter, occlusions, and varying lighting/reflectivity. Specify dataset strategy (augmentation/domain randomization) and an automated evaluation pipeline. Calibrate and time-synchronize multi-camera systems with tactile sensors; define data schemas and ROS 2/Isaac ROS interfaces.
Algorithm Implementation: Develop multi-camera fusion for 3D localization (triangulation, multi-view association) using e.g. EKF/UKF. Build multi-modal tracking that fuses object tracking with tactile signals. Optimize for real time (TensorRT, pruning/quantization). Implement robustness strategies for occlusions, small/reflective objects, and provide uncertainty estimates.
Simulation and Testing: Generate diverse scenarios in Isaac Sim with domain randomization; Evaluate calibration error, FPS/latency, and resource usage. Run ablations on fusion strategies, number/placement of cameras, and tactile on/off; study sim-to-real transfer.

What you bring to the table

Enrolled student
High motivation and initiative
Very good English language skills
Experience with Isaac Sim / Lab
Experience in computer vision and deep learning is an advantage
Independent, responsible, and structured working style

What you can expect

Pleasant working atmosphere
interesting and industry-relevant tasks
dynamic, interdisciplinary team
Insights into the future topic of automation
Good technical equipment

We value and promote the diversity of our employees' skills and therefore welcome all applications - regardless of age, gender, nationality, ethnic and social origin, religion, ideology, disability, sexual orientation and identity. Severely disabled persons are given preference in the event of equal suitability. Remuneration according to the general works agreement for employing assistant staff.

With its focus on developing key technologies that are vital for the future and enabling the commercial utilization of this work by business and industry, Fraunhofer plays a central role in the innovation process. As a pioneer and catalyst for groundbreaking developments and scientific excellence, Fraunhofer helps shape society now and in the future.

Interested? Apply online now. We look forward to getting to know you!

Ms. Jennifer Leppich

Recruiting

+49 711 970-1415

jennifer.leppich@ipa.fraunhofer.de

Fraunhofer Institute for Manufacturing Engineering and Automation IPA

www.ipa.fraunhofer.de

Requisition Number: 81682 Application Deadline:

Job Segment: Test Engineer, Testing, Computer Science, Software Engineer, Manufacturing Engineer, Engineering, Technology

Apply now »