Background:
FS Studio undertook a project to enhance an existing Unreal Engine 5 (UE5) office
simulation by implementing a system to export JSON objects containing pixel labels of
avatar head and feet positions for each frame and for a specific camera. The aim was to
extract and label specific positional data from avatars in the simulation, enabling accurate
spatial tracking in various environments.
This project involved complex challenges in developing non-native methods for extracting
data, managing distortion adjustments, and synchronizing labeling data with video frames.
The team also needed to maintain consistent avatar names across multiple camera views
and handle potential desync issues in the data export process.
Key Assumptions:
To achieve the project goals, FS Studio made the following key assumptions:
Weekly Check-ins for Progress Review:
FS Studio scheduled regular weekly meetings with the client to review progress,
address technical challenges, and ensure that milestones were met according to
the project timeline.
Distortion Adjustments and Screen Edges:
Distortion adjustments for pixel labels were expected to require tuning, particularly
near screen edges, where the distortions might have been more significant.
Clip Splitting and Desynchronization Risks:
Existing methods for clip splitting posed a potential risk of desynchronizing the
labeling data from video frames. To mitigate this, FS Studio considered altering the
export method to generate 10-second clips directly from image sequences, rather
than following the sequence > video > clip process. This change aimed to ensure
better synchronization and labeling accuracy.
Approximating 2D Bounding Boxes:
Since Unreal Engine 5 does not offer native methods to extract 2D bounding boxes,
FS Studio determined that approximations would be necessary. The team explored
options like using collision meshes, post-processing effects, or a combination of
both to approximate the required bounding boxes.
Scope of Services:
FS Studio provided a comprehensive suite of services, leveraging their expertise in Unreal
Engine 5, project management, and technical execution.
- Project Management:
FS Studio assigned a dedicated Project Manager to oversee the project. The Project
Manager coordinated the project kickoff, facilitated iterative design sessions, and handled
all technical working sessions. They also led weekly customer calls to ensure clear
communication and maintained a product backlog with individual work-tracking tickets to
track progress efficiently.
2. Core Team:
The project team included:
- Tech Artist (Unreal Engine 5): Responsible for implementing technical
solutions within UE5 and exporting data in the desired format. - Project Manager: Coordinated project milestones, client communication, and
overall timeline management.
Deliverables and Milestones:
The primary deliverable was an updated UE5 project with the following key features and
functionalities:
Feet and Head Labeling in JSON Format:
Feet average labeled in pixel coordinates: The system exported the pixel
coordinates for avatars’ feet, calculated as an average position.
- Feet average labeled in world coordinates at 0 height: Export of the feet’s world
coordinates was achieved, normalized at ground level (height 0). - Feet average labeled in map coordinates: The system also labeled avatars’ feet in
map coordinates, allowing further spatial tracking across environments. - 2D Bounding Box Extraction:
- 2D Bounding Box for Avatar Heads: A bounding box for avatar heads was
generated and exported, enabling accurate tracking of avatar movements. - Avatar 2D Bounding Box and Segmentation Mask: A segmentation mask was
added, further enhancing the ability to distinguish avatars within the simulation
environment.
Handling Distortion Adjustments:
FS Studio ensured that the system could handle and adjust for screen distortions. Special
care was taken with bounding boxes and labeling near screen edges where distortion
effects were most pronounced.
Consistency of Avatar Names Across Views:
A key requirement was ensuring avatar names remained consistent throughout the
different camera views. FS Studio ensured that each avatar’s name, bounding box, and
segmentation mask remained consistent regardless of the camera angle or perspective.
Outcome:
FS Studio successfully delivered the updated Unreal Engine 5 project, providing the client
with a robust solution for exporting JSON objects containing critical positional data and
bounding boxes for avatars. By addressing the complex challenges of distortion,
desynchronization, and non-native feature implementations, FS Studio showcased its
expertise in solving advanced technical problems within simulation environments.
The project’s success opened further avenues for using this labeling and bounding box
extraction method in future simulation projects, improving the accuracy of avatar tracking
and enabling advanced analytics across different camera perspectives.