当前位置:   article > 正文

python-ue4-metahuman-nerf:我创造了一个数字人!!_ue4-nerf

ue4-nerf

原文:https://zhuanlan.zhihu.com/p/561152870

目录

收起

1. 准备工作:制作 MetaHuman 角色

1.1 创建 MetaHuman 角色

1.2 Quixel Bridge 下载 MetaHuman

1.3 导出 MetaHuman 到 UE4

2. UE 渲染 MetaHuman 的多视点图片

2.1 如何在 UE 中手动渲染视频?

2.2 Python 自动化渲染

2.3 全部代码

3. 渲染结果

随着 ECCV 2020 NeRF 的出现,在计算机视觉、计算机图形学、三维重建等领域砸开了无数的坑位,一篇 NeRF 养活了多少科研工作者、大厂员工。但是要想训练这样一个 Neural Radiance Filed 却不是一件容易的事情,首先从数据来看,训练一个 NeRF 需要该场景的多视点图片,同时还需要知道每张图片所对应的相机参数,要想得到这样的数据,可谓是耗时耗力。那么,有没有一种方式,可以虚拟的仿真出一组 NeRF 的训练数据?

答案当然是有的,而且不止一种方式。下面本文将分享一种使用 UE4 来渲染带有相机参数的多视点图片的方法。在正式开始之前,首先来介绍一下 MetaHuman,它就是我们要渲染的对象。在计算机视觉、三维重建领域,人脸方向的研究一直是一个热点,而 MetaHuman 则是一个高保真的数字人开源框架,我们可以利用 MetaHuman 来设计出自己想要的数字人形象,然后渲染出其多视点图片,用作进一步的研究。

1. 准备工作:制作 MetaHuman 角色

在 UE4 中渲染 MetaHuman 之前,我们首先需要创造一个 Metahuman 角色,然后并将其导入到 UE4 中进行渲染,整体可分为以下几个步骤

  1. 使用 MetaHuman Creator 创建 MetaHuman 角色;也可以不创建新角色,直接使用自带的即可。
  2. 借助 Quixel Bridge 工具将创建的 MetaHuman 角色下载下来。
  3. 在 Quixel Bridge 中将 MetaHuman 角色导出到 UE4中。

下面分别对这几步进行说明:

1.1 创建 MetaHuman 角色

登入 MetaHuman Creator 官网:https://metahuman.unrealengine.com/,会看到如下界面,左侧的 CREATE METAHUMAN 是系统默认给出的 Metahuman 角色,可以直接在其基础上进行修改,创作一个新的 Metahuman,这会存在左侧的 MY METAHUMAN 目录中。

具体的操作界面如下,整体分为三部分:脸部、毛发、身体。我们可以对这三部分中的各个细节进行操作,大家可以自行尝试~

1.2 Quixel Bridge 下载 MetaHuman

我们刚刚创作好的 MetaHuman 角色是存在于官方的服务器上的,在对它进行操作之前,还需要借助 Quixel Bridge 工具将其下载下来。

Quixel Bridge 下载链接:https://quixel.com/bridge

在下载时候,我们使用 Epic Games 账号,后续用的也一直是这个账号。下载完成并登录之后,我们会看到如下界面

左侧导航栏中的 MetaHuman Presets 和 My MetaHuman 分别表示默认角色和我们前一步创建的角色。点击右上角绿色的下载按钮即可下载到本地。

1.3 导出 MetaHuman 到 UE4

在导出 MetaHuman 角色到 UE4 之前,我们需要配置:导出的目标引擎是什么、引擎的版本号、引擎的位置。如下图所示

之后,点击对应 MetaHuman 角色右上角的 Export 按钮即可将其导出到 UE 引擎中(注意:在导出时,要保证 UE 是打开的)。

2. UE 渲染 MetaHuman 的多视点图片

在利用 Python 自动化渲染符合要求的多视点图片之前,我们首先要搞明白的一件事情是:如何手动的在 UE4 中渲染视频序列(视频和图片本质上是一样的,图片就是视频一帧一帧截出来的)?

2.1 如何在 UE 中手动渲染视频?

step1:将 MetaHuman 导入到 UE4 的场景中,并将其定位到合适的位置。在场景中添加电影摄像机 CineCameraActor, 如下:

step2:创建过场动画,并添加关卡序列,如下

step3:依次点击,轨道 — Actor到Sequencer — CineCameraActor,创建 CineCameraActor 的轨道序列,如下

step4:在创建好的 CineCameraActor 轨道序列中添加关键帧(下图添加了 5 个关键帧,分别指定了相机的位置和旋转),使得相机按照既定方式运动,如下

step5:至此,视频序列创建完毕,点击 将影片渲染为视频序列或图片序列,完成导出

上述就是在 UE4 中渲染 MetaHuman 的整个过程,只需要鼠标点点就行了,比较简单。但是有一个严重的问题是:如果要渲染一个多视点 MetaHuman 数据集用作 NeRF 训练的话,我们是必须要知道相机的姿态的;而在上述过程中,我们只能准确的知道关键帧的相机姿态,其他帧的相机姿态都是被插值出来的。如果要手动定义数百上千个关键帧,又是相当繁琐的一项工作,因此我们下面考虑利用 python 来自动化的执行上述过程,自动化的定义关键帧,指定相机的运动。

2.2 Python 自动化渲染

首先,总结一下前一章节手动渲染的过程,整体上可以分为如下 4 个步骤:

  1. 导入 MetaHuman 角色,并将其定位的合适位置
  2. 创建关卡序列,并创建 CineCameraActor 轨道序列
  3. 设置相机参数,并指定相机的位置和旋转,定义相机的运动方式
  4. 导出渲染的视频、图片序列

上述 4 个步骤中,除了步骤一还需要我们手动操作以外,其余步骤全部都可以利用 python 脚本来实现。在 UE4 中使用 python 接口还需要进行一些简单的配置,具体操作可以参考:https://www.bilibili.com/video/BV1PE411d7z8

步骤一:角色导入以及人脸定位

由于我只想渲染 MetaHuman 的人脸区域,因此我需要将其余身体部分移动到相机视野之外。需要移动的身体部位有:Torso、Legs、Feet。

接着进行 MetaHuman 的定位,为了方便,我们可以将其 Face 位置定位在坐标原点(0,0,0),这是为了方便后面指定相机的位置与旋转。

步骤二:指定相机运动方式:位置和旋转

在这一步,我希望相机能够以 MetaHuman 的人脸(也就是坐标原点)为中心,做球形旋转运动。关于相机的外参,做如下设置:

  • 相机到坐标原点的距离(即:球半径)为35
  • 相机的俯仰角设置为如下角度:[-30, -15, 0, 15, 30]
  • 在每个俯仰角下,偏航角区间设置为:[-180, 180], 每间隔角度 10 定义一个关键点(即:拍摄一张照片)

渲染的相关的设置如下:

  • 相机的视场角设置为:100
  • 所渲染图片的分辨率:512
  • 帧与帧之间的延迟:1.5

相机内参的设置见:步骤三具体的渲染代码。

现在,我们将相机的运动数据(相机位置和旋转)、渲染的相关设置,存为一个 .json 文件,在步骤三渲染的时候直接读取这些数据即可。代码如下:

  1. import json
  2. import numpy as np
  3. # 得到相机的运动数据(位置、旋转)
  4. def get_camera_transform():
  5. radius = 35
  6. camara_transform = []
  7. for y_pitch in range(-30, 31, 15):
  8. for z_yaw in range(-180, 181, 10):
  9. location_x = radius * np.cos(z_yaw / 180 * np.pi) * np.cos(y_pitch / 180 * np.pi)
  10. location_y = radius * np.sin(z_yaw / 180 * np.pi) * np.cos(y_pitch / 180 * np.pi)
  11. location_z = -radius * np.sin(y_pitch / 180 * np.pi)
  12. print([-location_x, -location_y, location_z, 0, y_pitch, z_yaw])
  13. camara_transform.append([-location_x, -location_y, location_z, 0, y_pitch, z_yaw])
  14. return camara_transform
  15. # 将基础数据存为一个 .json 文件
  16. def make_json():
  17. camara_transform = get_camera_transform()
  18. data_dict = {
  19. f"film_fov": 100,
  20. "film_resolution": 512,
  21. "delay_every_frame": 1.5,
  22. "output_image_path": "F:/projects/Unreal projects/MetaHumanRender/RenderImages/Danielle/face_hair",
  23. "camera_data": {
  24. "camera_qRrq9DqY": {
  25. "type": "Camera",
  26. "camera_transform": camara_transform
  27. }
  28. }
  29. }
  30. data_json = json.dumps(data_dict, indent=4)
  31. with open("camera_data.json", 'w') as json_file:
  32. json_file.write(data_json)
  33. if __name__ == '__main__':
  34. make_json()

步骤三:读取相机运动数据,创建关卡序列添加关键帧

首先,读取上一节生成的.json文件,获取相机运动数据

  1. def read_json_file(path):
  2. file = open(path, "rb")
  3. file_json = json.load(file)
  4. export_path = file_json['output_image_path'] # image export path
  5. film_fov = file_json.get('film_fov', None) # camera film fov
  6. film_resolution = file_json.get("film_resolution", 512) # camera file resolution
  7. delay_every_frame = file_json.get("delay_every_frame", 3.0) # sequence run delay every frame
  8. camera_json = file_json['camera_data'] # camera data
  9. camera_transform_array = []
  10. camera_rotation_array = []
  11. camera_name_array = []
  12. for camera in camera_json:
  13. camera_transform = camera_json[camera]["camera_transform"]
  14. for index in range(len(camera_transform)):
  15. camera_transform_array.append(unreal.Transform(
  16. location=[camera_transform[index][0], camera_transform[index][1], camera_transform[index][2]],
  17. rotation=[camera_transform[index][3], camera_transform[index][4], camera_transform[index][5]],
  18. scale=[1.0, 1.0, 1.0]
  19. ))
  20. camera_rotation_array.append(unreal.Rotator(
  21. roll=camera_transform[index][3],
  22. pitch=camera_transform[index][4],
  23. yaw=camera_transform[index][5]
  24. ))
  25. camera_name_array.append(camera)
  26. return camera_transform_array, camera_rotation_array, camera_name_array, export_path, film_fov, film_resolution, delay_every_frame

接着创建关卡序列,并添加关键帧,每一个相机姿态对应一个关键帧。在这一部分,做要做了如下几个事情,具体细节可参考代码

  • 在场景中创建一个电影摄像机 CineCameraActor,并指定其位置和旋转。
  • 设置胶片或数字传感器的尺寸。
  • 相机的镜头的参数设置:最小焦距、最大焦距等。
  • 聚焦设置:焦距、光圈、视场角等。
  • 相机的绑定,相机的关键帧添加,对每一个相机姿态添加一个关键帧。

代码

  1. def create_sequence(asset_name, camera_transform_array, camera_rotation_array, camera_name_array, film_fov,
  2. package_path='/Game/'):
  3. # create a sequence
  4. sequence = unreal.AssetToolsHelpers.get_asset_tools().create_asset(asset_name, package_path, unreal.LevelSequence,
  5. unreal.LevelSequenceFactoryNew())
  6. sequence.set_display_rate(unreal.FrameRate(numerator=1, denominator=1))
  7. sequence.set_playback_start_seconds(0)
  8. sequence.set_playback_end_seconds(len(camera_transform_array))
  9. # add a camera cut track
  10. camera_cut_track = sequence.add_master_track(unreal.MovieSceneCameraCutTrack)
  11. print(camera_transform_array[0], '=======================================================================')
  12. print(camera_rotation_array[0].get_editor_property("roll"))
  13. # 添加关键帧
  14. for index in range(0, len(camera_name_array)):
  15. camera_cut_section = camera_cut_track.add_section()
  16. camera_cut_section.set_start_frame_bounded(6 * index)
  17. camera_cut_section.set_end_frame_bounded(6 * index + len(camera_transform_array))
  18. camera_cut_section.set_start_frame_seconds(6 * index)
  19. camera_cut_section.set_end_frame_seconds(6 * index + len(camera_transform_array))
  20. # Create a cine camera actor
  21. # 在世界编辑器中创建一个电影摄像机CineCameraActor,并指定位置和旋转
  22. camera_actor = unreal.EditorLevelLibrary().spawn_actor_from_class(unreal.CineCameraActor,
  23. unreal.Vector(0, 0, 0),
  24. unreal.Rotator(0, 0, 0))
  25. # 指定电影摄像机CineCameraActor的名称
  26. camera_actor.set_actor_label(camera_name_array[index])
  27. # 返回CineCamera组件
  28. camera_component = camera_actor.get_cine_camera_component()
  29. ratio = math.tan(film_fov / 360.0 * math.pi) * 2
  30. # step1:胶片背板设置
  31. filmback = camera_component.get_editor_property("filmback")
  32. # sensor_height:胶片或数字传感器的垂直尺寸,以毫米为单位
  33. filmback.set_editor_property("sensor_height", 60)
  34. # sensor_width:胶片或数字传感器的水平尺寸,以毫米为单位
  35. filmback.set_editor_property("sensor_width", 60)
  36. # step2:镜头设置
  37. lens_settings = camera_component.get_editor_property("lens_settings")
  38. lens_settings.set_editor_property("min_focal_length", 4.0)
  39. lens_settings.set_editor_property("max_focal_length", 1000)
  40. lens_settings.set_editor_property("min_f_stop", 1.2)
  41. lens_settings.set_editor_property("max_f_stop", 500)
  42. print(lens_settings)
  43. # step3:聚焦设置
  44. focal_length = 45 # 焦距
  45. current_aperture = 500 # 光圈
  46. camera_component.set_editor_property("current_focal_length", focal_length) # 设置当前焦距
  47. camera_component.set_editor_property("current_aperture", current_aperture) # 设置当前光圈
  48. fov = camera_component.get_editor_property("field_of_view")
  49. print(camera_component.get_editor_property("current_focal_length"))
  50. print(camera_component.get_editor_property("current_aperture"))
  51. print(f"fov - {fov}")
  52. # add a binding for the camera
  53. camera_binding = sequence.add_possessable(camera_actor)
  54. transform_track = camera_binding.add_track(unreal.MovieScene3DTransformTrack)
  55. transform_section = transform_track.add_section()
  56. transform_section.set_start_frame_bounded(6 * index)
  57. transform_section.set_end_frame_bounded(6 * index + len(camera_transform_array))
  58. transform_section.set_start_frame_seconds(6 * index)
  59. transform_section.set_end_frame_seconds(6 * index + len(camera_transform_array))
  60. # get channel for location_x location_y location_z
  61. channel_location_x = transform_section.get_channels()[0]
  62. channel_location_y = transform_section.get_channels()[1]
  63. channel_location_z = transform_section.get_channels()[2]
  64. # get key for rotation_y rotation_z
  65. channel_rotation_x = transform_section.get_channels()[3]
  66. channel_rotation_y = transform_section.get_channels()[4]
  67. channel_rotation_z = transform_section.get_channels()[5]
  68. # camera_transform = camera_transform_array[index]
  69. # camera_location_x = camera_transform.get_editor_property("translation").get_editor_property("x")
  70. # camera_location_y = camera_transform.get_editor_property("translation").get_editor_property("y")
  71. # camera_location_z = camera_transform.get_editor_property("translation").get_editor_property("z")
  72. for i in range(len(camera_transform_array)):
  73. camera_transform = camera_transform_array[i]
  74. camera_location_x = camera_transform.get_editor_property("translation").get_editor_property("x")
  75. camera_location_y = camera_transform.get_editor_property("translation").get_editor_property("y")
  76. camera_location_z = camera_transform.get_editor_property("translation").get_editor_property("z")
  77. camera_rotation = camera_rotation_array[i]
  78. camera_rotate_roll = camera_rotation.get_editor_property("roll")
  79. camera_rotate_pitch = camera_rotation.get_editor_property("pitch")
  80. camera_rotate_yaw = camera_rotation.get_editor_property("yaw")
  81. new_time = unreal.FrameNumber(value=i)
  82. channel_location_x.add_key(new_time, camera_location_x, 0.0)
  83. channel_location_y.add_key(new_time, camera_location_y, 0.0)
  84. channel_location_z.add_key(new_time, camera_location_z, 0.0)
  85. channel_rotation_x.add_key(new_time, camera_rotate_roll, 0.0)
  86. channel_rotation_y.add_key(new_time, camera_rotate_pitch, 0.0)
  87. channel_rotation_z.add_key(new_time, camera_rotate_yaw, 0.0)
  88. # if i % 5 == 0:
  89. # print(new_time, [camera_location_x, camera_location_y, camera_location_z, camera_rotate_roll, camera_rotate_pitch, camera_rotate_yaw])
  90. # add the binding for the camera cut section
  91. camera_binding_id = sequence.make_binding_id(camera_binding, unreal.MovieSceneObjectBindingSpace.LOCAL)
  92. camera_cut_section.set_camera_binding_id(camera_binding_id)
  93. # save sequence asset
  94. unreal.EditorAssetLibrary.save_loaded_asset(sequence, False)
  95. return sequence

步骤四:将关卡序列渲染为视频、图片序列

将创建的关卡序列渲染为电影序列,这部分主要就是设置一些渲染参数,对应手动渲染的 step5,具体参数的含义可参考 unreal 官方文档。代码如下:

  1. # sequence movie
  2. def render_sequence_to_movie(export_path, film_resolution, delay_every_frame, on_finished_callback):
  3. # 1) Create an instance of our UAutomatedLevelSequenceCapture and override all of the settings on it. This class is currently
  4. # set as a config class so settings will leak between the Unreal Sequencer Render-to-Movie UI and this object. To work around
  5. # this, we set every setting via the script so that no changes the user has made via the UI will affect the script version.
  6. # The users UI settings will be reset as an unfortunate side effect of this.
  7. capture_settings = unreal.AutomatedLevelSequenceCapture()
  8. # Set all POD settings on the UMovieSceneCapture
  9. # capture_settings.settings.output_directory = unreal.DirectoryPath("../../../unreal-render-release/Saved/VideoCaptures1/")
  10. capture_settings.settings.output_directory = unreal.DirectoryPath(export_path)
  11. # If you game mode is implemented in Blueprint, load_asset(...) is going to return you the C++ type ('Blueprint') and not what the BP says it inherits from.
  12. # Instead, because game_mode_override is a TSubclassOf<AGameModeBase> we can use unreal.load_class to get the UClass which is implicitly convertable.
  13. # ie: capture_settings.settings.game_mode_override = unreal.load_class(None, "/Game/AI/TestingSupport/AITestingGameMode.AITestingGameMode_C")
  14. capture_settings.settings.game_mode_override = None
  15. capture_settings.settings.output_format = "{camera}" + "_{frame}"
  16. capture_settings.settings.overwrite_existing = False
  17. capture_settings.settings.use_relative_frame_numbers = False
  18. capture_settings.settings.handle_frames = 0
  19. capture_settings.settings.zero_pad_frame_numbers = 4
  20. # If you wish to override the output framerate you can use these two lines, otherwise the framerate will be derived from the sequence being rendered
  21. capture_settings.settings.use_custom_frame_rate = False
  22. # capture_settings.settings.custom_frame_rate = unreal.FrameRate(24,1)
  23. capture_settings.settings.resolution.res_x = film_resolution
  24. capture_settings.settings.resolution.res_y = film_resolution
  25. capture_settings.settings.enable_texture_streaming = False
  26. capture_settings.settings.cinematic_engine_scalability = True
  27. capture_settings.settings.cinematic_mode = True
  28. capture_settings.settings.allow_movement = False # Requires cinematic_mode = True
  29. capture_settings.settings.allow_turning = False # Requires cinematic_mode = True
  30. capture_settings.settings.show_player = False # Requires cinematic_mode = True
  31. capture_settings.settings.show_hud = False # Requires cinematic_mode = True
  32. capture_settings.use_separate_process = False
  33. capture_settings.close_editor_when_capture_starts = False # Requires use_separate_process = True
  34. capture_settings.additional_command_line_arguments = "-NOSCREENMESSAGES" # Requires use_separate_process = True
  35. capture_settings.inherited_command_line_arguments = "" # Requires use_separate_process = True
  36. # Set all the POD settings on UAutomatedLevelSequenceCapture
  37. capture_settings.use_custom_start_frame = False # If False, the system will automatically calculate the start based on sequence content
  38. capture_settings.use_custom_end_frame = False # If False, the system will automatically calculate the end based on sequence content
  39. capture_settings.custom_start_frame = unreal.FrameNumber(0) # Requires use_custom_start_frame = True
  40. capture_settings.custom_end_frame = unreal.FrameNumber(0) # Requires use_custom_end_frame = True
  41. capture_settings.warm_up_frame_count = 0
  42. capture_settings.delay_before_warm_up = 0
  43. capture_settings.delay_before_shot_warm_up = 0
  44. capture_settings.delay_every_frame = delay_every_frame
  45. capture_settings.write_edit_decision_list = True
  46. # Tell the capture settings which level sequence to render with these settings. The asset does not need to be loaded,
  47. # as we're only capturing the path to it and when the PIE instance is created it will load the specified asset.
  48. # If you only had a reference to the level sequence, you could use "unreal.SoftObjectPath(mysequence.get_path_name())"
  49. capture_settings.level_sequence_asset = unreal.SoftObjectPath(sequence_asset_path)
  50. # To configure the video output we need to tell the capture settings which capture protocol to use. The various supported
  51. # capture protocols can be found by setting the Unreal Content Browser to "Engine C++ Classes" and filtering for "Protocol"
  52. # ie: CompositionGraphCaptureProtocol, ImageSequenceProtocol_PNG, etc. Do note that some of the listed protocols are not intended
  53. # to be used directly.
  54. # Right click on a Protocol and use "Copy Reference" and then remove the extra formatting around it. ie:
  55. # Class'/Script/MovieSceneCapture.ImageSequenceProtocol_PNG' gets transformed into "/Script/MovieSceneCapture.ImageSequenceProtocol_PNG"
  56. capture_settings.set_image_capture_protocol_type(
  57. unreal.load_class(None, "/Script/MovieSceneCapture.ImageSequenceProtocol_PNG"))
  58. # After we have set the capture protocol to a soft class path we can start editing the settings for the instance of the protocol that is internallyc reated.
  59. capture_settings.get_image_capture_protocol().compression_quality = 85
  60. # Finally invoke Sequencer's Render to Movie functionality. This will examine the specified settings object and either construct a new PIE instance to render in,
  61. # or create and launch a new process (optionally shutting down your editor).
  62. unreal.SequencerTools.render_movie(capture_settings, on_finished_callback)

2.3 全部代码

  1. import sys
  2. import unreal, os, json, math
  3. # sequence asset path
  4. sequence_asset_path = '/Game/Render_Sequence.Render_Sequence'
  5. # read json file
  6. def read_json_file(path):
  7. file = open(path, "rb")
  8. file_json = json.load(file)
  9. export_path = file_json['output_image_path'] # image export path
  10. film_fov = file_json.get('film_fov', None) # camera film fov
  11. film_resolution = file_json.get("film_resolution", 512) # camera file resolution
  12. delay_every_frame = file_json.get("delay_every_frame", 3.0) # sequence run delay every frame
  13. camera_json = file_json['camera_data'] # camera data
  14. camera_transform_array = []
  15. camera_rotation_array = []
  16. camera_name_array = []
  17. for camera in camera_json:
  18. camera_transform = camera_json[camera]["camera_transform"]
  19. for index in range(len(camera_transform)):
  20. camera_transform_array.append(unreal.Transform(
  21. location=[camera_transform[index][0], camera_transform[index][1], camera_transform[index][2]],
  22. rotation=[camera_transform[index][3], camera_transform[index][4], camera_transform[index][5]],
  23. scale=[1.0, 1.0, 1.0]
  24. ))
  25. camera_rotation_array.append(unreal.Rotator(
  26. roll=camera_transform[index][3],
  27. pitch=camera_transform[index][4],
  28. yaw=camera_transform[index][5]
  29. ))
  30. camera_name_array.append(camera)
  31. return camera_transform_array, camera_rotation_array, camera_name_array, export_path, film_fov, film_resolution, delay_every_frame
  32. def create_sequence(asset_name, camera_transform_array, camera_rotation_array, camera_name_array, film_fov,
  33. package_path='/Game/'):
  34. # create a sequence
  35. sequence = unreal.AssetToolsHelpers.get_asset_tools().create_asset(asset_name, package_path, unreal.LevelSequence,
  36. unreal.LevelSequenceFactoryNew())
  37. sequence.set_display_rate(unreal.FrameRate(numerator=1, denominator=1))
  38. sequence.set_playback_start_seconds(0)
  39. sequence.set_playback_end_seconds(len(camera_transform_array))
  40. # add a camera cut track
  41. camera_cut_track = sequence.add_master_track(unreal.MovieSceneCameraCutTrack)
  42. print(camera_transform_array[0], '=======================================================================')
  43. print(camera_rotation_array[0].get_editor_property("roll"))
  44. # 添加关键帧
  45. for index in range(0, len(camera_name_array)):
  46. camera_cut_section = camera_cut_track.add_section()
  47. camera_cut_section.set_start_frame_bounded(6 * index)
  48. camera_cut_section.set_end_frame_bounded(6 * index + len(camera_transform_array))
  49. camera_cut_section.set_start_frame_seconds(6 * index)
  50. camera_cut_section.set_end_frame_seconds(6 * index + len(camera_transform_array))
  51. # Create a cine camera actor
  52. # 在世界编辑器中创建一个电影摄像机CineCameraActor,并指定位置和旋转
  53. camera_actor = unreal.EditorLevelLibrary().spawn_actor_from_class(unreal.CineCameraActor,
  54. unreal.Vector(0, 0, 0),
  55. unreal.Rotator(0, 0, 0))
  56. # 指定电影摄像机CineCameraActor的名称
  57. camera_actor.set_actor_label(camera_name_array[index])
  58. # 返回CineCamera组件
  59. camera_component = camera_actor.get_cine_camera_component()
  60. ratio = math.tan(film_fov / 360.0 * math.pi) * 2
  61. # step1:胶片背板设置
  62. filmback = camera_component.get_editor_property("filmback")
  63. # sensor_height:胶片或数字传感器的垂直尺寸,以毫米为单位
  64. filmback.set_editor_property("sensor_height", 60)
  65. # sensor_width:胶片或数字传感器的水平尺寸,以毫米为单位
  66. filmback.set_editor_property("sensor_width", 60)
  67. # step2:镜头设置
  68. lens_settings = camera_component.get_editor_property("lens_settings")
  69. lens_settings.set_editor_property("min_focal_length", 4.0)
  70. lens_settings.set_editor_property("max_focal_length", 1000)
  71. lens_settings.set_editor_property("min_f_stop", 1.2)
  72. lens_settings.set_editor_property("max_f_stop", 500)
  73. print(lens_settings)
  74. # step3:聚焦设置
  75. focal_length = 45 # 焦距
  76. current_aperture = 500 # 光圈
  77. camera_component.set_editor_property("current_focal_length", focal_length) # 设置当前焦距
  78. camera_component.set_editor_property("current_aperture", current_aperture) # 设置当前光圈
  79. fov = camera_component.get_editor_property("field_of_view")
  80. print(camera_component.get_editor_property("current_focal_length"))
  81. print(camera_component.get_editor_property("current_aperture"))
  82. print(f"fov - {fov}")
  83. # add a binding for the camera
  84. camera_binding = sequence.add_possessable(camera_actor)
  85. transform_track = camera_binding.add_track(unreal.MovieScene3DTransformTrack)
  86. transform_section = transform_track.add_section()
  87. transform_section.set_start_frame_bounded(6 * index)
  88. transform_section.set_end_frame_bounded(6 * index + len(camera_transform_array))
  89. transform_section.set_start_frame_seconds(6 * index)
  90. transform_section.set_end_frame_seconds(6 * index + len(camera_transform_array))
  91. # get channel for location_x location_y location_z
  92. channel_location_x = transform_section.get_channels()[0]
  93. channel_location_y = transform_section.get_channels()[1]
  94. channel_location_z = transform_section.get_channels()[2]
  95. # get key for rotation_y rotation_z
  96. channel_rotation_x = transform_section.get_channels()[3]
  97. channel_rotation_y = transform_section.get_channels()[4]
  98. channel_rotation_z = transform_section.get_channels()[5]
  99. # camera_transform = camera_transform_array[index]
  100. # camera_location_x = camera_transform.get_editor_property("translation").get_editor_property("x")
  101. # camera_location_y = camera_transform.get_editor_property("translation").get_editor_property("y")
  102. # camera_location_z = camera_transform.get_editor_property("translation").get_editor_property("z")
  103. for i in range(len(camera_transform_array)):
  104. camera_transform = camera_transform_array[i]
  105. camera_location_x = camera_transform.get_editor_property("translation").get_editor_property("x")
  106. camera_location_y = camera_transform.get_editor_property("translation").get_editor_property("y")
  107. camera_location_z = camera_transform.get_editor_property("translation").get_editor_property("z")
  108. camera_rotation = camera_rotation_array[i]
  109. camera_rotate_roll = camera_rotation.get_editor_property("roll")
  110. camera_rotate_pitch = camera_rotation.get_editor_property("pitch")
  111. camera_rotate_yaw = camera_rotation.get_editor_property("yaw")
  112. new_time = unreal.FrameNumber(value=i)
  113. channel_location_x.add_key(new_time, camera_location_x, 0.0)
  114. channel_location_y.add_key(new_time, camera_location_y, 0.0)
  115. channel_location_z.add_key(new_time, camera_location_z, 0.0)
  116. channel_rotation_x.add_key(new_time, camera_rotate_roll, 0.0)
  117. channel_rotation_y.add_key(new_time, camera_rotate_pitch, 0.0)
  118. channel_rotation_z.add_key(new_time, camera_rotate_yaw, 0.0)
  119. # if i % 5 == 0:
  120. # print(new_time, [camera_location_x, camera_location_y, camera_location_z, camera_rotate_roll, camera_rotate_pitch, camera_rotate_yaw])
  121. # add the binding for the camera cut section
  122. camera_binding_id = sequence.make_binding_id(camera_binding, unreal.MovieSceneObjectBindingSpace.LOCAL)
  123. camera_cut_section.set_camera_binding_id(camera_binding_id)
  124. # save sequence asset
  125. unreal.EditorAssetLibrary.save_loaded_asset(sequence, False)
  126. return sequence
  127. # sequence movie
  128. def render_sequence_to_movie(export_path, film_resolution, delay_every_frame, on_finished_callback):
  129. # 1) Create an instance of our UAutomatedLevelSequenceCapture and override all of the settings on it. This class is currently
  130. # set as a config class so settings will leak between the Unreal Sequencer Render-to-Movie UI and this object. To work around
  131. # this, we set every setting via the script so that no changes the user has made via the UI will affect the script version.
  132. # The users UI settings will be reset as an unfortunate side effect of this.
  133. capture_settings = unreal.AutomatedLevelSequenceCapture()
  134. # Set all POD settings on the UMovieSceneCapture
  135. # capture_settings.settings.output_directory = unreal.DirectoryPath("../../../unreal-render-release/Saved/VideoCaptures1/")
  136. capture_settings.settings.output_directory = unreal.DirectoryPath(export_path)
  137. # If you game mode is implemented in Blueprint, load_asset(...) is going to return you the C++ type ('Blueprint') and not what the BP says it inherits from.
  138. # Instead, because game_mode_override is a TSubclassOf<AGameModeBase> we can use unreal.load_class to get the UClass which is implicitly convertable.
  139. # ie: capture_settings.settings.game_mode_override = unreal.load_class(None, "/Game/AI/TestingSupport/AITestingGameMode.AITestingGameMode_C")
  140. capture_settings.settings.game_mode_override = None
  141. capture_settings.settings.output_format = "{camera}" + "_{frame}"
  142. capture_settings.settings.overwrite_existing = False
  143. capture_settings.settings.use_relative_frame_numbers = False
  144. capture_settings.settings.handle_frames = 0
  145. capture_settings.settings.zero_pad_frame_numbers = 4
  146. # If you wish to override the output framerate you can use these two lines, otherwise the framerate will be derived from the sequence being rendered
  147. capture_settings.settings.use_custom_frame_rate = False
  148. # capture_settings.settings.custom_frame_rate = unreal.FrameRate(24,1)
  149. capture_settings.settings.resolution.res_x = film_resolution
  150. capture_settings.settings.resolution.res_y = film_resolution
  151. capture_settings.settings.enable_texture_streaming = False
  152. capture_settings.settings.cinematic_engine_scalability = True
  153. capture_settings.settings.cinematic_mode = True
  154. capture_settings.settings.allow_movement = False # Requires cinematic_mode = True
  155. capture_settings.settings.allow_turning = False # Requires cinematic_mode = True
  156. capture_settings.settings.show_player = False # Requires cinematic_mode = True
  157. capture_settings.settings.show_hud = False # Requires cinematic_mode = True
  158. capture_settings.use_separate_process = False
  159. capture_settings.close_editor_when_capture_starts = False # Requires use_separate_process = True
  160. capture_settings.additional_command_line_arguments = "-NOSCREENMESSAGES" # Requires use_separate_process = True
  161. capture_settings.inherited_command_line_arguments = "" # Requires use_separate_process = True
  162. # Set all the POD settings on UAutomatedLevelSequenceCapture
  163. capture_settings.use_custom_start_frame = False # If False, the system will automatically calculate the start based on sequence content
  164. capture_settings.use_custom_end_frame = False # If False, the system will automatically calculate the end based on sequence content
  165. capture_settings.custom_start_frame = unreal.FrameNumber(0) # Requires use_custom_start_frame = True
  166. capture_settings.custom_end_frame = unreal.FrameNumber(0) # Requires use_custom_end_frame = True
  167. capture_settings.warm_up_frame_count = 0
  168. capture_settings.delay_before_warm_up = 0
  169. capture_settings.delay_before_shot_warm_up = 0
  170. capture_settings.delay_every_frame = delay_every_frame
  171. capture_settings.write_edit_decision_list = True
  172. # Tell the capture settings which level sequence to render with these settings. The asset does not need to be loaded,
  173. # as we're only capturing the path to it and when the PIE instance is created it will load the specified asset.
  174. # If you only had a reference to the level sequence, you could use "unreal.SoftObjectPath(mysequence.get_path_name())"
  175. capture_settings.level_sequence_asset = unreal.SoftObjectPath(sequence_asset_path)
  176. # To configure the video output we need to tell the capture settings which capture protocol to use. The various supported
  177. # capture protocols can be found by setting the Unreal Content Browser to "Engine C++ Classes" and filtering for "Protocol"
  178. # ie: CompositionGraphCaptureProtocol, ImageSequenceProtocol_PNG, etc. Do note that some of the listed protocols are not intended
  179. # to be used directly.
  180. # Right click on a Protocol and use "Copy Reference" and then remove the extra formatting around it. ie:
  181. # Class'/Script/MovieSceneCapture.ImageSequenceProtocol_PNG' gets transformed into "/Script/MovieSceneCapture.ImageSequenceProtocol_PNG"
  182. capture_settings.set_image_capture_protocol_type(
  183. unreal.load_class(None, "/Script/MovieSceneCapture.ImageSequenceProtocol_PNG"))
  184. # After we have set the capture protocol to a soft class path we can start editing the settings for the instance of the protocol that is internallyc reated.
  185. capture_settings.get_image_capture_protocol().compression_quality = 85
  186. # Finally invoke Sequencer's Render to Movie functionality. This will examine the specified settings object and either construct a new PIE instance to render in,
  187. # or create and launch a new process (optionally shutting down your editor).
  188. unreal.SequencerTools.render_movie(capture_settings, on_finished_callback)
  189. # sequence end call back
  190. def on_render_movie_finished(success):
  191. unreal.log('on_render_movie_finished is called')
  192. if success:
  193. unreal.log('---end LevelSequenceTask')
  194. # check asset is exist
  195. def check_sequence_asset_exist(root_dir):
  196. sequence_path = os.path.join(root_dir, 'Content', 'Render_Sequence.uasset')
  197. if (os.path.exists(sequence_path)):
  198. unreal.log('---sequence is exist os path---')
  199. os.remove(sequence_path)
  200. else:
  201. unreal.log('---sequence is not exist os path---')
  202. if unreal.EditorAssetLibrary.does_asset_exist(sequence_asset_path):
  203. unreal.log('---sequence dose asset exist---')
  204. unreal.EditorAssetLibrary.delete_asset(sequence_asset_path)
  205. else:
  206. unreal.log('---sequence does not asset exist---')
  207. sequence_asset_data = unreal.AssetRegistryHelpers.get_asset_registry().get_asset_by_object_path(sequence_asset_path)
  208. if sequence_asset_data:
  209. unreal.log('---sequence is exist on content---')
  210. sequence_asset = unreal.AssetData.get_asset(sequence_asset_data)
  211. unreal.EditorAssetLibrary.delete_loaded_asset(sequence_asset)
  212. else:
  213. unreal.log('---sequence is not exit on content---')
  214. def main():
  215. # bind event
  216. on_finished_callback = unreal.OnRenderMovieStopped()
  217. on_finished_callback.bind_callable(on_render_movie_finished)
  218. # get json file
  219. root_dir = unreal.SystemLibrary.get_project_directory()
  220. json_path = os.path.join(root_dir, 'RawData', 'camera_data.json')
  221. # get json data
  222. camera_transform_array, camera_rotation_array, camera_name_array, export_path, film_fov, film_resolution, delay_every_frame = read_json_file(
  223. json_path)
  224. # check sequence is asset
  225. check_sequence_asset_exist(root_dir)
  226. # create sequence
  227. create_sequence('Render_Sequence', camera_transform_array, camera_rotation_array, camera_name_array, film_fov)
  228. # run sequence
  229. render_sequence_to_movie(export_path, film_resolution, delay_every_frame, on_finished_callback)

3. 渲染结果

上图即为渲染的多视点图片,每张图片对应的相机姿态也是确定的。

至此,在 UE4 中利用 python 实现自动化渲染的整个过程就结束了。在这里留下一个小问题:

UE4 中的相机姿态数据我们是无法直接使用的,如何将其转化为 NeRF 中的相机姿态数据呢?大家可以自行探索一下,后续文章再对这一部分的内容进行讲解。

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/2023面试高手/article/detail/83065
推荐阅读
相关标签
  

闽ICP备14008679号