Next (#384)
* feat/yoloface (#334) * added yolov8 to face_detector (#323) * added yolov8 to face_detector * added yolov8 to face_detector * Initial cleanup and renaming * Update README * refactored detect_with_yoloface (#329) * refactored detect_with_yoloface * apply review * Change order again * Restore working code * modified code (#330) * refactored detect_with_yoloface * apply review * use temp_frame in detect_with_yoloface * reorder * modified * reorder models * Tiny cleanup --------- Co-authored-by: tamoharu <133945583+tamoharu@users.noreply.github.com> * include audio file functions (#336) * Add testing for audio handlers * Change order * Fix naming * Use correct typing in choices * Update help message for arguments, Notation based wording approach (#347) * Update help message for arguments, Notation based wording approach * Fix installer * Audio functions (#345) * Update ffmpeg.py * Create audio.py * Update ffmpeg.py * Update audio.py * Update audio.py * Update typing.py * Update ffmpeg.py * Update audio.py * Rename Frame to VisionFrame (#346) * Minor tidy up * Introduce audio testing * Add more todo for testing * Add more todo for testing * Fix indent * Enable venv on the fly * Enable venv on the fly * Revert venv on the fly * Revert venv on the fly * Force Gradio to shut up * Force Gradio to shut up * Clear temp before processing * Reduce terminal output * include audio file functions * Enforce output resolution on merge video * Minor cleanups * Add age and gender to face debugger items (#353) * Add age and gender to face debugger items * Rename like suggested in the code review * Fix the output framerate vs. time * Lip Sync (#356) * Cli implementation of wav2lip * - create get_first_item() - remove non gan wav2lip model - implement video memory strategy - implement get_reference_frame() - implement process_image() - rearrange crop_mask_list - implement test_cli * Simplify testing * Rename to lip syncer * Fix testing * Fix testing * Minor cleanup * Cuda 12 installer (#362) * Make cuda nightly (12) the default * Better keep legacy cuda just in case * Use CUDA and ROCM versions * Remove MacOS options from installer (CoreML include in default package) * Add lip-syncer support to source component * Add lip-syncer support to source component * Fix the check in the source component * Add target image check * Introduce more helpers to suite the lip-syncer needs * Downgrade onnxruntime as of buggy 1.17.0 release * Revert "Downgrade onnxruntime as of buggy 1.17.0 release" This reverts commit f4a7ae6824fed87f0be50906bbc7e2d61d00617b. * More testing and add todos * Fix the frame processor API to at least not throw errors * Introduce dict based frame processor inputs (#364) * Introduce dict based frame processor inputs * Forgot to adjust webcam * create path payloads (#365) * create index payload to paths for process_frames * rename to payload_paths * This code now is poetry * Fix the terminal output * Make lip-syncer work in the preview * Remove face debugger test for now * Reoder reference_faces, Fix testing * Use inswapper_128 on buggy onnxruntime 1.17.0 * Undo inswapper_128_fp16 duo broken onnxruntime 1.17.0 * Undo inswapper_128_fp16 duo broken onnxruntime 1.17.0 * Fix lip_syncer occluder & region mask issue * Fix preview once in case there was no output video fps * fix lip_syncer custom fps * remove unused import * Add 68 landmark functions (#367) * Add 68 landmark model * Add landmark to face object * Re-arrange and modify typing * Rename function * Rearrange * Rearrange * ignore type * ignore type * change type * ignore * name * Some cleanup * Some cleanup * Opps, I broke something * Feat/face analyser refactoring (#369) * Restructure face analyser and start TDD * YoloFace and Yunet testing are passing * Remove offset from yoloface detection * Cleanup code * Tiny fix * Fix get_many_faces() * Tiny fix (again) * Use 320x320 fallback for retinaface * Fix merging mashup * Upload wave2lip model * Upload 2dfan2 model and rename internal to face_predictor * Downgrade onnxruntime for most cases * Update for the face debugger to render landmark 68 * Try to make detect_face_landmark_68() and detect_gender_age() more uniform * Enable retinaface testing for 320x320 * Make detect_face_landmark_68() and detect_gender_age() as uniform as … (#370) * Make detect_face_landmark_68() and detect_gender_age() as uniform as possible * Revert landmark scale and translation * Make box-mask for lip-syncer adjustable * Add create_bbox_from_landmark() * Remove currently unused code * Feat/uniface (#375) * add uniface (#373) * Finalize UniFace implementation --------- Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com> * My approach how todo it * edit * edit * replace vertical blur with gaussian * remove region mask * Rebase against next and restore method * Minor improvements * Minor improvements * rename & add forehead padding * Adjust and host uniface model * Use 2dfan4 model * Rename to face landmarker * Feat/replace bbox with bounding box (#380) * Add landmark 68 to 5 convertion * Add landmark 68 to 5 convertion * Keep 5, 5/68 and 68 landmarks * Replace kps with landmark * Replace bbox with bounding box * Reshape face_landmark5_list different * Make yoloface the default * Move convert_face_landmark_68_to_5 to face_helper * Minor spacing issue * Dynamic detector sizes according to model (#382) * Dynamic detector sizes according to model * Dynamic detector sizes according to model * Undo false commited files * Add lib syncer model to the UI * fix halo (#383) * Bump to 2.3.0 * Update README and wording * Update README and wording * Fix spacing * Apply _vision suffix * Apply _vision suffix * Apply _vision suffix * Apply _vision suffix * Apply _vision suffix * Apply _vision suffix * Apply _vision suffix, Move mouth mask to face_masker.py * Apply _vision suffix * Apply _vision suffix * increase forehead padding --------- Co-authored-by: tamoharu <133945583+tamoharu@users.noreply.github.com> Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
This commit is contained in:
Henry Ruhs
@@ -4,7 +4,7 @@ from functools import lru_cache
|
||||
import cv2
|
||||
import numpy
|
||||
|
||||
from facefusion.typing import Bbox, Kps, Frame, Mask, Matrix, Template
|
||||
from facefusion.typing import BoundingBox, FaceLandmark5, FaceLandmark68, VisionFrame, Mask, Matrix, Translation, Template, FaceAnalyserAge, FaceAnalyserGender
|
||||
|
||||
TEMPLATES : Dict[Template, numpy.ndarray[Any, Any]] =\
|
||||
{
|
||||
@@ -43,35 +43,41 @@ TEMPLATES : Dict[Template, numpy.ndarray[Any, Any]] =\
|
||||
}
|
||||
|
||||
|
||||
def warp_face_by_kps(temp_frame : Frame, kps : Kps, template : Template, crop_size : Size) -> Tuple[Frame, Matrix]:
|
||||
def warp_face_by_face_landmark_5(temp_vision_frame : VisionFrame, face_landmark_5 : FaceLandmark5, template : Template, crop_size : Size) -> Tuple[VisionFrame, Matrix]:
|
||||
normed_template = TEMPLATES.get(template) * crop_size
|
||||
affine_matrix = cv2.estimateAffinePartial2D(kps, normed_template, method = cv2.RANSAC, ransacReprojThreshold = 100)[0]
|
||||
crop_frame = cv2.warpAffine(temp_frame, affine_matrix, crop_size, borderMode = cv2.BORDER_REPLICATE, flags = cv2.INTER_AREA)
|
||||
return crop_frame, affine_matrix
|
||||
affine_matrix = cv2.estimateAffinePartial2D(face_landmark_5, normed_template, method = cv2.RANSAC, ransacReprojThreshold = 100)[0]
|
||||
crop_vision_frame = cv2.warpAffine(temp_vision_frame, affine_matrix, crop_size, borderMode = cv2.BORDER_REPLICATE, flags = cv2.INTER_AREA)
|
||||
return crop_vision_frame, affine_matrix
|
||||
|
||||
|
||||
def warp_face_by_bbox(temp_frame : Frame, bbox : Bbox, crop_size : Size) -> Tuple[Frame, Matrix]:
|
||||
source_kps = numpy.array([[ bbox[0], bbox[1] ], [bbox[2], bbox[1] ], [bbox[0], bbox[3] ]], dtype = numpy.float32)
|
||||
target_kps = numpy.array([[ 0, 0 ], [ crop_size[0], 0 ], [ 0, crop_size[1] ]], dtype = numpy.float32)
|
||||
affine_matrix = cv2.getAffineTransform(source_kps, target_kps)
|
||||
if bbox[2] - bbox[0] > crop_size[0] or bbox[3] - bbox[1] > crop_size[1]:
|
||||
def warp_face_by_bounding_box(temp_vision_frame : VisionFrame, bounding_box : BoundingBox, crop_size : Size) -> Tuple[VisionFrame, Matrix]:
|
||||
source_points = numpy.array([[bounding_box[0], bounding_box[1]], [bounding_box[2], bounding_box[1]], [bounding_box[0], bounding_box[3]]], dtype = numpy.float32)
|
||||
target_points = numpy.array([[ 0, 0 ], [ crop_size[0], 0 ], [ 0, crop_size[1] ]], dtype = numpy.float32)
|
||||
affine_matrix = cv2.getAffineTransform(source_points, target_points)
|
||||
if bounding_box[2] - bounding_box[0] > crop_size[0] or bounding_box[3] - bounding_box[1] > crop_size[1]:
|
||||
interpolation_method = cv2.INTER_AREA
|
||||
else:
|
||||
interpolation_method = cv2.INTER_LINEAR
|
||||
crop_frame = cv2.warpAffine(temp_frame, affine_matrix, crop_size, flags = interpolation_method)
|
||||
return crop_frame, affine_matrix
|
||||
crop_vision_frame = cv2.warpAffine(temp_vision_frame, affine_matrix, crop_size, flags = interpolation_method)
|
||||
return crop_vision_frame, affine_matrix
|
||||
|
||||
|
||||
def paste_back(temp_frame : Frame, crop_frame : Frame, crop_mask : Mask, affine_matrix : Matrix) -> Frame:
|
||||
def warp_face_by_translation(temp_vision_frame : VisionFrame, translation : Translation, scale : float, crop_size : Size) -> Tuple[VisionFrame, Matrix]:
|
||||
affine_matrix = numpy.array([[ scale, 0, translation[0] ], [ 0, scale, translation[1] ]])
|
||||
crop_vision_frame = cv2.warpAffine(temp_vision_frame, affine_matrix, crop_size)
|
||||
return crop_vision_frame, affine_matrix
|
||||
|
||||
|
||||
def paste_back(temp_vision_frame : VisionFrame, crop_vision_frame : VisionFrame, crop_mask : Mask, affine_matrix : Matrix) -> VisionFrame:
|
||||
inverse_matrix = cv2.invertAffineTransform(affine_matrix)
|
||||
temp_frame_size = temp_frame.shape[:2][::-1]
|
||||
inverse_crop_mask = cv2.warpAffine(crop_mask, inverse_matrix, temp_frame_size).clip(0, 1)
|
||||
inverse_crop_frame = cv2.warpAffine(crop_frame, inverse_matrix, temp_frame_size, borderMode = cv2.BORDER_REPLICATE)
|
||||
paste_frame = temp_frame.copy()
|
||||
paste_frame[:, :, 0] = inverse_crop_mask * inverse_crop_frame[:, :, 0] + (1 - inverse_crop_mask) * temp_frame[:, :, 0]
|
||||
paste_frame[:, :, 1] = inverse_crop_mask * inverse_crop_frame[:, :, 1] + (1 - inverse_crop_mask) * temp_frame[:, :, 1]
|
||||
paste_frame[:, :, 2] = inverse_crop_mask * inverse_crop_frame[:, :, 2] + (1 - inverse_crop_mask) * temp_frame[:, :, 2]
|
||||
return paste_frame
|
||||
temp_size = temp_vision_frame.shape[:2][::-1]
|
||||
inverse_mask = cv2.warpAffine(crop_mask, inverse_matrix, temp_size).clip(0, 1)
|
||||
inverse_vision_frame = cv2.warpAffine(crop_vision_frame, inverse_matrix, temp_size, borderMode = cv2.BORDER_REPLICATE)
|
||||
paste_vision_frame = temp_vision_frame.copy()
|
||||
paste_vision_frame[:, :, 0] = inverse_mask * inverse_vision_frame[:, :, 0] + (1 - inverse_mask) * temp_vision_frame[:, :, 0]
|
||||
paste_vision_frame[:, :, 1] = inverse_mask * inverse_vision_frame[:, :, 1] + (1 - inverse_mask) * temp_vision_frame[:, :, 1]
|
||||
paste_vision_frame[:, :, 2] = inverse_mask * inverse_vision_frame[:, :, 2] + (1 - inverse_mask) * temp_vision_frame[:, :, 2]
|
||||
return paste_vision_frame
|
||||
|
||||
|
||||
@lru_cache(maxsize = None)
|
||||
@@ -83,31 +89,48 @@ def create_static_anchors(feature_stride : int, anchor_total : int, stride_heigh
|
||||
return anchors
|
||||
|
||||
|
||||
def distance_to_bbox(points : numpy.ndarray[Any, Any], distance : numpy.ndarray[Any, Any]) -> Bbox:
|
||||
def create_bounding_box_from_landmark(face_landmark_68 : FaceLandmark68) -> BoundingBox:
|
||||
min_x, min_y = numpy.min(face_landmark_68, axis = 0)
|
||||
max_x, max_y = numpy.max(face_landmark_68, axis = 0)
|
||||
bounding_box = numpy.array([ min_x, min_y, max_x, max_y ]).astype(numpy.int16)
|
||||
return bounding_box
|
||||
|
||||
|
||||
def distance_to_bounding_box(points : numpy.ndarray[Any, Any], distance : numpy.ndarray[Any, Any]) -> BoundingBox:
|
||||
x1 = points[:, 0] - distance[:, 0]
|
||||
y1 = points[:, 1] - distance[:, 1]
|
||||
x2 = points[:, 0] + distance[:, 2]
|
||||
y2 = points[:, 1] + distance[:, 3]
|
||||
bbox = numpy.column_stack([ x1, y1, x2, y2 ])
|
||||
return bbox
|
||||
bounding_box = numpy.column_stack([ x1, y1, x2, y2 ])
|
||||
return bounding_box
|
||||
|
||||
|
||||
def distance_to_kps(points : numpy.ndarray[Any, Any], distance : numpy.ndarray[Any, Any]) -> Kps:
|
||||
def distance_to_face_landmark_5(points : numpy.ndarray[Any, Any], distance : numpy.ndarray[Any, Any]) -> FaceLandmark5:
|
||||
x = points[:, 0::2] + distance[:, 0::2]
|
||||
y = points[:, 1::2] + distance[:, 1::2]
|
||||
kps = numpy.stack((x, y), axis = -1)
|
||||
return kps
|
||||
face_landmark_5 = numpy.stack((x, y), axis = -1)
|
||||
return face_landmark_5
|
||||
|
||||
|
||||
def apply_nms(bbox_list : List[Bbox], iou_threshold : float) -> List[int]:
|
||||
def convert_face_landmark_68_to_5(landmark_68 : FaceLandmark68) -> FaceLandmark5:
|
||||
left_eye = numpy.mean(landmark_68[36:42], axis = 0)
|
||||
right_eye = numpy.mean(landmark_68[42:48], axis = 0)
|
||||
nose = landmark_68[30]
|
||||
left_mouth_end = landmark_68[48]
|
||||
right_mouth_end = landmark_68[54]
|
||||
face_landmark_5 = numpy.array([ left_eye, right_eye, nose, left_mouth_end, right_mouth_end ])
|
||||
return face_landmark_5
|
||||
|
||||
|
||||
def apply_nms(bounding_box_list : List[BoundingBox], iou_threshold : float) -> List[int]:
|
||||
keep_indices = []
|
||||
dimension_list = numpy.reshape(bbox_list, (-1, 4))
|
||||
dimension_list = numpy.reshape(bounding_box_list, (-1, 4))
|
||||
x1 = dimension_list[:, 0]
|
||||
y1 = dimension_list[:, 1]
|
||||
x2 = dimension_list[:, 2]
|
||||
y2 = dimension_list[:, 3]
|
||||
areas = (x2 - x1 + 1) * (y2 - y1 + 1)
|
||||
indices = numpy.arange(len(bbox_list))
|
||||
indices = numpy.arange(len(bounding_box_list))
|
||||
while indices.size > 0:
|
||||
index = indices[0]
|
||||
remain_indices = indices[1:]
|
||||
@@ -121,3 +144,19 @@ def apply_nms(bbox_list : List[Bbox], iou_threshold : float) -> List[int]:
|
||||
iou = width * height / (areas[index] + areas[remain_indices] - width * height)
|
||||
indices = indices[numpy.where(iou <= iou_threshold)[0] + 1]
|
||||
return keep_indices
|
||||
|
||||
|
||||
def categorize_age(age : int) -> FaceAnalyserAge:
|
||||
if age < 13:
|
||||
return 'child'
|
||||
elif age < 19:
|
||||
return 'teen'
|
||||
elif age < 60:
|
||||
return 'adult'
|
||||
return 'senior'
|
||||
|
||||
|
||||
def categorize_gender(gender : int) -> FaceAnalyserGender:
|
||||
if gender == 0:
|
||||
return 'female'
|
||||
return 'male'
|
||||
|
||||
Reference in New Issue
Block a user