jones/facefusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

3.0.0 (#748)

Browse Source 
* Cleanup after age modifier PR

* Cleanup after age modifier PR

* Use OpenVino 2024.2.0 for installer

* Prepare 3.0.0 for installer

* Fix benchmark suite, Introduce sync_item() for state manager

* Fix lint

* Render slide preview also in lower res

* Lower thread and queue count to avoid false usage

* Fix spacing

* Feat/jobs UI (#627)

* Jobs UI part1

* Change naming

* Jobs UI part2

* Jobs UI part3

* Jobs UI part4

* Jobs UI part4

* Jobs UI part5

* Jobs UI part6

* Jobs UI part7

* Jobs UI part8

* Jobs UI part9

* Jobs UI part10

* Jobs UI part11

* Jobs UI part12

* Fix rebase

* Jobs UI part13

* Jobs UI part14

* Jobs UI part15

* changes (#626)

* Remove useless ui registration

* Remove useless ui registration

* move job_list.py
replace [0] with get_first()

* optimize imports

* fix date None problem
add test job list

* Jobs UI part16

* Jobs UI part17

* Jobs UI part18

* Jobs UI part19

* Jobs UI part20

* Jobs UI part21

* Jobs UI part22

* move job_list_options

* Add label to job status checkbox group

* changes

* changes

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Update some dependencies

* UI helper to convert 'none'

* validate job (#628)

* changes

* changes

* add test

* changes

* changes

* Minor adjustments

* Replace is_json with is_file

* Handle empty and invalid json in job_list

* Handle empty and invalid json in job_list

* Handle empty and invalid json in job_list

* Work on the job manager UI

* Cosmetic changes on common helper

* Just make it work for now

* Just make it work for now

* Just make it work for now

* Streamline the step index lookups

* Hide footer

* Simplify instant runner

* Simplify instant runner UI and job manager UI

* Fix empty step choices

* Fix empty step choices

* Fix none values in UI

* Rework on benchmark (add warmup) and job list

* Improve ValueAndUnit

* Add step 1 of x output

* Cosmetic changes on the UI

* Fix invalid job file names

* Update preview

* Introducing has_step() and sorting out insert behaviour

* Introducing has_step() and sorting out insert behaviour

* Add [ none ] to some job id dropdowns

* Make updated dropdown values kinda perfect

* Make updated dropdown values kinda perfect

* Fix testing

* Minor improvement on UI

* Fix false config lookup

* Remove TensorRT as our models are not made for it

* Feat/cli commands second try rev2 (#640)

* Refactor CLI to commands

* Refactor CLI to commands part2

* Refactor CLI to commands part3

* Refactor CLI to commands part4

* Rename everything to facefusion.py

* Refactor CLI to commands part5

* Refactor CLI to commands part6

* Adjust testing

* Fix lint

* Fix lint

* Fix lint

* Refactor CLI to commands part7

* Extend State typing

* Fix false config lookup, adjust logical orders

* Move away from passing program part1

* Move away from passing program part2

* Move away from passing program part3

* Fix lint

* Move away from passing program part4

* ui-args update

* ui-args update

* ui-args update

* temporary type fix

* Move away from passing program part5

* remove unused

* creates args.py

* Move away from passing program part6

* Move away from passing program part7

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Minor optimizations

* Update commands in README

* Fix job-retry command

* Fix multi runs via UI

* add more job keys

* Cleanup codebase

* One method to create inference session (#641)

* One method to create inference session

* Remove warnings, as there are none

* Remember job id during processing

* Fix face masker config block

* Change wording

* Prevent age modifier from using CoreML

* add expression restorer (#642)

* add expression restorer

* fix import

* fix lint

* changes

* changes

* changes

* Host the final model for expression restorer

* Insert step on the given index

* UI workover (#644)

* UI workover part1

* Introduce ComponentOptions

* Only set Media components to None when visibility changes

* Clear static faces and reference faces between step processing

* Minor changes

* Minor changes

* Fix testing

* Enable test_sanitize_path_for_windows (#646)

* Dynamic download during job processing (#647)

* Fix face masker UI

* Rename run-headless to headless-run

* Feat/split frame processor UI (#649)

* Split frame processor UI

* Split frame processor UI part3, Refactor get_model_initializer

* Split frame processor UI part4

* Feat/rename frame processors (#651)

* Rename frame processors

* Rename frame processors part2

* Fix imports

 Conflicts:
	facefusion/uis/layouts/benchmark.py
	facefusion/uis/layouts/default.py

* Fix imports

* Cosmetic changes

* Fix multi threading for ROCm

* Change temp frames pattern

* Adjust terminal help

* remove expression restorer (#653)

* Expression restorer as processor (#655)

* add expression restorer

* changes

* Cleanup code

* Add TensorRT support back

* Add TensorRT support back

* Add TensorRT support back

* changes (#656)

* Change minor wording

* Fix face enhancer slider

* Add more typing

* Fix expression-restorer when using trim (#659)

* changes

* changes

* Rework/model and inference pool part2 (#660)

* Rework on model and inference pool

* Introduce inference sources and pools part1

* Introduce inference sources and pools part2

* Introduce inference sources and pools part3

* Introduce inference sources and pools part4

* Introduce inference sources and pools part5

* Introduce inference sources and pools part6

* Introduce inference sources and pools part6

* Introduce inference sources and pools part6

* Introduce inference sources and pools part7

* Introduce inference sources and pools part7

* Introduce inference sources and pools part8

* Introduce inference sources and pools part9

* Introduce inference sources and pools part10

* Introduce inference sources and pools part11

* Introduce inference sources and pools part11

* Introduce inference sources and pools part11

* Introduce inference sources and pools part12

* Reorganize the face masker UI

* Fix trim in UI

* Feat/hashed sources (#668)

* Introduce source helper

* Remove post_check() and just use process_manager

* Remove post_check() part2

* Add hash based downloads

* Add hash based downloads part2

* Add hash based downloads part3

* Add hash based downloads part4

* Add hash based downloads part5

* Add hash based downloads part6

* Add hash based downloads part7

* Add hash based downloads part7

* Add hash based downloads part8

* Remove print

* Prepare 3.0.0 release

* Fix UI

* Release the check when really done

* Update inputs for live portrait

* Update to 3.0.0 releases, extend download postfix

* Move files to the right place

* Logging for the hash and source validation

* Changing logic to handle corrupt sources

* Fix typo

* Use names over get_inputs(), Remove set_options() call

* Age modifier now works for CoreML too

* Update age_modifier.py

* Add video encoder h264_videotoolbox and hevc_videotoolbox

* Face editor add eye gaze & remove open factor sliders (#670)

* changes

* add eye gaze

* changes

* cleanup

* add eyebrow control

* changes

* changes

* Feat/terminal UI (#671)

* Introduce terminal to the UI

* Introduce terminal to the UI part2

* Introduce terminal to the UI part2

* Introduce terminal to the UI part2

* Calc range step to avoid weird values

* Use Sequence for ranges

* Use Sequence for ranges

* changes (#673)

* Use Sequence for ranges

* Finalize terminal UI

* Finalize terminal UI

* Webcam cosmetics, Fix normalize fps to accept int

* Cosmetic changes

* Finalize terminal UI

* Rename leftover typings

* Fix wording

* Fix rounding in metavar

* Fix rounding in metavar

* Rename to face classifier

* Face editor lip moves (#677)

* changes

* changes

* changes

* Fix rounding in metavar

* Rename to face classifier

* changes

* changes

* update naming

---------

Co-authored-by: henryruhs <info@henryruhs.com>

* Fix wording

* Feat/many landmarker + face analyser breakdown (#678)

* Basic multi landmarker integration

* Simplify some method names

* Break into face_detector and face_landmarker

* Fix cosmetics

* Fix testing

* Break into face_attributor and face_recognizer

* Clear them all

* Clear them all

* Rename to face classifier

* Rename to face classifier

* Fix testing

* Fix stuff

* Add face landmarker model to UI

* Add face landmarker model to UI part2

* Split the config

* Split the UI

* Improvement from code review

* Improvement from code review

* Validate args also for sub parsers

* Remove clear of processors in process step

* Allow finder control for the face editor

* Fix lint

* Improve testing performance

* Remove unused file, Clear processors from the UI before job runs

* Update the installer

* Uniform set handler for swapper and detector in the UI

* Fix example urls

* Feat/inference manager (#684)

* Introduce inference manager

* Migrate all to inference manager

* clean ini

* Introduce app context based inference pools

* Fix lint

* Fix typing

* Adjust layout

* Less border radius

* Rename app context names

* Fix/live portrait directml (#691)

* changes (#690)

* Adjust naming

* Use our assets release

* Adjust naming

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Add caches to gitignore

* Update dependencies and drop CUDA 11.8 support (#693)

* Update dependencies and drop CUDA 11.8 support

* Play save and keep numpy 1.x.x

* Improve TensorRT optimization

* changes

* changes

* changes

* changes

* changes

* changes

* changes

* changes

* changes

* Reuse inference sessions (#696)

* Fix force-download command

* Refactor processors to forward() (#698)

* Install tensorrt when selecting cuda

* Minor changes

* Use latest numpy

* Fix limit system memory

* Implement forward() for every inference (#699)

* Implement forward() for every inference

* Implement forward() for every inference

* Implement forward() for every inference

* Implement forward() for every inference

* changes

* changes

* changes

* changes

* Feat/fairface (#710)

* Replace gender_age model with fair face (#709)

* changes

* changes

* changes

* age dropdown to range-slider

* Cleanup code

* Cleanup code

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Extend installer to set library paths for cuda and tensorrt (#707)

* Extend installer to set library paths for cuda and tensorrt

* Add refresh of conda env

* Remove invalid commands

* Set the conda env according to operating system

* Update for ROCm 6.2

* fix installer

* Aktualisieren von installer.py

* Add missing face selector keys

* Try to keep original LD_LIBRARY_PATH

* windows support installer

* Final touch to the installer

* Remove spaces

* Simplidy collect_model_downloads()

* Fix force download for once and forever

* Housekeeping (#715)

* changes

* changes

* changes

* Fix performance part1

* Fix mixed states (#689)

* Fix mixed states

* Add missing sync for job args

* Move UnionStateXXX to base typing

* Undo

* Remove UnionStateXXX

* Fix app context performance lookup (#717)

* Restore performance for inswapper

* Mover upper() to the logger

* Undo debugging

* Move TensorRT installation to docs

* Sort out log level typing, Add log level UI dropdown (#719)

* Fix inference pool part1

* Validate conda library paths existence

* Default face selector order to large-small

* Fix inference pool context according to execution provider (#720)

* Fix app context under Windows

* CUDA and TensorRT update for the installer

* Remove concept of static processor modules

* Revert false commit

* Change event order makes a difference

* Fix multi model context in inference pool (#721)

* Fix multi model context in inference pool

* Fix multi model context in inference pool part2

* Use latest gradio to avoid fastapi bug

* Rework on the Windows Installer

* Use embedding converter (#724)

* changes (#723)

* Upload models to official assets repo

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Rework on the Windows Installer part2

* Resolve subprocess calls (#726)

* Experiment

* Resolve subprocess calls to cover edge cases like broken PATH

* Adjust wording

* Simplify code

* Rework on the Windows Installer part3

* Rework on the Windows Installer part4

* Numpy fix for older onnxruntime

* changes (#729)

* Add space

* Add MacOS installer

* Use favicon

* Fix disabled logger

* Layout polishing (#731)

* Update dependencies, Adjust many face landmarker logic

* Cosmetics changes

* Should be button

* Introduce randomized action button

* Fix update of lip syncer and expression restorer

* Stop sharing inference session this prevents flushing VRAM

* Fix test

* Fix urls

* Prepare release

* Vanish inquirer

* Sticky preview does not work on portrait images

* Sticky preview only for landscape images and videos

* remove gradio tunnel env

* Change wording and deeplinks

* increase peppa landmark score offset

* Change wording

* Graceful exit install.py

* Just adding a required

* Cannot use the exit_helper

* Rename our model

* Change color of face-landmark-68/5

* Limit liveportrait (#739)

* changes

* changes

* changes

* Cleanup

* Cleanup

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* limit expression restorer

* change expression restorer 0-100 range

* Use 256x icon

* changes

* changes

* changes

* changes

* Limit face editor rotation (#745)

* changes (#743)

* Finish euler methods

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Use different coveralls badge

* Move about wording

* Shorten scope in the logger

* changes

* changes

* Shorten scope in the logger

* fix typo

* Simplify the arcface converter names

* Update preview

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
This commit is contained in:
Henry Ruhs
2024-09-20 17:27:50 +02:00
committed by GitHub
parent 57016d7c77
commit 13761af044
171 changed files with 11598 additions and 5115 deletions
Download Patch File Download Diff File Expand all files Collapse all files

125
facefusion/face_helper.py
View File

@@ -1,10 +1,11 @@
from typing import Any, Tuple, List
from cv2.typing import Size
from functools import lru_cache
from typing import List, Sequence, Tuple
import cv2
import numpy
from cv2.typing import Size
from facefusion.typing import BoundingBox, FaceLandmark5, FaceLandmark68, VisionFrame, Mask, Matrix, Translation, WarpTemplate, WarpTemplateSet, FaceAnalyserAge, FaceAnalyserGender
from facefusion.typing import Anchors, Angle, BoundingBox, Distance, FaceDetectorModel, FaceLandmark5, FaceLandmark68, Mask, Matrix, Points, Scale, Score, Translation, VisionFrame, WarpTemplate, WarpTemplateSet
WARP_TEMPLATES : WarpTemplateSet =\
{
@@ -86,7 +87,7 @@ def paste_back(temp_vision_frame : VisionFrame, crop_vision_frame : VisionFrame,
@lru_cache(maxsize = None)
def create_static_anchors(feature_stride : int, anchor_total : int, stride_height : int, stride_width : int) -> numpy.ndarray[Any, Any]:
def create_static_anchors(feature_stride : int, anchor_total : int, stride_height : int, stride_width : int) -> Anchors:
y, x = numpy.mgrid[:stride_height, :stride_width][::-1]
anchors = numpy.stack((y, x), axis = -1)
anchors = (anchors * feature_stride).reshape((-1, 2))
@@ -94,14 +95,50 @@ def create_static_anchors(feature_stride : int, anchor_total : int, stride_heigh
return anchors
def create_bounding_box_from_face_landmark_68(face_landmark_68 : FaceLandmark68) -> BoundingBox:
def create_rotated_matrix_and_size(angle : Angle, size : Size) -> Tuple[Matrix, Size]:
rotated_matrix = cv2.getRotationMatrix2D((size[0] / 2, size[1] / 2), angle, 1)
rotated_size = numpy.dot(numpy.abs(rotated_matrix[:, :2]), size)
rotated_matrix[:, -1] += (rotated_size - size) * 0.5 #type:ignore[misc]
rotated_size = int(rotated_size[0]), int(rotated_size[1])
return rotated_matrix, rotated_size
def create_bounding_box(face_landmark_68 : FaceLandmark68) -> BoundingBox:
min_x, min_y = numpy.min(face_landmark_68, axis = 0)
max_x, max_y = numpy.max(face_landmark_68, axis = 0)
bounding_box = numpy.array([ min_x, min_y, max_x, max_y ]).astype(numpy.int16)
bounding_box = normalize_bounding_box(numpy.array([ min_x, min_y, max_x, max_y ]))
return bounding_box
def distance_to_bounding_box(points : numpy.ndarray[Any, Any], distance : numpy.ndarray[Any, Any]) -> BoundingBox:
def normalize_bounding_box(bounding_box : BoundingBox) -> BoundingBox:
x1, y1, x2, y2 = bounding_box
x1, x2 = sorted([ x1, x2 ])
y1, y2 = sorted([ y1, y2 ])
return numpy.array([ x1, y1, x2, y2 ])
def transform_points(points : Points, matrix : Matrix) -> Points:
points = points.reshape(-1, 1, 2)
points = cv2.transform(points, matrix) #type:ignore[assignment]
points = points.reshape(-1, 2)
return points
def transform_bounding_box(bounding_box : BoundingBox, matrix : Matrix) -> BoundingBox:
points = numpy.array(
[
[ bounding_box[0], bounding_box[1] ],
[ bounding_box[2], bounding_box[1] ],
[ bounding_box[2], bounding_box[3] ],
[ bounding_box[0], bounding_box[3] ]
])
points = transform_points(points, matrix)
x1, y1 = numpy.min(points, axis = 0)
x2, y2 = numpy.max(points, axis = 0)
return normalize_bounding_box(numpy.array([ x1, y1, x2, y2 ]))
def distance_to_bounding_box(points : Points, distance : Distance) -> BoundingBox:
x1 = points[:, 0] - distance[:, 0]
y1 = points[:, 1] - distance[:, 1]
x2 = points[:, 0] + distance[:, 2]
@@ -110,14 +147,21 @@ def distance_to_bounding_box(points : numpy.ndarray[Any, Any], distance : numpy.
return bounding_box
def distance_to_face_landmark_5(points : numpy.ndarray[Any, Any], distance : numpy.ndarray[Any, Any]) -> FaceLandmark5:
def distance_to_face_landmark_5(points : Points, distance : Distance) -> FaceLandmark5:
x = points[:, 0::2] + distance[:, 0::2]
y = points[:, 1::2] + distance[:, 1::2]
face_landmark_5 = numpy.stack((x, y), axis = -1)
return face_landmark_5
def convert_face_landmark_68_to_5(face_landmark_68 : FaceLandmark68) -> FaceLandmark5:
def scale_face_landmark_5(face_landmark_5 : FaceLandmark5, scale : Scale) -> FaceLandmark5:
face_landmark_5_scale = face_landmark_5 - face_landmark_5[2]
face_landmark_5_scale *= scale
face_landmark_5_scale += face_landmark_5[2]
return face_landmark_5_scale
def convert_to_face_landmark_5(face_landmark_68 : FaceLandmark68) -> FaceLandmark5:
face_landmark_5 = numpy.array(
[
numpy.mean(face_landmark_68[36:42], axis = 0),
@@ -129,41 +173,38 @@ def convert_face_landmark_68_to_5(face_landmark_68 : FaceLandmark68) -> FaceLand
return face_landmark_5
def apply_nms(bounding_box_list : List[BoundingBox], iou_threshold : float) -> List[int]:
keep_indices = []
dimension_list = numpy.reshape(bounding_box_list, (-1, 4))
x1 = dimension_list[:, 0]
y1 = dimension_list[:, 1]
x2 = dimension_list[:, 2]
y2 = dimension_list[:, 3]
areas = (x2 - x1 + 1) * (y2 - y1 + 1)
indices = numpy.arange(len(bounding_box_list))
while indices.size > 0:
index = indices[0]
remain_indices = indices[1:]
keep_indices.append(index)
xx1 = numpy.maximum(x1[index], x1[remain_indices])
yy1 = numpy.maximum(y1[index], y1[remain_indices])
xx2 = numpy.minimum(x2[index], x2[remain_indices])
yy2 = numpy.minimum(y2[index], y2[remain_indices])
width = numpy.maximum(0, xx2 - xx1 + 1)
height = numpy.maximum(0, yy2 - yy1 + 1)
iou = width * height / (areas[index] + areas[remain_indices] - width * height)
indices = indices[numpy.where(iou <= iou_threshold)[0] + 1]
def estimate_face_angle(face_landmark_68 : FaceLandmark68) -> Angle:
x1, y1 = face_landmark_68[0]
x2, y2 = face_landmark_68[16]
theta = numpy.arctan2(y2 - y1, x2 - x1)
theta = numpy.degrees(theta) % 360
angles = numpy.linspace(0, 360, 5)
index = numpy.argmin(numpy.abs(angles - theta))
face_angle = int(angles[index] % 360)
return face_angle
def apply_nms(bounding_boxes : List[BoundingBox], face_scores : List[Score], score_threshold : float, nms_threshold : float) -> Sequence[int]:
normed_bounding_boxes = [ (x1, y1, x2 - x1, y2 - y1) for (x1, y1, x2, y2) in bounding_boxes ]
keep_indices = cv2.dnn.NMSBoxes(normed_bounding_boxes, face_scores, score_threshold = score_threshold, nms_threshold = nms_threshold)
return keep_indices
def categorize_age(age : int) -> FaceAnalyserAge:
if age < 13:
return 'child'
elif age < 19:
return 'teen'
elif age < 60:
return 'adult'
return 'senior'
def get_nms_threshold(face_detector_model : FaceDetectorModel, face_detector_angles : List[Angle]) -> float:
if face_detector_model == 'many':
return 0.1
if len(face_detector_angles) == 2:
return 0.3
if len(face_detector_angles) == 3:
return 0.2
if len(face_detector_angles) == 4:
return 0.1
return 0.4
def categorize_gender(gender : int) -> FaceAnalyserGender:
if gender == 0:
return 'female'
return 'male'
def merge_matrix(matrices : List[Matrix]) -> Matrix:
merged_matrix = numpy.vstack([ matrices[0], [ 0, 0, 1 ] ])
for matrix in matrices[1:]:
matrix = numpy.vstack([ matrix, [ 0, 0, 1 ] ])
merged_matrix = numpy.dot(merged_matrix, matrix)
return merged_matrix[:2, :]