* Cleanup after age modifier PR

* Cleanup after age modifier PR

* Use OpenVino 2024.2.0 for installer

* Prepare 3.0.0 for installer

* Fix benchmark suite, Introduce sync_item() for state manager

* Fix lint

* Render slide preview also in lower res

* Lower thread and queue count to avoid false usage

* Fix spacing

* Feat/jobs UI (#627)

* Jobs UI part1

* Change naming

* Jobs UI part2

* Jobs UI part3

* Jobs UI part4

* Jobs UI part4

* Jobs UI part5

* Jobs UI part6

* Jobs UI part7

* Jobs UI part8

* Jobs UI part9

* Jobs UI part10

* Jobs UI part11

* Jobs UI part12

* Fix rebase

* Jobs UI part13

* Jobs UI part14

* Jobs UI part15

* changes (#626)

* Remove useless ui registration

* Remove useless ui registration

* move job_list.py
replace [0] with get_first()

* optimize imports

* fix date None problem
add test job list

* Jobs UI part16

* Jobs UI part17

* Jobs UI part18

* Jobs UI part19

* Jobs UI part20

* Jobs UI part21

* Jobs UI part22

* move job_list_options

* Add label to job status checkbox group

* changes

* changes

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Update some dependencies

* UI helper to convert 'none'

* validate job (#628)

* changes

* changes

* add test

* changes

* changes

* Minor adjustments

* Replace is_json with is_file

* Handle empty and invalid json in job_list

* Handle empty and invalid json in job_list

* Handle empty and invalid json in job_list

* Work on the job manager UI

* Cosmetic changes on common helper

* Just make it work for now

* Just make it work for now

* Just make it work for now

* Streamline the step index lookups

* Hide footer

* Simplify instant runner

* Simplify instant runner UI and job manager UI

* Fix empty step choices

* Fix empty step choices

* Fix none values in UI

* Rework on benchmark (add warmup) and job list

* Improve ValueAndUnit

* Add step 1 of x output

* Cosmetic changes on the UI

* Fix invalid job file names

* Update preview

* Introducing has_step() and sorting out insert behaviour

* Introducing has_step() and sorting out insert behaviour

* Add [ none ] to some job id dropdowns

* Make updated dropdown values kinda perfect

* Make updated dropdown values kinda perfect

* Fix testing

* Minor improvement on UI

* Fix false config lookup

* Remove TensorRT as our models are not made for it

* Feat/cli commands second try rev2 (#640)

* Refactor CLI to commands

* Refactor CLI to commands part2

* Refactor CLI to commands part3

* Refactor CLI to commands part4

* Rename everything to facefusion.py

* Refactor CLI to commands part5

* Refactor CLI to commands part6

* Adjust testing

* Fix lint

* Fix lint

* Fix lint

* Refactor CLI to commands part7

* Extend State typing

* Fix false config lookup, adjust logical orders

* Move away from passing program part1

* Move away from passing program part2

* Move away from passing program part3

* Fix lint

* Move away from passing program part4

* ui-args update

* ui-args update

* ui-args update

* temporary type fix

* Move away from passing program part5

* remove unused

* creates args.py

* Move away from passing program part6

* Move away from passing program part7

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Minor optimizations

* Update commands in README

* Fix job-retry command

* Fix multi runs via UI

* add more job keys

* Cleanup codebase

* One method to create inference session (#641)

* One method to create inference session

* Remove warnings, as there are none

* Remember job id during processing

* Fix face masker config block

* Change wording

* Prevent age modifier from using CoreML

* add expression restorer (#642)

* add expression restorer

* fix import

* fix lint

* changes

* changes

* changes

* Host the final model for expression restorer

* Insert step on the given index

* UI workover (#644)

* UI workover part1

* Introduce ComponentOptions

* Only set Media components to None when visibility changes

* Clear static faces and reference faces between step processing

* Minor changes

* Minor changes

* Fix testing

* Enable test_sanitize_path_for_windows (#646)

* Dynamic download during job processing (#647)

* Fix face masker UI

* Rename run-headless to headless-run

* Feat/split frame processor UI (#649)

* Split frame processor UI

* Split frame processor UI part3, Refactor get_model_initializer

* Split frame processor UI part4

* Feat/rename frame processors (#651)

* Rename frame processors

* Rename frame processors part2

* Fix imports

 Conflicts:
	facefusion/uis/layouts/benchmark.py
	facefusion/uis/layouts/default.py

* Fix imports

* Cosmetic changes

* Fix multi threading for ROCm

* Change temp frames pattern

* Adjust terminal help

* remove expression restorer (#653)

* Expression restorer as processor (#655)

* add expression restorer

* changes

* Cleanup code

* Add TensorRT support back

* Add TensorRT support back

* Add TensorRT support back

* changes (#656)

* Change minor wording

* Fix face enhancer slider

* Add more typing

* Fix expression-restorer when using trim (#659)

* changes

* changes

* Rework/model and inference pool part2 (#660)

* Rework on model and inference pool

* Introduce inference sources and pools part1

* Introduce inference sources and pools part2

* Introduce inference sources and pools part3

* Introduce inference sources and pools part4

* Introduce inference sources and pools part5

* Introduce inference sources and pools part6

* Introduce inference sources and pools part6

* Introduce inference sources and pools part6

* Introduce inference sources and pools part7

* Introduce inference sources and pools part7

* Introduce inference sources and pools part8

* Introduce inference sources and pools part9

* Introduce inference sources and pools part10

* Introduce inference sources and pools part11

* Introduce inference sources and pools part11

* Introduce inference sources and pools part11

* Introduce inference sources and pools part12

* Reorganize the face masker UI

* Fix trim in UI

* Feat/hashed sources (#668)

* Introduce source helper

* Remove post_check() and just use process_manager

* Remove post_check() part2

* Add hash based downloads

* Add hash based downloads part2

* Add hash based downloads part3

* Add hash based downloads part4

* Add hash based downloads part5

* Add hash based downloads part6

* Add hash based downloads part7

* Add hash based downloads part7

* Add hash based downloads part8

* Remove print

* Prepare 3.0.0 release

* Fix UI

* Release the check when really done

* Update inputs for live portrait

* Update to 3.0.0 releases, extend download postfix

* Move files to the right place

* Logging for the hash and source validation

* Changing logic to handle corrupt sources

* Fix typo

* Use names over get_inputs(), Remove set_options() call

* Age modifier now works for CoreML too

* Update age_modifier.py

* Add video encoder h264_videotoolbox and hevc_videotoolbox

* Face editor add eye gaze & remove open factor sliders (#670)

* changes

* add eye gaze

* changes

* cleanup

* add eyebrow control

* changes

* changes

* Feat/terminal UI (#671)

* Introduce terminal to the UI

* Introduce terminal to the UI part2

* Introduce terminal to the UI part2

* Introduce terminal to the UI part2

* Calc range step to avoid weird values

* Use Sequence for ranges

* Use Sequence for ranges

* changes (#673)

* Use Sequence for ranges

* Finalize terminal UI

* Finalize terminal UI

* Webcam cosmetics, Fix normalize fps to accept int

* Cosmetic changes

* Finalize terminal UI

* Rename leftover typings

* Fix wording

* Fix rounding in metavar

* Fix rounding in metavar

* Rename to face classifier

* Face editor lip moves (#677)

* changes

* changes

* changes

* Fix rounding in metavar

* Rename to face classifier

* changes

* changes

* update naming

---------

Co-authored-by: henryruhs <info@henryruhs.com>

* Fix wording

* Feat/many landmarker + face analyser breakdown (#678)

* Basic multi landmarker integration

* Simplify some method names

* Break into face_detector and face_landmarker

* Fix cosmetics

* Fix testing

* Break into face_attributor and face_recognizer

* Clear them all

* Clear them all

* Rename to face classifier

* Rename to face classifier

* Fix testing

* Fix stuff

* Add face landmarker model to UI

* Add face landmarker model to UI part2

* Split the config

* Split the UI

* Improvement from code review

* Improvement from code review

* Validate args also for sub parsers

* Remove clear of processors in process step

* Allow finder control for the face editor

* Fix lint

* Improve testing performance

* Remove unused file, Clear processors from the UI before job runs

* Update the installer

* Uniform set handler for swapper and detector in the UI

* Fix example urls

* Feat/inference manager (#684)

* Introduce inference manager

* Migrate all to inference manager

* clean ini

* Introduce app context based inference pools

* Fix lint

* Fix typing

* Adjust layout

* Less border radius

* Rename app context names

* Fix/live portrait directml (#691)

* changes (#690)

* Adjust naming

* Use our assets release

* Adjust naming

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Add caches to gitignore

* Update dependencies and drop CUDA 11.8 support (#693)

* Update dependencies and drop CUDA 11.8 support

* Play save and keep numpy 1.x.x

* Improve TensorRT optimization

* changes

* changes

* changes

* changes

* changes

* changes

* changes

* changes

* changes

* Reuse inference sessions (#696)

* Fix force-download command

* Refactor processors to forward() (#698)

* Install tensorrt when selecting cuda

* Minor changes

* Use latest numpy

* Fix limit system memory

* Implement forward() for every inference (#699)

* Implement forward() for every inference

* Implement forward() for every inference

* Implement forward() for every inference

* Implement forward() for every inference

* changes

* changes

* changes

* changes

* Feat/fairface (#710)

* Replace gender_age model with fair face (#709)

* changes

* changes

* changes

* age dropdown to range-slider

* Cleanup code

* Cleanup code

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Extend installer to set library paths for cuda and tensorrt (#707)

* Extend installer to set library paths for cuda and tensorrt

* Add refresh of conda env

* Remove invalid commands

* Set the conda env according to operating system

* Update for ROCm 6.2

* fix installer

* Aktualisieren von installer.py

* Add missing face selector keys

* Try to keep original LD_LIBRARY_PATH

* windows support installer

* Final touch to the installer

* Remove spaces

* Simplidy collect_model_downloads()

* Fix force download for once and forever

* Housekeeping (#715)

* changes

* changes

* changes

* Fix performance part1

* Fix mixed states (#689)

* Fix mixed states

* Add missing sync for job args

* Move UnionStateXXX to base typing

* Undo

* Remove UnionStateXXX

* Fix app context performance lookup (#717)

* Restore performance for inswapper

* Mover upper() to the logger

* Undo debugging

* Move TensorRT installation to docs

* Sort out log level typing, Add log level UI dropdown (#719)

* Fix inference pool part1

* Validate conda library paths existence

* Default face selector order to large-small

* Fix inference pool context according to execution provider (#720)

* Fix app context under Windows

* CUDA and TensorRT update for the installer

* Remove concept of static processor modules

* Revert false commit

* Change event order makes a difference

* Fix multi model context in inference pool (#721)

* Fix multi model context in inference pool

* Fix multi model context in inference pool part2

* Use latest gradio to avoid fastapi bug

* Rework on the Windows Installer

* Use embedding converter (#724)

* changes (#723)

* Upload models to official assets repo

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Rework on the Windows Installer part2

* Resolve subprocess calls (#726)

* Experiment

* Resolve subprocess calls to cover edge cases like broken PATH

* Adjust wording

* Simplify code

* Rework on the Windows Installer part3

* Rework on the Windows Installer part4

* Numpy fix for older onnxruntime

* changes (#729)

* Add space

* Add MacOS installer

* Use favicon

* Fix disabled logger

* Layout polishing (#731)

* Update dependencies, Adjust many face landmarker logic

* Cosmetics changes

* Should be button

* Introduce randomized action button

* Fix update of lip syncer and expression restorer

* Stop sharing inference session this prevents flushing VRAM

* Fix test

* Fix urls

* Prepare release

* Vanish inquirer

* Sticky preview does not work on portrait images

* Sticky preview only for landscape images and videos

* remove gradio tunnel env

* Change wording and deeplinks

* increase peppa landmark score offset

* Change wording

* Graceful exit install.py

* Just adding a required

* Cannot use the exit_helper

* Rename our model

* Change color of face-landmark-68/5

* Limit liveportrait (#739)

* changes

* changes

* changes

* Cleanup

* Cleanup

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* limit expression restorer

* change expression restorer 0-100 range

* Use 256x icon

* changes

* changes

* changes

* changes

* Limit face editor rotation (#745)

* changes (#743)

* Finish euler methods

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Use different coveralls badge

* Move about wording

* Shorten scope in the logger

* changes

* changes

* Shorten scope in the logger

* fix typo

* Simplify the arcface converter names

* Update preview

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
This commit is contained in:
Henry Ruhs
2024-09-20 17:27:50 +02:00
committed by GitHub
parent 57016d7c77
commit 13761af044
171 changed files with 11598 additions and 5115 deletions

View File

@@ -0,0 +1,217 @@
from typing import Tuple
import cv2
import numpy
from facefusion import inference_manager, state_manager
from facefusion.download import conditional_download_hashes, conditional_download_sources
from facefusion.face_helper import create_rotated_matrix_and_size, estimate_matrix_by_face_landmark_5, transform_points, warp_face_by_translation
from facefusion.filesystem import resolve_relative_path
from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.typing import Angle, BoundingBox, DownloadSet, FaceLandmark5, FaceLandmark68, InferencePool, ModelSet, Prediction, Score, VisionFrame
MODEL_SET : ModelSet =\
{
'2dfan4':
{
'hashes':
{
'2dfan4':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/2dfan4.hash',
'path': resolve_relative_path('../.assets/models/2dfan4.hash')
}
},
'sources':
{
'2dfan4':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/2dfan4.onnx',
'path': resolve_relative_path('../.assets/models/2dfan4.onnx')
}
},
'size': (256, 256)
},
'peppa_wutz':
{
'hashes':
{
'peppa_wutz':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/peppa_wutz.hash',
'path': resolve_relative_path('../.assets/models/peppa_wutz.hash')
}
},
'sources':
{
'peppa_wutz':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/peppa_wutz.onnx',
'path': resolve_relative_path('../.assets/models/peppa_wutz.onnx')
}
},
'size': (256, 256)
},
'fan_68_5':
{
'hashes':
{
'fan_68_5':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/fan_68_5.hash',
'path': resolve_relative_path('../.assets/models/fan_68_5.hash')
}
},
'sources':
{
'fan_68_5':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/fan_68_5.onnx',
'path': resolve_relative_path('../.assets/models/fan_68_5.onnx')
}
}
}
}
def get_inference_pool() -> InferencePool:
_, model_sources = collect_model_downloads()
model_context = __name__ + '.' + state_manager.get_item('face_landmarker_model')
return inference_manager.get_inference_pool(model_context, model_sources)
def clear_inference_pool() -> None:
model_context = __name__ + '.' + state_manager.get_item('face_landmarker_model')
inference_manager.clear_inference_pool(model_context)
def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]:
model_hashes =\
{
'fan_68_5': MODEL_SET.get('fan_68_5').get('hashes').get('fan_68_5')
}
model_sources =\
{
'fan_68_5': MODEL_SET.get('fan_68_5').get('sources').get('fan_68_5')
}
if state_manager.get_item('face_landmarker_model') in [ 'many', '2dfan4' ]:
model_hashes['2dfan4'] = MODEL_SET.get('2dfan4').get('hashes').get('2dfan4')
model_sources['2dfan4'] = MODEL_SET.get('2dfan4').get('sources').get('2dfan4')
if state_manager.get_item('face_landmarker_model') in [ 'many', 'peppa_wutz' ]:
model_hashes['peppa_wutz'] = MODEL_SET.get('peppa_wutz').get('hashes').get('peppa_wutz')
model_sources['peppa_wutz'] = MODEL_SET.get('peppa_wutz').get('sources').get('peppa_wutz')
return model_hashes, model_sources
def pre_check() -> bool:
download_directory_path = resolve_relative_path('../.assets/models')
model_hashes, model_sources = collect_model_downloads()
return conditional_download_hashes(download_directory_path, model_hashes) and conditional_download_sources(download_directory_path, model_sources)
def detect_face_landmarks(vision_frame : VisionFrame, bounding_box : BoundingBox, face_angle : Angle) -> Tuple[FaceLandmark68, Score]:
face_landmark_2dfan4 = None
face_landmark_peppa_wutz = None
face_landmark_score_2dfan4 = 0.0
face_landmark_score_peppa_wutz = 0.0
if state_manager.get_item('face_landmarker_model') in [ 'many', '2dfan4' ]:
face_landmark_2dfan4, face_landmark_score_2dfan4 = detect_with_2dfan4(vision_frame, bounding_box, face_angle)
if state_manager.get_item('face_landmarker_model') in [ 'many', 'peppa_wutz' ]:
face_landmark_peppa_wutz, face_landmark_score_peppa_wutz = detect_with_peppa_wutz(vision_frame, bounding_box, face_angle)
if face_landmark_score_2dfan4 > face_landmark_score_peppa_wutz - 0.2:
return face_landmark_2dfan4, face_landmark_score_2dfan4
return face_landmark_peppa_wutz, face_landmark_score_peppa_wutz
def detect_with_2dfan4(temp_vision_frame: VisionFrame, bounding_box: BoundingBox, face_angle: Angle) -> Tuple[FaceLandmark68, Score]:
model_size = MODEL_SET.get('2dfan4').get('size')
scale = 195 / numpy.subtract(bounding_box[2:], bounding_box[:2]).max().clip(1, None)
translation = (model_size[0] - numpy.add(bounding_box[2:], bounding_box[:2]) * scale) * 0.5
rotated_matrix, rotated_size = create_rotated_matrix_and_size(face_angle, model_size)
crop_vision_frame, affine_matrix = warp_face_by_translation(temp_vision_frame, translation, scale, model_size)
crop_vision_frame = cv2.warpAffine(crop_vision_frame, rotated_matrix, rotated_size)
crop_vision_frame = conditional_optimize_contrast(crop_vision_frame)
crop_vision_frame = crop_vision_frame.transpose(2, 0, 1).astype(numpy.float32) / 255.0
face_landmark_68, face_heatmap = forward_with_2dfan4(crop_vision_frame)
face_landmark_68 = face_landmark_68[:, :, :2][0] / 64 * 256
face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(rotated_matrix))
face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(affine_matrix))
face_landmark_score_68 = numpy.amax(face_heatmap, axis = (2, 3))
face_landmark_score_68 = numpy.mean(face_landmark_score_68)
face_landmark_score_68 = numpy.interp(face_landmark_score_68, [ 0, 0.9 ], [ 0, 1 ])
return face_landmark_68, face_landmark_score_68
def detect_with_peppa_wutz(temp_vision_frame : VisionFrame, bounding_box : BoundingBox, face_angle : Angle) -> Tuple[FaceLandmark68, Score]:
model_size = MODEL_SET.get('peppa_wutz').get('size')
scale = 195 / numpy.subtract(bounding_box[2:], bounding_box[:2]).max().clip(1, None)
translation = (model_size[0] - numpy.add(bounding_box[2:], bounding_box[:2]) * scale) * 0.5
rotated_matrix, rotated_size = create_rotated_matrix_and_size(face_angle, model_size)
crop_vision_frame, affine_matrix = warp_face_by_translation(temp_vision_frame, translation, scale, model_size)
crop_vision_frame = cv2.warpAffine(crop_vision_frame, rotated_matrix, rotated_size)
crop_vision_frame = conditional_optimize_contrast(crop_vision_frame)
crop_vision_frame = crop_vision_frame.transpose(2, 0, 1).astype(numpy.float32) / 255.0
crop_vision_frame = numpy.expand_dims(crop_vision_frame, axis = 0)
prediction = forward_with_peppa_wutz(crop_vision_frame)
face_landmark_68 = prediction.reshape(-1, 3)[:, :2] / 64 * model_size[0]
face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(rotated_matrix))
face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(affine_matrix))
face_landmark_score_68 = prediction.reshape(-1, 3)[:, 2].mean()
face_landmark_score_68 = numpy.interp(face_landmark_score_68, [ 0, 0.95 ], [ 0, 1 ])
return face_landmark_68, face_landmark_score_68
def conditional_optimize_contrast(crop_vision_frame : VisionFrame) -> VisionFrame:
crop_vision_frame = cv2.cvtColor(crop_vision_frame, cv2.COLOR_RGB2Lab)
if numpy.mean(crop_vision_frame[:, :, 0]) < 30: # type:ignore[arg-type]
crop_vision_frame[:, :, 0] = cv2.createCLAHE(clipLimit = 2).apply(crop_vision_frame[:, :, 0])
crop_vision_frame = cv2.cvtColor(crop_vision_frame, cv2.COLOR_Lab2RGB)
return crop_vision_frame
def estimate_face_landmark_68_5(face_landmark_5 : FaceLandmark5) -> FaceLandmark68:
affine_matrix = estimate_matrix_by_face_landmark_5(face_landmark_5, 'ffhq_512', (1, 1))
face_landmark_5 = cv2.transform(face_landmark_5.reshape(1, -1, 2), affine_matrix).reshape(-1, 2)
face_landmark_68_5 = forward_fan_68_5(face_landmark_5)
face_landmark_68_5 = cv2.transform(face_landmark_68_5.reshape(1, -1, 2), cv2.invertAffineTransform(affine_matrix)).reshape(-1, 2)
return face_landmark_68_5
def forward_with_2dfan4(crop_vision_frame : VisionFrame) -> Tuple[Prediction, Prediction]:
face_landmarker = get_inference_pool().get('2dfan4')
with conditional_thread_semaphore():
prediction = face_landmarker.run(None,
{
'input': [ crop_vision_frame ]
})
return prediction
def forward_with_peppa_wutz(crop_vision_frame : VisionFrame) -> Prediction:
face_landmarker = get_inference_pool().get('peppa_wutz')
with conditional_thread_semaphore():
prediction = face_landmarker.run(None,
{
'input': crop_vision_frame
})[0]
return prediction
def forward_fan_68_5(face_landmark_5 : FaceLandmark5) -> FaceLandmark68:
face_landmarker = get_inference_pool().get('fan_68_5')
with conditional_thread_semaphore():
face_landmark_68_5 = face_landmarker.run(None,
{
'input': [ face_landmark_5 ]
})[0][0]
return face_landmark_68_5