jones/facefusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

3.1.0 (#839)

Browse Source 
* Replace audio whenever set via source

* add H264_qsv&HEVC_qsv (#768)

* Update ffmpeg.py

* Update choices.py

* Update typing.py

* Fix spaces and newlines

* Fix return type

* Introduce hififace swapper

* Disable stream for expression restorer

* Webcam polishing part1 (#796)

* Cosmetics on ignore comments

* Testing for replace audio

* Testing for restore audio

* Testing for restore audio

* Fix replace_audio()

* Remove shortest and use fixed video duration

* Remove shortest and use fixed video duration

* Prevent duplicate entries to local PATH

* Do hard exit on invalid args

* Need for Python 3.10

* Fix state of face selector

* Fix OpenVINO by aliasing GPU.0 to GPU

* Fix OpenVINO by aliasing GPU.0 to GPU

* Fix/age modifier styleganex 512 (#798)

* fix

* styleganex template

* changes

* changes

* fix occlusion mask

* add age modifier scale

* change

* change

* hardcode

* Cleanup

* Use model_sizes and model_templates variables

* No need for prepare when just 2 lines of code

* Someone used spaces over tabs

* Revert back [0][0]

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>

* Feat/update gradio5 (#799)

* Update to Gradio 5

* Remove overrides for Gradio

* Fix dark mode for Gradio

* Polish errors

* More styles for tabs and co

* Make slider inputs and reset like a unit

* Make slider inputs and reset like a unit

* Adjust naming

* Improved color matching (#800)

* aura fix

* fix import

* move to vision.py

* changes

* changes

* changes

* changes

* further reduction

* add test

* better test

* change name

* Minor cleanup

* Minor cleanup

* Minor cleanup

* changes (#801)

* Switch to official assets repo

* Add __pycache__ to gitignore

* Gradio pinned python-multipart to 0.0.12

* Update dependencies

* Feat/temp path second try (#802)

* Terminate base directory from temp helper

* Partial adjust program codebase

* Move arguments around

* Make `-j` absolete

* Resolve args

* Fix job register keys

* Adjust date test

* Finalize temp path

* Update onnxruntime

* Update dependencies

* Adjust color for checkboxes

* Revert due terrible performance

* Fix/enforce vp9 for webm (#805)

* Simple fix to enforce vp9 for webm

* Remove suggest methods from program helper

* Cleanup ffmpeg.py a bit

* Update onnxruntime (second try)

* Update onnxruntime (second try)

* Remove cudnn_conv_algo_search tweaks

* Remove cudnn_conv_algo_search tweaks

* changes

* add both mask instead of multiply

* adaptive color correction

* changes

* remove model size requirement

* changes

* add to facefusion.ini

* changes

* changes

* changes

* Add namespace for dfm creators

* Release five frame enhancer models

* Remove vendor from model name

* Remove vendor from model name

* changes

* changes

* changes

* changes

* Feat/download providers (#809)

* Introduce download providers

* update processors download method

* add ui

* Fix CI

* Adjust UI component order, Use download resolver for benchmark

* Remove is_download_done()

* Introduce download provider set, Remove choices method from execution, cast all dict keys() via list()

* Fix spacing

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>

* Fix model paths for 3.1.0

* Introduce bulk-run (#810)

* Introduce bulk-run

* Make bulk run bullet proof

* Integration test for bulk-run

* new alignment

* Add safer global named resolve_file_pattern() (#811)

* Allow bulk runner with target pattern only

* changes

* changes

* Update Python to 3.12 for CI (#813)

* changes

* Improve NVIDIA device lookups

* Rename template key to deepfacelive

* Fix name

* Improve resolve download

* Rename bulk-run to batch-run

* Make deep swapper inputs universal

* Add more deepfacelive models

* Use different morph value

* Feat/simplify hashes sources download (#814)

* Extract download directory path from assets path

* Fix lint

* Fix force-download command, Fix urls in frame enhancer

* changes

* fix warp_face_by_bounding_box dtype error

* DFM Morph (#816)

* changes

* Improve wording, Replace [None], SideQuest: clean forward() of age modifier

* SideQuest: clean forward() of face enhancer

---------

Co-authored-by: henryruhs <info@henryruhs.com>

* Fix preview refresh after slide

* Add more deepfacelive models (#817)

* Add more deepfacelive models

* Add more deepfacelive models

* Fix deep swapper sizes

* Kill accent colors, Number input styles for Chrome

* Simplify thumbnail-item looks

* Fix first black screen

* Introduce model helper

* ci.yml: Add macOS on ARM64 to the testing (#818)

* ci.yml: Add macOS on ARM64 to the testing

* ci.yml: uses: AnimMouse/setup-ffmpeg@v1

* ci.yml: strategy: matrix: os: macos-latest,

* - name: Set up FFmpeg

* Update .github/workflows/ci.yml

* Update ci.yml

---------

Co-authored-by: Henry Ruhs <info@henryruhs.com>

* Show/hide morph slider for deep swapper (#822)

* remove dfl_head and update dfl_whole_face template

* Add deep swapper models by Mats

* Add deep swapper models by Druuzil

* Add deep swapper models by Rumateus

* Implement face enhancer weight for codeformer, Side Quest: has proces… (#823)

* Implement face enhancer weight for codeformer, Side Quest: has processor checks

* Fix typo

* Fix face enhancer blend in UI

* Use static model set creation

* Add deep swapper models by Jen

* Introduce create_static_model_set() everywhere (#824)

* Move clear over to the UI (#825)

* Fix model key

* Undo restore_audio()

* Switch to latest XSeg

* Switch to latest XSeg

* Switch to latest XSeg

* Use resolve_download_url() everywhere, Vanish --skip-download flag

* Fix resolve_download_url

* Fix space

* Kill resolve_execution_provider_keys() and move fallbacks where they belong

* Kill resolve_execution_provider_keys() and move fallbacks where they belong

* Remove as this does not work

* Change TempFrameFormat order

* Fix CoreML partially

* Remove duplicates (Rumateus is the creator)

* Add deep swapper models by Edel

* Introduce download scopes (#826)

* Introduce download scopes

* Limit download scopes to force-download command

* Change source-paths behaviour

* Fix space

* Update README

* Rename create_log_level_program to create_misc_program

* Fix wording

* Fix wording

* Update dependencies

* Use tolerant for video_memory_strategy in benchmark

* Feat/ffmpeg with progress (#827)

* FFmpeg with progress bar

* Fix typing

* FFmpeg with progress bar part2

* Restore streaming wording

* Change order in choices and typing

* Introduce File using list_directory() (#830)

* Feat/local deep swapper models (#832)

* Local model support for deep swapper

* Local model support for deep swapper part2

* Local model support for deep swapper part3

* Update yet another dfm by Druuzil

* Refactor/choices and naming (#833)

* Refactor choices, imports and naming

* Refactor choices, imports and naming

* Fix styles for tabs, Restore toast

* Update yet another dfm by Druuzil

* Feat/face masker models (#834)

* Introduce face masker models

* Introduce face masker models

* Introduce face masker models

* Register needed step keys

* Provide different XSeg models

* Simplify model context

* Fix out of range for trim frame, Fix ffmpeg extraction count (#836)

* Fix out of range for trim frame, Fix ffmpeg extraction count

* Move restrict of trim frame to the core, Make sure all values are within the range

* Fix and merge testing

* Fix typing

* Add region mask for deep swapper

* Adjust wording

* Move FACE_MASK_REGIONS to choices

* Update dependencies

* Feat/download provider fallback (#837)

* Introduce download providers fallback, Use CURL everywhre

* Fix CI

* Use readlines() over readline() to avoid while

* Use readlines() over readline() to avoid while

* Use readlines() over readline() to avoid while

* Use communicate() over wait()

* Minor updates for testing

* Stop webcam on source image change

* Feat/webcam improvements (#838)

* Detect available webcams

* Fix CI, Move webcam id dropdown to the sidebar, Disable warnings

* Fix CI

* Remove signal on hard_exit() to prevent exceptions

* Fix border color in toast timer

* Prepare release

* Update preview

* Update preview

* Hotfix progress bar

---------

Co-authored-by: DDXDB <38449595+DDXDB@users.noreply.github.com>
Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
Co-authored-by: Christian Clauss <cclauss@me.com>
This commit is contained in:
Henry Ruhs
2024-12-24 12:46:56 +01:00
committed by GitHub
parent ec12f679bf
commit 7a09479fb5
102 changed files with 3699 additions and 2112 deletions
Download Patch File Download Diff File Expand all files Collapse all files

172
facefusion/face_masker.py
View File

@@ -1,72 +1,103 @@
from functools import lru_cache
from typing import Dict, List, Tuple
from typing import List, Tuple
import cv2
import numpy
from cv2.typing import Size
from facefusion import inference_manager
from facefusion.download import conditional_download_hashes, conditional_download_sources
import facefusion.choices
from facefusion import inference_manager, state_manager
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.filesystem import resolve_relative_path
from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.typing import DownloadSet, FaceLandmark68, FaceMaskRegion, InferencePool, Mask, ModelSet, Padding, VisionFrame
from facefusion.typing import DownloadScope, DownloadSet, FaceLandmark68, FaceMaskRegion, InferencePool, Mask, ModelSet, Padding, VisionFrame
MODEL_SET : ModelSet =\
{
'face_occluder':
@lru_cache(maxsize = None)
def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
return\
{
'hashes':
'xseg_1':
{
'face_occluder':
'hashes':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/dfl_xseg.hash',
'path': resolve_relative_path('../.assets/models/dfl_xseg.hash')
}
'face_occluder':
{
'url': resolve_download_url('models-3.1.0', 'xseg_1.hash'),
'path': resolve_relative_path('../.assets/models/xseg_1.hash')
}
},
'sources':
{
'face_occluder':
{
'url': resolve_download_url('models-3.1.0', 'xseg_1.onnx'),
'path': resolve_relative_path('../.assets/models/xseg_1.onnx')
}
},
'size': (256, 256)
},
'sources':
'xseg_2':
{
'face_occluder':
'hashes':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/dfl_xseg.onnx',
'path': resolve_relative_path('../.assets/models/dfl_xseg.onnx')
}
'face_occluder':
{
'url': resolve_download_url('models-3.1.0', 'xseg_2.hash'),
'path': resolve_relative_path('../.assets/models/xseg_2.hash')
}
},
'sources':
{
'face_occluder':
{
'url': resolve_download_url('models-3.1.0', 'xseg_2.onnx'),
'path': resolve_relative_path('../.assets/models/xseg_2.onnx')
}
},
'size': (256, 256)
},
'size': (256, 256)
},
'face_parser':
{
'hashes':
'bisenet_resnet_18':
{
'face_parser':
'hashes':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/bisenet_resnet_34.hash',
'path': resolve_relative_path('../.assets/models/bisenet_resnet_34.hash')
}
'face_parser':
{
'url': resolve_download_url('models-3.1.0', 'bisenet_resnet_18.hash'),
'path': resolve_relative_path('../.assets/models/bisenet_resnet_18.hash')
}
},
'sources':
{
'face_parser':
{
'url': resolve_download_url('models-3.1.0', 'bisenet_resnet_18.onnx'),
'path': resolve_relative_path('../.assets/models/bisenet_resnet_18.onnx')
}
},
'size': (512, 512)
},
'sources':
'bisenet_resnet_34':
{
'face_parser':
'hashes':
{
'url': 'https://github.com/facefusion/facefusion-assets/releases/download/models-3.0.0/bisenet_resnet_34.onnx',
'path': resolve_relative_path('../.assets/models/bisenet_resnet_34.onnx')
}
},
'size': (512, 512)
'face_parser':
{
'url': resolve_download_url('models-3.0.0', 'bisenet_resnet_34.hash'),
'path': resolve_relative_path('../.assets/models/bisenet_resnet_34.hash')
}
},
'sources':
{
'face_parser':
{
'url': resolve_download_url('models-3.0.0', 'bisenet_resnet_34.onnx'),
'path': resolve_relative_path('../.assets/models/bisenet_resnet_34.onnx')
}
},
'size': (512, 512)
}
}
}
FACE_MASK_REGIONS : Dict[FaceMaskRegion, int] =\
{
'skin': 1,
'left-eyebrow': 2,
'right-eyebrow': 3,
'left-eye': 4,
'right-eye': 5,
'glasses': 6,
'nose': 10,
'mouth': 11,
'upper-lip': 12,
'lower-lip': 13
}
def get_inference_pool() -> InferencePool:
@@ -79,24 +110,33 @@ def clear_inference_pool() -> None:
def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]:
model_hashes =\
{
'face_occluder': MODEL_SET.get('face_occluder').get('hashes').get('face_occluder'),
'face_parser': MODEL_SET.get('face_parser').get('hashes').get('face_parser')
}
model_sources =\
{
'face_occluder': MODEL_SET.get('face_occluder').get('sources').get('face_occluder'),
'face_parser': MODEL_SET.get('face_parser').get('sources').get('face_parser')
}
model_hashes = {}
model_sources = {}
model_set = create_static_model_set('full')
if state_manager.get_item('face_occluder_model') == 'xseg_1':
model_hashes['xseg_1'] = model_set.get('xseg_1').get('hashes').get('face_occluder')
model_sources['xseg_1'] = model_set.get('xseg_1').get('sources').get('face_occluder')
if state_manager.get_item('face_occluder_model') == 'xseg_2':
model_hashes['xseg_2'] = model_set.get('xseg_2').get('hashes').get('face_occluder')
model_sources['xseg_2'] = model_set.get('xseg_2').get('sources').get('face_occluder')
if state_manager.get_item('face_parser_model') == 'bisenet_resnet_18':
model_hashes['bisenet_resnet_18'] = model_set.get('bisenet_resnet_18').get('hashes').get('face_parser')
model_sources['bisenet_resnet_18'] = model_set.get('bisenet_resnet_18').get('sources').get('face_parser')
if state_manager.get_item('face_parser_model') == 'bisenet_resnet_34':
model_hashes['bisenet_resnet_34'] = model_set.get('bisenet_resnet_34').get('hashes').get('face_parser')
model_sources['bisenet_resnet_34'] = model_set.get('bisenet_resnet_34').get('sources').get('face_parser')
return model_hashes, model_sources
def pre_check() -> bool:
download_directory_path = resolve_relative_path('../.assets/models')
model_hashes, model_sources = collect_model_downloads()
return conditional_download_hashes(download_directory_path, model_hashes) and conditional_download_sources(download_directory_path, model_sources)
return conditional_download_hashes(model_hashes) and conditional_download_sources(model_sources)
@lru_cache(maxsize = None)
@@ -114,7 +154,8 @@ def create_static_box_mask(crop_size : Size, face_mask_blur : float, face_mask_p
def create_occlusion_mask(crop_vision_frame : VisionFrame) -> Mask:
model_size = MODEL_SET.get('face_occluder').get('size')
face_occluder_model = state_manager.get_item('face_occluder_model')
model_size = create_static_model_set('full').get(face_occluder_model).get('size')
prepare_vision_frame = cv2.resize(crop_vision_frame, model_size)
prepare_vision_frame = numpy.expand_dims(prepare_vision_frame, axis = 0).astype(numpy.float32) / 255
prepare_vision_frame = prepare_vision_frame.transpose(0, 1, 2, 3)
@@ -126,7 +167,8 @@ def create_occlusion_mask(crop_vision_frame : VisionFrame) -> Mask:
def create_region_mask(crop_vision_frame : VisionFrame, face_mask_regions : List[FaceMaskRegion]) -> Mask:
model_size = MODEL_SET.get('face_parser').get('size')
face_parser_model = state_manager.get_item('face_parser_model')
model_size = create_static_model_set('full').get(face_parser_model).get('size')
prepare_vision_frame = cv2.resize(crop_vision_frame, model_size)
prepare_vision_frame = prepare_vision_frame[:, :, ::-1].astype(numpy.float32) / 255
prepare_vision_frame = numpy.subtract(prepare_vision_frame, numpy.array([ 0.485, 0.456, 0.406 ]).astype(numpy.float32))
@@ -134,7 +176,7 @@ def create_region_mask(crop_vision_frame : VisionFrame, face_mask_regions : List
prepare_vision_frame = numpy.expand_dims(prepare_vision_frame, axis = 0)
prepare_vision_frame = prepare_vision_frame.transpose(0, 3, 1, 2)
region_mask = forward_parse_face(prepare_vision_frame)
region_mask = numpy.isin(region_mask.argmax(0), [ FACE_MASK_REGIONS[region] for region in face_mask_regions ])
region_mask = numpy.isin(region_mask.argmax(0), [ facefusion.choices.face_mask_region_set.get(face_mask_region) for face_mask_region in face_mask_regions ])
region_mask = cv2.resize(region_mask.astype(numpy.float32), crop_vision_frame.shape[:2][::-1])
region_mask = (cv2.GaussianBlur(region_mask.clip(0, 1), (0, 0), 5).clip(0.5, 1) - 0.5) * 2
return region_mask
@@ -150,7 +192,8 @@ def create_mouth_mask(face_landmark_68 : FaceLandmark68) -> Mask:
def forward_occlude_face(prepare_vision_frame : VisionFrame) -> Mask:
face_occluder = get_inference_pool().get('face_occluder')
face_occluder_model = state_manager.get_item('face_occluder_model')
face_occluder = get_inference_pool().get(face_occluder_model)
with conditional_thread_semaphore():
occlusion_mask : Mask = face_occluder.run(None,
@@ -162,7 +205,8 @@ def forward_occlude_face(prepare_vision_frame : VisionFrame) -> Mask:
def forward_parse_face(prepare_vision_frame : VisionFrame) -> Mask:
face_parser = get_inference_pool().get('face_parser')
face_parser_model = state_manager.get_item('face_parser_model')
face_parser = get_inference_pool().get(face_parser_model)
with conditional_thread_semaphore():
region_mask : Mask = face_parser.run(None,