Cyril Vallez 4303d88c09 Add Phi4 multimodal (#36939) ...

* raw start

* update

* update

* add to imports

* update

* up

* simplify configs

* clean configs

* style

* typos

* Update convert_phi4_multimodal_weights_to_hf.py

* Update convert_phi4_multimodal_weights_to_hf.py

* fix

* up

* up

* up

* Update convert_phi4_multimodal_weights_to_hf.py

* Update convert_phi4_multimodal_weights_to_hf.py

* up

* up

* up

* Update feature_extraction_phi4_multimodal.py

* up

* up

* up

* up

* up

* simplify configs

* typo

* cut code

* typo

* typo

* typo

* re

* typo

* up

* up

* up

* add tests

* fix

* fix

* Update test_modeling_phi4_multimodal.py

* up

* Update test_modeling_phi4_multimodal.py

* doc

* fix

* up

* up

* up

* up

* up

* up

* simplify

* up

* simplify

* config docstrings

* cleanup

* clean

* typo

* typo

* fix

* Update phi4_multimodal.md

* fix

* fix

* Update test_modeling_phi4_multimodal.py

* update

* simplify reshapes and permutes

* up

* simplify special tokens

* simplify processor a lot

* Update processing_phi4_multimodal.py

* Update processing_phi4_multimodal.py

* switch to fast processor

* image processor

* Update image_processing_phi4_multimodal_fast.py

* add lora extraction to converter

* Update convert_phi4_multimodal_weights_to_hf.py

* Update __init__.py

* add AudioInput type in audio_utils

* rewrite feature_extraction: support torch batched FFT

* input_audio_embeds -> audio_input_features, input_image_embeds -> image_pixel_values

* test update

* not mono channel warning update

* remove auto maps from processor

* kargs dispatch in processor

* simplify kwargs dispatch

* simplify merging

* remove default sampling rate

* style

* Update test_modeling_phi4_multimodal.py

* update doc

* doc

* torch only feature extractor

* make fake tokens adjustable

* Update feature_extraction_phi4_multimodal.py

* fix

* Update processing_phi4_multimodal.py

* simplify mask

* last touch

* fix copies

* style

* Update audio_utils.py

* style

* Update feature_extraction_phi4_multimodal.py

* Update __init__.py

* docstrings

* copies

* fix all checks

* back to fix-copies

* trigger CIs

* Update feature_extraction_phi4_multimodal.py

* improve tests with multimodal inputs

* trigger CIs

---------

Co-authored-by: Eustache Le Bihan <eulebihan@gmail.com>

2025-03-25 09:55:21 +01:00

..

test_module

AutoImageProcessor (#20111)

2022-11-08 19:54:41 +00:00

tf_ops

Check TF ops for ONNX compliance (#10025)

2021-02-15 07:55:10 -05:00

add_pipeline_model_mapping_to_test.py

update ruff version (#30932)

2024-05-22 06:40:15 +02:00

check_bad_commit.py

Fix utils/check_bad_commit.py (for auto ping in CI) (#34943)

2024-11-28 15:34:38 +01:00

check_build.py

Use deformable_detr kernel from the Hub (#36853)

2025-03-21 13:08:47 +01:00

check_config_attributes.py

Shieldgemma2 (#36678)

2025-03-20 15:14:38 +01:00

check_config_docstrings.py

Add TimmWrapper (#34564)

2024-12-11 12:40:30 +00:00

check_copies.py

chore: fix typos in utils module (#36668)

2025-03-13 15:12:44 +00:00

check_doc_toc.py

update ruff version (#30932)

2024-05-22 06:40:15 +02:00

check_docstrings.py

Add Phi4 multimodal (#36939)

2025-03-25 09:55:21 +01:00

check_doctest_list.py

update ruff version (#30932)

2024-05-22 06:40:15 +02:00

check_dummies.py

Add model visual debugger (#36798)

2025-03-20 17:37:29 +01:00

check_inits.py

chore: fix typos in utils module (#36668)

2025-03-13 15:12:44 +00:00

check_model_tester.py

Add a new script to check model testers' config (#22063)

2023-03-13 19:11:19 +01:00

check_modular_conversion.py

[modular] Sort modular skips (#36304)

2025-03-20 10:55:12 +00:00

check_repo.py

Add Phi4 multimodal (#36939)

2025-03-25 09:55:21 +01:00

check_self_hosted_runner.py

Tiny fix for check_self_hosted_runner.py (#24052)

2023-06-06 18:17:41 +02:00

check_tf_ops.py

Check TF ops for ONNX compliance (#10025)

2021-02-15 07:55:10 -05:00

create_dependency_mapping.py

Modular Conversion --fix_and_overwrite on Windows (#36583)

2025-03-06 13:12:30 +00:00

create_dummy_models.py

CI: fix efficientnet pipeline timeout and prevent future similar issues due to large image size (#33123)

2024-08-27 11:58:27 +01:00

custom_init_isort.py

chore: fix typos in utils module (#36668)

2025-03-13 15:12:44 +00:00

deprecate_models.py

chore: fix typos in utils module (#36668)

2025-03-13 15:12:44 +00:00

download_glue_data.py

update ruff version (#30932)

2024-05-22 06:40:15 +02:00

extract_warnings.py

update github actions packages' version to suppress warnings (#30249)

2024-04-15 15:08:09 +02:00

get_ci_error_statistics.py

Add artifact name in job step to maintain job / artifact correspondence (#28682)

2024-01-31 15:58:17 +01:00

get_github_job_time.py

Make Slack CI reporting stronger (#21823)

2023-02-28 17:12:44 +01:00

get_modified_files.py

exclude deleted files in the fixup script (#21436)

2023-02-03 12:57:02 -05:00

get_previous_daily_ci.py

Ping team members for new failed tests in daily CI (#34171)

2024-10-17 16:11:52 +02:00

get_test_info.py

CI: fix efficientnet pipeline timeout and prevent future similar issues due to large image size (#33123)

2024-08-27 11:58:27 +01:00

important_models.txt

ENH: [CI] Add new workflow to run slow tests of important models on push main if they are modified (#29235)

2024-04-12 10:01:28 +02:00

models_to_deprecate.py

update ruff version (#30932)

2024-05-22 06:40:15 +02:00

modular_model_converter.py

Support custom dosctrings in modular (#36726)

2025-03-18 14:00:54 -04:00

not_doctested.txt

Use deformable_detr kernel from the Hub (#36853)

2025-03-21 13:08:47 +01:00

notification_service_doc_tests.py

Refactor doctest (#30210)

2024-04-15 13:20:36 +02:00

notification_service_quantization.py

Revive Nightly/Past CI (#31159)

2024-06-20 18:57:24 +02:00

notification_service.py

Remove old benchmark code (#35730)

2025-01-21 17:56:43 +00:00

past_ci_versions.py

(Re-)Enable Nightly + Past CI (#22393)

2023-03-30 21:06:35 +02:00

patch_helper.py

[Patch helper] update to not have to checkout main (#34006)

2024-10-09 09:21:46 +02:00

pr_slow_ci_models.py

notify new model merged to main (#36375)

2025-02-24 17:53:18 +01:00

print_env.py

Print more library versions in CI (#17384)

2022-06-02 10:24:16 +02:00

process_bad_commit_report.py

Tiny update after #34383 (#34404)

2024-10-28 12:01:05 +01:00

process_circleci_workflow_test_reports.py

Aggeregate test summary files in CircleCI workflow runs (#34989)

2024-12-16 11:06:17 +01:00

process_test_artifacts.py

fix the parallel number of CI nodes when it is smaller than number of tests (#33276)

2024-09-03 16:53:21 +02:00

release.py

Remove research projects (#36645)

2025-03-11 13:47:38 +00:00

set_cuda_devices_for_ci.py

Fix Cohere CI (#31263)

2024-06-10 15:16:58 +02:00

slow_documentation_tests.txt

Update CodeLlama references (#30218)

2024-05-09 22:57:52 +02:00

sort_auto_mappings.py

update ruff version (#30932)

2024-05-22 06:40:15 +02:00

split_doctest_jobs.py

chore: fix typos in utils module (#36668)

2025-03-13 15:12:44 +00:00

split_model_tests.py

consistent job / pytest report / artifact name correspondence (#30392)

2024-04-24 22:32:42 +02:00

tests_fetcher.py

chore: fix typos in utils module (#36668)

2025-03-13 15:12:44 +00:00

update_metadata.py

chore: fix typos in utils module (#36668)

2025-03-13 15:12:44 +00:00

update_tiny_models.py

Mention model_info.id instead of model_info.modelId (#32106)

2024-07-22 14:14:47 +01:00