Tanay Mehta 12f043eaea Fix MarianTokenizer to remove metaspace character in decode (#26091) ...

* add: check to remove metaspace from marian tokenizer

* fix: metaspace character being removed from everywhere

* fix: remove redundant check at top

* add: test for marian tokenizer decode fix

* fix: simplified the test

2023-09-12 21:53:31 +02:00

..

__init__.py

Move test model folders (#17034)

2022-05-03 14:42:02 +02:00

test_modeling_flax_marian.py

Update quality tooling for formatting (#21480)

2023-02-06 18:10:56 -05:00

test_modeling_marian.py

Marian: post-hack-fix correction (#25459)

2023-08-16 11:49:29 +01:00

test_modeling_tf_marian.py

Speed up TF tests by reducing hidden layer counts (#24595)

2023-06-30 16:30:33 +01:00

test_tokenization_marian.py

Fix MarianTokenizer to remove metaspace character in decode (#26091)

2023-09-12 21:53:31 +02:00