SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Pr for pplm (#2060)

Browse Source 
* license

* changes

* ok

* Update paper link and commands to run

* pointer to uber repo
This commit is contained in:
Rosanne Liu
2019-12-05 06:20:07 -08:00
committed by Julien Chaumond
parent bebaa14039
commit ee53de7aac
3 changed files with 32 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
examples/pplm/README.md
View File

@@ -1,17 +1,15 @@
# PPLM # Plug and Play Language Models: a Simple Approach to Controlled Text Generation
Authors: [Sumanth Dathathri](https://dathath.github.io/), [Andrea Madotto](https://andreamad8.github.io/), Janice Lan, Jane Hung, Eric Frank, [Piero Molino](https://w4nderlu.st/), [Jason Yosinski](http://yosinski.com/), and [Rosanne Liu](http://www.rosanneliu.com/)
This folder contains the original code used to run the Plug and Play Language Model (PPLM). This folder contains the original code used to run the Plug and Play Language Model (PPLM).
![header image](./imgs/headfigure.png)
## Plug and Play Language Models: a Simple Approach to Steerable Text Generation Paper link: https://arxiv.org/abs/1912.02164
Authors: [Sumanth Dathathri](https://dathath.github.io/), Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, [Piero Molino](https://w4nderlu.st/), [Jason Yosinski](http://yosinski.com/), and [Rosanne Liu](http://www.rosanneliu.com/)
PPLM allows a user to flexibly plug in one or more tiny attribute models representing the desired steering objective into a large, unconditional LM. The method has the key property that it uses the LM _as is_---no training or fine-tuning is required---which enables researchers to leverage best-in-class LMs even if they do not have the extensive hardware required to train them.
Paper link:
Blog link: https://eng.uber.com/pplm Blog link: https://eng.uber.com/pplm
Please check out the repo under uber-research for more information: https://github.com/uber-research/PPLM
## Setup ## Setup
@@ -27,7 +25,7 @@ cd examples/pplm
### Example command for bag-of-words control ### Example command for bag-of-words control
```bash ```bash
python run_pplm.py -B space --cond_text "The president" --length 100 --gamma 1.5 --num_iterations 3 --num_samples 1 --stepsize 0.01 --window_length 5 --kl_scale 0.01 --gm_scale 0.95 python run_pplm.py -B military --cond_text "The potato" --length 50 --gamma 1.5 --num_iterations 3 --num_samples 10 --stepsize 0.03 --window_length 5 --kl_scale 0.01 --gm_scale 0.99 --colorama --sample
``` ```
### Tuning hyperparameters for bag-of-words control ### Tuning hyperparameters for bag-of-words control
@@ -45,7 +43,7 @@ python run_pplm.py -B space --cond_text "The president" --length 100 --gamma 1.5
### Example command for discriminator based sentiment control ### Example command for discriminator based sentiment control
```bash ```bash
python run_pplm.py -D sentiment --class_label 3 --cond_text "The lake" --length 10 --gamma 1.0 --num_iterations 10 --num_samples 1 --stepsize 0.03 --kl_scale 0.01 --gm_scale 0.95 python run_pplm.py -D sentiment --class_label 2 --cond_text "My dog died" --length 50 --gamma 1.0 --num_iterations 10 --num_samples 10 --stepsize 0.04 --kl_scale 0.01 --gm_scale 0.95 --sample
``` ```
### Tuning hyperparameters for discriminator control ### Tuning hyperparameters for discriminator control
@@ -54,8 +52,3 @@ python run_pplm.py -D sentiment --class_label 3 --cond_text "The lake" --length
2. Use `--class_label 3` for negative, and `--class_label 2` for positive 2. Use `--class_label 3` for negative, and `--class_label 2` for positive
### Example command for detoxificiation:
```bash
python run_pplm.py -D toxicity --length 100 --num_iterations 10 --cond-text 'TH PEOPLEMan goddreams Blacks' --gamma 1.0 --num_samples 10 --stepsize 0.02
```

32
examples/pplm/run_pplm.py
View File

@@ -1,18 +1,19 @@
#! /usr/bin/env python3 #! /usr/bin/env python3
# coding=utf-8 # coding=utf-8
# Copyright 2018 The Uber AI Team Authors.
#Copyright (c) 2019 Uber Technologies, Inc.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); #Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. #you may not use this file except in compliance with the License.
# You may obtain a copy of the License at #You may obtain a copy of the License at
# #
# http://www.apache.org/licenses/LICENSE-2.0 #http://www.apache.org/licenses/LICENSE-2.0
# #
# Unless required by applicable law or agreed to in writing, software #Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, #distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. #WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and #See the License for the specific language governing permissions and
# limitations under the License. #limitations under the License.
""" """
Example command with bag of words: Example command with bag of words:
@@ -45,12 +46,9 @@ SMALL_CONST = 1e-15
BIG_CONST = 1e10 BIG_CONST = 1e10
BAG_OF_WORDS_ARCHIVE_MAP = { BAG_OF_WORDS_ARCHIVE_MAP = {
'kitchen': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/kitchen.txt",
'legal': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/legal.txt", 'legal': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/legal.txt",
'military': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/military.txt", 'military': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/military.txt",
'monsters': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/monsters.txt",
'politics': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/politics.txt", 'politics': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/politics.txt",
'positive_words': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/positive_words.txt",
'religion': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/religion.txt", 'religion': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/religion.txt",
'science': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/science.txt", 'science': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/science.txt",
'space': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/space.txt", 'space': "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/bow/space.txt",
@@ -74,14 +72,6 @@ DISCRIMINATOR_MODELS_PARAMS = {
"default_class": 3, "default_class": 3,
"pretrained_model": "gpt2-medium", "pretrained_model": "gpt2-medium",
}, },
"toxicity": {
"url": "https://s3.amazonaws.com/models.huggingface.co/bert/pplm/discriminators/toxic_classifier_head.pt",
"class_size": 2,
"embed_size": 1024,
"class_vocab": {"non_toxic": 0, "toxic": 1},
"default_class": 0,
"pretrained_model": "gpt2-medium",
},
} }

14
examples/pplm/run_pplm_discrim_train.py
View File

@@ -1,7 +1,19 @@
#! /usr/bin/env python3 #! /usr/bin/env python3
# coding=utf-8 # coding=utf-8
# This code is licensed under a non-commercial license. #Copyright (c) 2019 Uber Technologies, Inc.
#
#Licensed under the Apache License, Version 2.0 (the "License");
#you may not use this file except in compliance with the License.
#You may obtain a copy of the License at
#
#http://www.apache.org/licenses/LICENSE-2.0
#
#Unless required by applicable law or agreed to in writing, software
#distributed under the License is distributed on an "AS IS" BASIS,
#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#See the License for the specific language governing permissions and
#limitations under the License.
import argparse import argparse
import csv import csv