Tensorflow 预处理将字符串分割为字符

Question

我想编写使用

TextVectorization

预处理层，但将字符串拆分为字符。

data = tf.constant(
    [
        "The Brain is wider than the Sky",
        "For put them side by side",
        "The one the other will contain",
        "With ease and You beside",
    ]
)
# Instantiate TextVectorization with "int" output_mode
text_vectorizer = preprocessing.TextVectorization(output_mode="int")
# Index the vocabulary via `adapt()`
text_vectorizer.adapt(data)

TextVectorization

类有

split

参数，它可以是一个函数。 在纯Python上我想写这样的东西：

text_vectorizer = preprocessing.TextVectorization(output_mode="int",split=lambda x:list(x)))

但是在 TensorFlow 世界中我应该怎么写呢？

Answer 1

尝试使用

tf.strings.regex_replace

将每个序列先转换为单个字符串，然后再次应用

tf.strings.regex_replace

将字符串拆分为字符。接下来，使用

tf.strings.strip

删除每个字符串的前导和尾随空格。最后，分割并返回你的字符串：

import tensorflow as tf

def split_chars(input_data):
  s = tf.strings.regex_replace(input_data, ' ', '')
  tf.print('Single string --> ', s)
  s = tf.strings.regex_replace(s, '', ' ')
  tf.print('Characters --> ', s)
  s = tf.strings.strip(s)
  tf.print('Stripped --> ', s)
  s = tf.strings.split(s, sep = ' ')
  tf.print('Split --> ', s)
  return s

data = tf.constant(
    [
        "The Brain is wider than the Sky",
        "For put them side by side",
        "The one the other will contain",
        "With ease and You beside",
    ]
)
input_text_processor = tf.keras.layers.TextVectorization(split = split_chars)

input_text_processor.adapt(data)
tf.print(f"Vocabulary --> {input_text_processor.get_vocabulary()}")

Single string -->  ["thebrainiswiderthanthesky" "forputthemsidebyside" "theonetheotherwillcontain" "witheaseandyoubeside"]
Characters -->  [" t h e b r a i n i s w i d e r t h a n t h e s k y " " f o r p u t t h e m s i d e b y s i d e " " t h e o n e t h e o t h e r w i l l c o n t a i n " " w i t h e a s e a n d y o u b e s i d e "]
Stripped -->  ["t h e b r a i n i s w i d e r t h a n t h e s k y" "f o r p u t t h e m s i d e b y s i d e" "t h e o n e t h e o t h e r w i l l c o n t a i n" "w i t h e a s e a n d y o u b e s i d e"]
Split -->  [['t', 'h', 'e', ..., 's', 'k', 'y'], ['f', 'o', 'r', ..., 'i', 'd', 'e'], ['t', 'h', 'e', ..., 'a', 'i', 'n'], ['w', 'i', 't', ..., 'i', 'd', 'e']]
Vocabulary --> ['', '[UNK]', 'e', 't', 'i', 'h', 's', 'n', 'o', 'd', 'a', 'r', 'y', 'w', 'b', 'u', 'l', 'p', 'm', 'k', 'f', 'c']

Answer 2

按照

此处

所述使用split="character"

这将对输出中的每个单独字符进行矢量化，将换行符或空格等空白字符保留为词汇项本身。

text_vectorizer = preprocessing.TextVectorization(output_mode="int",split="character")