El seguimientoword2ngrams La función extrae 3 gramos de caracteres de una palabra:

>>> x = 'foobar'
>>> n = 3
>>> [x[i:i+n] for i in range(len(x)-n+1)]
['foo', 'oob', 'oba', 'bar']

Esta publicación muestra la extracción de ngrams de caracteres para una sola palabra,Implementación rápida de caracteres n-gramas usando python.

Pero, ¿qué pasa si tengo oraciones y quiero extraer los ngrams de caracteres,¿Existe un método más rápido que no sea iterativamente llamar alword2ngram()?

¿Cuál será la versión regex de lograr lo mismo?word2ngram ysent2ngram ¿salida? ¿Sería más rápido?

He intentado:

import string, random, time
from itertools import chain

def word2ngrams(text, n=3):
  """ Convert word into character ngrams. """
  return [text[i:i+n] for i in range(len(text)-n+1)]

def sent2ngrams(text, n=3):
    return list(chain(*[word2ngrams(i,n) for i in text.lower().split()]))

def sent2ngrams_simple(text, n=3):
    text = text.lower()
    return [text[i:i+n] for i in range(len(text)-n+1) if not " " in text[i:i+n]]

# Generate 10000 random strings of length 100.
sents = [" ".join([''.join(random.choice(string.ascii_uppercase) for j in range(10)) for i in range(100)]) for k in range(100)]

start = time.time()
x = [sent2ngrams(i) for i in sents]
print time.time() - start        

start = time.time()
y = [sent2ngrams_simple(i) for i in sents]
print time.time() - start        

print x==y

[fuera]:

0.0205280780792
0.0271739959717
True

EDITADO

El método regex se ve elegante pero funciona más lento que llamar de forma iterativaword2ngram():

import string, random, time, re
from itertools import chain

def word2ngrams(text, n=3):
  """ Convert word into character ngrams. """
  return [text[i:i+n] for i in range(len(text)-n+1)]

def sent2ngrams(text, n=3):
    return list(chain(*[word2ngrams(i,n) for i in text.lower().split()]))

def sent2ngrams_simple(text, n=3):
    text = text.lower()
    return [text[i:i+n] for i in range(len(text)-n+1) if not " " in text[i:i+n]]

def sent2ngrams_regex(text, n=3):
    rgx = '(?=('+'\S'*n+'))'
    return re.findall(rgx,text)

# Generate 10000 random strings of length 100.
sents = [" ".join([''.join(random.choice(string.ascii_uppercase) for j in range(10)) for i in range(100)]) for k in range(100)]

start = time.time()
x = [sent2ngrams(i) for i in sents]
print time.time() - start        

start = time.time()
y = [sent2ngrams_simple(i) for i in sents]
print time.time() - start        

start = time.time()
z = [sent2ngrams_regex(i) for i in sents]
print time.time() - start  

print x==y==z

[fuera]:

0.0211708545685
0.0284190177917
0.0303599834442
True