trigram-utils

Some trigram language statistics utility functions.

They have their own repo here to make sure wooorm/trigrams (trigram information for the Universal Declaration of Human Rights) and wooorm/franc (language detection) use the same cleaning and classification methods.

Installation

npm:

1

$ npm install trigram-utils

Component:

1

$ component install wooorm/trigram-utils

Bower:

1

$ bower install trigram-utils

Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

var trigramUtils = require('trigram-utils');

trigramUtils.clean(' t@rololol '); // 't rololol'

trigramUtils.trigrams(' t@rololol ');
// [ ' t ', 't r', ' ro', 'rol', 'olo', 'lol', 'olo', 'lol', 'ol ' ]

trigramUtils.asDictionary(' t@rololol ');
/*
 * {
 *   'ol ': 1,
 *   'lol': 2,
 *   'olo': 2,
 *   'rol': 1,
 *   ' ro': 1,
 *   't r': 1,
 *   ' t ': 1
 * }
 */

var tuples = trigramUtils.asTuples(' t@rololol ');
/*
 * [
 *   ['ol ': 1],
 *   ['rol': 1],
 *   [' ro': 1],
 *   ['t r': 1],
 *   [' t ': 1],
 *   ['lol': 2],
 *   ['olo': 2]
 * ]
 */

trigramUtils.tuplesAsDictionary(tuples.slice(-3));
/*
 * {
 *   'olo': 2,
 *   'lol': 2,
 *   ' t ': 1
 * }
 */

API

trigramUtils.clean(string)

Cleans a given string: strips certain (for language detection) useless punctuation, symbols, and numbers. Concatenated extraneous white space, trims, lowercases, and pads (with one space on both sides).

trigramUtils.trigrams(string)

Gets cleaned trigrams (see wooorm/n-gram).

trigramUtils.asDictionary(string)

Gets cleaned trigrams as a dictionary: the keys of the objects are trigrams, the values are occurrence counts.

trigramUtils.asTuples(string)

Gets cleaned trigrams with occurrence counts as a tuple: the first indice (0) being the trigram, the second (1) the occurrence count.

trigramUtils.tuplesAsDictionary(Array.\>)

Transforms an array of trigram–occurrence tuples (as returned by trigramUtils.asTuples(string)) as a dictionary (see trigramUtils.asDictionary(string));

License

MIT © Titus Wormer