This guide will get you up and running with Spokestack for React Native, and you’ll be hearing and responding to your users in no time.

Prerequisites

We assume that you have a working react-native 0.60+ project (with npm and cocoapods already installed), and are adding Spokestack to that project. If you’re starting from scratch, please refer to the react-native getting started guide and then come back here!

iOS

iOS 13+, CocoaPods v1.6.0+.

Android

Android SDK 21+.

Installation

Using npm:

npm install --save react-native-spokestack

or using yarn:

yarn add react-native-spokestack

Then follow the instructions for each platform to link react-native-spokestack to your project:

iOS installation

platform :ios, '13.0'

target 'SpokestackExample' do
  use_frameworks!
  #...

  # X Remove or comment out these lines X
  use_flipper!
  post_install do |installer|
    flipper_post_install(installer)
  end
  # XX

$ npx pod-install

<key>NSMicrophoneUsageDescription</key>
<string>This app uses the microphone to hear voice commands</string>
<key>NSSpeechRecognitionUsageDescription</key>
<string>This app uses speech recognition to process voice commands</string>

#import <AVFoundation/AVFoundation.h>

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
  AVAudioSession *session = [AVAudioSession sharedInstance];
  [session setCategory:AVAudioSessionCategoryPlayAndRecord
     mode:AVAudioSessionModeDefault
  options:AVAudioSessionCategoryOptionDefaultToSpeaker | AVAudioSessionCategoryOptionAllowAirPlay | AVAudioSessionCategoryOptionAllowBluetoothA2DP | AVAudioSessionCategoryOptionAllowBluetooth
    error:nil];
  [session setActive:YES error:nil];

  // ...

Android installation

// ...
  ext {
    // Minimum SDK is 21
    minSdkVersion = 21
// ...
  dependencies {
    // Minimium gradle is 3.0.1+
    // The latest React Native already has this
    classpath("com.android.tools.build:gradle:3.5.3")

    <!-- For TTS -->
    <uses-permission android:name="android.permission.INTERNET" />
    <!-- For wake word and ASR -->
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    <!-- For ensuring no downloads happen over cellular, unless forced -->
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />

Configuration

There are many options for configuring Spokestack. This example begins capturing audio when Spokestack.start() is called and uses a Voice Activity Detection (VAD) component to detect speech and send it through an automated speech recognition system, in this case the platform’s built-in speech recognizer. In other words, we’re configuring this app to always actively listen, and no wake word detection is performed. See the configuration guide for more information about pipeline building options. Beginning in version 4.0.0, it’s easy to change the pipeline configuration through the use of the profile property.

We’ll configure the NLU (Natural Language Understanding) component, which uses Spokestack models. To test things out, try one of our free NLU models on your Spokestack account page, such as our example Minecraft NLU model, model metadata, and NLU vocabulary.

First, we’ll need a Spokestack API key and secret, available for free once you’ve created an account. Create a key in the account section.

import Spokestack from 'react-native-spokestack'

/**
 * Initialize the Spokestack speech pipeline.
 *
 * This example configures a voice-triggered speech recongnizer
 * For all configuration options, see https://github.com/spokestack/react-native-spokestack
 * For more examples of speech recognizer configuration, see https://github.com/spokestack/spokestack-android#configuration
 *
 * The first 2 arguments are your Spokestack credentials
 * available for free from https://spokestack.io.
 * Avoid hardcoding these in your app.
 * There are several ways to include
 * environment variables in your code.
 *
 * Using process.env:
 * https://babeljs.io/docs/en/babel-plugin-transform-inline-environment-variables/
 *
 * Using a local .env file ignored by git:
 * https://github.com/goatandsheep/react-native-dotenv
 * https://github.com/luggit/react-native-config
 */
Spokestack.initialize(
  process.env.SPOKESTACK_CLIENT_ID,
  process.env.SPOKESTACK_CLIENT_SECRET,
  {
    pipeline: {
      // The default is PTT_NATIVE_ASR
      profile: Spokestack.PipelineProfile.VAD_NATIVE_ASR
    },
    nlu: {
      // These are only required if using Spokestack.classify
      model: require('./nlu.tflite'),
      metadata: require('./metadata.json'),
      vocab: require('./vocab.txt')
    }
  }
)

Including model files in your app bundle requires a small change to your React Native bundler config. See this guide for more info.

Hearing your users

Call Spokestack.start() to begin listening for speech. Spokestack.stop() will stop listening completely.

To manually start speech recognition, without waiting for activation from a wake word or voice trigger, call Spokestack.activate() after calling Spokestack.start().

Call Spokestack.deactivate() to stop speech recognition. The speech pipeline will continue listening for a wake word or voice trigger, if applicable.

// Start and stop the speech pipeline. All methods can be called repeatedly.
await Spokestack.start() // start speech pipeline. can only start after initialize is called.
await Spokestack.stop() // stop speech pipeline
await Spokestack.activate() // manually activate the speech pipeline. The speech pipeline is now actively listening for speech to recognize.
await Spokestack.deactivate() // manually deactivate the speech pipeline. The speech pipeline is now passively waiting for an activation trigger.

Listen to speech events to receive the speech recognition results, be informed of errors and debugging events, or get notified when the pipeline activates or deactivates. For example, you may want to disable any buttons or show a special “listening” indicator while recording.

componentDidMount() {
  Spokestack.addListener('recognize', ({ transcript }) => console.log(transcript))
  Spokestack.addListener('activate', () => console.log('activated'))
  Spokestack.addListener('deactivate', () => console.log('deactivated'))
  Spokestack.addListener('error', ({ error }) => console.error(error))
  Spokestack.addListener('trace', ({ message }) => console.error(message)) // For debugging
}

// Remember to remove listeners when they are no longer needed
componentWillUnmount() {
  Spokestack.removeAllListeners()
}

Understanding your users

The "recognize" event will give you the raw text of what the user just said via the transcript property. Translating that utterance into an action in your app is the job of an NLU component. Spokestack leaves the choice of NLU up to you, but we do offer our own full-featured NLU component for Spokestack based on years of research and lessons learned from working with other services. Our NLU runs directly on your user’s device, instead of calling back to the cloud. Call Spokestack.classify() to use your NLU.

const result = await Spokestack.classify(utterance)
console.log(result)
// { "intent": "request.lights.activate", "slots": [{ "location": { "value": "room", "type": "selset" } }], "confidence":" 0.9999888" }

Classification results are in a map containing an intent. An intent-based classifier will regularize all sorts of related language into a single canonical intent, e.g. “let’s go” and “please cease” get classified as start and stop, respectively. The slots are the data, or parameters on which your app can take action. See our NLU guide for information on NLU in general and Spokestack’s implementation.

Talking back to your users

If you want full hands-free and eyes-free interaction, you’ll also want to deliver responses via voice. This requires a text-to-speech (TTS) component, and Spokestack has one of these too! Call it with the text you want to synthesize, and it will send you an event with an audio file containing the synthesized voice speaking your text utterance.

// Get a URL to a mp3 audio file of the processed speech
const url = await Spokestack.synthesize(utterance)
audioPlayer.play(url)

Spokestack includes one free voice. An upgraded account is needed for custom voices. For more information on TTS and details about controlling pronunciation, see the TTS concept guide.

Conclusion

That’s all there is to it! Your app is now configured to accept and respond to voice commands. Hopefully, this guide gives you an idea of the power of Spokestack. Spokestack has everything you need to start building rich voice experiences in your apps in an elegant way. If you’re interested in advanced topics (such as configuring the pipeline’s sensitivity or adding your own custom wake word), check out our other Concepts and Design guides.