Android Deep Dive

Into the depths of the Android SDK

In this section, we will look at some of the advanced features provided by Slang's Android SDK and how they can be used by apps to get the most out of the Slang platform.

Advanced Actions

Asynchronous handling of intent actions

Normally when handling an intent, the "action" method is expected to be synchronous, i.e., Slang expects the action to be completed when the method returns. And based on the return value (SUCCESS or FAILURE) it will trigger the Completion prompt as described below. But if the action needs to perform asynchronous processing (such as a network fetch) before the action can be completed, it can make use of the session helper methods - waitForActionCompletion & notifyActionCompleted to yield control back to Slang and asynchronously notify it of the completion status.

SlangBuddyOptions options = new SlangBuddyOptions.Builder()
    .setIntentAction(new SlangIntentAction() {
        @Override
        public Status action(SlangIntent slangIntent, final SlangSession slangSession) {
            switch (intent.getName()) {
                case "roaming":
                // Inform Slang to expect async operations
                slangSession.waitForActionCompletion();
        
                // Start a background thread
                new Thread(new Runnable() {
                    @Override
                    public void run() {
                        // Notify Slang its done
                        slangSession.notifyActionCompleted(SUCCESS);
                        // Completion prompt would be spoken out if any has
                        // been configured
                    }
                }).run();
                
                // return back and indicate SUCCESS but at this point
                // the completion prompt wont be spoken
                return SUCCESS;
                
            ... 
        }
    })
    ... // rest of the SlangBuddyOptions    

Customizing Prompts

Completion Prompts

When an intent is detected by Slang, it will call the "action" method of the registered SlangIntentAction implementation. After the "action" method is done with its handling, Slang will speak out the Completion prompt associated with this intent if the handler returns SUCCESS.

private static class MyIntentHandler implements SlangIntentAction {
    @Override
    public Status action(SlangIntent intent, SlangSession session) {
        switch (intent.getName()) {
            case "balance":
                // app specific logic to handle this intent
                if (handle_balance(intent)) {
                    // This will trigger the "completion" prompt
                    // of the "balance" intent as configured in the
                    // Slang console (eg: "Showing your balance")
                    return SUCCESS;
                } else {
                    // This will suppress the "completion" prompt 
                    // of the "balance" intent
                    return FAILURE;
                }                
                ...
        }
    }
}

By default it will use the Completion prompt configured in the Slang Console for that intent. The "action" method can override the prompt if necessary.

            case "balance":
                // app specific logic to handle this intent
                if (handle_balance(intent)) {
                    // This will trigger the "completion" prompt
                    // of the "balance" intent
                    
                    // Override the prompt
                    List<String> prompts = new ArrayList<String>();
                    
                    prompts.add("Here is your balance");
                    slangIntent.setCompletionStatement(
                        new SlangStatement(prompts, null)
                    );                    
                    return SUCCESS;
                } else {
                    // This will suppress the "completion" prompt 
                    //of the "balance" intent
                    return FAILURE;
                }                
                ...

Setting the Greeting and Clarification prompt

By default Slang will show a prompt on its surface when the Slang Trigger is pressed ("Hi! How may I help you"). Similarly when Slang does not understand what the user is saying, it will ask for the user to retry with a default prompt ("Sorry, I didn't understand that, could you please say it again?). These prompts can be overridden in the Slang Console.

Prompting for missing information

Sometimes, when a user speaks an utterance that corresponds to an intent, it is possible that some important information could be left out of the utterance. For example, for flight ticket booking intent, the required pieces of information would be the source city, the destination city and the date of travel. However, the user could end up speaking an utterance with incomplete information, like this:

"Show me flights from Bangalore to Delhi"

In this case, the user has not specified the date of travel, which is required information for processing the intent and the app should prompt the user to provide the missing information. There are two ways in which this can be achieved through Slang:

1. Configure entity prompts in Slang Console (Recommended): In this method, the app's configuration in the Slang Console should contain appropriate messages to prompt the user for each entity. You can configure these messages for each entity under the Prompts section in the Console as shown below.

These values can be accessed in the app and used to prompt the users to provide the missing information in the following way:

public Status action(SlangIntent intent, SlangSession session) {
    SlangEntity startDate = intent.getEntity("start_date");
    if (!startDate.isResolved()) {
        intent.setCompletionStatement(startDate.getStatement());
    }
    return SUCCESS;
}

2. Override completion statement fully within the app: With this method, the app can fully override the completion statement without relying on the app configuration in Slang Console. Here is how this can be done:

public Status action(SlangIntent intent, SlangSession session) {
    SlangEntity startDate = intent.getEntity("start_date");
    if (!startDate.isResolved()) {
        intent.setCompletionStatement(getStartDateStatement());
    }
    return SUCCESS;
}

private SlangStatement getStartDateStatement() {
    HashMap <Locale, String> localeStmts = new HashMap<>(1);
    localeStmts.put(SlangLocale.LOCALE_ENGLISH_IN, "Please specify date of travel");
    // Add statements for all locales supported by the app
    return SlangMessage.create(localeStmts);
}

Error Handling

Handling Slang errors

When Slang is initialized in the application (typically in the onCreate method the Application class), the SDK performs a handshake with our servers and verifies that the Buddy credentials are valid and then retrieve details about the Buddy. But if things go wrong during this process (eg the Buddy credentials are invalid or the network is unreachable), then the app needs to be notified of this. This is done via the SlangBuddy.Listener handler. To register the Listener, instantiate one and pass it to the SlangBuddyOptions at initialization time.

SlangBuddyOptions options = new SlangBuddyOptions.Builder()    
    .setListener(new SlangBuddy.Listener() {
        @Override
        public void onInitialized() {
             // Slang has been successfully initialized               
        }

        @Override
        public void onInitializationFailed(SlangBuddy.InitializationError initializationError) {
            // Slang failed to initialize.
        }
    })
    ... // rest of the SlangBuddyOptions

Handling unresolved utterances

By default, Slang processes all user utterance and attempts to detect the intent. On successful intent detection, it calls the "action" method of the registered SlangIntentAction instance. For utterances that did not resolve into intents, Slang will ask the user to retry. But sometimes the app may want to get notified about these unresolved utterances (and also all utterances). To get notified, pass an instance of the SlangUtteranceAction class to the SlangBuddyOptions at initialization time.

SlangBuddyOptions options = new SlangBuddyOptions.Builder()
    .setUtteranceAction(new SlangUtteranceAction() {
        @Override
        public void onUtteranceDetected(String userUtterance, SlangSession slangSession) {
            // User spoke something. Slang has still not attempted to
            // resolve it at this point                      
        }

        @Override
        public void onUtteranceUnresolved(String s, SlangSession slangSession) {
            // User spoke sometheing. Slang could not detect any intent            
        }
    })
    ... // rest of the SlangBuddyOptions

Customizing the UI

Changing the spoken voice (Talkback)

By default the voice spoken is a female voice that best suits the current locale

. To change the voice to a male voice instead, use the below API

    // Pass the options needed for Slang
    SlangBuddyOptions options = new SlangBuddyOptions.Builder()
        .setTalkbackVoice(TalkbackVoice.DEFAULT_MALE)
        ... // rest of the options
        .build();

Turning on/off Slang Trigger

Sometimes it might be required to turn off Slang (for example, when in the login page). To do this, you can use the "show/hide" APIs. Note that it's important to make sure that Slang has initialized before calling these APIs.

if (SlangBuddy.isInitialized()) {
    SlangBuddy.getBuiltinUI().hide();
}

if (SlangBuddy.isInitialized()) {
    SlangBuddy.getBuiltinUI().show(MyActivity.this);
}

Customizing default Slang Trigger/Surface

By default, Slang uses a green theme for its assets and use a green mic button and Slang branded icon. Use the SlangBuiltInUI object (which can be retrieved using SlangBuddy.getBuiltinUI()) to customize it.

Note that some of these customizations might be available only under the paid plan. In such cases, Slang will throw a SlangBuddy.InsufficientPrivilegeException if used under free plans.

// To change background color of surface
SlangBuddy.getBuiltinUI().setBackgroundColor(R.color.warm_grey);

// To change the color of the text on the surface
SlangBuddy.getBuiltinUI().setTextColor(R.color.white);

// To change the image for the trigger
SlangBuddy.getBuiltinUI().setImageResource(R.drawable.your_image);

// To set the position of the trigger
SlangBuddy.getBuiltinUI().setPosition(
    SlangBuiltinUI.SlangTriggerPosition.LEFT_BOTTOM,
    <offset X>,
    <offset Y>,
    false    
);

// To prevent the trigger from being draggable. Default is true
SlangBuddy.getBuiltinUI().setIsDraggable(false);

// To not show the slang banner
SlangBuddy.getBuiltinUI().hideSlangBanner();

Implementing a fully custom UI

In certain scenarios, you may wish to bypass Slang's built-in UI completely. For example,

• To build a fully customized voice experience for a more seamless integration with the app

• To run Slang in a "headless" mode and directly access Slang's NLU capabilities, such as with a "master" app that drives other apps through deep-links and broadcast intents

In such scenarios, you could bypass Slang's built-in UI by writing a UI provider class that implements the SlangUIProvider interface and passing it to SlangBuddyOptions at initialization time.

Slang will invoke the various UI callbacks in the UI provider class based on its state (listening, utterance detected, etc). In turn, it's the responsibility of the UI provider class to notify Slang about user events (start speech recognition, cancel speech recognition, etc) through the SlangUIDelegate object that's available via the SlangUIProvider.onCreate() method.

SlangBuddyOptions options = new SlangBuddyOptions.Builder()
        .setUIProvider(new SlangUIProvider() {
            @Override
            public void onDisabled(SlangException e) {
                // When Slang is not enabled (eg API key invalid)
            }

            @Override
            public void onCreate(Activity activity, SlangUIDelegate slangUIDelegate) {
                // When Slang has been initialized successfully (eg when the
                // app starts)
                
                // Use the SlangUIDelegate object to inform Slang about 
                // various events
            }

            @Override
            public void onUserSessionStarted() {
                // A new Slang session is going to start
            }

            @Override
            public void onListeningStarted() {
                // Slang is listening to the user
            }

            @Override
            public void onUserTextAvailable(String s, Locale locale) {
                // Slang has detected user speech. The locale of the current
                // language will be passed to it
            }

            @Override
            public void onListeningEnded() {
                // Slang is no longer listening (end of speech detected)                
            }

            @Override
            public void onListeningTimedOut() {
                // Slang timedout when listening for user speech 
                // (ie no speech was detected)
            }

            @Override
            public void onListeningError(String s, Locale locale) {
                // Some error occurred while listening 
            }

            @Override
            public void onSlangProcessingStarted() {
                // Slang is now processing the detected utterance
            }

            @Override
            public void onSlangProcessingEnded() {
                // Slang is done processing the detected utterance
            }

            @Override
            public void onSlangProcessingError(String s, Locale locale) {
                // Slang hit an error when processing the utterance
            }

            @Override
            public void onSlangTextAvailable(String s, Locale locale) {
                // Slang wants to say something to the user. The text that
                // Slang wants to say is available in the parameter
            }

            @Override
            public void onUserSessionEnded() {
                // Slang session ended
            }

            @Override
            public void onDismissed() {
                // The UI needs to be dismissed
            }

            @Override
            public void onHelpRequested() {
                // Slang noticed that the user requested for "help"
            }

            @Override
            public void onHelpRequested(Set<String> intentsFilter) {
                // Slang noticed that the user requsted for "help" and
                // the provider should restrict the help strings to those 
                // of the the given intents.
            }

            @Override
            public void onDestroy() {
                // Slang is no longer initialized
            }
        })
        ... // Rest of the SlangBuddyOptions

The UI provider class should notify Slang about user events at appropriate times to drive the life-cycle correctly.

public class MyUIProvider implements SlangUIProvider {
    private SlangUIDelegate mUIDelegate;
    
    @Override
    public void onCreate(Activity activity, SlangUIDelegate slangUIDelegate) {
        mUIDelegate = slangUIDelegate;
    }
    
    // ... rest of the SlangUIProvider methods
    
    public void onStartButtonClicked() {
        mUIDelegate.notifyUserStartedSession();
    }
    
    public void onCancelButtonClicked() {
        mUIDelegate.notifyUserCanceled();
    }
    
    public void onBackButtonPressed() {
        mUIDelegate.notifyUserPressedBackButton();
    }
    
    public void onLocaleChanged() {
        Locale currentLocale = getCurrentLocale();
        mUIDelegate.notifyUserChangedLocale(currentLocale);
    }
}

Others

Detecting the current locale

Sometimes, the app may need to know the current locale that Slang is using to perform it's processing. The current locale can be queried through the SlangBuddyContext.getCurrentLocale() API.

Locale currentLocale = SlangBuddy.getContext().getCurrentLocale();

Getting access to Current Activity

When processing an intent, the handler might sometimes need to get access to the current activity on the screen. The handler could potentially make different decisions based on what screen is currently active. Slang provides a convenience method to get access to the current activity using the SlangSession object

SlangBuddyOptions options = new SlangBuddyOptions.Builder()
    .setIntentAction(new SlangIntentAction() {
        @Override
        public Status action(SlangIntent slangIntent, SlangSession slangSession) {
           Activity currentActivity = slangSession.getCurrentActivity();
           // rest of the processing
           ...
           
           // Inform Slang you handled the intent successfully
           return SUCCESS;
        }
    })

Staging and Production Environment

When creating and configuring a Buddy in the Slang Console, one can save the Buddy into the "Staging" environment or "Production" environment. By default, Slang will attempt to use the Buddy configuration that is available in the "Production" environment. But if one is trying to test the configuration in "Staging", they need to let Slang know it should use that.

SlangBuddyOptions options = new SlangBuddyOptions.Builder()
    .setEnvironment(Environment.STAGING)
    ... // rest of the SlangBuddyOptions

Refer to the Building Your First Buddy section under the Console page for details of how to save the Buddy configuration to Staging and Production environments.