Speech Recognition?

General chat related to ScummVM, adventure gaming, and so on.

Moderator: ScummVM Team

Post Reply
FlvZ2gWPdj9w71Fd97it
Posts: 5
Joined: Tue Jan 07, 2014 7:18 pm

Speech Recognition?

Post by FlvZ2gWPdj9w71Fd97it » Wed Jan 08, 2014 5:12 pm

Some older games had text-based interfaces. Would it be possible to use speech recognition with newer games too? Perhaps as a ScummVM add-on or feature?

Are there any speech recognition or standardized keyboard inputs/shortcuts for ScummVM or the supported games?

digitall
ScummVM Developer
Posts: 1013
Joined: Thu Aug 02, 2012 1:40 pm

Post by digitall » Wed Jan 08, 2014 9:12 pm

To be honest, this has been on my personal task list for a while... With SDL providing sound capture function, we should be able to add a simple voice command backend to emulate keyboard and mouse control... maybe based on CVoiceControl: http://www.kiecza.net/daniel/linux/

Not sure how functional the result would be, but could be an interesting experiment for someone... or maybe as a future GSoC task.

digitall
ScummVM Developer
Posts: 1013
Joined: Thu Aug 02, 2012 1:40 pm

Post by digitall » Wed Jan 08, 2014 9:13 pm

https://arjo129.wordpress.com/experiments/%C2%B5speech/ could also provide a lightweight basis for a limited vocabulary command recogniser...

User avatar
icanntspell
Posts: 95
Joined: Mon May 18, 2009 12:14 pm
Location: The Netherlands
Contact:

Post by icanntspell » Fri Jan 10, 2014 6:36 am

Android seems to have something build in for this.

http://developer.android.com/reference/ ... nizer.html

It sounds like a nice idea, but I don't see myself playing LSL using this :) At least not in public :D

User avatar
md5
ScummVM Developer
Posts: 2261
Joined: Thu Nov 03, 2005 9:31 pm
Location: Athens, Greece

Post by md5 » Fri Jan 10, 2014 9:37 am

Even if there is speech recognition, I doubt it'll be THAT useful, as you have to move around with the mouse anyway. It will save some typing, but all the mouse-related controls won't be affected.

digitall
ScummVM Developer
Posts: 1013
Joined: Thu Aug 02, 2012 1:40 pm

Post by digitall » Tue Jan 14, 2014 9:06 pm

I was mainly going to try doing this for general code twiddling fun, but I wondered about the utility for profoundly physically disabled gamers ie. using a http://tetramouse.com/ say...

I was intending to allow "MOUSE-HOLD. UP. DOWN. LEFT. RIGHT. SMALL. CLICK. RIGHT-CLICK. DRAG. RELEASE. MOUSE-RELEASE." as a commandset for controlling mouse movement and control by a limited vocab recogniser and then keyboard control by "KEYBOARD-ON. KEYBOARD-OFF" and a more general purpose recogniser and/or a way of voice-typing.

FlvZ2gWPdj9w71Fd97it
Posts: 5
Joined: Tue Jan 07, 2014 7:18 pm

Post by FlvZ2gWPdj9w71Fd97it » Sat Feb 15, 2014 9:57 pm

It's possible to control the mouse with things like VoiceFinger or Windows Speech Recognition macros. Though, my experience is that doing so is quite slow. "A New Beginning" does have a very helpful keyboard shortcut to show all hotspots (space bar). When used with WSR or VF it's possible to play without very slowly scanning the screen hunting for hotspots.

Given the limited vocabulary necessary for most adventure games it seems possible to leverage Windows Speech Recogition (on Windows) or other speech engines since the verbs, hotspot names, and objects should be known to ScummVM. Perhaps there could be a module or plugin system to work with different speech systems.

Even just adding keyboard input or shortcuts for verbs and objects would be a big help. Those can often be spoken using WSR or SphinxKeys.

AbleGamers is an organization with some great guidelines for tips to improve accessibility.

User avatar
DrMcCoy
ScummVM Developer
Posts: 596
Joined: Sat Dec 17, 2005 1:33 pm
Location: Braunschweig, Germany
Contact:

Post by DrMcCoy » Sat Feb 15, 2014 10:11 pm

FlvZ2gWPdj9w71Fd97it wrote:since the verbs, hotspot names, and objects should be known to ScummVM
That's highly game/engine specific. Some engines do, some probably don't. And there's nothing in ScummVM to abstract that, to bind the hotspot concepts of different engine together and pass it somewhere.

For example, for the gob games, ScummVM (or rather, the gob engine in ScummVM; the info doesn't get passed to any common code) only knows about general hotspots. When the mouse enters a hotspot, when it leaves a hotspot, or when a mouse button is clicked inside a hotspot, a script is triggered. Those scripts then do things like displaying the hotspot name. Opaquely, without the engine code knowing anything about that from a high-level perspective. The engine code doesn't know what those hotspot are, what they mean to the game.

Potentially, an engine could go even further: the scripts could just periodically query the current mouse position and state. Where the hotspots are could also be handled by the scripts. I don't know if any of our engines does that already, but it's certainly possible.

FlvZ2gWPdj9w71Fd97it
Posts: 5
Joined: Tue Jan 07, 2014 7:18 pm

Post by FlvZ2gWPdj9w71Fd97it » Mon Feb 17, 2014 3:26 am

Thanks for the quick reply. I didn't realize ScummVM had isolated engine code. Though, I don't think the problem is insurmountable. Creating such an abstraction to work with the most widely used engine would be my first step. Unfortunately my day job leaves my hands too tired to get into the source at this time.

Regardless, I look forward to whatever future accessibility improvements do get incorporated.

User avatar
md5
ScummVM Developer
Posts: 2261
Joined: Thu Nov 03, 2005 9:31 pm
Location: Athens, Greece

Post by md5 » Mon Feb 17, 2014 12:24 pm

FlvZ2gWPdj9w71Fd97it: We're happy for the work you're putting in ScummVM :) would you mind changing your nick to something that's readable by human beings? Thanks!

Serious Callers Only
Got a warning
Posts: 166
Joined: Thu Feb 25, 2010 7:44 am

Post by Serious Callers Only » Tue Feb 18, 2014 12:36 pm

Extracting that information from when it can be extracted into a common parser would be great indeed.

Serious Callers Only
Got a warning
Posts: 166
Joined: Thu Feb 25, 2010 7:44 am

Post by Serious Callers Only » Wed Feb 19, 2014 5:45 pm

Although it would probably require some expansion of older games primitive pathfinding systems maybe, to be used completely hands off. And then there are the puzzles that require positioning...
A invasive change to the wrong engines, although i wonder if the parser based AGI/SCI wouldn't be easier.

Post Reply