Overview

First step - Setup the complete flow.

  • Using ffmpeg, listen to a port for the stream from python code. (server) ffmpeg -protocol_whitelist file,udp,rtp -i something.sdp -ar str(SAMPLE_RATE) -ac 1 -f s16le - -loglevel quiet

  • Using ffmpeg streaming hellohowareyou.mp3 file in loop to server to mimic mic input. (local) ffmpeg -stream_loop -1 -re -i /Users/harims/code/archived/vosk_code/voice_processing/input/audiofiles/hellohowareyou.mp3 -ar 8000 -f mulaw -f rtp rtp://<Server IP>:<Port>

  • Setup Mosquitto mqqt sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa sudo apt-get update sudo apt install mosquitto

  • From server push the identified sentence into mqqt.

      import paho.mqtt.client as mqtt

  def on_pre_connect(client,data):
      return

  # The callback for when the client receives a CONNACK response from the server.
  def on_connect(client, userdata, flags, rc):
      print("Connected with result code "+str(rc))

  client = mqtt.Client()
  client.on_pre_connect=on_pre_connect
  client.connect("localhost", 1883, 60)

  def sendtomqqt(data):
      client.publish("identifiedtext", data)

  • Subscribe to this message and invoke Python function to process.

      import paho.mqtt.client as mqtt
  def on_pre_connect(client,data):
      return
  # The callback for when the client receives a CONNACK response from the server.
  def on_connect(client, userdata, flags, rc):
      print("Connected with result code "+str(rc))

      # Subscribing in on_connect() means that if we lose the connection and
      # reconnect then subscriptions will be renewed.
      client.subscribe("identifiedtext")

  # The callback for when a PUBLISH message is received from the server.
  def on_message(client, userdata, msg):
      print(msg.topic+" "+str(msg.payload))

  client = mqtt.Client()
  client.on_pre_connect=on_pre_connect
  client.on_connect = on_connect
  client.on_message = on_message

  client.connect("localhost", 1883, 60)

  # Blocking call that processes network traffic, dispatches callbacks and
  # handles reconnecting.
  # Other loop*() functions are available that give a threaded interface and a
  # manual interface.
  client.loop_forever()
  • Setup NLP based systems like deeppavlov.
  • Get the output and push that to mqqt.
  • Subscribe to this message and execute functionality. Push the result code to mqqt back.

  • Subscribe to this message and invoke Python function to prepare response and generate sound file using TTS or Vosk-tts

      tts --text "Text for TTS" --model_name "tts_models/en/ek1/tacotron2" --vocoder_name "vocoder_models/universal/libri-tts/wavegrad" --out_path ~/sound.wav
  • Using ffmpeg stream it back to source.

To stream mic - ffmpeg -f avfoundation -i ":0" -acodec libmp3lame -ab 32k -ac 1 -f rtp rtp://192.168.1.148:8009