Meeting minimal InterCom

Contents

1 Description

InterCom is a desktop application that captures, sends, receives, and plays audio (and video), in real time.

2 Supported platforms

InterCom should run on any platform, such as Linux, Windows, and macOS, where Python and sounddevice are available. sounddevice is a wrapper for PortAudio [1], which is ultimately responsible for communicating with the audio driver (in Linux: ALSA, PulseAudio, JACK or Pipewire, in Windows: MME, DirectSound, WASAPI, or WDM-KS, in macOS: Core Audio).

3 Real-time audio processing alternatives

Using sounddevice (and in general, in all real-time audio-processing applications), we have two alternatives for implementing an application such as InterCom:

3.1 Loop-based implementation

Roughtly, InterCom can be divided into 6 steps:

1  # Loop-based algorithm (to be called in a loop) 
2  def record_IO_and_play(chunk_size): 
3    chunk = record(chunk_size) # (1) 
4    packed_chunk = pack(chunk) # (2) 
5    send(packed_chunk) # (3) 
6    packed_chunk = receive() # (4) 
7    chunk = unpack(packed_chunk) # (5) 
8    play(chunk) # (6)

where:

1. The record(chunk_size) method captures a chunk (a fragment) of frames¹. In sounddevice, this operation can be² carried on by the read() method. As can be find out in wire4.py³ and also in the documentation of sounddevice if we read only the frames that are available in the soundcard’s buffer, this generates a non-blocking operation and the chunk size depends on the instant of time in which this method is called. Otherwise, if we specify a number of frames bigger than the number of frames available, the operation will⁴ be blocking and I/O-bound (the calling process sleeps until the required chunk size is returned).
2. pack(chunk) process the chunk to create a packet (or a sequence of packets), a structure that can be transmitted through the Internet using the Datagram Model. In general, this is a CPU-bounded (CPU-intensive) operation because the payload of the packet can be compressed in order to minimize the transmission bit-rate.
3. send(packed_chunk) sends the packet to our interlocutor. When datagrams are used (which is our case), this step is neither non-blocking nor CPU-bound (the CPU usage is very low), as long as the number of packets/second is small and the sizes of the payloads are also small, as it is expected in InterCom (audio).
4. receive(), waits (blocking the calling process) for an incoming packet, and therefore, this operation is IO-bound. However, most socket APIs [2] offeer a non-blocking option where when a packet is not available in the kernel’s buffer associated with the corresponding socket after a predetermined amount of time, some kind of exception is generated, and, in this case, it is resposabability of the programmer to generate an “alternative” chunk (in our case, for example, a chunk filled with zeros that will not produce any sound when it is played).
5. unpack(packed_chunk) is (like the method pack(chunk)) a CPU-intensive step that transforms a possibly compressed chunk into a chunk of “playable” audio.
6. play(chunk) renders the chunk. In general, this is an I/O-bound blocking action. However, if play() is called at the same pace as record(), and the record and play parameters are exactly the same (as usually happens in InterCom), the playing of the chunk should return immediately because the time that play() needs to complete would match exactly the time that the record() method requires (see wire4b.py).

3.2 Timer-based algorithm

The to-be-called-in-a-loop implementation depicted in the previous section works fine, but finally InterCom uses a to-be-called-by-an-interruption implementation because it is allows to run another task in parallel for free.

In this algorithm, the task dedicated to record and play the chunks of audio is called periodically (probably, using some timer usually provided by the sound hardware). This procedure guarantees a gliches-free audio-IO when constant chunk-sizes are used, because the timer interruption coincides exactly with the instant of time in which the record() and the play() methods could be used without blocking.

The current implementation of InterCom uses the timer-based (interruption-based) algorithm. Therefore, you are not going to find the methods sounddevice.read() and sounddevice.write() in the minimal.py module.

4 Deliverables

You should understand the meaning, purpose and usage of all of the parameters of minimal.py. Do the following experiments. Report every thing in a document, and in the laboratory, show to your teacher the results (and the document):

1. Run minimial.py using localhost as the destination for your chunks of audio (notice that this is the default configuration for the parameters --destination_address and --destination_port). Check that you can listen to yourself after some delay, and that the quality of the sound is good (it is recommended to used a headset). If you think that the quality is not high enough, comment this with your teacher. Remember that the parameters --show_stats, --show_samples, and --show_spectrum, can provide some information about what is happening. Determine the delay.
2. Always using the command line (do not modify the source code for this), change the default value of the parameter --frames_per_chunk, which is 1024. Do you notice any variation in the latency? Determine it. And in the quality of the audio?
3. Modify the parameter --frames_per_second. Again, do you notice any changes in latency or audio quality? Determine the new latency. Determine also the maximum frequency that you will capture (depending on –frames_per_second).
4. With one of your group mates, try to communicate using the same LAN (you will need to modify the parameter --destination_address). Notice that some LANs, such as the provided by the UAL could filter your InterCom traffic. Good alternative options are you home LAN or an ad-hoc LAN created with a mobile (telephone). Determine the new latency.
5. Repeat the last experiment when two (or more) group mates transmit to you their chunks. What is happening?
6. Finally, are you able to communicate with an interlocutor that is in a different LAN than you (for example, when two InterCom instances are running in different home networks)? Describe the experiment, even if it does not work.

5 Resources