Detect duplicate calls and do not stream #164

davidmckenzie · 2017-01-30T02:34:17Z

Low priority feature request :)

My local fire agency does some weird things with channel patching - my neighbouring districts conventional P25 channels are often patched together during quiet times. If I was to leave both channels enabled then I would get double calls on the streaming server. They also sometimes do this with trunked talkgroups in another district, however they don't seem to use the native channel patching thing like other agencies do.

As a potential solution, could the system look at the LDUs? If more than one channel is currently active, monitor the LDUs over a short period. If more than n voice LDUs are identical to the other active channel, stop playing the audio and delete the temporary stream recording on one channel, with a log event added.

DSheirer · 2017-01-30T13:27:26Z

I've been thinking about this issue for large, multi-site systems. In order to get maximum monitoring coverage over the system, you have to monitor the control channel for each site, but that produces duplicate calls when the same call is broadcast on both sites.

I think I need to implement a System-level call deconfliction module. That way, when you have two or more sites from the same system decoding, the system-level module decides which site will monitor the call and then the other sites could simply ignore the call.

Do you think this type of enhancement would solve the issue you're seeing?

davidmckenzie · 2017-01-30T19:47:11Z

That's a solid idea, definitely worth pursuing. Wouldn't solve the issue I'm facing specifically, but it's a very fringe case and not likely to be an issue for anyone else (hence the super low priority). The duplicate calls in my scenario are technically across multiple systems, rather than just multiple sites.

I could add all of the conventional channels into the same system in the config, but I'm not sure how it would behave since the individual sites know nothing of each other.

E.g. if you had two calls start simultaneously on different channels/sites that weren't actually patched, but both had a TGID of '0001'.

It's a tough problem to solve, and I wouldn't want you to get bogged down solving a problem that likely only affects me, so I'd say absolutely go with the more logical solution to duplicate calls. Would be a quicker win and would improve the app for a lot more people.

…s. Decreases periodicity of processing the LibUsb timeout handler from 9 milliseconds to 5 milliseconds to reduce chance that buffers are not being serviced in time.

ImagoTrigger · 2017-02-05T17:02:38Z

I'm seeing this issue or something similar as well.

Using the latest version (6bc92e8) the patch group calls are now being streamed (broadcasted) multiple times, whereas before (6a9e522) it seemed the alias was still not being pulled correctly and the patch group call would not be streamed at all, let alone duplicated.

As hinted by David, only patch groups setup a certain way (hard patch / soft patch / etc) exhibit this problem.

10:51:30.276 INFO m.d.p25.P25DecoderState - Add patch group NAC:402 TSBK2 MOTOROLA ADD PATCH GROUP 2CFE [2CFE, 2D00]
10:51:30.511 INFO m.d.p25.P25DecoderState - Add patch group NAC:402 TSBK2 MOTOROLA ADD PATCH GROUP 0322 [032A, 0322]
10:51:30.574 INFO m.d.p25.P25DecoderState - Add patch group NAC:402 TSBK2 MOTOROLA ADD PATCH GROUP 2CFE [2D00, 2CFE]
10:51:30.772 INFO m.d.p25.P25DecoderState - Add patch group NAC:402 TSBK1 MOTOROLA ADD PATCH GROUP 0322 [0322, 032A]

It's really been bothering me ;-) I can provide access to the machine if needed.

ImagoTrigger · 2017-02-05T17:14:25Z

After the patch groups are deleted (when they "split the channels" every day at 11AM) the calls are still streamed multiple times until the SDRTrunk application is restarted.

DSheirer · 2017-02-08T01:53:10Z

I've been thinking about this for the past 2 days. I think that a good solution may be to timestamp the patch groups when they are created and update the timestamp each time we receive a Patch Group Add message, indicating that the patch is still valid. When a patch is in effect, there seems to be a stream of several messages per second while the patch group is active.

Before each channel grant/call, I'll check all of the patches registered in the decoder and remove any patches that haven't been updated in the previous 60 seconds. By removing the patch before I allocate a channel for the call event, there's little chance that a legacy patch group would be erroneously matched to a call event.

Right now, I'm depending on receiving the patch group delete message before I tear down a patch registered with the decoder.

Thoughts?

I'm still not sure what may be causing the streaming duplications. My thought is that there are two channels decoding the same call or channel, producing 2 recordings, which get streamed.

Can either of you log and capture the Call Event Log when this occurs and see if there is more than 1 call event (probably on different channels/frequencies) for a given call and talkgroup? That'll help me to narrow down where the issue is.

Denny

davidmckenzie · 2017-02-08T02:23:21Z

Yeah the systems I'm monitoring don't patch the TGs through any system level function - it's like they just plug in a repeater. This is likely due to it being a hybrid system of both conventional P25 towers and a trunked P25 network.

A single call to one of these "patched" channels might show up as a "Patch Group Call" or just a "Group Call" on the trunked tower to a specific TGID, and the conventional towers will show a "Group Call" with either no TGID in the logs, or with the standard TGID of 0001.

TGIDs that are patched together in the one trunked system work fine, it's just when they patch conventional channels with each other or with TGIDs that things get freaky.

ImagoTrigger · 2017-02-08T17:10:56Z

thank you for looking into this!

…

On Feb 7, 2017 7:53 PM, "Denny Sheirer" ***@***.***> wrote: I've been thinking about this for the past 2 days. I think that a good solution may be to timestamp the patch groups when they are created and update the timestamp each time we receive a Patch Group Add message, indicating that the patch is still valid. When a patch is in effect, there seems to be a stream of several messages per second while the patch group is active. Before each channel grant/call, I'll check all of the patches registered in the decoder and remove any patches that haven't been updated in the previous 60 seconds. By removing the patch before I allocate a channel for the call event, there's little chance that a legacy patch group would be erroneously matched to a call event. Right now, I'm depending on receiving the patch group delete message before I tear down a patch registered with the decoder. Thoughts? I'm still not sure what may be causing the streaming duplications. My thought is that there are two channels decoding the same call or channel, producing 2 recordings, which get streamed. Can either of you log and capture the Call Event Log when this occurs and see if there is more than 1 call event (probably on different channels/frequencies) for a given call and talkgroup? That'll help me to narrow down where the issue is. Denny — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#164 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADLfRBxEKrg8poBLsqicy-A7oc0131pPks5raSAIgaJpZM4LxAHE> .

davidmckenzie · 2017-02-08T21:15:19Z

FWIW - my workaround for this is to have multiple streams in Broadcastify. Not ideal, but at least I'm not getting any more feed reports of dupe messages :D

DSheirer · 2017-02-11T00:40:40Z

@davidmckenzie @ImagoTrigger If you both have a chance, could you please confirm if this issue of duplicate streaming calls is resolved with the latest beta 3 code?

davidmckenzie · 2017-02-11T01:00:33Z

It is for the erroneous duplicate calls, but not for the "by design" duplicate calls that happen due to manual patching of conventional channels. I'm pretty happy so far with my workaround of splitting the two conventional channels to different Broadcastify streams, so would be cool with this being closed :)

ImagoTrigger · 2017-02-11T18:42:27Z

I wrote a program to confirm the recordings were not getting duplicated in any output other then when streaming (from SDRTrunk).

use strict;
use Win32::MediaPlayer;
use Time::HiRes qw(usleep);
use Win32API::File 0.08 qw( :ALL );
use Win32::GUI;
 
 my $sdrdir = "R:/scanner/SDRTrunk/recordings/";
 mkdir $sdrdir;
 
my $winmm = new Win32::MediaPlayer;

my $main = Win32::GUI::Window->new(
    -name    => "Main",
    -title   => "Win32-GUI:Test",
    -left    => 100,
    -top     => 100,
    -width   => 1,
    -height  => 1,
  );

  sub Main_Terminate() {
    print "Main window terminated\n";
    return -1;
  }


 	$main->Show();
  	$main->Text("Starting...");
  	my %armer = ();
	open(ARMER,"R:/scanner/armer2.txt");
	my @rows = <ARMER>;
	close ARMER;
	foreach my $row (@rows) {
		my ($hex,$alpha) = split(/\t/,$row);
		$hex = sprintf("%04X",$hex);
		$armer{$hex} = $alpha;
	}
  	my %radios = ();
	open(ARMER,"R:/scanner/radios.txt");
	my @rows = <ARMER>;
	close ARMER;
	foreach my $row (@rows) {
		if ($row =~ /(\d+?) (.*)/) {
			my $hex = sprintf("%06X",$1);
			$radios{$hex} = $2;
		}
	}	

while(1) {
	my %files = get_sorted_files("$sdrdir");
	foreach my $key (sort{$files{$a} <=> $files{$b}} keys %files) {
		next if ($key eq "$sdrdir." || $key eq "$sdrdir..");

		#make sure SDRTrunk is done writing the .wav
		my $handle = CreateFile( "$key", GENERIC_ALL()|FILE_APPEND_DATA(),FILE_SHARE_READ()|FILE_SHARE_DELETE(), [], OPEN_EXISTING(), 0, [] ) or next;
	 	CloseHandle( $handle);
	 	
	 	print "Playing $key\n";
	 		my $meta = $key;
	 		my $to; my $from = "?";
	 		if ($key =~ /1_2_TRAFFIC__(.*).wav/) {
	 			$meta = $1;
	 			if ($meta =~ /TO_(.*?)_/) {
	 				$to = $armer{$1};
	 				chomp $to;
	 				$to = $1 if (!$to);
	 			}
	 			if ($meta =~ /FROM_(.*?)_/) {
 					$from = $radios{$1};
	 				chomp $from;
	 				$from = $1 if (!$from);
	 			}
	 		}
	 		
			$main->Text("TO:$to FROM:$from");
			my $wut = $winmm->load($key);    
			$winmm->play;     
			#wait for the file to play
       			my $max = -1;
			while (1) {
				my $lastmax = 0;
				my $pos = $winmm->pos;
				if ($max < $pos) {
					$max = $pos;
					$lastmax = 1;
				}
				last if (!$lastmax);
				usleep 65000;
			};
    			$winmm->close;
			unlink $key; #deletes the .wav
	}
	$main->Text("Idle...");
	usleep 100000;
};

Win32::GUI::Dialog();

sub get_sorted_files {
   my $path = shift;
   opendir my($dir), $path or die "can't opendir $path: $!";
   my %hash = map {$_ => (stat($_))[9] || undef} # avoid empty list
           map  { "$path$_" }
           readdir $dir;
   closedir $dir;
   return %hash;
}

confirmed.

ImagoTrigger · 2017-02-11T18:44:29Z

I will try building the latest code now and let you know. Thanks!!!!!!

ImagoTrigger · 2017-02-12T14:30:41Z

seems to NOT be fixed!

…

On Fri, Feb 10, 2017 at 7:00 PM, davidmckenzie ***@***.***> wrote: It is for the erroneous duplicate calls, but not for the "by design" duplicate calls that happen due to manual patching of conventional channels. I'm pretty happy so far with my workaround of splitting the two conventional channels to different Broadcastify streams, so would be cool with this being closed :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADLfRPebVGVSey4ZzBAI786Zj6B90eXnks5rbQgygaJpZM4LxAHE> .

ImagoTrigger · 2017-02-12T15:01:37Z

actually no. still streaming patch group calls more than once. ;-( also seeing more patch group adds in the event list than before.

.. this issue goes away for a while when i restart sdrtrunk...

.. does not happen during off-peak hours ("channels are merged")

..seems to be improved in beta3-4...

..makes it hard to reproduce...

DSheirer · 2017-02-12T15:27:30Z

I fixed an issue where (#1) calls were being streamed multiple times to the same channel because the call contained 2 or more aliases and each of the aliases had the same stream specified.

However, the issue of (#2) detecting duplicates when you're decoding the same call across two separate channels is still unresolved.

If the duplicate streamed calls issue you're still seeing #1 or #2?

ImagoTrigger · 2017-02-12T15:36:10Z

#1 - it was repeating just the patched group traffic when i listened this morning after sdrtrunk was running fine for 12+ hours. the issue is improved because now if i restart sdrtrunk it goes away. i believe there still exists an issue when the actual moment the patch group is setup and taken down on the site.

DSheirer · 2017-02-12T15:48:37Z

Ok, just confirmed it's still an issue. Working on a fix.

DSheirer · 2017-02-12T16:02:02Z

@ImagoTrigger I just merged a fix to master ... could you please try this and see if the duplicate calls on the stream issue is resolved? thanks!

ImagoTrigger · 2017-02-12T21:54:11Z

Will do.

ImagoTrigger · 2017-02-13T17:23:17Z

it is fixed. i ran it thru the setup and tear down of the patch groups at 4am and 11am with no dupes. excellent work.

DSheirer · 2017-02-14T01:35:12Z

Thanks for verifying this!! I'll close out this issue.

davidmckenzie mentioned this issue Feb 6, 2017

Support stereo streaming #175

Open

DSheirer added this to the Build 0.3.0 milestone Feb 11, 2017

DSheirer added bug medium labels Feb 11, 2017

DSheirer closed this as completed Feb 14, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect duplicate calls and do not stream #164

Detect duplicate calls and do not stream #164

davidmckenzie commented Jan 30, 2017

DSheirer commented Jan 30, 2017

davidmckenzie commented Jan 30, 2017

ImagoTrigger commented Feb 5, 2017

ImagoTrigger commented Feb 5, 2017

DSheirer commented Feb 8, 2017

davidmckenzie commented Feb 8, 2017 •

edited

Loading

ImagoTrigger commented Feb 8, 2017 via email

davidmckenzie commented Feb 8, 2017

DSheirer commented Feb 11, 2017

davidmckenzie commented Feb 11, 2017

ImagoTrigger commented Feb 11, 2017 •

edited

Loading

ImagoTrigger commented Feb 11, 2017 •

edited

Loading

ImagoTrigger commented Feb 12, 2017 via email •

edited

Loading

ImagoTrigger commented Feb 12, 2017 •

edited

Loading

DSheirer commented Feb 12, 2017

ImagoTrigger commented Feb 12, 2017

DSheirer commented Feb 12, 2017

DSheirer commented Feb 12, 2017

ImagoTrigger commented Feb 12, 2017

ImagoTrigger commented Feb 13, 2017

DSheirer commented Feb 14, 2017

Detect duplicate calls and do not stream #164

Detect duplicate calls and do not stream #164

Comments

davidmckenzie commented Jan 30, 2017

DSheirer commented Jan 30, 2017

davidmckenzie commented Jan 30, 2017

ImagoTrigger commented Feb 5, 2017

ImagoTrigger commented Feb 5, 2017

DSheirer commented Feb 8, 2017

davidmckenzie commented Feb 8, 2017 • edited Loading

ImagoTrigger commented Feb 8, 2017 via email

davidmckenzie commented Feb 8, 2017

DSheirer commented Feb 11, 2017

davidmckenzie commented Feb 11, 2017

ImagoTrigger commented Feb 11, 2017 • edited Loading

ImagoTrigger commented Feb 11, 2017 • edited Loading

ImagoTrigger commented Feb 12, 2017 via email • edited Loading

ImagoTrigger commented Feb 12, 2017 • edited Loading

DSheirer commented Feb 12, 2017

ImagoTrigger commented Feb 12, 2017

DSheirer commented Feb 12, 2017

DSheirer commented Feb 12, 2017

ImagoTrigger commented Feb 12, 2017

ImagoTrigger commented Feb 13, 2017

DSheirer commented Feb 14, 2017

davidmckenzie commented Feb 8, 2017 •

edited

Loading

ImagoTrigger commented Feb 11, 2017 •

edited

Loading

ImagoTrigger commented Feb 11, 2017 •

edited

Loading

ImagoTrigger commented Feb 12, 2017 via email •

edited

Loading

ImagoTrigger commented Feb 12, 2017 •

edited

Loading