AirNav Systems Forum

AirNav RadarBox and RadarBox24.com => AirNav RadarBox and RadarBox24.com Discussion => Topic started by: DaveReid on May 17, 2010, 11:41:35 AM

Title: Log data analysis results
Post by: DaveReid on May 17, 2010, 11:41:35 AM
I've now completed my analysis of the logs and socket capture files that users have kindly sent to me - 6 files in total containing nearly three-quarters of a million records, which I would consider to be a representative sample.

All 6 files contain multiple instances of the previously identified problem - data (callsigns, altitudes, squawks, etc) being assigned to the wrong aircraft.

I don't propose to do any further analysis as the aim was simply to establish whether or not my experience of anomalies with my own box was unique, and I'm now satisfied that it isn't.  The likelihood of all 7 of us coincidentally having rogue boxes is infinitesimal, so it seems highly likely that the issue affects all RadarBoxes.

That's unfortunate - I'd been hoping to offer RadarBox as an option to one of my clients for a tracking project (clients always love options!) but clearly now that won't be possible.

I'd stress that my conclusions only apply to professional use of RB where data integrity is important.  For everyday amateur enthusiast use the typical symptoms (sudden jumps in altitude, two aircraft showing the same Flight ID, duplicated squawks, etc) may be considered acceptable - obviously that's up to individual users' judgement.

Thanks again to everyone who participated.

Dave
Title: Re: Log data analysis results
Post by: tarbat on May 17, 2010, 12:25:53 PM
Dave, were you able to come to any general conclusions?  For example:

1. Were the aircraft affected both ADS/B and non-ADS/B equipped aircraft?

2. Did the aircraft affected generally have very similar ModeS hex codes, so could this still be caused by corrupted data being received?

3. I asked over on the other forum (http://radarspotters.eu/forum/index.php/topic,2459.msg19185.html#msg19185) (before being banned again), but do you know which message types can be parity checked, and which can't?  In other words, could any data corruption be filtered out by Radarbox using the parity bits?  Also see http://www.airnavsystems.com/forum/index.php?topic=1641.msg28678#msg28678

4. Roughly what percentage of data received exhibits these problems?

5. Is this the same problem identified in one of the competing product, where the option bits are set?  See http://www.kinetic-avionics.co.uk/forums/viewtopic.php?f=5&t=5691&p=42860&hilit=parity#p42860
Title: Re: Log data analysis results
Post by: DaveReid on May 17, 2010, 01:11:28 PM
I don't have time to do any more detailed analysis but my findings match those of other users previously, in that the hex codes for the pairs of aircraft that are being confused normally only differ by a few bits.

Whether that's caused by data corruption or not is something only AirNav can answer, and so far they have been conspicuously absent from this discussion.  It seems odd that, if a 24-bit address is being incorrectly decoded, it always happens to match that of another aircraft that's being picked up at the same time - I would have thought that data corruption would produce random address errors.

Re parity-checking, all DFs incorporate check bits using one of two different schemas, depending on which DF is being considered.

As I said in my first post, I've done all the work I need to now, having ascertained the answer to the question re RadarBox's suitability for professional applications.
Title: Re: Log data analysis results
Post by: tarbat on May 17, 2010, 01:26:09 PM
Okay Dave, I'm sure we all appreciate your efforts.

I still think Edgy31's theory is the most plausible explanation I've heard so far - at least the parts that I understand.  That affects the SBS-1 receiver as well, and is described over on the SBS-1 support forum at http://www.kinetic-avionics.co.uk/forums/viewtopic.php?f=5&t=5691&p=42860&hilit=parity#p42860

My own theory is that this is a generic problem affecting any ModeS receiver that is not able to receive the transmissions from the ground station requesting the aircraft data in the first place, and so not able to check parity on all message types.

It seems odd that, if a 24-bit address is being incorrectly decoded, it always happens to match that of another aircraft that's being picked up at the same time - I would have thought that data corruption would produce random address errors.

Maybe the Radarbox software always attempts to match these types of ModeS message with an aircraft already in the MyFlights list, and so you're only seeing examples where the corrupted 24-bit address does match one in the list.  There may be many other corrupted 24-bit addresses that simply get thrown away because there is no match to an aircraft already in the list.
Title: Re: Log data analysis results
Post by: Runway 31 on May 17, 2010, 01:34:39 PM
Does these defect mean that no SBS receiver is entirely suitable for professional applications or has the SBS-1 been updated to correct this defect?.

Also would someone like me who is no where near an airport have thios problem if I cannot see aircraft on the ground?
Title: Re: Log data analysis results
Post by: Deadcalm on May 17, 2010, 01:49:45 PM
What's probably more important - does Airnav recognise this as a problem, now, and are they prepared to investigate and fix it, if it is?

Thanks, Dave, for your efforts.

DC
Title: Re: Log data analysis results
Post by: AirNav Support on May 17, 2010, 02:46:27 PM
We have paid attention to the post and unlike what Dave has said we have read his post and looked through details we had and past support queries etc.. We decided however to wait and see whether anything more was found.

Our Professional customers (which we have quite a few Airlines, ATC and Agencies) do use the RadarBox but do not use the .rbl files or the 30003 port data. They use a different interface altogether and we haven't had issues with them regarding this.

However we will investigate this further, this was investigated in the past however the actual investigation of what is wrong isn't easy at all. This case could be GIGO (Garbage in Garbage out)
Title: Re: Log data analysis results
Post by: DaveReid on May 17, 2010, 02:47:53 PM
I still think Edgy31's theory is the most plausible explanation I've heard so far - at least the parts that I understand.  That affects the SBS-1 receiver as well, and is described over on the SBS-1 support forum at http://www.kinetic-avionics.co.uk/forums/viewtopic.php?f=5&t=5691&p=42860&hilit=parity#p42860

Well my understanding (and probably Edgy's too) of the issues is probably a bit better now that it was two years ago when that dialogue took place :-)

Quote
My own theory is that this is a generic problem affecting any ModeS receiver that is not able to receive the transmissions from the ground station requesting the aircraft data in the first place, and so not able to check parity on all message types.

That would make sense were it not for the fact that I only started to notice this phenomenon when I changed the source for my EGLL arrivals website from SBS-1 to RadarBox. 

If the problem was generic, I would presumably have been seeing examples of aircraft landing at Heathrow at FL380 for the past 4 years. 

Although I said I wouldn't do any more analysis, I've just done a quick and dirty count - out of 673,000 or so LHR arrivals in my database, the only ones that have shown silly altitudes on finals are those from last week during the period when RadarBox was supplying the data.

So clearly there is a way of avoiding assigning data to the wrong hex code, but I suspect that's something each vendor has to work out for themselves.
Title: Re: Log data analysis results
Post by: tarbat on May 17, 2010, 03:20:25 PM
I do wonder whether this only happens in areas of high traffic volumes with the possibility of more data collisions.  I've run testmodes.exe (a Radarbox native capture utility) for a couple of hours now, and ported all the data into Excel.  I see no altitude anomolies, etc.  I am in an area of VERY low volume at the moment though :(
Title: Re: Log data analysis results
Post by: DaveReid on May 17, 2010, 03:45:56 PM
Our Professional customers (which we have quite a few Airlines, ATC and Agencies) do use the RadarBox but do not use the .rbl files or the 30003 port data. They use a different interface altogether and we haven't had issues with them regarding this.

That's good news.

If you are saying that there is an alternative way of getting data out of RadarBox that's reliable then I would welcome details and, if satisfied, will happily put RadarBox back onto the shortlist for use in developing professional solutions.
Title: Re: Log data analysis results
Post by: tarbat on May 17, 2010, 03:53:55 PM
Okay, I've now found examples of this problem in the native capture:

$PTA,400620,NPT521P,,,,,,,,,,17
$PTA,400620,,,,,,,,,,,11
$PTA,400620,,16000,,,,,,,A,,4
$PTA,400620,,-1200,,,,,12.0127,0,A,,17
$PTA,400620,,,,,,,,,A,6005,21
$PTA,400620,,-1200,,,,,12.0127,0,A,,17
$PTA,400620,,16000,,,,,,,A,,20
$PTA,400620,,16000,,,,,,,A,,20
$PTA,400620,,,,,,,,,A,6005,21
$PTA,400620,,-1200,,,,,12.0127,0,A,,17
$PTA,400620,,,,,,,,,,,11
$PTA,400620,,-1200,,,,,12,0,A,,17
$PTA,400620,,16000,,,,,,,A,,0
$PTA,400620,,-1200,,,,,12,0,A,,17
$PTA,400620,,,,,,,,,,,11
$PTA,400620,,-1200,,,,,12,0,A,,17
$PTA,400620,,16000,,,,,,,A,,20
$PTA,400620,,16000,,,,,,,A,,20
$PTA,400620,,16000,,,,,,,A,,20
$PTA,400620,,-1200,,,,,12,0,A,,17
$PTA,400620,,,,,,,,,,,11
$PTA,400620,,16000,,,,,,,A,,0
$PTA,400620,,-1200,,,,,12,0,A,,17
$PTA,400620,,16000,,,,,,,A,,4
$PTA,400620,,16000,,,,,,,A,,4
$PTA,400620,,-1200,,,,,12,0,A,,17
In this example, it appears to be getting a negative altitude (-1200) from DF17 messages.  The altitude from DF4 and DF20 looks okay.

Hopefully this will help Airnav diagnose the problem and develop a fix.
Title: Re: Log data analysis results
Post by: DaveReid on May 17, 2010, 04:06:34 PM
Hmmm, I'm not sure that's the same problem, bearing in mind what we know about the ADS-B fit on ATPs - unless you're suggesting that there's a submarine close to you with a similar hex code :-)

Edit:  Out of interest, what do those other parameter values (12 and 12.0127) in the DF17 represent ?

And is 6005 a valid squawk in your part of the world ?  My squawk list may be out-of-date, but it says "6001-6006 Not allocated".
Title: Re: Log data analysis results
Post by: Runway 31 on May 17, 2010, 04:09:14 PM
Tarbat, Silly question from someone who hasnt a clue, is this the fault of radarbox or duff info coming from the aircraft?.
Title: Re: Log data analysis results
Post by: tarbat on May 17, 2010, 04:14:09 PM
Hmmm, I'm not sure that's the same problem, bearing in mind what we know about the ADS-B fit on ATPs - unless you're suggesting that there's a submarine close to you with a similar hex code :-)

You may be right.  Unfortunately it was the only anomoly I could find :(   In your testing, do you know if it was particular DF messages that were causing the anomolies?
Title: Re: Log data analysis results
Post by: DaveReid on May 17, 2010, 04:24:49 PM
In your testing, do you know if it was particular DF messages that were causing the anomolies?

I don't know, I've only got access to the socket and RBL logs, not the raw data.
Title: Re: Log data analysis results
Post by: tarbat on May 17, 2010, 04:27:42 PM
Tarbat, Silly question from someone who hasnt a clue, is this the fault of radarbox or duff info coming from the aircraft?.

That's the big question, and it could be either.  It could even be a combination of both factors - duff data not being filtered out by the Radarbox software.
Title: Re: Log data analysis results
Post by: Runway 31 on May 17, 2010, 04:31:44 PM
Got an entry in my log today for G-JAKF an R44 with start altitude of 01700 ft and an end altitude of 18800 ft.
Title: Re: Log data analysis results
Post by: shaun on May 17, 2010, 04:40:00 PM
This might be related: on the 2010/05/16 I remember this plane alternating between flight ID 353 and TSO354 it might even be in the RBL i sent.
Title: Re: Log data analysis results
Post by: DaveReid on May 17, 2010, 04:42:48 PM
Got an entry in my log today for G-JAKF an R44 with start altitude of 01700 ft and an end altitude of 18800 ft.

Welcome to the club :-)

I'd hazard a guess that if you look in your log at around the same time as the R44 you will also find one of the following:

A320 G-MONX 400541
A319 G-EUOI 400941
A319 G-EZNM 400C41
Title: Re: Log data analysis results
Post by: DaveReid on May 17, 2010, 04:48:23 PM
This might be related: on the 2010/05/16 I remember this plane alternating between flight ID 353 and TSO354 it might even be in the RBL i sent.

No, it wasn't in the log that you kindly sent, but VP-BYP did visit LHR yesterday, arriving as "353" at 16:07 and departing as TSO354 at 18:54.
Title: Re: Log data analysis results
Post by: shaun on May 17, 2010, 04:52:11 PM
OK, thanks

Shaun
Title: Re: Log data analysis results
Post by: Runway 31 on May 17, 2010, 04:56:47 PM
Dave, checked my log and the 3 aircraft you mentioned are not on it.

Alan
Title: Re: Log data analysis results
Post by: DaveReid on May 17, 2010, 05:02:52 PM
Dave, checked my log and the 3 aircraft you mentioned are not on it.

I missed one other possibility: BE20 G-MAMD 400F41.  That's assuming it was only a single-bit error, if it was 2 or 3 bits out there are lots more possibles.
Title: Re: Log data analysis results
Post by: Runway 31 on May 17, 2010, 05:20:00 PM
Nope, G-MAMD not seen either, may just be an aircraft error.

Alan
Title: Re: Log data analysis results
Post by: eggplant on May 17, 2010, 09:06:44 PM
I have several Robinson R44 helicopters logged around 36 to 37000 feet. Although I'm no expert I think that is beyond their known ceiling. Perhaps the owners/operators had installed nitrous oxide systems or such like  :-)
Title: Re: Log data analysis results
Post by: Marpleman on May 17, 2010, 10:23:19 PM
I have several Robinson R44 helicopters logged around 36 to 37000 feet. Although I'm no expert I think that is beyond their known ceiling. Perhaps the owners/operators had installed nitrous oxide systems or such like  :-)

Safe limit is to 14,000 feet for the R44, so obviously JATO versions ;-)

rich
Title: Re: Log data analysis results
Post by: DaveReid on May 18, 2010, 07:19:34 AM
I have several Robinson R44 helicopters logged around 36 to 37000 feet. Although I'm no expert I think that is beyond their known ceiling. Perhaps the owners/operators had installed nitrous oxide systems or such like  :-)

Safe limit is to 14,000 feet for the R44, so obviously JATO versions ;-)

Slightly OT, but an interesting snippet from the R44 Type Certificate says that, although the Density Altitude Limit is 14,000ft, the aircraft isn't allowed to fly more than 9,000' AGL "to allow landing within 5 minutes in case of fire  ".

Does the FAA know something about Robinsons that we don't ?  :-)
Title: Re: Log data analysis results
Post by: Marpleman on May 18, 2010, 08:48:52 AM


Slightly OT, but an interesting snippet from the R44 Type Certificate says that, although the Density Altitude Limit is 14,000ft, the aircraft isn't allowed to fly more than 9,000' AGL "to allow landing within 5 minutes in case of fire  ".

Does the FAA know something about Robinsons that we don't ?  :-)


I noticed that also - not to good a sales pitch really is it? 

But then again , you know how quickly "hair dryers" can cut out from over-heating?

Rich
Title: Re: Log data analysis results
Post by: DaveReid on May 18, 2010, 01:06:37 PM
Our Professional customers (which we have quite a few Airlines, ATC and Agencies) do use the RadarBox but do not use the .rbl files or the 30003 port data. They use a different interface altogether and we haven't had issues with them regarding this.

That's good news.

If you are saying that there is an alternative way of getting data out of RadarBox that's reliable then I would welcome details and, if satisfied, will happily put RadarBox back onto the shortlist for use in developing professional solutions.

Still awaiting aforesaid details !

In the meantime it occurs to me that, if it's true what you're saying and you have other versions of RadarBox that don't suffer from the "hex code confusion" bug, then why on earth aren't you using the same data handling routines in the RB version that we're all running ?

It does seem strange, though, that you are blaming the external data output interface for the hex code errors, I would have thought that they originated further upstream, in the decoding process.

And are you absolutely sure that it's only the amateur version of RadarBox that has this bug ?  I can understand any vendor's reluctance to acknowledge defects in their software but, as any developer will tell you, honesty is always the best policy.

Anyway, taking your statement at face value, can you please tell me how can I obtain an evaluation copy of the bug-free professional version of the RB software to confirm that that's the case and to see if it would be suitable for my clients ?

I'm hoping that all the work I've done on analysis over the last couple of days won't turn out to have been a waste of time.  Having said that, pending a bug-fix I've at least saved myself the potential embarrassment of recommending a solution that I would later regret.
Title: Re: Log data analysis results
Post by: AirNav Support on May 18, 2010, 03:06:36 PM
DaveReid,

The Professional version costs extra as you can imagine and we will only provide the files etc if you are interested to purchase.

If you are please contact us directly, Professional trials etc are done via email and need agreements and we will not be forced into giving one over the forum.

Also just to clarify our clients have not come back with that issue regarding the Pro version that doesn't mean to say it is not there.

Lastly, if you want an answer from support or development please contact them directly. As I am sure you are aware the forum is not a guaranteed support forum for us, we get many posts and some get missed etc.. The support ticket system is assured a reply and that’s why its there.
Title: Re: Log data analysis results
Post by: DaveReid on May 18, 2010, 04:08:08 PM
Email sent re evaluation of the Professional version.
Title: Re: Log data analysis results
Post by: shaun on May 18, 2010, 05:01:12 PM
Maybe this is related?
Title: Re: Log data analysis results
Post by: Runway 31 on May 18, 2010, 05:10:15 PM
Nothing related Shaun, just the crew havent done the flight ID for the flight, hence no company logo.

Alan
Title: Re: Log data analysis results
Post by: shaun on May 18, 2010, 05:25:25 PM
I can't see what's wrong.
Title: Re: Log data analysis results
Post by: DaveG on May 18, 2010, 05:37:04 PM
What is wrong is two things.

First, your issue does not relate to the current discussion in this thread.

Second the reason your not seeing a company logo is because there is no valid FlightID for that record.  Notice it has 0000000, if it had something like RYR1075 then you'd see a RyanAir logo, if BAW95 then a British Airways logo.  The crew of your aircraft have not entered into their system a FlightID.
Title: Re: Log data analysis results
Post by: DaveReid on May 20, 2010, 04:45:04 PM
Email sent re evaluation of the Professional version.

My thanks to AirNav for the evaluation copy of RadarBox Professional.

Unfortunately my tests have shown that it, too, suffers from the "hex code confusion" bug (whereby flight details for one aircraft are recorded against a different aircraft whose hex code differs by a few bits). 

An example this afternoon was G-BYAT B752 400573 TOM71E showing details belonging to G-CIVB B744 400551 BAW287.
Title: Re: Log data analysis results
Post by: malc41 on May 20, 2010, 07:22:17 PM
Dave

Which version of software did this start with?

Title: Re: Log data analysis results
Post by: malc41 on May 20, 2010, 08:19:48 PM
Dave

Not sure if this is the same problem that you experience?  From todays log

400EBE  GBYWB                    G-BYWI  G115  Vt Aerospace Ltd     2010/05/20 09:05:07 
400EBE  GBYWG                    G-BYWI  G115  Vt Aerospace Ltd     2010/05/20 13:52:29 
400EBE  GBYWI                    G-BYWI  G115  Vt Aerospace Ltd     2010/05/20 13:12:36
also

400EEE  GBYUJ                    G-BYVR  G115  Vt Aerospace Ltd     2010/05/20 14:03:18 
400EEE  GBYVR                    G-BYVR  G115  Vt Aerospace Ltd     2010/05/20 08:17:51
Title: Re: Log data analysis results
Post by: DaveReid on May 20, 2010, 08:55:05 PM
Not sure if this is the same problem that you experience?  From todays log

400EBE  GBYWB                    G-BYWI  G115  Vt Aerospace Ltd     2010/05/20 09:05:07 
400EBE  GBYWG                    G-BYWI  G115  Vt Aerospace Ltd     2010/05/20 13:52:29 
400EBE  GBYWI                    G-BYWI  G115  Vt Aerospace Ltd     2010/05/20 13:12:36
also

400EEE  GBYUJ                    G-BYVR  G115  Vt Aerospace Ltd     2010/05/20 14:03:18 
400EEE  GBYVR                    G-BYVR  G115  Vt Aerospace Ltd     2010/05/20 08:17:51

That does indeed look odd, but it's a different issue (probably related to the aircraft database) whereas the bug I'm concerned about relates solely to the data coming from the aircraft (hex code, callsign, squawk, altitude, etc) and isn't related to database lookups.

In answer to your other question about what version of the software the "hex code confusion" bug started with - I have no idea, in fact I'm not even sure whether it's a software or a firmware bug, to be honest.

I could do trials with previous RB versions, but there isn't really much point as most people are on 3.13 or 4.03 now.
Title: Re: Log data analysis results
Post by: malc41 on May 20, 2010, 09:02:34 PM
Dave

Thanks for that, it was just in the back of my mind that this issue has been around for some time.

Title: Re: Log data analysis results
Post by: DaveReid on May 24, 2010, 07:25:46 AM
Thanks for that, it was just in the back of my mind that this issue has been around for some time.

So it would appear - no criticism of the beta-testers intended :-)

To be fair, many enthusiast RadarBox users probably wouldn't be too worried about two aircraft in the list with the same squawk or callsign, or a sudden unexplained jump in altitude (although, as you say, a number of individuals have posted examples).

It's likely to be professional users of RadarBox (and now, it would appear, AirNav Flight Data) who are most affected - the ADS-B application I'm currently working on would certainly be screwed up by, for example, an A320 reappearing at FL380 5 minutes after it has landed, so RadarBox is no longer an option that I'm prepared to offer the client.

Hopefully that will change soon - I would expect that AirNav are reviewing their decoding algorithms even as we speak ...

Dave
Title: Re: Log data analysis results
Post by: AirNav Support on May 24, 2010, 07:37:33 AM
Yes we are taking a look at this and will be added to the bugs page.
Title: Re: Log data analysis results
Post by: tarbat on May 24, 2010, 09:50:57 AM
I note that on the bug tracking system, you say:
"Occurs mainly in high message areas where Mode-S confusion can occur, mainly as an aircraft lands"

I'm not sure that is true.  Firstly, I'm in an area with low message volumes.  And I've taken port 30003 data, and analysed in Excel.  Out of 45,000 messages, I found two cases of mixed up altitudes.  One aircraft was at FL235 when it suddenly reported an altitude of FL390, and the other was at FL105 when it suddenly reported an altitude of FL390.

In both cases, the ModeS hex codes were different by two bits.
400619 (FL235) vs 400409 (FL390)
010000000000011000011001
010000000000010000001001

400AD1 (FL105) vs 400A99 (FL390)
010000000000101011010001
010000000000101010011001
Title: Re: Log data analysis results
Post by: DaveReid on May 24, 2010, 11:09:17 AM
I don't understand the reference to aircraft landing, either.

The bug appears to affect aircraft in all phases of flight, although of course it's easier to spot when there is a wide discrepancy in altitude between the two aircraft, e.g. if one is landing and one is en-route.
Title: Re: Log data analysis results
Post by: AirNav Support on May 24, 2010, 12:55:39 PM
Its just related to the ones we have caught during testing. Don't worry its all written down in detail elsewhere.