AirNav Systems Forum
AirNav RadarBox and RadarBox24.com => AirNav RadarBox and RadarBox24.com Discussion => Topic started by: Marpleman on May 26, 2010, 09:09:35 PM
-
Hi All
I am posting this on behalf of the Database Updater team, as we thought it was about time we gave you an update regarding the steps being taken to tackle the ongoing issues concerning the integrity of the records held in the AirNav central database.
Firstly, a couple of points to make regarding the programme;
Several weeks ago, AirNav approached the main board asking for volunteers to offer their time and shared experience of our collective hobby in an attempt to provide a mechanism for “cleaning up” the information contained in the database, and to provide a platform to process additional records, and deal with erroneous/incomplete records.
Please note - we are not here to debate if this is the correct approach to take. This topic of debate has been aired on several threads over the past few months, with as yet no viable, workable solution, that fits for all parties.
Please therefore do not litter this thread bemoaning the steps we have or will be taking.
The members who volunteered for the task have spent the last few weeks sharing ideas and trailing a “database updater” to give such a mechanism as described above.
We have also been in close contact with Support and Development, exchanging ideas, some workable, some not, to get this up and running.
At the present time, we have made considerable inroads in “data clensing”, but it’s a vast exercise!
At present we have close to 70,000 individual records on a central database, and are steadily working our way through this by aircraft manufacturer/type.
We are also adding new records as they become apparent.
We also fully appreciate that this is by no means any where near a complete data set, but at least it’s a start
To validate the integrity of the records on the central database, we are using several independent databases either in the public domain, or which we individually have access to.
A major topic of debate between us has revolved around the most effective way of dealing with this, together with the addition of many, many more records that we are aware of. At the moment, we are discussing with AirNav the best solution to deal with this – please be patient with us on this.
The basic mechanism, in the words of “Support” as to how this works, is as follows;
There are two separate databases interacting -
Central RadarBox Database - Stores all the Aircraft details populated by the updater team, plus the initial record data set.
Local Database - Once an aircraft with no details is picked up, it searchers the Mode-S in the central database and pulls back all the details and populates the local database.
There is still some work ongoing revolving the exact approach taken here – we will update asap on this process, as we are also desparately trying to tackle the known issue of photographic record validity, which much of this hinges on!
We want to be in a position to open a thread where the forum members can post details of records that are;
- showing erroneous data
- pulling back blank data fields in MyFlights
- new aircraft not on the database (in this case more likely covered by the point above)
Again, there are many different approaches to this. Until we reach a more suitable method that is currently being developed, we suggest offering two approaches as follows;
1 – a separate thread will be opened entitled “Database update requests” where members can post single records that require our attention.
2 – as a “launch” project, I am willing to accept csv/excel file with multiple records acquiring our attention. These files must be forwarded to my personal e-mail as can be found below my “user name” on the thread, or in my user profile (”Marpleman”). This worked extremely well with an individual members data made up of mode-s codes picked up over the last month, which were not producing any other data fields in “MyFlights”.
Any files forwarded will be collated by myself, and managed by the team. If we cannot clear the record to a suitable standard for addition to the database, we will throw it back to you guys for ideas/info.
There is an excel file attached to this post, detailing the fields we will be entering onto the database.
These are as follows –
“ms” – mode s code
“ar” – registration/serial number(mil)
“cn” – construction number
“ai” – aircraft ICAO
“at” – aircraft type
“co” – company/operator
There is an example of an Air Berlin A320 fully completed on the attachment to give a general idea.
We need at least the “ms” or “ar” fields to be completed for us to investigate.
If the query relates to incorrect data, please provide a brief description as to the nature of this in the “narrative” field – i.e incorrect cn etc etc
Please give as much information as you can to facilitate this. From our perspective, the more the better.
There are no doubt many things I’ve failed to mention in this post – if so sorry – please ask below and we will try to answer any queries, but I must stress, we are trying to improve the database and do not want to get involved in arguments regarding approach/responsibility etc etc.
At the end of the day we are volunteers. We’ve all at some point complained about incorrect records/photo’s etc. We will undoubtedly make mistakes along the way, but hopefully we can all help to improve the integrity of the database.
Thanks for reading this.
Regards
Rich
-
Sorry - forgot to mention, if anyone e-mails an update file to me . please could they pm me, and rename the file "database update file " and add their username.
i.e. database update file Marpleman.xls/csv
Cheers
Rich
-
We want to be in a position to open a thread where the forum members can post details of records that are;
- showing erroneous data
- pulling back blank data fields in MyFlights
- new aircraft not on the database (in this case more likely covered by the point above)
Does that imply that you/AirNav will be supplying an updated NavData.db3 before this exercise starts ?
Presumably there isn't any point in users reporting errors in the current NavData file, since these are likely to have been found already by the research team ?
-
We want to be in a position to open a thread where the forum members can post details of records that are;
- showing erroneous data
- pulling back blank data fields in MyFlights
- new aircraft not on the database (in this case more likely covered by the point above)
Does that imply that you/AirNav will be supplying an updated NavData.db3 before this exercise starts ?
Presumably there isn't any point in users reporting errors in the current NavData file, since these are likely to have been found already by the research team ?
Hi Dave
The update team don't have the capacity to do this at present, and I'm afraid I cannot comment on behalf of AirNav.
There is still a very high % of records we haven't got close to looking at yet if I'm being honest, so we haven't found them by a long chalk
Regards
Rich
-
Marpleman ~
I've attached my current blank records for your perusal.
Thanks for your attention!
Jim
-
Marpleman ~
I've attached my current blank records for your perusal.
Thanks for your attention!
Jim
Thanks Jim
Will pm you when we've taken a look
Rich
-
This may be stating the obvious, but about 85% of those "unknowns" aren't unknowns at all, but aircraft that are current on the US register - for example the first one, ABFD82, is CRJ2 7718 N8718E of Northwest Airlink.
A visit here: http://registry.faa.gov/aircraftinquiry/NNum_Inquiry.aspx might be a good idea.
-
This may be stating the obvious, but about 85% of those "unknowns" aren't unknowns at all, but aircraft that are current on the US register - for example the first one, ABFD82, is CRJ2 7718 N8718E of Northwest Airlink.
A visit here: http://registry.faa.gov/aircraftinquiry/NNum_Inquiry.aspx might be a good idea.
Hi Dave
Agreed entirely
At the moment we have a slight "issue" in data capture to the central database that AirNav are working on.
This is causing some frustration from my perspective at present, but when sorted will give us a much more expedient option of making these "known".
Thanks for your input
Regards
Rich
-
Hi Marpleman,
Have just read your opening message in this thread - a very comprehensive and encouraging update as to what is being done to improve the database - many thanks.
I have just run a filtered report from MyFlights since May 1st. It generated 58 records where the Mode-S didn't tie up with any Reg and a further 56 records where a reg tie-up occurred but the aircraft ICAO type showed as '...'. (And I only run ANRB a couple of hours a day from an internal aerial in a 'dip - and always online for supposed updates'!)
Don't have the time to enter all this missing data manually into an Excel file to send you - is there a way to export my relevant records into either an excel or csv file automatically?
Cheers
Chris
-
Hi Marpleman,
Have just read your opening message in this thread - a very comprehensive and encouraging update as to what is being done to improve the database - many thanks.
I have just run a filtered report from MyFlights since May 1st. It generated 58 records where the Mode-S didn't tie up with any Reg and a further 56 records where a reg tie-up occurred but the aircraft ICAO type showed as '...'. (And I only run ANRB a couple of hours a day from an internal aerial in a 'dip - and always online for supposed updates'!)
Don't have the time to enter all this missing data manually into an Excel file to send you - is there a way to export my relevant records into either an excel or csv file automatically?
Cheers
Chris
Hi Chris
I'll pm you in the next hour if that's ok?
thanks
Rich
-
Hi Chris
I'll pm you in the next hour if that's ok?
thanks
Rich
Hi Rich,
Look forward to your PM - may not be able to reply tonight - but have long BH weekend in prospect to get to grips with any suggestions!
Chris
-
Rich:
Attached is my current list of blank records. It's nearly doubled in size since the last submission.
Thanks for your attention!
Jim
-
Attached is my current list of blank records. It's nearly doubled in size since the last submission.
I'm still confused as to what we're looking at here - again, details for over 80% of those "blanks" are just a mouse-click away.
For example, the first on the list, A4BD40: http://registry.faa.gov/aircraftinquiry/NNum_Results.aspx?NNumbertxt=N404UA
I can't help thinking that you're making your job even harder than it need be by ignoring readily-available (free) resources, or am I missing something ?
-
Dave, not sure who your reply is directed at but I'll add a comment for it anyway.
Finding the details of airframes is not the major problem when we're updating records in the database, as you've said there are lots of location online to find the details, and some for free.
The problem is time, the time it takes to update each single record multiplied by the amount of records needing checking/updating.
No doubt about it the current system of updating the database is good for single or small batches but drives you eyes crazy if doing loads. Suggestion have been made to Airnav Dev regarding possible solutions as a way forward. until then its a case of updating till your eyes have had it ;-)
Dave, on another(ish) topic, do you have an empty Navdata file by any chance?
-
The problem is time, the time it takes to update each single record multiplied by the amount of records needing checking/updating.
No doubt about it the current system of updating the database is good for single or small batches but drives you eyes crazy if doing loads. Suggestion have been made to Airnav Dev regarding possible solutions as a way forward. until then its a case of updating till your eyes have had it ;-)
I absolutely agree - but I'd have thought that, with all the resources and skills at AirNav's disposal, putting together a routine to import registers in bulk would be a no-brainer.
It's hard to avoid the conclusion that the support you're getting from your "employer" is pretty pitiful.
Dave, on another(ish) topic, do you have an empty Navdata file by any chance?
No, but I can easily put one together, give me a day or two.
-
I absolutely agree - but I'd have thought that, with all the resources and skills at AirNav's disposal, putting together a routine to import registers in bulk would be a no-brainer.
It's hard to avoid the conclusion that the support you're getting from your "employer" is pretty pitiful.
Only one thing to add, Not employer, volunteer, big difference especially on some of these forums, lol.
No, but I can easily put one together, give me a day or two.
Thanks Dave, just need one to try something out.
-
Dave. The problem is not the airnav support, it is the lack of info that is one click away. GAS is fairly comprehensive for Europe/North America. But it does not cover china, south america or the middle east well. We sped through the BA/bmi/Lufthansa fleets. With your one click can you find me full details for the AirAsia Indonesia fleet, or VivaAerobus, or China United Airlines. We have to piece these together with many fragments from many different websites. Add to that over 60New airliners roll out of the factories each month. Many many more are reregistered/deregistered. These are the things slowing us down. not the level of support we are receiving.
-
The moral is, if you see aircraft without details and its not on GAS, then use the box on the GAS website to update the details there, including boeing/douglas line numbers to the c/n. We will then have the one click checking tool, which will speed the process up.
-
Dave. The problem is not the airnav support, it is the lack of info that is one click away. GAS is fairly comprehensive for Europe/North America. But it does not cover china, south america or the middle east well. We sped through the BA/bmi/Lufthansa fleets. With your one click can you find me full details for the AirAsia Indonesia fleet, or VivaAerobus, or China United Airlines. We have to piece these together with many fragments from many different websites. Add to that over 60 New airliners roll out of the factories each month. Many many more are reregistered/deregistered. These are the things slowing us down. not the level of support we are receiving.
Yes, I'm probably more familiar than most with the issues involved in keeping a Mode S database up-to-date :-)
But it's hard to square, on one hand, your assertion that AirNav are supporting you properly with, on the other hand, lists of "unknowns" that are 80% US civil registered aircraft.
AirNav are, or should be, perfectly capable of furnishing you with a routine that will update your central database with N-registered aircraft automatically, and yet we're told that they expect you to enter even those aircraft details painstakingly, one at a time.
Most of us, myself included, are in aviation as either a hobby or career because we love what we do, but it always worries me when I see what looks like cynical exploitation of people's enthusiasm and goodwill.
-
No, but I can easily put one together, give me a day or two.
Thanks Dave, just need one to try something out.
http://www.civilaircraftregisters.org/Mode_S_Resources/NavDataEmpty.db3 (rename before use!)
HTH
Dave
-
No, but I can easily put one together, give me a day or two.
Thanks Dave, just need one to try something out.
http://www.civilaircraftregisters.org/Mode_S_Resources/NavDataEmpty.db3 (rename before use!)
HTH
Dave
Cheers
Dave
-
[
Most of us, myself included, are in aviation as either a hobby or career because we love what we do, but it always worries me when I see what looks like cynical exploitation of people's enthusiasm and goodwill.
That's a tad harsh Dave?
I wouldn't class anything that keeps me away from "her indoors" as "cynical exploitation" - more like "greatfully received opportunity" :-)
AirNav have a "wishlist" that the team have compiled - I think perhaps other commercial priorities have precluded these items from becoming fully functional at present, but when they are, we'll be flying.
In a perverse way, I'm actually enjoying the task also ;-)
By the way, appreciate the links/requests you keep providing us with - they're very useful!
Rgds
Rich
-
Hello Dave
Actually I find updating very enjoyable and informative. It has certainly broadened my knowledge of airlines etc so I don't feel exploited at all.
Andrew
-
"exploitation", yep too many hours, not enough pay and bad social life because of it, Hmm wait that's not Airnav that my real job. Must think about that one! anyone got the 6 ball numbers for this weekend lotto draw.
-
That's a tad harsh Dave?
I wouldn't class anything that keeps me away from "her indoors" as "cynical exploitation" - more like "greatfully received opportunity" :-)
Hmmm... An analogy. I make an offer to a keen (possibly deluded) gardener to come and mow my lawn and in return I allow him to clean my gutters, would this also not be exploitation ?
I have to concur with Dave.
-
That's a tad harsh Dave?
I wouldn't class anything that keeps me away from "her indoors" as "cynical exploitation" - more like "greatfully received opportunity" :-)
Hmmm... An analogy. I make an offer to a keen (possibly deluded) gardener to come and mow my lawn and in return I allow him to clean my gutters, would this also not be exploitation ?
I have to concur with Dave.
Please yourself
Unfortunately I don't know anything about gardening
-
[/quote]
AirNav have a "wishlist" that the team have compiled - I think perhaps other commercial priorities have precluded these items from becoming fully functional at present, but when they are, we'll be flying.
[/quote]
Does anyone know what this 'wishlist' contains and has anyone seen it (other than AirNav staff)?
Herein lies the problem. I appreciate AirNav are a commercial organisation and they need the big Boeing bucks to maintain a healthy profit margin. But this 'wait for a big announcement' culture is hacking everyone off.
I think the cynical exploitation statement is a little harsh but the little guys like me are beginning to feel marginalised by what has been going on recently. I know we don't have the financial clout of someone like Boeing but I think all of us who have spent our hard earned cash on a RB deserve much better than we have been given.
-
Southwest.
This wishlist has nothing to do with the functionality of RadarBox and its simply just for the database updaters app.
If you have an specfic issue that you think we are not looking at please contact us. We are a company which listens to our customer base, our efforts on the forum to talk to customers shows this.
-
Airvav did post their list a few months ago if i remember right, now stop your whingeing and look for it.
from pat
-
AirNav have a "wishlist" that the team have compiled - I think perhaps other commercial priorities have precluded these items from becoming fully functional at present, but when they are, we'll be flying.
[/quote]
Does anyone know what this 'wishlist' contains and has anyone seen it (other than AirNav staff)?
Herein lies the problem. I appreciate AirNav are a commercial organisation and they need the big Boeing bucks to maintain a healthy profit margin. But this 'wait for a big announcement' culture is hacking everyone off.
I think the cynical exploitation statement is a little harsh but the little guys like me are beginning to feel marginalised by what has been going on recently. I know we don't have the financial clout of someone like Boeing but I think all of us who have spent our hard earned cash on a RB deserve much better than we have been given.
[/quote]
Ok -probably my fault here!
AirNavs comment is correct - the "wishlist" was the updaters wishlist purely to make our task more simplified and quicker.
They provided the framework for a bog standard database updater, which we played around with for a week or so.
The team then reached a concensus as to what we need as additional functionality on this updater - this then goes on the wishlist, and as and when AirNav have the ability to look at our suggestions, they do.
Sorry if I confused annyone!
Rich
-
Airvav did post their list a few months ago if i remember right, now stop your whingeing and look for it.
from pat
Swordfish - Didn't you have any toys as a child?
Rich - Thanks for taking the time to answer in a civil manner.
Air Nav team - You know what I want, it's better route information but I do appreciate there are other things that will come before that.
-
Hello,
I run Airnav suite 4 with PC-HFDL and Posfix. Unfortunately most of the plots I receive & process do not indicate neither routes nor registration. So my plots show mostly flight numbers only.
When I check the supplied Airnav data base for these details these flights & details simply lack there. I receive mainly Russian flights - AFL & SU.
Does AirNav intend to update soon its data accordingly doing justice to the fact that surely a good many German users with the same problem are among its customers ??
If unable or unwilling to do so soon
is there the possibility to carry out the necessary updates to its data base by myself ?
If so, how is this done ?
Thanks in advance.
Regards,
R. Fechter
-
Loophunter, we are working on it. Please forward on the hex code of any fllights with no registration and we will apply them. AFL and SU are definately on the database but forward the hex code of the aircraft concerned and we will add them.
Alan
-
Attached is a file containing my blanks received since June 3rd when I made my last submission.
Thanks!
Jim
-
Attached is a file containing my blanks received since June 3rd when I made my last submission.
Thanks!
Jim
Cheers Jim
Appreciate your efforts
Rich
-
I did a very involved import of the FAA database. I'd say it's better than 95% complete/accurate at this point. The only data lacking or incorrect would be aircraft that have been newly registered since I downloaded the FAA data; de/re-registrations; and aircraft using the incorrect code for their registration. Any of these things get updated as I log them. Every day I typically log 4 to 6 N-reg codes that I need to enter the details for.
Often though they are unassigned codes/registrations. For these I just enter the code and matching reg, "unk" for AT, and "Not Assigned/Reserved", "Reserved by SBS Program Office" or "Reserved by (so and so)"
for AN and AC. For non N-reg I just enter Unknown for AN and AC unless they are display a flight ID that gives a possible identification. I currently have 315 logged N-regs classfied with an 'unassigned' or 'reserved' entry.
There have been a few aircraft using the wrong codes that I have identified except for the registration. Comparing flight IDs and altitudes to what is showing on Flightaware has allowed this.
In my database I also included all the reserved registrations but have not entered all the mode-s codes for them. Many of the logged planes with no details are now using their previously reserved registrations.
Some other data is sort of missing because of how the FAA makes the database available. Generally this is only for less comon aircraft. For 'AN' some have an FAA code that corresponds to a particular airframe. I update this with the actual make and model when I log them. An example from today is 138527H. This is the code for Boeing 777-FS2. There are currently only four of these, all operated by Fed Ex. At the time of my import the FAA data listed Boeing Company as the owner. Another example is 05623YO for N311RV. It's listed as a Testement J/Dobb PA Vans RV-10. There is one one of these. I updated all these records in my DB today.
If someone would like my data to work with let me know.
-
If someone would like my data to work with let me know.
Hi freqhopping
Interesting post,and thanks for the insight
We've deliberated over the FAA database within the updater team with, if I'm being totally honest some differing opinions and some obvious concerns,some of which you've commented on above (operaters/owners etc).
Although we've been undertaking the updating task now for several weeks (but what seems like an eternity!) we have primarily focused on dealing with ADS-B equipped aircraft as a starting point, and dealing with other specific "one-offs" as and when requested.
This doesn't mean we underestimate the requirements for other aircraft (military,biz jets, light aircraft)
One of the problems we're faced with, as several other forum members have commented on, is that there is a wealth of information "out there" giving us the info we need, and I think from memory Dave Reid (sorry if it wasn't you Dave?) has made reference to certain members "unknown" hex code listings , as not being unknown, but just not on the database!
Problem is, at the moment we don't have the facility to upload bulk info, however it is being worked on.
When we have access to this, we will be in a much better position to utilise this sort of data, together with other reputable sources out there.
We do however need to undertake this process in a carefully considered manner so as not to "undo" a lot of good work already completed.
Personally, I'd much rather have a central database with as many records on as possible, even if not checked/verified, as at least this would result in fewer blank populations of fields on peoples screens. We could still undertake the checking/validation, and carry on accordingly.
Members would then be able to continue advising us of errors etc in the normal process, and in the intervening period, we'd get far less requests for checking of fields not populating.
We also have some issues in the process by which the updated info on the central database is pulled through to peoples de's - it clearly isn't working as we'd like it to at present.Again, this is being looked into.
I'm hopeful we can revisit this scenario soon, but I'm sure you'll appreciate we need to get other areas and improved functionality into our process before we look at this sort of usage.
Would be interested to get feedback on peoples constructive views on this, as at the end of the day, what we're doing will hopefully benefit all users?
Regards
Rich