AirNav Radar
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
 


Author Topic: Database Updater Team progress report  (Read 20090 times)

0 Members and 1 Guest are viewing this topic.

Marpleman

  • Hero Member
  • *****
  • Posts: 738
  • Proper aeroplanes!
Database Updater Team progress report
« on: May 26, 2010, 09:09:35 PM »
Hi All

I am posting this on behalf of the Database Updater team, as we thought it was about time we gave you an update regarding the steps being taken to tackle the ongoing issues concerning the integrity of the records held in the AirNav central database.

Firstly, a couple of points to make regarding the programme;

Several weeks ago, AirNav approached the main board asking for volunteers to offer their time and shared experience of our collective hobby in an attempt to provide a mechanism for “cleaning up” the information contained in the database, and to provide a platform to process additional records, and deal with erroneous/incomplete records.

Please note - we are not here to debate if this is the correct approach to take. This topic of debate has been aired on several threads over the past few months, with as yet no viable, workable solution, that fits for all parties.

Please therefore do not litter this thread bemoaning the steps we have or will be taking.


The members who volunteered for the task have spent the last few weeks sharing ideas and trailing a “database updater” to give such a mechanism as described above.

We have also been in close contact with Support and Development, exchanging ideas, some workable, some not, to get this up and running.

At the present time, we have made considerable inroads in “data clensing”, but it’s a vast exercise!

At present we have close to 70,000 individual records on a central database, and are steadily working our way through this by aircraft manufacturer/type.

We are also adding new records as they become apparent.

We also fully appreciate that this is by no means any where near a complete data set, but at least it’s a start

To validate the integrity of the records on the central database, we are using several independent databases either in the public domain, or which we individually have access to.

A major topic of debate between us has revolved around the most effective way of dealing with this, together with the addition of many, many more records that we are aware of. At the moment, we are discussing with AirNav the best solution to deal with this – please be patient with us on this.

The basic mechanism, in the words of “Support” as to how this works, is as follows;

There are two separate databases interacting -

Central RadarBox Database - Stores all the Aircraft details populated by the updater team, plus the initial record data set.

Local Database  - Once an aircraft with no details is picked up, it searchers the Mode-S in the central database and pulls back all the details and populates the local database.

There is still some work ongoing revolving the exact approach taken here – we will update asap on this process, as we are also desparately trying to tackle the known issue of photographic record validity, which much of this hinges on!


We want to be in a position to open a thread where the forum members can post details of records that are;

-   showing erroneous data
-   pulling back blank data fields in MyFlights
-   new aircraft not on the database (in this case more likely covered by the point above)

Again, there are many different approaches to this. Until we reach a more suitable method that is currently being developed, we suggest offering two approaches as follows;

1 – a separate thread will be opened entitled “Database update requests” where members can post single records that require our attention.

2 – as a “launch” project, I am willing to accept csv/excel file with multiple records acquiring our attention. These files must be forwarded to my personal e-mail as can be found below my “user name” on the thread, or in my user profile (”Marpleman”). This worked extremely well with an individual members data made up of mode-s codes picked up over the last month, which were not producing any other data fields in “MyFlights”.
 Any files forwarded will be collated by myself, and managed by the team. If we cannot clear the record to a suitable standard for addition to the database, we will throw it back to you guys for ideas/info.


There is an excel file attached to this post, detailing the fields we will be entering onto the database.

These are as follows –

“ms” – mode s code
“ar” – registration/serial number(mil)
“cn” – construction number
“ai” – aircraft ICAO
“at” – aircraft type
“co” – company/operator

There is an example of an Air Berlin A320 fully completed on the attachment to give a general idea.

We need at least the “ms” or “ar” fields to be completed for us to investigate.

If the query relates to incorrect data, please provide a brief description as to the nature of this in the “narrative” field – i.e incorrect cn etc etc

Please give as much information as you can to facilitate this. From our perspective, the more the better.

There are no doubt many things I’ve failed to mention in this post – if so sorry – please ask below and we will try to answer any queries, but I must stress, we are trying to improve the database and do not want to get involved in arguments regarding approach/responsibility etc etc.
 
At the end of the day we are volunteers. We’ve all at some point complained about incorrect records/photo’s etc. We will undoubtedly make mistakes along the way, but hopefully we can all help to improve the integrity of the database.

Thanks for reading this.

Regards

Rich

Marpleman

  • Hero Member
  • *****
  • Posts: 738
  • Proper aeroplanes!
Re: Database Updater Team progress report
« Reply #1 on: May 26, 2010, 09:46:09 PM »
Sorry - forgot to mention, if anyone e-mails an update file to me . please could they pm me, and rename the file "database update file " and add their username.

i.e.   database update file Marpleman.xls/csv

Cheers

Rich

DaveReid

  • Hero Member
  • *****
  • Posts: 1815
    • Heathrow last 100 ADS-B arrivals
Re: Database Updater Team progress report
« Reply #2 on: May 26, 2010, 10:57:22 PM »
We want to be in a position to open a thread where the forum members can post details of records that are;

-   showing erroneous data
-   pulling back blank data fields in MyFlights
-   new aircraft not on the database (in this case more likely covered by the point above)

Does that imply that you/AirNav will be supplying an updated NavData.db3 before this exercise starts ?

Presumably there isn't any point in users reporting errors in the current NavData file, since these are likely to have been found already by the research team ?
This post has been scanned for any traces of negativity, bias, sarcasm and general anti-social behaviour

Marpleman

  • Hero Member
  • *****
  • Posts: 738
  • Proper aeroplanes!
Re: Database Updater Team progress report
« Reply #3 on: May 27, 2010, 09:03:06 AM »
We want to be in a position to open a thread where the forum members can post details of records that are;

-   showing erroneous data
-   pulling back blank data fields in MyFlights
-   new aircraft not on the database (in this case more likely covered by the point above)

Does that imply that you/AirNav will be supplying an updated NavData.db3 before this exercise starts ?

Presumably there isn't any point in users reporting errors in the current NavData file, since these are likely to have been found already by the research team ?

Hi Dave

The update team don't have the capacity to do this at present, and I'm afraid I cannot comment on behalf of AirNav.

There is still a very high % of records we haven't got close to looking at yet if I'm being honest, so we haven't found them by a long chalk

Regards

Rich

54901 - Jim

  • Hero Member
  • *****
  • Posts: 566
    • N9JHB.com
Re: Database Updater Team progress report
« Reply #4 on: May 27, 2010, 11:38:56 AM »
Marpleman ~

I've attached my current blank records for your perusal.

Thanks for your attention!

Jim
KOSH - Oshkosh, Wisconsin USA

Marpleman

  • Hero Member
  • *****
  • Posts: 738
  • Proper aeroplanes!
Re: Database Updater Team progress report
« Reply #5 on: May 27, 2010, 01:23:57 PM »
Marpleman ~

I've attached my current blank records for your perusal.

Thanks for your attention!

Jim


Thanks Jim

Will pm you when we've taken a look

Rich

DaveReid

  • Hero Member
  • *****
  • Posts: 1815
    • Heathrow last 100 ADS-B arrivals
Re: Database Updater Team progress report
« Reply #6 on: May 27, 2010, 01:46:59 PM »
This may be stating the obvious, but about 85% of those "unknowns" aren't unknowns at all, but aircraft that are current on the US register - for example the first one, ABFD82, is CRJ2 7718 N8718E of Northwest Airlink.

A visit here: http://registry.faa.gov/aircraftinquiry/NNum_Inquiry.aspx might be a good idea.
This post has been scanned for any traces of negativity, bias, sarcasm and general anti-social behaviour

Marpleman

  • Hero Member
  • *****
  • Posts: 738
  • Proper aeroplanes!
Re: Database Updater Team progress report
« Reply #7 on: May 27, 2010, 03:01:00 PM »
This may be stating the obvious, but about 85% of those "unknowns" aren't unknowns at all, but aircraft that are current on the US register - for example the first one, ABFD82, is CRJ2 7718 N8718E of Northwest Airlink.

A visit here: http://registry.faa.gov/aircraftinquiry/NNum_Inquiry.aspx might be a good idea.

Hi Dave

Agreed entirely

At the moment we have a slight "issue" in data capture to the central database that AirNav are working on.

This is causing some frustration from my perspective at present, but when sorted will give us a much more expedient option of making these "known".

Thanks for your input

Regards

Rich

Pinza

  • Full Member
  • ***
  • Posts: 248
Re: Database Updater Team progress report
« Reply #8 on: May 27, 2010, 07:31:41 PM »
Hi Marpleman,

Have just read your opening message in this thread - a very comprehensive and encouraging update as to what is being done to improve the database - many thanks.

I have just run a filtered report from MyFlights since May 1st. It generated 58 records where the Mode-S didn't tie up with any Reg and a further 56 records where a reg tie-up occurred but the aircraft ICAO type showed as '...'. (And I only run ANRB a couple of hours a day from an internal aerial in a 'dip - and always online for supposed updates'!)

Don't have the time to enter all this missing data manually into an Excel file to send you - is there a way to export my relevant records into either an excel or csv file automatically?

Cheers
Chris




Marpleman

  • Hero Member
  • *****
  • Posts: 738
  • Proper aeroplanes!
Re: Database Updater Team progress report
« Reply #9 on: May 27, 2010, 07:59:03 PM »
Hi Marpleman,

Have just read your opening message in this thread - a very comprehensive and encouraging update as to what is being done to improve the database - many thanks.

I have just run a filtered report from MyFlights since May 1st. It generated 58 records where the Mode-S didn't tie up with any Reg and a further 56 records where a reg tie-up occurred but the aircraft ICAO type showed as '...'. (And I only run ANRB a couple of hours a day from an internal aerial in a 'dip - and always online for supposed updates'!)

Don't have the time to enter all this missing data manually into an Excel file to send you - is there a way to export my relevant records into either an excel or csv file automatically?

Cheers
Chris





Hi Chris

I'll pm you in the next hour if that's ok?

thanks

Rich

Pinza

  • Full Member
  • ***
  • Posts: 248
Re: Database Updater Team progress report
« Reply #10 on: May 27, 2010, 08:14:03 PM »
Quote

Hi Chris

I'll pm you in the next hour if that's ok?

thanks

Rich

Hi Rich,

Look forward to your PM - may not be able to reply tonight - but have long BH weekend in prospect to get to grips with any suggestions!

Chris

54901 - Jim

  • Hero Member
  • *****
  • Posts: 566
    • N9JHB.com
Re: Database Updater Team progress report
« Reply #11 on: June 03, 2010, 03:38:03 PM »
Rich:

Attached is my current list of blank records.  It's nearly doubled in size since the last submission.

Thanks for your attention!

Jim
KOSH - Oshkosh, Wisconsin USA

DaveReid

  • Hero Member
  • *****
  • Posts: 1815
    • Heathrow last 100 ADS-B arrivals
Re: Database Updater Team progress report
« Reply #12 on: June 03, 2010, 04:02:30 PM »
Attached is my current list of blank records.  It's nearly doubled in size since the last submission.

I'm still confused as to what we're looking at here - again, details for over 80% of those "blanks" are just a mouse-click away.

For example, the first on the list, A4BD40: http://registry.faa.gov/aircraftinquiry/NNum_Results.aspx?NNumbertxt=N404UA

I can't help thinking that you're making your job even harder than it need be by ignoring readily-available (free) resources, or am I missing something ?
This post has been scanned for any traces of negativity, bias, sarcasm and general anti-social behaviour

DaveG

  • Hero Member
  • *****
  • Posts: 543
Re: Database Updater Team progress report
« Reply #13 on: June 03, 2010, 04:15:31 PM »
Dave, not sure who your reply is directed at but I'll add a comment for it anyway.

Finding the details of airframes is not the major problem when we're updating records in the database, as you've said there are lots of location online to find the details, and some for free.

The problem is time, the time it takes to update each single record multiplied by the amount of records needing checking/updating.

No doubt about it the current system of updating the database is good for single or small batches but drives you eyes crazy if doing loads.  Suggestion have been made to Airnav Dev regarding possible solutions as a way forward.  until then its a case of updating till your eyes have had it ;-)

Dave, on another(ish) topic, do you have an empty Navdata file by any chance?
Cornwall, UK

DaveReid

  • Hero Member
  • *****
  • Posts: 1815
    • Heathrow last 100 ADS-B arrivals
Re: Database Updater Team progress report
« Reply #14 on: June 03, 2010, 05:02:24 PM »
The problem is time, the time it takes to update each single record multiplied by the amount of records needing checking/updating.

No doubt about it the current system of updating the database is good for single or small batches but drives you eyes crazy if doing loads.  Suggestion have been made to Airnav Dev regarding possible solutions as a way forward.  until then its a case of updating till your eyes have had it ;-)

I absolutely agree - but I'd have thought that, with all the resources and skills at AirNav's disposal, putting together a routine to import registers in bulk would be a no-brainer. 

It's hard to avoid the conclusion that the support you're getting from your "employer" is pretty pitiful.

Quote
Dave, on another(ish) topic, do you have an empty Navdata file by any chance?

No, but I can easily put one together, give me a day or two.
This post has been scanned for any traces of negativity, bias, sarcasm and general anti-social behaviour