Data Tools Wizard User's Guide

Frequency Manager Suite

Data Tools Wizard
User’s Guide
Copyright © 2015 Jeff Knapp N8GJL sdrs.freqmgr@cox.net

Introduction

The FMSuite Data Tools Wizard (called DT in the rest of this guide) is a utility program designed to download and import frequency databases for use within the Frequency Manager + Scanner (called FM in the rest of this guide).  It gives you a choice of importing six internet databases, importing the default SDR# Frequency Manager’s database, importing a generic file, and exporting an FM database to a generic file.

System Requirements

Basically if you can run SDR Sharp, you can use Data Tools.
  • Operating Systems: Windows 7, Windows 8.x, and Windows 10.  The binaries are compiled for a 32-bit environment for the broadest compatibility but run equally well in an x64 environment.
  • Processor: 1-gigahertz processor or faster; 2 or more cores is recommended.  Hyper-threading/Multi-threading should be enabled for the best performance. NOTE: SDR# and my plugins and applications are multi-threaded, which means they perform multiple tasks simultaneously.  As a result you may have performance problems if your processor cannot execute at least 4 threads simultaneously.
  • Memory: 1 gigabyte or more total in the computer.
  • Hard Disk: The software requires about 415KB.
  • .NET: Version 4.6 runtime as required by SDR Sharp.
  • SDR#: Revision 1408 or newer.

Important Note
SDR# is a hobby project created by other people and I am not involved in it.  I have no control over changes made to SDR#.  I am completely at their mercy as to how their changes affect DT.  They are under no obligation to inform me of changes; I won’t know if their changes will break my software until the software is run with the new version of SDR# and you tell me there is a problem.  As a result there will be a period after they make a breaking change in which my software may not work.  Please be assured I will make any necessary changes, and release updates, as soon as practical after SDR# is changed and I have identified the issue.

Using Data Tools

Data Tools is a standalone program separate from SDR# and its plugins.  It offers a way to download frequency databases from the internet and import them into an FM database.  It also permits you to import data from a generic data file format and to export an FM database to a generic file format.  Data Tools operates as a wizard; you make choices, fill in some blanks, and it takes care of the rest.

The Data Tools Wizard

Selecting a database action

Download a database from the Internet

Data Tools already knows about several popular frequency databases on the Internet and lets you download and import them into a Frequency Manager + Scanner database.

The download function has 4 steps.  The amount and type of information required for a download can vary according to the desired database.  To move between steps, simply click the Back and Next buttons.
  1. Select an Internet database to download. 
  2. Select an FM database in which to store the data.  If you specify a filename that doesn’t exist you will be prompted for permission to create it.

  3. However if the FM database exists, and it already contains data from the selected internet database, an additional choice is presented to you:  Append new data to the existing data, or Replace the existing data with fresh data from the download.

  4. Specify the Internet download address (the URL) of the database. DT comes with these addresses already supplied; however many databases change their URLs periodically, so if you type the new URL here it will be saved for the next time you wish to download the data.
  5. The download will start and a progress bar will display.  When the data is fully downloaded and imported the wizard’s “Finished” page will display.

Some of the databases require additional information before you can download them.

  • The Classaxe download needs to know which of three databases you want to download:  Europe, North America, or World Wide.  The last choice combines the first two choices into one file and will take longer to download.
  • The FMList download needs to know your latitude and longitude so that the database can be customized for your location.  DT, by default, uses the latitude and longitude of the Royal Observatory Greenwich in the UK.  If this is acceptable you can proceed with the download.  However if you know your latitude and longitude you can type them into the designated fields in decimal format; but Data Tools will look up your latitude and longitude for you if you click the “Find my Location” button.  Your location is determined by one of the free Internet databases that accept an Internet IP address; your computer’s IP will be passed to the database and the latitude and longitude will be retrieved.


Important Note
The location is approximate because these free services cannot derive the latitude and longitude down to your specific house, office, or school; instead it locates the equipment where your neighborhood connects to the internet provider.  This might be 100 feet away or several miles away depending on your internet provider, but it will be close enough for the FMList database to be accurate for this purpose.

Important Note
The ability to customize the database to your latitude and longitude is dependent on that feature being offered by the owners of the FMList database.  At the time the Data Tools Wizard was developed it was a planned feature by its author and was not generally available for use.
When the FMList download progress page displays you may see the text “Download Requested.  This may take a few minutes.”
  • All database downloads check the download URL before requesting the download.  In this manner DT tries to let you know of any issues with the URL before committing to a download.
However the MWList download may require much more time to respond to the URL request than other databases so you may see additional text on the URL wizard page that says “Validating the URL.  This website may take a few minutes, so please be patient.” 
When the MWList download progress page displays you may see the text “Download Requested.  This may take a few minutes” before the progress bar starts to increment.
Why not RadioReference.com?
In all honesty – the free time I have.  Downloading from RR is possible but very involved, and I’ve decided to make that a future enhancement in favor of getting this version’s improvements to you more quickly.

Importing a Generic File

The choice “Import a Generic File” permits you to take a delimited file from any source and import it into an FM database.  There are necessarily some restrictions which are quite common with generic files, and they are described here to help you format your file properly.

  • Delimiters are the characters in a generic file that separate one data field from the next.  DT supports 4 kinds of delimiter characters:
    • Commas
    • Semicolons
    • Tabs
    • Custom, whereby you specify the character.  The most common custom character is the pipe character (“|”)
  • If comma delimiters are used, text fields containing commas must be surrounded by the double-quote character (").  For example if the City of a radio station is  Las Vegas, NV  the field in the Comma-delimited generic file should look like this:

,"Las Vegas, NV",
With other data fields before and after the City.  Numeric and date fields do not require surrounding double-quote characters but do require a delimiter between fields.
  • Any one frequency’s data must be all in one line of content in the generic file.  The end of the line must be marked with a carriage return and a line feed.
  • The fields in your generic file must be in a specific sequence and each field must be a specific data type.  See the Appendix for a table that describes the field sequence and data types.
  • Most of the fields in the file are optional; if you have no data for a field you may simply leave it with no content between the field delimiters.  However you must at least have content for Frequency, Center, Mode, and Filter Bandwidth.
  • You may import a list of Groups by combining them into a list of Groups separated by a caret character (“^”).  For example you might have the local Police helicopters in your source data assigned to the groups “Police” and “Aero”.  To import these into an FM database, format as the two words separated by a caret:
,”Police^Aero”,

As the generic data is read from the file, each line is checked to ensure it is acceptable.  Any lines that are not acceptable will be displayed in an error message containing up to 10 of such lines.  Any line in which the Frequency value cannot be interpreted as a number will be skipped.

One more thing – having the first line of your file filled with the names of the fields is optional.  DT doesn’t require it, and will ignore it if it is found.

Important Note
Do not use Microsoft Excel to edit CSV files.  Excel will not surround text fields with double quotes, and will “helpfully” garble your data by converting large numbers like frequencies into scientific notation and by removing leading zeros from other numbers.  If you must edit a CSV file try one of the several free CSV editing programs available on the Internet.

Exporting a database

When exporting an existing FM database you perform these steps:
  1. Select the FM database that you wish to export.
  2. In the Export Options step of the wizard, specify the file to export the data into.  The other options here are:
    • Your choice of Delimiter:
      • Commas
      • Semicolons
      • Tabs
      • Custom (define the custom character you wish to use)
    • Include Column Headings will put the column names in the first row of the file.
    • Enclose text in double quotes:  this is highly recommended for the best compatibility with other software.
    • Remove Special Characters:  this too is highly recommended because it will remove characters from your text that might cause problems with the software that ultimately imports the file.  For example double quotes inside text are converted to single quotes.
  3. After clicking Next on the Export Options page your file will be created.
The export includes the Group(s) you have defined for each frequency.  If you have defined more than one Group for a frequency all of the groups will be listed in the Groups column of the exported data as a single list; the Group names in the list will be separated by the caret symbol (^).  For example:
,”Clark County Misc^Transporation^Aero”,
The caret symbol is used instead of commas or other common punctuation in order to avoid confusing the application into which you import this exported data; if commas or other common punctuation were used, and if you did not choose Enclose text in double quotes, the groups would throw off the column alignment of the remaining columns on each row and your data import would not work.

Edit default values for missing data

Sometimes a downloadable database will not contain all the data you would like. Using this option you may define default values for missing data before performing a download. When the database is downloaded, missing data from the internet will be replaced with your default values.

Select this option and then click the Next button. A window will appear which looks similar to the Edit window of the Frequency Manager + Scanner. You may enter your default values and click Save to record them. All values are optional; if a value is left blank then the corresponding value will be blank if the incoming data does not contain that value.
As with the Edit window, there are Basic and Extended tabs into which you may enter data. Click Save to save the data. Click Erase All to delete all default values and return them to their empty state. And click Cancel to exit out of the window without saving any changes.
After saving your defaults you are returned to the “I want to” page of the Data Tools Wizard. You may now select a database for importing. In this example we have chosen the AOKI database and have clicked the Next button.
If you put a checkmark in “Apply default values for missing data”, any defaults you created earlier will be applied when the data is imported.

Appendix

Generic File Import Fields and Data Types

A generic file import must have all of these fields, even if blank or empty. Fields marked as “Required” must have values in them.
Field Nbr
Field Name
Data Type
Size
Notes
1
Frequency
Numbers
Up to 11 digits
In Hertz; zero or numbers. Required.
2
Center
Numbers
Up to 11 digits
In Hertz; zero or numbers. Required.
3
Call Sign
Text
Up to 20 characters
Letters and numbers
4
Description
Text
Up to 32,767 characters
Letters, numbers, punctuation
5
Protocol
Text
Up to 20 characters
Letters, numbers, punctuation
6
Service
Text
Up to 20 characters
Letters, numbers, punctuation
7
Groups
Text
Zero or more characters
Letters, numbers, punctuation; group names must be separated by a caret character (“^”)
8
City
Text
Up to 50 characters
Letters, numbers, punctuation
9
Country
Text
Up to 20 characters
Letters, numbers, punctuation
10
Language
Text
Up to 20 characters
Letters, numbers, punctuation
11
Latitude
Text
Up to 20 characters
Letters, numbers, minus sign, decimal point
12
Longitude
Text
Up to 20 characters
Letters, numbers, minus sign, decimal point
13
Azimuth
Text
Up to 4 characters
Letters, numbers, minus sign, decimal point
14
Target
Text
Up to 20 characters
Letters, numbers, minus sign, decimal point
15
Days
Text
Up to 20 characters
Letters, numbers, minus sign, decimal point
16
Start Time
Text
4 digits
Typically 0000 through 2359
17
Stop Time
Text
4 digits
Typically 0000 through 2359
18
Power
Numbers
Up to 7 digits
Zero or numbers only. The scale depends on the source of the data.
19
Notes
Text
Up to 32,767 characters
Letters, numbers, punctuation
20
Flagged
Numbers
1 digit
Zero or 1
21
Locked
Numbers
1 digit
Zero or 1
22
Heard
Numbers
1 digit
Zero or 1
23
Date
Text
0 to 20 characters
Letters, numbers, punctuation. This is normally the last date the station was monitored.
24
Mode
Text
Up to 20 characters
From the options provided by SDR#. Required.
25
Shift Enabled
Numbers
1 digit
Zero or 1
26
Shift
Numbers
Digits
In Hertz, in the range supported by SDR#
27
Filter Type
Text
Up to 20 characters
From the options provided by SDR#. Required.
28
Filter Bandwidth
Numbers
Digits
In Hertz, in the range supported by SDR#. Required.
29
Filter Order
Numbers
Digits
In the range supported by SDR#. Required.
30
Squelch Enabled
Numbers
1 digit
Zero or 1
31
Squelch
Numbers
0 to 3 digits
In the range of 0 to 100
32
CW Shift
Numbers
Digits
In Hertz, in the range supported by SDR#
33
Step Size
Numbers
Digits
In Hertz, from the list supported by SDR#. Required.
34
Source
Text
Zero or more characters
Give the source of the data a name

Troubleshooting

I wish I could say that there will never be any challenges, but in reality stuff happens.  Please try the following remedies if you have these problems.

Error Messages

Problem
Potential Solution(s)
The URL could not be verified; please check it for accuracy.
Before requesting a download, DT tests the URL to make sure it is good.  This message appears when that test fails.  Obtain the correct URL and type it into the relevant text box; it will be saved for future use if it is correct.
The Latitude & Longitude finder website returned "404 Not Found". It is recommended that you uninstall and reinstall the Data Tools Wizard to restore the finder's URL.
The IP-to-LatLong URL was incorrect. 
An error ocurred while getting the Latitude & Longitude: timed out.  Try again another time.
The IP-to-LatLong website took too long to respond.
An error ocurred while getting the Latitude & Longitude: followed by the text of the error
An unexpected error was received when attempting to look up your Latitude and Longitude.  Resolve the error and try again.
An error occurred while attempting to download the data.  The error is: followed by the text of the error
An unexpected error was received when attempting to download the internet database.  Resolve the error and try again.
An error ocurred:  The downloaded file was empty.
The download was successful but the file that was downloaded had no content.
No data was downloaded.  Check the URL for correctness or try again later.
The website URL was genuine but did not have a file to download.
The downloaded file was empty or could not be converted into data.
The download worked but the resulting file was empty or contained invalid or unexpected data.
Line X has less than N columns.
The generic file being imported has too few columns.  This can happen because the file has too few columns overall, or because a character such as a carriage return has split a single line into 2 lines.  Open the file in a text editor and examine line X to see what the problem is.
Line X has more than N columns.
The generic file being imported has too many columns.  This can happen because some field had a delimiter character in it (for example a comma in a name or description) and the line was split into an additional column.  Open the file in a text editor and examine line X to see what the problem is.  You may need to remove delimiters from inside text fields, or surround text fields with double quotes, in order to get an accurate import.
Line X has an invalid ‘Y’; line was skipped due to the error: Z
The generic file being imported has invalid data in line X, column Y.  Error message Z explains the problem with the data.
Line X has data that cannot be interpreted in column ‘Y' : ‘Z
The generic file being imported has invalid data in line X, column Y.  Error message Z explains the problem with the data.
An error occurred while attempting to import the data.  The error is: followed by the text of the error
This generic file import error typically occurs because the specified file is in an incompatible format.
An error occurred while attempting to import the data. Some data is in an unexpected format or data is in the wrong column. Check your file and make sure that there are no column headings,that only text is in text columns, that only numbers are in numeric columns, etc.
The generic file being imported had field formatting errors.
An error ocurred:  The zip file did not contain the desired file: followed by the name of the expected file
Some database downloads, like HFCC, send a zip file containing one or more data files.  The named file did not exist in the zip.
An error occurred while parsing the HFCC files: followed by the text of the error
One or more of the HFCC files could not be read accurately.
The file contains data but it is not in a compatible format.  Unable to import.
Generally occurs when an SDR# Frequency Manager database (NOT the Frequency Manager + Scanner database) is not in the expected format.
The data is in an unexpected format.  Unable to import.
The file to import is in an unfamiliar format.

Known Issues

None.

Notices

  • “SDR Sharp”, “SDR#”, and the SDR# software are Copyright © Youssef TOUIL 2012.
  • “FreqMgr”, “Frequency Manager + Scanner”, “Frequency Manager Suite” and software distributed with the Frequency Manager Suite are Copyright © 2013-2015 Jeff Knapp.

7 comments:

  1. How do I create my own databases and can I copy and paste from radioreference.com?

    ReplyDelete
    Replies
    1. Hi Joseph, thanks for writing.

      There are several ways to create a database:
      • An empty database is created for you the first time you run FMS. To add frequencies, just tune to a frequency and click the Edit button; this will open a dialog into which you can type the relevant information. This is the slowest and most labor-intensive method.
      • You can go into FMS Preferences and check the box titled “Log newly found frequencies to the database” in the Scan Rules tab. This will add each new frequency to a new database record to which you can add more info later via the Browse window. This method is faster but still requires you to hand-enter a lot of data.
      • You can import lists of data from various sources including Radio Reference. The Data Tools Wizard already knows about several popular internet databases but most of those are shortwave, medium wave, and FM. If you want local VHF/UHF frequencies you may be able to find an existing list or download a list of data from RR. In these cases you need to edit the list into a CSV or Tab-delimited file which you can later import using the Generic Import feature of the Data Tools Wizard. Instructions are in the Data Tools Wizard User’s Guide. Take note that the data you import must be in a specific format; all the listed fields are required and most can be blank – but those fields highlighted as “Required” must have valid data in them.

      As for copy and paste from RR, that is a ton of work for you. Try to download a list from them instead and reformat it to match the requirements of the Generic Import tool.

      I hope this helps; let me know if I can do anything else. Thanks for your interest and thanks for using my software.

      Cheers,
      Jeff

      Delete
  2. Where is the download link for the Data Tools Wizard? Can't find it. Thanks!

    ReplyDelete
    Replies
    1. Hi Davis. It is installed as part of the Frequency Manager Suite. You should have a shortcut for it already on your computer; if you can't find the shortcut, the app is in the same folder as SDR#.

      Thanks for using my software. Cheers,
      Jeff

      Delete
  3. hello there,
    How can I export reading of SDR# to a file ?

    ReplyDelete
    Replies
    1. Hi. I'm not sure I understand your question. Do you mean that you want to export the stations that SDR# stops on when scanning with my scanner plugin?

      Delete
  4. Absolutely CAN NOT get this application to launch. Tried reinstalling FMSuite as well as .NET 5.0.0 Desktop runtime.

    Application crashes immediately with the error message "FMSuite.DataTools.exe has stopped working"
    Problem Event Name: APPCRASH
    Application Name: FMSuite.DataTools.exe
    Application Version: 2.2.1.0
    Application Timestamp: 60c8ffcd
    Fault Module Name: KERNELBASE.dll
    Fault Module Version: 6.1.7601.24545
    Fault Module Timestamp: 5e0eb6bd
    Exception Code: e0434352
    Exception Offset: 000000000000b87d
    OS Version: 6.1.7601.2.1.0.256.48
    Locale ID: 1033
    Additional Information 1: 8d4e
    Additional Information 2: 8d4e57bd40baf532f2c22e095c09a8c0
    Additional Information 3: 4f0b
    Additional Information 4: 4f0b288510137abea100542dd3fb4879

    ReplyDelete