Frequency Manager Suite
Data Tools Wizard
User’s Guide
Copyright © 2015 Jeff Knapp N8GJL sdrs.freqmgr@cox.net
Introduction
The FMSuite Data Tools Wizard
(called DT in the rest of this guide) is a utility program designed to
download and import frequency databases for use within the Frequency Manager
+ Scanner (called FM in the rest of this guide). It gives you a choice
of importing six internet databases, importing the default SDR# Frequency Manager’s
database, importing a generic file, and exporting an FM database to a generic
file.
System Requirements
Basically if you can run SDR Sharp, you can use Data Tools.
- Operating Systems: Windows 7, Windows 8.x, and Windows 10. The binaries are compiled for a 32-bit environment for the broadest compatibility but run equally well in an x64 environment.
- Processor: 1-gigahertz processor or faster; 2 or more cores is recommended. Hyper-threading/Multi-threading should be enabled for the best performance. NOTE: SDR# and my plugins and applications are multi-threaded, which means they perform multiple tasks simultaneously. As a result you may have performance problems if your processor cannot execute at least 4 threads simultaneously.
- Memory: 1 gigabyte or more total in the computer.
- Hard Disk: The software requires about 415KB.
- .NET: Version 4.6 runtime as required by SDR Sharp.
- SDR#: Revision 1408 or newer.
Important
Note
|
|
SDR# is a hobby
project created by other people and I am not involved in it. I
have no control over changes made to SDR#. I am completely at
their mercy as to how their changes affect DT. They are under
no obligation to inform me of changes; I won’t know if their changes
will break my software until the software is run with the new version
of SDR# and you tell me there is a problem. As a result there
will be a period after they make a breaking change in which my
software may not work. Please be assured I will make any necessary
changes, and release updates, as soon as practical after SDR# is changed
and I have identified the issue.
|
Using Data Tools
Data Tools is a standalone program separate from SDR# and
its plugins. It offers a way to download frequency databases from the
internet and import them into an FM database. It also permits you to import
data from a generic data file format and to export an FM database to a generic
file format. Data Tools operates as a wizard; you make choices, fill in
some blanks, and it takes care of the rest.
The Data Tools Wizard
Selecting a database action
Download a database from the Internet
Data Tools already knows about several popular frequency databases on the Internet
and lets you download and import them into a Frequency Manager + Scanner database.
The download function has 4 steps. The amount and
type of information required for a download can vary according to the desired
database. To move between steps, simply click the Back and Next buttons.
- Select an Internet database to download.
- Select an FM database in which to store the data. If you specify a filename that doesn’t exist you will be prompted for permission to create it.
- Specify the Internet download address (the URL) of the database. DT comes with these addresses already supplied; however many databases change their URLs periodically, so if you type the new URL here it will be saved for the next time you wish to download the data.
- The download will start and a progress bar will display. When the data is fully downloaded and imported the wizard’s “Finished” page will display.
However if the FM database exists, and it already contains data from the selected internet database, an
additional choice is presented to you: Append new data to the existing
data, or Replace the existing data with fresh data from the download.
Some of the databases require additional information before
you can download them.
- The Classaxe download needs to know which of three databases you want to download: Europe, North America, or World Wide. The last choice combines the first two choices into one file and will take longer to download.
- The FMList download needs to know your latitude and longitude so that the database can be customized for your location. DT, by default, uses the latitude and longitude of the Royal Observatory Greenwich in the UK. If this is acceptable you can proceed with the download. However if you know your latitude and longitude you can type them into the designated fields in decimal format; but Data Tools will look up your latitude and longitude for you if you click the “Find my Location” button. Your location is determined by one of the free Internet databases that accept an Internet IP address; your computer’s IP will be passed to the database and the latitude and longitude will be retrieved.
Important
Note
|
|
The location is
approximate because these free services cannot derive the latitude and
longitude down to your specific house, office, or school; instead it
locates the equipment where your neighborhood connects to the internet
provider. This might be 100 feet away or several miles away depending
on your internet provider, but it will be close enough for the FMList
database to be accurate for this purpose.
|
Important
Note
|
|
The ability to
customize the database to your latitude and longitude is dependent on
that feature being offered by the owners of the FMList database.
At the time the Data Tools Wizard was developed it was a planned feature
by its author and was not generally available for use.
|
When the FMList download
progress page displays you may see the text “Download Requested. This
may take a few minutes.”
- All database downloads check the download URL before requesting the download. In this manner DT tries to let you know of any issues with the URL before committing to a download.
However the MWList download
may require much more time to respond to the URL request than other databases
so you may see additional text on the URL wizard page that says “Validating
the URL. This website may take a few minutes, so please be patient.”
When the MWList download
progress page displays you may see the text “Download Requested. This
may take a few minutes” before the progress bar starts to increment.
Why not RadioReference.com?
In all honesty – the free time I have. Downloading
from RR is possible but very involved, and I’ve decided to make that a future
enhancement in favor of getting this version’s improvements to you more quickly.
Importing a Generic File
The choice “Import a Generic File” permits you to take
a delimited file from any source and import it into an FM database. There
are necessarily some restrictions which are quite common with generic files,
and they are described here to help you format your file properly.
- Delimiters are the characters in a generic file that
separate one data field from the next. DT supports 4 kinds of delimiter
characters:
- Commas
- Semicolons
- Tabs
- Custom, whereby you specify the character. The most common custom character is the pipe character (“|”)
- If comma delimiters are used, text fields containing commas must be surrounded by the double-quote character ("). For example if the City of a radio station is Las Vegas, NV the field in the Comma-delimited generic file should look like this:
,"Las Vegas, NV",
With other data fields before and after
the City. Numeric and date fields do not require surrounding double-quote
characters but do require a delimiter between fields.
- Any one frequency’s data must be all in one line of content in the generic file. The end of the line must be marked with a carriage return and a line feed.
- The fields in your generic file must be in a specific sequence and each field must be a specific data type. See the Appendix for a table that describes the field sequence and data types.
- Most of the fields in the file are optional; if you have no data for a field you may simply leave it with no content between the field delimiters. However you must at least have content for Frequency, Center, Mode, and Filter Bandwidth.
- You may import a list of Groups by combining them into a list of Groups separated by a caret character (“^”). For example you might have the local Police helicopters in your source data assigned to the groups “Police” and “Aero”. To import these into an FM database, format as the two words separated by a caret:
,”Police^Aero”,
As the generic data is read from the file, each line is
checked to ensure it is acceptable. Any lines that are not acceptable
will be displayed in an error message containing up to 10 of such lines.
Any line in which the Frequency value cannot be interpreted as a number will
be skipped.
One more thing – having the first line of your file filled
with the names of the fields is optional. DT doesn’t require it, and will
ignore it if it is found.
Important
Note
|
|
Do not use Microsoft
Excel to edit CSV files. Excel will not surround text fields with
double quotes, and will “helpfully” garble your data by converting large
numbers like frequencies into scientific notation and by removing leading
zeros from other numbers. If you must edit a CSV file try one
of the several free CSV editing programs available on the Internet.
|
Exporting a database
When exporting an existing FM database you perform these
steps:
- Select the FM database that you wish to export.
- In the Export Options step of the wizard, specify the
file to export the data into. The other options here are:
- Your choice of Delimiter:
- Commas
- Semicolons
- Tabs
- Custom (define the custom character you wish to use)
- Include Column Headings will put the column names in the first row of the file.
- Enclose text in double quotes: this is highly recommended for the best compatibility with other software.
- Remove Special Characters: this too is highly recommended because it will remove characters from your text that might cause problems with the software that ultimately imports the file. For example double quotes inside text are converted to single quotes.
- Your choice of Delimiter:
- After clicking Next on the Export Options page your file will be created.
The export includes the Group(s)
you have defined for each frequency. If you have defined more than one
Group for a frequency all of the groups will be listed in the Groups column
of the exported data as a single list; the Group names in the list will be separated
by the caret symbol (^). For example:
,”Clark County Misc^Transporation^Aero”,
The caret symbol is used instead
of commas or other common punctuation in order to avoid confusing the application
into which you import this exported data; if commas or other common punctuation
were used, and if you did not choose Enclose text in double quotes, the
groups would throw off the column alignment of the remaining columns on each
row and your data import would not work.
Edit default values for missing data
Sometimes a downloadable database will not contain all the data you would like. Using this option you may define default values for missing data before performing a download. When the database is downloaded, missing data from the internet will be replaced with your default values.
Select this option and then click the Next button. A window will appear which looks similar to the Edit window of the Frequency Manager + Scanner. You may enter your default values and click Save to record them. All values are optional; if a value is left blank then the corresponding value will be blank if the incoming data does not contain that value.
As with the Edit window, there are Basic and Extended tabs into which you may enter data. Click Save to save the data. Click Erase All to delete all default values and return them to their empty state. And click Cancel to exit out of the window without saving any changes.
After saving your defaults you are returned to the “I want to” page of the Data Tools Wizard. You may now select a database for importing. In this example we have chosen the AOKI database and have clicked the Next button.
If you put a checkmark in “Apply default values for missing data”, any defaults you created earlier will be applied when the data is imported.
Appendix
Generic File Import Fields and Data Types
A generic file import must have all of these fields, even if blank or empty. Fields marked as “Required” must have values in them.
Field Nbr
|
Field
Name
|
Data
Type
|
Size
|
Notes
|
1
|
Frequency
|
Numbers
|
Up to 11 digits
|
In Hertz; zero or numbers. Required.
|
2
|
Center
|
Numbers
|
Up to 11 digits
|
In Hertz; zero or numbers. Required.
|
3
|
Call Sign
|
Text
|
Up to 20 characters
|
Letters and numbers
|
4
|
Description
|
Text
|
Up to 32,767 characters
|
Letters, numbers,
punctuation
|
5
|
Protocol
|
Text
|
Up to 20 characters
|
Letters, numbers,
punctuation
|
6
|
Service
|
Text
|
Up to 20 characters
|
Letters, numbers, punctuation
|
7
|
Groups
|
Text
|
Zero or more characters
|
Letters, numbers,
punctuation; group names must be separated by a caret character (“^”)
|
8
|
City
|
Text
|
Up to 50 characters
|
Letters, numbers,
punctuation
|
9
|
Country
|
Text
|
Up to 20 characters
|
Letters, numbers, punctuation
|
10
|
Language
|
Text
|
Up to 20 characters
|
Letters, numbers,
punctuation
|
11
|
Latitude
|
Text
|
Up to 20 characters
|
Letters, numbers, minus
sign, decimal point
|
12
|
Longitude
|
Text
|
Up to 20 characters
|
Letters, numbers, minus
sign, decimal point
|
13
|
Azimuth
|
Text
|
Up to 4 characters
|
Letters, numbers, minus
sign, decimal point
|
14
|
Target
|
Text
|
Up to 20 characters
|
Letters, numbers, minus
sign, decimal point
|
15
|
Days
|
Text
|
Up to 20 characters
|
Letters, numbers, minus
sign, decimal point
|
16
|
Start Time
|
Text
|
4 digits
|
Typically 0000 through 2359
|
17
|
Stop Time
|
Text
|
4 digits
|
Typically 0000 through 2359
|
18
|
Power
|
Numbers
|
Up to 7 digits
|
Zero or numbers only. The
scale depends on the source of the data.
|
19
|
Notes
|
Text
|
Up to 32,767 characters
|
Letters, numbers,
punctuation
|
20
|
Flagged
|
Numbers
|
1 digit
|
Zero or 1
|
21
|
Locked
|
Numbers
|
1 digit
|
Zero or 1
|
22
|
Heard
|
Numbers
|
1 digit
|
Zero or 1
|
23
|
Date
|
Text
|
0 to 20 characters
|
Letters, numbers,
punctuation. This is normally the last date the station was monitored.
|
24
|
Mode
|
Text
|
Up to 20 characters
|
From the options provided
by SDR#. Required.
|
25
|
Shift Enabled
|
Numbers
|
1 digit
|
Zero or 1
|
26
|
Shift
|
Numbers
|
Digits
|
In Hertz, in the range
supported by SDR#
|
27
|
Filter Type
|
Text
|
Up to 20 characters
|
From the options provided
by SDR#. Required.
|
28
|
Filter Bandwidth
|
Numbers
|
Digits
|
In Hertz, in the range
supported by SDR#. Required.
|
29
|
Filter Order
|
Numbers
|
Digits
|
In the range supported by
SDR#. Required.
|
30
|
Squelch Enabled
|
Numbers
|
1 digit
|
Zero or 1
|
31
|
Squelch
|
Numbers
|
0 to 3 digits
|
In the range of 0 to 100
|
32
|
CW Shift
|
Numbers
|
Digits
|
In Hertz, in the range
supported by SDR#
|
33
|
Step Size
|
Numbers
|
Digits
|
In Hertz, from the list
supported by SDR#. Required.
|
34
|
Source
|
Text
|
Zero or more characters
|
Give the source of the data
a name
|
Troubleshooting
I wish I could say that
there will never be any challenges, but in reality stuff happens. Please
try the following remedies if you have these problems.
Error Messages
Problem
|
Potential Solution(s)
|
The URL could not be verified; please check it
for accuracy.
|
Before requesting a download, DT tests the URL
to make sure it is good. This message appears when that test fails.
Obtain the correct URL and type it into the relevant text box; it will
be saved for future use if it is correct.
|
The Latitude & Longitude finder website returned "404
Not Found". It is recommended that you uninstall and reinstall
the Data Tools Wizard to restore the finder's URL.
|
The IP-to-LatLong URL was incorrect.
|
An error ocurred while getting the Latitude &
Longitude: timed out. Try again another time.
|
The IP-to-LatLong website took too long to respond.
|
An error ocurred while getting the Latitude &
Longitude: followed by the text of the error
|
An unexpected error was received when attempting
to look up your Latitude and Longitude. Resolve the error and
try again.
|
An error occurred while attempting to download
the data. The error is: followed by the text of the error
|
An unexpected error was received when attempting
to download the internet database. Resolve the error and try again.
|
An error ocurred: The downloaded file was
empty.
|
The download was successful but the file that was
downloaded had no content.
|
No data was downloaded. Check the URL for
correctness or try again later.
|
The website URL was genuine but did not have a
file to download.
|
The downloaded file was empty or could not be converted
into data.
|
The download worked but the resulting file was
empty or contained invalid or unexpected data.
|
Line X has less than N columns.
|
The generic file being imported has too few columns.
This can happen because the file has too few columns overall, or because
a character such as a carriage return has split a single line into 2
lines. Open the file in a text editor and examine line X
to see what the problem is.
|
Line X has more than N columns.
|
The generic file being imported has too many columns.
This can happen because some field had a delimiter character in it (for
example a comma in a name or description) and the line was split into
an additional column. Open the file in a text editor and examine
line X to see what the problem is. You may need to remove
delimiters from inside text fields, or surround text fields with double
quotes, in order to get an accurate import.
|
Line X has an invalid ‘Y’; line was skipped
due to the error: Z
|
The generic file being imported has invalid data
in line X, column Y. Error message Z explains
the problem with the data.
|
Line X has data that cannot be interpreted
in column ‘Y' : ‘Z’
|
The generic file being imported has invalid data
in line X, column Y. Error message Z explains
the problem with the data.
|
An error occurred while attempting to import the
data. The error is: followed by the text of the error
|
This generic file import error typically occurs
because the specified file is in an incompatible format.
|
An error occurred while attempting to import the
data. Some data is in an unexpected format or data is in the wrong column.
Check your file and make sure that there are no column headings,that
only text is in text columns, that only numbers are in numeric columns,
etc.
|
The generic file being imported had field formatting
errors.
|
An error ocurred: The zip file did not contain
the desired file: followed by the name of the expected file
|
Some database downloads, like HFCC, send a zip
file containing one or more data files. The named file did not
exist in the zip.
|
An error occurred while parsing the HFCC files:
followed by the text of the error
|
One or more of the HFCC files could not be read
accurately.
|
The file contains data but it is not in a compatible
format. Unable to import.
|
Generally occurs when an SDR# Frequency Manager
database (NOT the Frequency Manager + Scanner database) is not in the
expected format.
|
The data is in an unexpected format. Unable
to import.
|
The file to import is in an unfamiliar format.
|
Known Issues
None.
Notices
- “SDR Sharp”, “SDR#”, and the SDR# software are Copyright © Youssef TOUIL 2012.
- “FreqMgr”, “Frequency Manager + Scanner”, “Frequency Manager Suite” and software distributed with the Frequency Manager Suite are Copyright © 2013-2015 Jeff Knapp.
How do I create my own databases and can I copy and paste from radioreference.com?
ReplyDeleteHi Joseph, thanks for writing.
DeleteThere are several ways to create a database:
• An empty database is created for you the first time you run FMS. To add frequencies, just tune to a frequency and click the Edit button; this will open a dialog into which you can type the relevant information. This is the slowest and most labor-intensive method.
• You can go into FMS Preferences and check the box titled “Log newly found frequencies to the database” in the Scan Rules tab. This will add each new frequency to a new database record to which you can add more info later via the Browse window. This method is faster but still requires you to hand-enter a lot of data.
• You can import lists of data from various sources including Radio Reference. The Data Tools Wizard already knows about several popular internet databases but most of those are shortwave, medium wave, and FM. If you want local VHF/UHF frequencies you may be able to find an existing list or download a list of data from RR. In these cases you need to edit the list into a CSV or Tab-delimited file which you can later import using the Generic Import feature of the Data Tools Wizard. Instructions are in the Data Tools Wizard User’s Guide. Take note that the data you import must be in a specific format; all the listed fields are required and most can be blank – but those fields highlighted as “Required” must have valid data in them.
As for copy and paste from RR, that is a ton of work for you. Try to download a list from them instead and reformat it to match the requirements of the Generic Import tool.
I hope this helps; let me know if I can do anything else. Thanks for your interest and thanks for using my software.
Cheers,
Jeff
Where is the download link for the Data Tools Wizard? Can't find it. Thanks!
ReplyDeleteHi Davis. It is installed as part of the Frequency Manager Suite. You should have a shortcut for it already on your computer; if you can't find the shortcut, the app is in the same folder as SDR#.
DeleteThanks for using my software. Cheers,
Jeff
hello there,
ReplyDeleteHow can I export reading of SDR# to a file ?
Hi. I'm not sure I understand your question. Do you mean that you want to export the stations that SDR# stops on when scanning with my scanner plugin?
DeleteAbsolutely CAN NOT get this application to launch. Tried reinstalling FMSuite as well as .NET 5.0.0 Desktop runtime.
ReplyDeleteApplication crashes immediately with the error message "FMSuite.DataTools.exe has stopped working"
Problem Event Name: APPCRASH
Application Name: FMSuite.DataTools.exe
Application Version: 2.2.1.0
Application Timestamp: 60c8ffcd
Fault Module Name: KERNELBASE.dll
Fault Module Version: 6.1.7601.24545
Fault Module Timestamp: 5e0eb6bd
Exception Code: e0434352
Exception Offset: 000000000000b87d
OS Version: 6.1.7601.2.1.0.256.48
Locale ID: 1033
Additional Information 1: 8d4e
Additional Information 2: 8d4e57bd40baf532f2c22e095c09a8c0
Additional Information 3: 4f0b
Additional Information 4: 4f0b288510137abea100542dd3fb4879