Data Logging Application
The main objective of this application, called Data Logging Application, is to store data (online while they are generated) in a TDMS file so that one can view data at the later stages. This application is capable of storing data available on the National Instrument Shared Variable Engine (NI SVE). Transferring data from other sources to NI SVE, for example using OPC clients, you can record data from other resources.
Along with this application, “Data Viewer”, which is an independent application, is delivered (I hope I can finish it soon). Data Viewer allows you to view and plot logged data while or after recording.
Have a look here to see how you can test the application.
Here is a picture of the user interface.
- Stand-alone application that can be executed on an independent PC - no need to be on a computational PC
- Storing data in a binary-size file (of TDMS format) that is readable in other applications such as Excel, MATLAB, Python and LabVIEW
- Capable of storing data of 36 data types
- Storing data with timestamp and adjustable sampling frequency (rate)
- Storing data in a rich documented file with possibility of adding attribute and tracking appendices
- Viewing and plotting stored data on-line (while logging) and off-line – using Data Viewer application
- Automatically saving and restoring data logging setting as customizable projects
- No need for additional hardware – The sole piece of hardware required is a PC
- Windows 7 or higher
- Minimum 2GB RAM
- NI Run-time Engine 2014
The data logging applications supports 36 data types including scalars, 1-dimensional, and 2-dimensional arrays. You can find supported data types for data logging in Table 1 in the user guide.
To give you a brief overview, it can store scalars as well as 1 or 2 dimensional arrays of the following data types: boolean, int/uint(8,16,32,64 bits), single-precision(32 bits) or double precision(64 bits) floating point and string.
To start with logging, follow the instruction below:
- Set the locations for the TDMS log file and the database file. If the database for a selected library is not available, the application can make a base database for data logging
- Select the file name.
- Initialize the application. Deployed Libraries are then shown.
- Select one deployed library.
- Check the database status. If the database is available, skip step 5.
- Create database. This is disabled if the database is available.
- Select a deployed library and variables that you want to log.
- Set the sample rate and the server name.
- (Optional) Add more information to the log file.
- Start recording.
- Stop recording when you want. The logged file is found in the location you specified in step 0.
Here, I would like to pinpoint some cool features of this application. To get to know the entire features, you need to have a look at the user guide either in /support/documentation or after installation using menu: Help -> User Giude
In development, there are occasions that you need to have many windows open. Some applications like a data logging application, can run in background and you do not need to see them. Actually, I personally like to free up space in the windows task bar if I am logging data for long time, and make the screen look neater.
Minimizing the application will place the icon in system tray. Double clicking the system tray icon you will get the UI back at the screen.
You can disable the system tray feature using menu: View -> System Tray
The data logging relies on a database which contains information for the selected variables. This is to have faster logging. The database holds the name and data type for each selected variable.
Database is not more than a text file saved as csv or an excel sheet.
The database templates are found using the menu bar: Help -> Database Template
The application looks for database with the same name as the deployed library with extensions “.xlsx”, “.xls”, and “.csv”, in order. If more than one database is found, the first found database is used but the following message is shown to the user.
The application warns you if a database cannot be found for the selected deployed library (process). To facilitate making a database, you can simply press Create Database. This will create a database for all variables in the designated folder as specified in the folder settings.
Note: In case of a change in the deployed library, you can recreate the database using menu: Tools -> Database -> Replace the existing database.
Datebase format can be xlsx or csv. It can be specified using menu: Settings -> Advanced Settings -> Database Type. Also, you can specify the delimiter for the csv format using Settings -> Advanced Settings -> CSV Delimiter.
Note: The database idea will be important when you realize that you can create the library using SV Creation Wizard when you have the database ready. Then, maintaining shared variable list and logging is done using a single database.
One of the limitations of the application is that you can selected variables from one deployed library. A solution to log from several processes that can be deployed on different servers is to create one new library and bind variables to those that you want.
This is possible to do using the database. See the templates for the database using the menu bar: Help -> Database Template
A file is fragmented when data are not stored in sequentially/continuously. File fragmentation may result in large file size and may degrade performance (e.g. may slow down reading process).
In the data logging application, there are two approaches to avoid/cure fragmented files.
- File Defragment. This option defrags a TDMS file by removing fragmentation, and making the log file contiguous. There is an Auto Defrag option in the Preference in Setting Pane (alternatively: Ctrl+2 or menu bar: Settings -> Preference). If it is ON, the TDMS file is defragged after stopping recording.
It is also possible to defrag a selected TDMS file using menu: go to Tools -> Defrag
- Adjustable Buffer Size. With small buffer size, it is more likely to have fragmented files. Therefore, increasing the buffer size will increase the size of data chunks that are written in memory blocks. In turn, file fragmentation is smaller. The buffer size can be adjusted in Preference in Setting Pane (alternatively: Ctrl+2 or menu bar: Settings -> Preference).
Note: With large buffer size, the Data Viewer (currently, another independent application) in online mode is updated when the new buffer of data is written in TDMS file.
If the user switch of Auto Defrag mode and the buffer Size is less than 50, the following warning is shown to user before recording.
The application is made robust with respect to exceptions and errors.
The custom error list can be found under /support/error lists.
If you start recording on an existing file, new data are appended to the file. In the file properties, there are 2 properties related to this.
- Appendix_no which shows the number of re-use of the file
- DateTime_App_{i} which shows the timestamp for one specific append
See the below image. It shows that the log file has been appended 2 times after the first use which was on 6/8/2018 15:36:49.382. The timestamps for appends are also given.
Here is the list of short keys.
See this Wiki page.