Mass data import

There was an issue with importing all UK post codes from a csv file, about 3m post codes. Importing using rapidstart services (excel import) can cause buffer overflow messages. excel itself has row/size limitations. Increasing MaxNoOfXMLRecordsToSend in config file ClientUserSettings.config from default value 5000 to e.g. 20000 is no problem and can help. Also changing MaxUploadSize in server config file CustomSettings.config is an option (also available via nav service admin console). Better choice for mass data import are dataports (older nav versions) and xmlports.

Another option is to develope a report, which imports the file contents and loops through the lines. quite simple, no memory issues.

create a new report, add following code to trigger OnPreReport():

OnPreReport()
// variables
 PostCode, Record, Post Code
 file, File
 fileName, Text, 250
 line, Text, 1024
 dlg, Dialog
 idx, Integer
 txtValue, Text, 100
// code
 PostCode.DELETEALL(FALSE);
 COMMIT;
 dlg.OPEN('#1###### #2######'); // show progress dialog
 idx := 1;

// downloaded post code file from https://www.doogal.co.uk/PostcodeDownloads.php
// as test file, size: 500k, 2.1m lines, some of them contain obsolete post codes
 fileName := 'c:\temp\England postcodes.csv';

 file.WRITEMODE := FALSE;
 file.TEXTMODE := TRUE;
 file.OPEN(fileName);
 file.READ(line); // skip header line
 WHILE file.READ(line) > 0 DO BEGIN
   // skip obsolete post codes: 2. value = No
   IF SELECTSTR(2,line) = 'Yes' THEN BEGIN 
     PostCode.Code := SELECTSTR(1,line);
     PostCode.VALIDATE(City,GetValue(SELECTSTR(15,line)));
     PostCode.County := GetValue(SELECTSTR(8,line));
     PostCode."Country/Region Code" := 'GB';
     PostCode.INSERT;
     dlg.UPDATE(1,idx);
     dlg.UPDATE(2,PostCode.Code);
     idx += 1;
   END;
 END;

 file.CLOSE;
 dlg.CLOSE;

 MESSAGE(FORMAT(idx) + ' post codes imported.');

LOCAL GetValue(txtValue : Text[100]) : Text
 txtValue := DELCHR(txtValue,'<','"'); // remove leading "  
 txtValue := DELCHR(txtValue,'>','"'); // remove trailing "
 IF STRLEN(txtValue) > 30 THEN // fields City,County are Text[30]
   txtValue := COPYSTR(txtValue,1,30); // cut text, leading 30 chars

 EXIT(txtValue);

report runs with 1.5m valid lines/records about 5 min.

cheers

Copying data to-and-from between Excel & Navision

Great posting from Ashwini Tripathi about copying data between Nav and Excel with Nav Standard Processes:

One of my reader has requested to show him how to export data from Nav Journal to Excel, perform correction and import back to Navision. So let us see how can we perform this and what are the limit…

Source: Copying data to-and-fro between Excel & Navision

Simple Read/Write Excel Data

In a Nav forum there was a question, how to read data from an excel document, change the data in Nav, write data back to the excel document without changing the existing cell formatting. With table ExcelBuffer and the helper assemblies delivered with Nav this cannot be done, because the Write functions e.g. SetCellValueText work with decorators.

So i developed assembly XlsLib. With that assembly read and write actions from an excel document are done very easy. The provided methods are very simple. You can read the cell values, the basic number formats and save it to table Excel Buffer. When writing only the value as is is written back to the cell. No further formatting is read or written.That can also be used, when you work with templates.

Main functions:

XlsWorksheet.GetValue(rowId, colId, value, type):
– Get a value from cell rowId/colId,
– given value is of type text,
– type is of datatype integer, can be 0-3 according option field NumberFormat in table ExcelBuffer, Type=4 means Unknown

XlsWorksheet.SetValue(rowId, colId, value):
– Sets a value in cell rowId/colId. value can be of any type.

Here is a sample code:

TestXlsLib()
// variables
//  XlsWorkbook DotNet XlsLib.Workbook.'XlsLib, Version=1.0.0.0, Culture=neutral, PublicKeyToken=38613a2311532c9a'
// XlsWorksheet DotNet XlsLib.Worksheet.'XlsLib, Version=1.0.0.0, Culture=neutral, PublicKeyToken=38613a2311532c9a'
// value Text
// type Integer

// open excel workbook
XlsWorkbook := XlsWorkbook.Workbook('c:\temp\test.xlsx');
// select sheet 'Sheet2'
XlsWorksheet := XlsWorksheet.Worksheet(XlsWorkbook,'Sheet2');

// read cell values from row=1, column=1-3
XlsWorksheet.GetValue(1, 1, value, type); // e.g. 1234
XlsWorksheet.GetValue(1, 2, value, type); // e.g. 10.10.1990
XlsWorksheet.GetValue(1, 3, value, type); // e.g. 99.12

// Write some values
XlsWorksheet.SetValue(1, 1, 5432);  // an integer value again
XlsWorksheet.SetValue(1, 2, 010175D); // again a date value
XlsWorksheet.SetValue(1, 3, 34.45); // a decimal value

// read values again to check changes
XlsWorksheet.GetValue(1, 1, value, type);
XlsWorksheet.GetValue(1, 2, value, type);
XlsWorksheet.GetValue(1, 3, value, type);

// Save: Save to loaded file filename
//XlsWorkbook.Save();
// SaveAs: create a copy, save result to new file
XlsWorkbook.SaveAs('c:\temp\test2.xlsx');
// Close Excel App !!
XlsWorkbook.Close();

// if an error occurs ...
IF XlsWorkbook.Error <> '' THEN
  MESSAGE(XlsWorkbook.Error);

// release memory
clear(XlsWorkbook);

You can download XlsLib here.

Hint: When downloading that file, it can be, that you have troubles using it because of security issues, e.g. dll is not visible in list of assemblies, you get an error message like “Cannot open xlslib, because one or more dependencies are not met”, methods are not visible or something else. then start windows explorer, goto folder were you copied the lib files, view file properties. if you see a notice at the bottom of the file properties window like “file is blocked because downloaded from internet, grant access?”, then click on “grant access”.

cheers

Working with Excel Documents in Nav

nav-xls

Dynamics Nav is shipped with the opportunity to export every report as Excel document. Additional there is Table ExcelBuffer for exporting each kind of data. An example for this table’s usage is the action “Export to Excel” in Page “G/L Budget”.

In table ExcelBuffer there are used some very helpful .net assemblies shipped with Dynamics Nav to cover the interaction with excel.

In Nav forums there are quite often questions about importing or exporting data from/to Excel. So i wrote a sample, where you can see, how to work with these assemblies.

xls1

  • An excel file is opened
  • Adding a new sheet
  • Write a value to the new sheet
  • Close the WorkbookWriter Object to close the current Excel session
  • Open the Excel document in Excel with standard Excel Interop assembly

xls2

The variables: this is the Nav 2013 version, so here is used Microsoft.Dynamics.Nav.OpenXml vs. 7.0. In Nav 2015 use vs. 8.0 instead.