Bite Size C# – XLS File Merger

This will be the last Bite Size C# post revolving around CSV and XLS files (for a while at least). The next one will be on a different and probably a bit more exciting topic.

Today we will look at my XLS File Merger, which resides in the Core.XLS namespace of my Core.dll library. It is a static class with one static method MergeFiles. The purpose of this method is to merge 2 or more XLS files into a single file, i.e. taking the worksheets of 2 or more files and combing them into a single file. The MergeFiles method takes 2 arguments: a list of locations of files to be merged (filePathList) and the destination file that will be created (outputFile).

Here is the code:

using System;
using Microsoft.Office.Interop.Excel;

namespace Core.XLS
{
    public static class XlsFileMerge
    {
        public static void MergeFiles(string[] filePathList, string outputFile)
        {
            var app = new Application {Visible = false};

            app.Workbooks.Add("");
            foreach (var file in filePathList)
            {
                app.Workbooks.Add(file);
            }


            for (var i = 2; i <= app.Workbooks.Count; i++)
            {
                var count = app.Workbooks[i].Worksheets.Count;

                app.Workbooks[i].Activate();
                for (var j = 1; j <= count; j++)
                {
                    var ws = (_Worksheet) app.Workbooks[i].Worksheets[j];
                    if (ws.UsedRange.Rows.Count <= 1) continue;
                    ws.Select(Type.Missing);
                    ws.Cells.Select();


                    var sel = (Range) app.Selection;
                    sel.Copy(Type.Missing);

                    var sheet = (_Worksheet) app.Workbooks[1].Worksheets.Add(
                        Type.Missing, Type.Missing, Type.Missing, Type.Missing
                                                 );
                    sheet.Name = ws.Name;
                    sheet.Paste(Type.Missing, Type.Missing);
                }
            }
            app.Workbooks[1].SaveAs(@outputFile);
            app.Quit();
        }
    }
}
Bite Size C# – XLS File Merger

Roaming Robot Update

robot1

My Roaming robot has recently been giving me some problems, specifically regarding the power distribution between the 2 motors (with one wheel sporadicly turning faster than the other). I initially thought that one of the motors might have been damaged so I replaced both, only to have the problem remain. After some testing and replacing both the Pololu Dual MC33926 Motor Driver Shield and the Arduino Uno R3, I determined that the problem lay with the Pololu Motor shield. I am not sure if one of the other sensors or actuators was causing some interference on the shield or if it is simply a design flaw in the shield, but I found it impossible to balance the power of the 2 motors. 

I thus decided to replace the Pololu Dual MC33926 Motor Driver Shield with a Pololu DRV8833 Dual Motor Driver Carrier, and this resolved the problem I was experiencing. 

Here the Pololu Dual MC33926 Motor Driver Shield and the Pololu DRV8833 Dual Motor Driver Carrier can be seen side by side (the  Pololu DRV8833 Dual Motor Driver Carrier being the much smaller of the 2): 

motordrivers

I mounted the Motor driver on a bread board, attached on top of an Adafruit Proto-shield. This has the added benefit of having screw terminals for the Arduino pins, so that the jumper cables can no longer accidentally become dislodged.

Proto      protodriver

To install the new DRV8833 Motor Driver I had to switch some of the pin usage on the Arduino in order to free up some PWM digital pins needed by the motor driver. The new Layout is as follows:

Analog Pins:

  • A0 
  • A1 
  • A2 – Servo 
  • A3
  • A4 – IR Sensor
  • A5 – Photo-resistor

Digital Pins:

  • 0
  • 1
  • 2 – LED
  • 3 – Motor Driver B IN 1
  • 5 – Motor Driver B IN 2
  • 6 – Motor Driver A IN 1
  • 7 – Ultrasonic Sensor
  • 8 – Right Trigger Switch
  • 9 – Motor Driver A IN 2
  • 10 
  • 11 – Left Trigger Switch
  • 12 
  • 13

Here are the updated drawings illustrating the wiring of the different components:

robot 1_bb

robot part 2_bb

robot part 3_bb2


I also replaced some of the wiring to the Ultrasonic sensor as they had become worn due to the “neck movement” of the servo pointing the sensor in different directions.

I also ran the power for the Ultrasonic sensor straight from the Arduino and no longer via the Proto-board, this was simply to give the cables a bit more play for the “neck movement” performed by the servo.

After these changes the little guy is working perfectly again.

I will be posting a video of this robot in action in the near future.

Here is the updated code for the robot:


#include "Servo.h" 

#define IR_PIN A4
#define SERVO_PIN A2
#define PING_PIN 7
#define BUMPER_LEFT_PIN 11
#define BUMPER_RIGHT_PIN 8
#define LDR_PIN A5
#define LED_PIN 2
#define IR_DROP 500 //Distance considered a potential drop
#define MIN_LIGHT 300 //Level of light to turn on LED for light
#define BIN_1  3
#define BIN_2  5
#define AIN_1  6
#define AIN_2  9
#define MAX_PWM_VOLTAGE  150

int IRDist = 0; 
int bpLeft = 0; 
int bpRight = 0; 
int LDRValue = 0;
const int dangerThresh = 16;// (in cm) used for obstacle avoidance
int leftDistance, rightDistance; //distances on either side
Servo panMotor; //'neck' servo 
long duration; //time it takes to recieve PING signal

void setup()
{
  Serial.begin(9600); //used for serial communication for debugging 
  pinMode(BUMPER_LEFT_PIN,INPUT);
  pinMode(BUMPER_RIGHT_PIN,INPUT);
  pinMode(LED_PIN, OUTPUT);
  pinMode(BIN_1, OUTPUT);
  pinMode(BIN_2, OUTPUT);
  pinMode(AIN_1, OUTPUT);
  pinMode(AIN_2, OUTPUT);
}

void moveStop()
{
  digitalWrite(BIN_1, LOW);
  digitalWrite(BIN_2, LOW);
  digitalWrite(AIN_1, LOW);
  digitalWrite(AIN_2, LOW);
}

void moveForward()
{
  digitalWrite(BIN_1, LOW);
  analogWrite(BIN_2, MAX_PWM_VOLTAGE);
  analogWrite(AIN_1, MAX_PWM_VOLTAGE);
  digitalWrite(AIN_2, LOW);
  delay(300);
}

void turnLeft()
{
  digitalWrite(BIN_1, LOW);
  analogWrite(BIN_2, MAX_PWM_VOLTAGE);
  digitalWrite(AIN_1, LOW);
  analogWrite(AIN_2, MAX_PWM_VOLTAGE);
  delay(300); 
}

void turnRight()
{
  analogWrite(BIN_1, MAX_PWM_VOLTAGE);
  digitalWrite(BIN_2, LOW);
  analogWrite(AIN_1, MAX_PWM_VOLTAGE);
  digitalWrite(AIN_2, LOW);
  delay(300);
}

void moveBack()
{
   analogWrite(BIN_1, MAX_PWM_VOLTAGE);
   digitalWrite(BIN_2, LOW);
   digitalWrite(AIN_1, LOW);
   analogWrite(AIN_2, MAX_PWM_VOLTAGE);
   delay(500); 
}

int checkDrop()
{
  pinMode(IR_PIN,INPUT);
  IRDist = analogRead(IR_PIN);    
  //Serial.println(IRDist);
  if(IRDist < IR_DROP) 
  { 
    return 1; //Drop present (determined with IR distance sensor)
  }
  else
  {
    return 0;
  }
}

void lightDarkness()
{
 LDRValue = analogRead(LDR_PIN);
 if(LDRValue < MIN_LIGHT)
  {
      digitalWrite(LED_PIN,HIGH); //It is dark, turn on the light
  }
  else
  {
    digitalWrite(LED_PIN,LOW);
  }
}

int checkLeftBumper()
{
  bpLeft = digitalRead(BUMPER_LEFT_PIN);
  if(bpLeft == HIGH)
  {  
    //Serial.println("Left Bumper");
    //HIT!
    return 1;
  }
  else
  {
    return 0;
  }
}

int checkRightBumper()
{
   bpRight = digitalRead(BUMPER_RIGHT_PIN);
  if(bpRight == HIGH)
  {
    //Serial.println("Right Bumper");
    //HIT!
    return 1;
  }
  else
  {
    return 0;
  }
}

void compareDistance()
{
  if (leftDistance>rightDistance) //if left is less obstructed 
  {
    turnLeft();
  }
  else if (rightDistance>leftDistance) //if right is less obstructed
  {
    turnRight();
  }
   else //if they are equally obstructed
  {
    moveBack();
  }
}

long ping()
{
  // Send Ping pulse
  pinMode(PING_PIN, OUTPUT);
  digitalWrite(PING_PIN, LOW);
  delayMicroseconds(2);
  digitalWrite(PING_PIN, HIGH);
  delayMicroseconds(5);
  digitalWrite(PING_PIN, LOW);
  
  //Get duration it takes to receive echo of Ping sent
  pinMode(PING_PIN, INPUT);
  duration = pulseIn(PING_PIN, HIGH);
  
  //Convert duration into distance (cm)
  return duration / 29 / 2;
}

void loop()
{
  
  lightDarkness();
  if(checkDrop() == 1)
  {
    moveBack();
    moveBack();
    moveBack();
    turnRight();
  }
  else
  if(checkLeftBumper() == 1)
  {
    moveBack();
    turnRight();
  } 
  else if(checkRightBumper() == 1)
  {
    moveBack();
    turnLeft();
  } 
  else
  {
  moveStop();
  int distanceFwd = ping();
  //Serial.println(distanceFwd);
  if (distanceFwd>dangerThresh) //if path is clear
  {
   moveForward();
  }
  else //if path is blocked
  {
    moveStop();
    
    panMotor.attach(SERVO_PIN);
    panMotor.write(20); //look right

    delay(400);
    
    rightDistance = ping(); //scan to the right
    panMotor.write(160); //look left

    delay(550);
    
    leftDistance = ping(); //scan to the left
    panMotor.write(90); //look to center

    delay(400);
    
    panMotor.detach();
    compareDistance();
  }
  }
}
Roaming Robot Update

Pololu Zumo Robot

I recently ordered a Zumo robot shield manufactured by Pololu and found it to be a great little robot.

It comes preassembled and has various integrated sensors and actuators, including a 3-axis accelerometer, a 3-axis magnetometer, a Buzzer, a motor driver, an IR reflective sensor array, amongst others.

It comes with 2 75:1 HP micro metal gear motors, so it has a fair amount of power for such a small robot. The robot is within the 10cm x 10cm size limit of most robot sumo competitions, and with its very low centre of gravity combined with its speed and power it would be a serious competitor.

sideFront

An Arduino Uno R3 acts as the robots’ brain and simply slots onto the top of the robot, exactly like any other shield. All programs are then loaded onto the Arduino Uno as per normal and Pololu provides various example programs, including line following, maze solving and robot sumo programs.

  

The robot has an expansion area through which some pins are exposed that can be used to integrate some additional sensors and actuators.

I am looking forward to customising this robot and seeing how much more I can expand it.

I will write more about what I am doing with this robot in a future post and I will also be posting a video of the robot in action on my Youtube Channel in the near future.

Pololu Zumo Robot

Bite Size C# – XLS Creator

Today we will look at my XLS Creator Class. It is very similar to my CSV Creator class discussed in an earlier post (BITE SIZE C# – CSV FILE CREATOR), except that it creates a well formed XLS file instead of a CSV file.

It is a static class contained in the Core.XLS namespace and contains 1 static method ExportToExcel. Like the CSV Creator class it uses a generic list of objects to construct a XLS file. The method also takes 2 string values xlsNameWithExt and sheetName.

xlsNameWithExt which is used to define the name and location to save the created XLS file and sheetName which is used to set the sheet name the data will be inserted on.

Here is the code:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Text;

namespace Core.XLS
{
  public static class XlsCreator
  {
  public static void ExportToExcel<T>(List<T> list, string xlsNameWithExt, string sheetName)
  {
  var columnCount = 0;

  var StartTime = DateTime.Now;

  var rowData = new StringBuilder();

  var properties = typeof (T).GetProperties(BindingFlags.Public | BindingFlags.Instance);

  rowData.Append("<Row ss:StyleID=\"s62\">");
  foreach (var p in properties)
  {
  if (p.Name.ToUpper() == "ENTITYSTATE" || p.Name.ToUpper() == "ENTITYKEY")
    continue;

  if (p.PropertyType.Name != "EntityCollection`1" && p.PropertyType.Name != "EntityReference`1")
  {
    var type = "String";
    columnCount++;
    rowData.Append("<Cell><Data ss:Type=\"" + type + "\">" + p.Name + "</Data></Cell>");
  }
  else
    break;
  }
  rowData.Append("</Row>");

  foreach (var item in list)
  {
  rowData.Append("<Row>");
  for (var x = 0; x < columnCount; x++) //each (PropertyInfo p in properties)
  {
    var o = properties[x].GetValue(item, null);
    var value = o == null ? "" : o.ToString();
    rowData.Append("<Cell><Data ss:Type=\"String\">" + value + "</Data></Cell>");
  }
  rowData.Append("</Row>");
  }

  var sheet = @"<?xml version=""1.0""?>
    <?mso-application progid=""Excel.Sheet""?>
    <Workbook xmlns=""urn:schemas-microsoft-com:office:spreadsheet""
    xmlns:o=""urn:schemas-microsoft-com:office:office""
    xmlns:x=""urn:schemas-microsoft-com:office:excel""
    xmlns:ss=""urn:schemas-microsoft-com:office:spreadsheet""
    xmlns:html=""http://www.w3.org/TR/REC-html40"">
    <DocumentProperties xmlns=""urn:schemas-microsoft-com:office:office"">
    <Author>MSADMIN</Author>
    <LastAuthor>MSADMIN</LastAuthor>
    <Created>2011-07-12T23:40:11Z</Created>
    <Company>Microsoft</Company>
    <Version>12.00</Version>
    </DocumentProperties>
    <ExcelWorkbook xmlns=""urn:schemas-microsoft-com:office:excel"">
    <WindowHeight>6600</WindowHeight>
    <WindowWidth>12255</WindowWidth>
    <WindowTopX>0</WindowTopX>
    <WindowTopY>60</WindowTopY>
    <ProtectStructure>False</ProtectStructure>
    <ProtectWindows>False</ProtectWindows>
    </ExcelWorkbook>
    <Styles>
    <Style ss:ID=""Default"" ss:Name=""Normal"">
    <Alignment ss:Vertical=""Bottom""/>
    <Borders/>
    <Font ss:FontName=""Calibri"" x:Family=""Swiss"" ss:Size=""11"" ss:Color=""#000000""/>
    <Interior/>
    <NumberFormat/>
    <Protection/>
    </Style>
    <Style ss:ID=""s62"">
    <Font ss:FontName=""Calibri"" x:Family=""Swiss"" ss:Size=""11"" ss:Color=""#000000""
    ss:Bold=""1""/>
    </Style>
    </Styles>
    <Worksheet ss:Name=""" + sheetName + @""">
    <Table ss:ExpandedColumnCount=""" + (properties.Count() + 1) + @""" ss:ExpandedRowCount=""" +
    (list.Count() + 1) + @""" x:FullColumns=""1""
    x:FullRows=""1"" ss:DefaultRowHeight=""15"">
    " + rowData + @"
    </Table>
    <WorksheetOptions xmlns=""urn:schemas-microsoft-com:office:excel"">
    <PageSetup>
    <Header x:Margin=""0.3""/>
    <Footer x:Margin=""0.3""/>
    <PageMargins x:Bottom=""0.75"" x:Left=""0.7"" x:Right=""0.7"" x:Top=""0.75""/>
    </PageSetup>
    <Print>
    <ValidPrinterInfo/>
    <HorizontalResolution>300</HorizontalResolution>
    <VerticalResolution>300</VerticalResolution>
    </Print>
    <Selected/>
    <Panes>
    <Pane>
      <Number>3</Number>
      <ActiveCol>2</ActiveCol>
    </Pane>
    </Panes>
    <ProtectObjects>False</ProtectObjects>
    <ProtectScenarios>False</ProtectScenarios>
    </WorksheetOptions>
    </Worksheet>
    <Worksheet ss:Name=""Sheet2"">
    <Table ss:ExpandedColumnCount=""1"" ss:ExpandedRowCount=""1"" x:FullColumns=""1""
    x:FullRows=""1"" ss:DefaultRowHeight=""15"">
    </Table>
    <WorksheetOptions xmlns=""urn:schemas-microsoft-com:office:excel"">
    <PageSetup>
    <Header x:Margin=""0.3""/>
    <Footer x:Margin=""0.3""/>
    <PageMargins x:Bottom=""0.75"" x:Left=""0.7"" x:Right=""0.7"" x:Top=""0.75""/>
    </PageSetup>
    <ProtectObjects>False</ProtectObjects>
    <ProtectScenarios>False</ProtectScenarios>
    </WorksheetOptions>
    </Worksheet>
    <Worksheet ss:Name=""Sheet3"">
    <Table ss:ExpandedColumnCount=""1"" ss:ExpandedRowCount=""1"" x:FullColumns=""1""
    x:FullRows=""1"" ss:DefaultRowHeight=""15"">
    </Table>
    <WorksheetOptions xmlns=""urn:schemas-microsoft-com:office:excel"">
    <PageSetup>
    <Header x:Margin=""0.3""/>
    <Footer x:Margin=""0.3""/>
    <PageMargins x:Bottom=""0.75"" x:Left=""0.7"" x:Right=""0.7"" x:Top=""0.75""/>
    </PageSetup>
    <ProtectObjects>False</ProtectObjects>
    <ProtectScenarios>False</ProtectScenarios>
    </WorksheetOptions>
    </Worksheet>
    </Workbook>";

  using (var sw = new StreamWriter(xlsNameWithExt))
  {
  sw.Write(sheet);
  }
  }
  }
}
Bite Size C# – XLS Creator

Bite Size C# – XLS to CSV Converter

Let us have a look at my XLS to CSV Converter class today.

It is a static class, which is located in the Core.CSV namespace. It contains one static method CovertExcelToCsv which takes 3 parameters excelFilePath, csvOutputFile and worksheetNumber. The excelFilePath parameter is used to input the source Excel file, csvOutputFile is used to input the destination CSV file and lastly worksheetNumber is used to define which worksheet to convert, as CSV files do not have worksheets only one worksheet can be converted to CSV file at a time. The worksheetNumber parameter has a default value of 1 in the event that no value is passed in.

using System;
using System.Data;
using System.Data.OleDb;
using System.IO;

namespace Core.CSV
{
 public static class XlsToCsvConverter
 {
  public static void CovertExcelToCsv(string excelFilePath, string csvOutputFile, int worksheetNumber = 1)
  {
   if (!File.Exists(excelFilePath)) throw new FileNotFoundException(excelFilePath);
   if (File.Exists(csvOutputFile)) throw new ArgumentException("File exists: " + csvOutputFile);

   // Connection string
   var connectionStr =
    String.Format(
     "Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties=\"Excel 8.0;IMEX=1;HDR=NO\"",
     excelFilePath);
   var cnn = new OleDbConnection(connectionStr);

   // Get schema information and then data in spreadsheet
   var dt = new DataTable();
   try
   {
    cnn.Open();
    var schemaTable = cnn.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
    if (schemaTable != null && schemaTable.Rows.Count < worksheetNumber)
     throw new ArgumentException("The worksheet number requested does not exist.");
    if (schemaTable != null)
    {
     var worksheet = schemaTable.Rows[worksheetNumber - 1]["table_name"].ToString().Replace("'", "");
     var sql = String.Format("select * from [{0}]", worksheet);
     var da = new OleDbDataAdapter(sql, cnn);
     da.Fill(dt);
    }
   }
   catch (Exception ex)
   {
    //Insert logging of exception here
    throw;
   }
   finally
   {
    // Close connection to the spreadsheet
    cnn.Close();
   }

   // Write out CSV data
   using (var wtr = new StreamWriter(csvOutputFile))
   {
    foreach (DataRow row in dt.Rows)
    {
     var firstLine = true;
     foreach (DataColumn col in dt.Columns)
     {
      if (!firstLine)
      {
       wtr.Write(",");
      }
      else
      {
       firstLine = false;
      }
      var data = row[col.ColumnName].ToString().Replace("\"", "");
      wtr.Write(String.Format("{0}", data));
     }
     wtr.WriteLine();
    }
   }
  }
 }
}
Bite Size C# – XLS to CSV Converter

Bite Size SQL – Basic Query Optimisation

How SQL queries are written can have a huge impact on how quickly and efficiently they execute. In this post I will explain some very basic methods that can be used to optimise SQL queries.

  • When writing select queries, only select the columns you need. In general using “SELECT *” is very bad practice.
  • Avoid using sub queries, rather use joins. What is meant by this is instead of using:
        SELECT COLUMNA FROM TABLEA WHERE COLUMNB IS IN (SELECT COLUMNB FROM TABLEB)
        Rather use:
        SELECT COLUMNA FROM TABLEA AS TA INNER JOIN TABLEB AS TB ON TA.COLUMNB = TB.COLUMNB
  • Use WHERE to limit the result set to only what you need. There is no point in pulling 10000 records if you are only looking for records, for example in a certain date range.
  • Use WITH NOLOCK with SELECT queries when feasible, this can prevent deadlocks which can cause serious performance issues. However note that using WITH NOLOCK can result in a non-accurate result set being returned which contains non committed records, so keep this in mind when using. But when querying a production system database this risk might be outweighed by the risk of deadlocks occuring.

One last point on stored procedures, this is not really an optimisation point as much as a best practice point. When developing stored procedures always write them to return result sets of the same dimensions. I have worked on numerous legacy systems were this was not the case and it results in a complete nightmare, especially with projects that utilise an ORM.

Bite Size SQL – Basic Query Optimisation

Dark Carnival Geek Box

I recently decided to give the Dark Carnival Geek Box (http://www.darkcarnival.co.za/) a try. It is a South African offering based on international offerings such as Loot Crate, 1Up box, etc. whereby you pay a monthly subscription fee (usually between $20 to $40(USD)) and in return receive a package every month that contains a few surprises. It usually contains 4 to 10 items consisting of anything from T-shirts to toys, all with a geeky theme. The Dark Carnival Geek Box is priced at R505 (ZAR), which included packing and delivery fees. At time of writing the price equates to about $41 (USD), so as these surprise box services go the cost is at the high-end of the spectrum. But as the other before mentioned services do not deliver to South Africa, there is very little competition in the market for a service like this.

The box is a normal cardboard shipping box with a sticker on top, as opposed to the specially designed boxes used with the international services. geek box closed Let us have a look of what we get inside.

geek box open

The box contained 6 items:

Hatbatman bag  MaskStickBig Hero 6

tickets

A plushy hat, a canvas bag, a small mask, a batman selfie stick, a Big Hero 6 bobble head and 2 tickets to GeekFest 2015. The quality of the plushy hat and mask were pretty poor. However the bobble head, selfie stick and canvas bag were all good quality (note however that as far as I can see none of the items are officially licensed goods). The tickets for GeekFest (which cost R80 (ZAR) each, approximately $6.50 (USD), which is an event hosted by Dark Carnival, were a nice touch.

My favourite item was the bobble head, and least favourite was the mask.

That said the value offering is far from what the international offerings provide, where in some cases you can get up to double the value as compared to the price. With the Dark Carnival Geek Box you might barely get your money back in terms of value, maybe I would have felt differently if I had been able to go to GeekFest. I might try the service again in a few months to see if the offering has improved, but for now would I recommend the Dark Carnival Geek Box? No, I can’t say I would.

box2

Dark Carnival Geek Box