How To Save Data from Arduino To A .csv File Using Processing

 

Introduction

Spreadsheets are awesome! They are a ubiquitous and powerful file structure that can cause headache and joy in equally heart-stopping intensities. Within the context of a larger project, you may want to save Arduino data (from sensors, for example) to a .csv file for further analysis, visualization, or whatever spreadsheet bonanzas you can dream up. The following two sketches save data from Arduino to a .csv file on your computer, complete with a timestamp with gloriously specific column names (year, month, day, hour, minute, second…and you can add milliseconds!).

This 2-sketch solution takes advantage of the built-in Table class in Processing. Tables are also awesome, as they are basically spreadsheets in the Processing world. Read all about them (this link has some handy methods for working with tables beyond what this tutorialprovides): http://processing.org/reference/javadoc/core/processing/data/Table.html

Here is an overview of what this tutorial accomplishes:

Arduino to .csv graphic

This is a beginning-level tutorial with one prerequisite! Arduino and Processing have to shake hands. I know it’s a lot to ask, but cooperation is at the root of any good spreadsheet. Here’s a link to a tutorial: http://www.hackerscapes.com/2014/10/lab-6-processing/

First, Download and Install the following:

Step 1: Read Data, Send to Processing via Serial Port

First, set up the Arduino side. Here’s a a gist-y sketch:

Step 2: Use Processing to Receive Data from Arduino, Write data to a Table, and Save Table to a .csv
Now we will handle the processing side.

Additional Resources

23 thoughts on “How To Save Data from Arduino To A .csv File Using Processing”

  1. Well, it looks like the same thing happened to my comment. Maybe the browser thought it was an HTML tag.

    I’m assuming the include statement should be to SoftwareSerial.h.

  2. Hi Joel,

    Apologies for the delay in replying. Thanks for the comment! You are correct about line 27, I’ve updated the code. I hope it works for you, please comment if you see any other errors.

  3. Arduino code:
    Line 35: typo, sensorPin3
    Line 49: float sensorVals[] = {0,0,0}, but in 59-61 and 68-72 are used different undefined variables.

    Processing code:
    Line 29: Table dataTable, but further “table” is used.

    Variable ‘val’ undefined, suppose to be String.

    And for me it doesn’t work, it gives “NullPointerException”, pointing to the line 67

  4. I am still getting the error on line 27 #include SoftwareSerial.h

    it says #include expects “filename” or

    also, line 52 void setup(){
    returns the error expected ‘,’ or ‘;’ before void. Sorry, I am new to this so it is throwing me for a loop.

  5. for arduino code:
    SoftwareSerial.h should be between.

    I some how debugged the code for arduino.
    but unable to solve problem in processing code.
    I assigned val as string but after that it is showing error as Null pointer exception.
    kindly help me out to get out of this problem.

    1. I think i have the NullPointerException solution.

      on line 29 when the Table is declared it should be:
      Table dataTable = new Table();

  6. For all of you having problems with the processing code, you can use this version.

    I have introduce a try catch enviroment inside the savefile so regarding any reading error the code doesn’t stop.

    Table also is define as new table so there should not be anymore the issue with Null pointer.

    Cheers,

    Javi

    Copy here:

    /*
    Saving Values from Arduino to a .csv File Using Processing – Pseduocode

    This sketch provides a basic framework to read data from Arduino over the serial port and save it to .csv file on your computer.
    The .csv file will be saved in the same folder as your Processing sketch.
    This sketch takes advantage of Processing 2.0’s built-in Table class.
    This sketch assumes that values read by Arduino are separated by commas, and each Arduino reading is separated by a newline character.
    Each reading will have it’s own row and timestamp in the resulting csv file. This sketch will write a new file a set number of times. Each file will contain all records from the beginning of the sketch’s run.
    This sketch pseduo-code only. Comments will direct you to places where you should customize the code.
    This is a beginning level sketch.

    The hardware:
    * Sensors connected to Arduino input pins
    * Arduino connected to computer via USB cord

    The software:
    *Arduino programmer
    *Processing (download the Processing software here: https://www.processing.org/download/
    *Download the Software Serial library from here: http://arduino.cc/en/Reference/softwareSerial

    Created 12 November 2014
    By Elaine Laguerta
    http://url/of/online/tutorial.cc

    */

    import processing.serial.*;
    Serial myPort; //creates a software serial port on which you will listen to Arduino
    Table dataTable; //table where we will read in and store values. You can name it something more creative!

    int numReadings = 100; //keeps track of how many readings you’d like to take before writing the file.
    int readingCounter = 0; //counts each reading to compare to numReadings.

    String fileName, val;
    void setup()
    {
    String portName = Serial.list()[2];
    //CAUTION: your Arduino port number is probably different! Mine happened to be 1. Use a “handshake” sketch to figure out and test which port number your Arduino is talking on. A “handshake” establishes that Arduino and Processing are listening/talking on the same port.
    //Here’s a link to a basic handshake tutorial: https://processing.org/tutorials/overview/

    myPort = new Serial(this, portName, 9600); //set up your port to listen to the serial port
    dataTable = new Table();

    dataTable.addColumn(“id”); //This column stores a unique identifier for each record. We will just count up from 0 – so your first reading will be ID 0, your second will be ID 1, etc.

    //the following adds columns for time. You can also add milliseconds. See the Time/Date functions for Processing: https://www.processing.org/reference/
    dataTable.addColumn(“year”);
    dataTable.addColumn(“month”);
    dataTable.addColumn(“day”);
    dataTable.addColumn(“hour”);
    dataTable.addColumn(“minute”);
    dataTable.addColumn(“second”);

    //the following are dummy columns for each data value. Add as many columns as you have data values. Customize the names as needed. Make sure they are in the same order as the order that Arduino is sending them!
    dataTable.addColumn(“sensor1”);
    dataTable.addColumn(“sensor2”);

    }

    void serialEvent(Serial myPort){
    try {
    val = myPort.readStringUntil(‘\n’); //The newline separator separates each Arduino loop. We will parse the data by each newline separator.
    if (val!= null) { //We have a reading! Record it.
    val = trim(val); //gets rid of any whitespace or Unicode nonbreakable space
    println(val); //Optional, useful for debugging. If you see this, you know data is being sent. Delete if you like.
    float sensorVals[] = float(split(val, ‘,’)); //parses the packet from Arduino and places the valeus into the sensorVals array. I am assuming floats. Change the data type to match the datatype coming from Arduino.

    TableRow newRow = dataTable.addRow(); //add a row for this new reading
    newRow.setInt(“id”, dataTable.lastRowIndex());//record a unique identifier (the row’s index)

    //record time stamp
    newRow.setInt(“year”, year());
    newRow.setInt(“month”, month());
    newRow.setInt(“day”, day());
    newRow.setInt(“hour”, hour());
    newRow.setInt(“minute”, minute());
    newRow.setInt(“second”, second());

    //record sensor information. Customize the names so they match your sensor column names.
    newRow.setFloat(“sensor1”, sensorVals[0]);
    newRow.setFloat(“sensor2”, sensorVals[1]);

    readingCounter++; //optional, use if you’d like to write your file every numReadings reading cycles

    //saves the table as a csv in the same folder as the sketch every numReadings.
    if (readingCounter % numReadings ==0)//The % is a modulus, a math operator that signifies remainder after division. The if statement checks if readingCounter is a multiple of numReadings (the remainder of readingCounter/numReadings is 0)
    {
    fileName = str(year()) + str(month()) + str(day()) + str(dataTable.lastRowIndex()); //this filename is of the form year+month+day+readingCounter
    saveTable(dataTable, fileName, “csv”); //Woo! save it to your computer. It is ready for all your spreadsheet dreams.
    }
    }
    }
    catch(RuntimeException e) {
    e.printStackTrace();
    }
    }

    void draw()
    {
    //visualize your sensor data in real time here! In the future we hope to add some cool and useful graphic displays that can be tuned to different ranges of values.
    }

    1. thic code is worked, but why my output file have no extention on it ? just a blank file. and should rename it manually into csv before open it, and also the file name is no “0” number, ex: It should show 085620190405 (hours[08]+minutes[56]+year[2019]+month[04]+day[05]), but it just show 856201945. can you help ?

  7. Okay guys, the big problem with the code is that it doesn’t take into account the fact that Processing doesn’t truly understand how to handle numbers, int or float or otherwise. If you are using an array to feed into the Processing code, assign it as an array of strings. When I made that change, the code worked perfectly.
    (of course including the other revisions listed above also helped get the code to a finished state)

    1. Could you please post the final version of the code that you came up with? I am trying to get it to work but can’t seem to figure it out. Thanks 🙂

  8. How about if I ma not using sensor? I will instead use a switch on pin 8, 9, 10, 11 of arduino input. I just want to record the high and low state of the 4 channels.
    I edited the below code, pls help to check if any correction.
    Thanks.
    import processing.serial.*;

    int Oven_P1_1 = 8;
    int Oven_P1_2 = 9;
    int Oven_P5_1 = 10;
    int Oven_P5_2 = 11;

    Serial myPort;
    Table table;

    int numReadings = 100;
    int readingCounter = 0

    String filename, val;

    void setup() {

    String portName = Serial.list()[5];
    myPort = new Serial(this, portName, 9600);
    table = new Table();
    table.addColumn(“id”);
    table.addColumn(“year”);
    table.addColumn(“month”);
    table.addColumn(“day”);
    table.addColumn(“hour”);
    table.addColumn(“minute”);
    table.addColumn(“second”);
    table.addColumn(“Oven_P1_1”);
    table.addColumn(“Oven_P1_2”);
    table.addColumn(“Oven_P5_1”);
    table.addColumn(“Oven_P5_2”);
    }

    void serialEvent(Serial myPort)
    try{
    val = myPort.readStringUntil(‘\n’);
    if (val!= null) {
    val = trim(val);
    println(val);
    //float sensorVals[] = float(split(val, ‘,’)); // i dont need floats because I just need high or low input in 4 channel., what other function should I use.
    TableRow newRow = table.addRow();
    newRow.setInt(“id”, table.lastRowIndex());
    newRow.setInt(“year”, year());
    newRow.setInt(“month”, month());
    newRow.setInt(“day”, day());
    newRow.setInt(“hour”, hour());
    newRow.setInt(“minute”, minute());
    newRow.setInt(“second”, second”);
    newRow.setInt(“Oven_P1_1”, ovenVals[0]);
    newRow.setInt(“Oven_P1_2”, ovenVals[1]);
    newRow.setInt(“Oven_P5_1”, ovenVals[2]);
    newRow.setInt(“Oven_P5_2”, ovenVals[3]);

    readingCounter++;

    if (reading Counter %numReadings ==0)
    {
    saveTable(table, “data/OvenStatus.csv”);
    }
    }
    }

    void draw(){

    }

  9. Hey,
    can somebody help? I am very new to this and in line 36-40 it tells me: the variable “val” does not exist.

    How can I fix this?

    Thank you very much,

    Jas

  10. please can someone send me the updated code for outputting reading from arduino sensor to csv format..thanks..

  11. I have run the code Bong. But no file is being written in the sketch folder. need help.

    Cheers,
    Usman

Leave a Reply

Your email address will not be published. Required fields are marked *