A DIY security video camera recorder (part 2) - beaglebone black host

2015-05-04 12.13.58In part 1 of this series I demonstrated how ffmpeg can be used to dump live video streams from IP cameras to a hard drive. In this post I will show how a Beaglebone Black board can be used to build a cheap and energy efficient video recorder.

Given that the recorder is not re-coding any of the camera streams the CPU requirements are quite moderate. The processor only collects the streaming data and copies them to a hard drive. Hence using a Mac or PC as a host platform is quite an over-kill. Also from an energy and cost point of view a smaller size computer is a much better fit. Embedded systems like the Rasberry Pi or Beaglebon Black should be perfectly capable of handling this task at a fraction of the cost.  So in the next few section I am going to walk you through the process of setting up such a IP-DVR system.

Hardware

Before we get into the details here is the hardware we will use:

  • Beagle Bone Black, 5 V power supply and >4 GB microSD card
  • Samsung IP camera
  • 3 Port USB Hub with Ethernet - I am using a "YUANXIN X-2104 Compact 3-Port USB 2.0 Hub w/ RJ45 Port + LED Indicator - Black + Grey"
  • 150GB - 1.5TGB USB 2.0 Harddisk
  • WiFi USB Dongle - 2.4 GHz 150Mbps Wireless IEEE802.11N Nano USB Adapter TP-LINK TL-WN725N

The Beaglebone Black (BBB) comes with a single USB 2.0 host and Ethernet port. For our setup we need at least 2 x Ethernet ports. One to connect the camera (LAN Port) and one for the connection to the internet (WAN port). The easiest way to add the second Ethernet port is the buy a USB-RJ45 Ethernet adapter. However this would occupy the USB 2.0 port so we also need a USB hub to add an additional port for the hard drive.

Fortunately there are combined 3-Port USB 2.0 Hub with RJ45 Port offered at your local computer store or on-line. They give you flexibility as you will have spare USB ports to add WIFI or other USB devices in the future.

Software

The following description assumes the latest BBB Debian Linux installation. If you are still using Angstrom please install the Debian Linux. No worries, it is very easy to do and there are many description out there that walk you through the steps. I just give an outline here assuming you use a Windows machine to write the Micro SD card.

Pull the lates image from the official site: http://beagleboard.org/latest-images. Then unpack the image with 7-Zip. You can now write the img-file with a utility like Win32DiskImager or WinFlashTool to your microSD card. Once the memory card is written you insert it into the BBB. Connect a USB hub with a keyboard and mouse and hook a Monitor to it. You are now ready to power the BBB up.

After a while the Linux environment should be visible on the screen.

If you use a SD card that offers more than 4GB of space like me you also want to increase the  BBB Image Disk Space. First make sure you boot from the mircoSD card, then adjust the Linux partition with fdisk.

execute the following single letter commands inside the fdisk utility:

1) Delete the second partition: d, 2
2) Create New Primary Linux Partition: n, p, 2
3) Write new partition table to disk: w
4) Exit fdisk: q

reboot and then adjust file system to use the extended partition:

reboot a second time so the change can take effect. You should now see the difference using the disk free utility:

Connect the BBB to the internet and update the package list:

Before we start to install additional drivers we need to make sure the build tools and header files are installed:

USB-Ethernet Driver

Now that we have an up-to-date Linux system we can start to install the drivers of the additional hardware. I will start with the USB-Ethernet interface first. My 3-Port USB 2.0 Hub with Ethernet Port uses the ASIX controller. To install the drivers follow the steps outlined below. Download the driver source from the ASIX's web site, unpack the source code and compile it. Then install the driver:

Reboot and check in the /var/log/messages file to see if the module gets properly loaded:

and for a the Ethernet interface assigned to the port:

Now add the interface to your configuration file. In my case it is eth1:

the configuration file should look something like this:

Plug in an Ethernet cable connected to your network and activate the interface:

The device should now fetch an IP address and once successful you are connected to the Internet.

The USB-WiFi Driver (optional, only required if you need support for WIFI)

The next driver we are going to install is for the USB TP-LINK TL-WN725N Wifi Dongle.
Important to know: there are TWO versions of the "TL-WN725N", one needs the RTL8188CUS driver, the other needs this RTL8188EU. See wikidevi.com/wiki/TP-LINK_TL-WN725N_v1 and wikidevi.com/wiki/TP-LINK_TL-WN725N_v2 for differences.

These instructions are for the RTL8188EU version. Before you start confirm that your TP-LINK adapter is based on a RTL8188EU Realtek controller. Plug in the adapter and checkwith the following command:

For my system I get the following response:

To install the Realtek driver follow the steps outlined below. Download the driver source from git, unpack the source code and compile it. Then install the driver:

Check that the driver is properly loaded:

you  should be able to see the loaded driver with

Check the interface and wireless signals

For me the system assigned the interface wlan2. I now edited the network interfaces file vi /etc/network/interfaces and add the wlan2

The system is now configured such that we can bring up the wireless connection:

You can take down eth1 (ifdown eth1) before bring up (ifup) wlan2.

 

Webmin

I was looking for a tool similar to OpenWRT's LUCI that allows me to administrate the BBB Debian Linux from a web browser. I settled on Webmin. Please leave  a comment if you have a better recommendation.

Webmin allows you to manage all to most common Linux services like retworks, firewalls, routing, sql, HTTP....... from simple web forms.

We are going to install Webmin via APT. The first step is to edit the /etc/apt/sources.list file on your system and add the lines :

Now APT can find the webmin repository. Run update to load the modified configuration:

Due to a known bug I had to change in the  /etc/apt/apt.conf.d/02compress-indexes file from  Acquire::GzipIndexes "true" to false:

Update the database and install webmin with the following commands:

You can now point your browser to https://myhost:10000 and if everything is properly installed log into your system.

To connect the camera with a dynamic IP address we need to install the DHCP server:

I then used  firehol shell script to configure the firewall:

You can let firebol guess your system and generate an example configuration

Here is my /etc/firehol/firehol.conf

Now run firehol

If everything went well your firewall is now configured.

ffmpeg

It is now time to install ffmpeg tool.

You can again use APT to install ffmpeg. However this will get you a fairly dated version ffmpeg. Just add in your /etc/apt/sources.list

If you want the latest version follow the next few steps to compile from source code.

Now that all the tools are in place you can just follow the instruction of part 1 to capture and dump the video to the hard drive.

Conclusion

In this blog we learned to build a simple BBB based IP video recorder. With Debian Linux as the baseline I demonstrated how to install all the additional hardware drivers, tools and software such that the resulting server is able to record IP based video streams to a hard drive for a cost of less than $100 dollars.

Credits

To all the web sites, blog and forum posts I missed to record so that I could credit them here. To debian.org, elinux.org, webmin.com, ffmpeg.org, firehol.org and all the other contributing opensource communities that jointly created all the software used in this post.

 

A DIY security video camera recorder (part 1) - the missing newspaper

2015-04-28 10.56.31-3Over the last few month 1-2 a week our newspaper was missing. This, most of the mornings, really ticket me off as it disrupted my breakfast routine. A while later I got a letter from the courier announcing that in our neighborhood newspapers get stolen.

I need my morning newspaper even though there is hardly any news in there. I considered cancelling the subscription but realized that I was not ready to let go of this habit. I also grew curious who would steal newspapers when most of the news was freely available on-line. Maybe one of my lovely neighbors had a dark side. So I took the challenge and set out to solve this mystery.

I ordered a Samsung SNH-P6410BN in-door and Samsung SNH-E6440BN outdoor wireless IP security camera. These are really great Hi-Def cameras with a wide field of view and good picture quality during daylight as well as at night.

The camera comes with pretty decent cloud support and an Android and iPhone app. It can store the video feeds to microSD cards that can be inserted into the cameras.  However I did not want to fiddle with SD-cards as the camera is connected to my network and I hoped that additional software could help me out.
I was intrigued by 3rd party Video Software like BlueIris that could in fact connect to my cameras video streams.

While setting up my camera with BlueIris I realized that the camera is able to stream video in any of the following resolutions:

This is no where documented in the product manual that shipped with the camera.  Anyway, I managed to open the camera streams in VLC and watch the live feed using a URL like this:

However I was mostly interested in dumping these streams to a hard drive so that I could zap through them with VLC. That is when I started to look into ffmpeg.

What a tool this is! The Swiss-Army knife of Video. Is there something that cannot be done with this utility? However, the flexibility comes at a price. There are so many options and parameters that exploring them feels like an entry riddle to a secret society. That said, the ffmpeg user community is large and there are many example and posts that are available on the internet to get you started.
So here is a simple command that dumps my stream:

This line dumps the input video stream into the file VideoDump.mp4. The option "-vcodec copy" makes sure that the stream is not re-coded so the process hardly takes any CPU resources. This worked really well. However the files started to become big as a 1920x1080 stream uses about 1 GByte of hard disk space per hour. Is there a way to split the files into 1hour segments? You bet!

The following command line will dump segments of 1 hour:

The segment time is specified with the option "-segment_time 3600" that takes seconds as the argument. The option "-segment_time_delta 0.03" allows the utility some 0.03 second flexibiliy where to cut a segment so it can do it at a key frame. The option " -reset_timestamps 1" will reset the time inside a segment to start at 00:00. Also the segments get numbered. The segment files are named like this:

The amount of video data I collected was rapidly growing and I was interested to start and turn off the capturing at define moments. Sure ffmpeg can do this:

Adding the option "-ss 00:00:00 -t 02:30:00" will capture 2.5 hours of footage and then stop. I then used a cron job on my Mac to start the stream capturing.

In summary ffmpeg works great with the Samsung IP cameras and allows to capture videos exactly during the hours of interest.

So coming back to the missing newspaper, you wonder who the culprit was? Well to my surprise there was no thief. The footage clearly revealed that the newspaper never got delivered the days it was "stolen".

Logging Weather Data to the Cloud

In this post I want to report some of the finding from a 4 month experiment with a Spark Core. The core is uploading pressure measurement values into the cloud every five minutes. It then loggs them into a Google Spreadsheet. I use the Spark Temboo library and service after I failed to reliably poll the core from Google.

Anyway the Spreadsheet has now more than 30,000 entries. The chart generation is a bit slow but everything is still running stable to this day. A plot of the 4 month is shown below:

Temboo

After a measurement is taken the core goes to deep sleep to conserve energy. An STM103 internal HW timer interrupt is used to wake-up in time for the next measurement.

Over the four month I have seen only a few corrupted values. These are most likely transmission errors due to network connectivity issues.

PCB Design Workshop

Last weekend I attended a cool 1 day workshop "Designing a Circuit Board" given by Matt Berggren. Matt is a member of supplyframe San Francisco with many years of experience as a PCB designer and instructor.

The class is positioned as "Learn to Build a PCB from the ground up..." targeting hackers and professionals that want to tip their toes into the water of hardware design. This was the 4th installment of this class that always fills up withing a very short time.

Matt is an engaging presenter. He spent the morning explaining the fundamentals of PCB boards and technology. The class does't assuming an EE degree so Matt is careful to explain the fundamentals and terms relevant for PCB design. In fact he is quite a master in catering to the beginners and more advanced members of his audience.

Here is an overview of the topics Matt covered in the morning:

  • Circuit Board basics and terminology - layers, cores, finishes, objects (pads, vias, components, lands, land patterns, footprints, vias), multilayer vs. single-layer, layer types, etc.
  • Some electrical basics related to boards - dielectrics, copper considerations, current carrying, impedance control, parasitic capacitance (don't freak out, we'll explain all of this)

The afternoon was mostly spent hands-on working with Eagle. Here is what we covered:

  • Build our first schematic + PCB + CAM (simple USB power supply that generates some very basic voltages we can use on the bench to do stuff - very simple USB->LDO->Connector)
  • Looked at Eagle and learn some important Eagle terminology, basics of the menus, command line interface, shortcuts, etc + basic workflow (should prepare you for doing this on your own)

Cudo and thanks to Matt. He runs these classes for free at weekends. His audience's backgrounds are very divers. Some are complete electronic novices and some are pros. Matt masters this challenge by providing a lot of well selected practical technical information with entertaining anecdotes. Great teaching job.

Updated Temboo library available

I finally found some time to fix the problems with the Spark Temboo library. The following issues were fixed:

  • mib(), max() definition clash between the STL library and the Spark Wiring math library.
  • Misc makefile fixes to build the library from a shell

The updated source code is available at Github under https://github.com/bentuino/temboo.git Thanks to all the notes and hits I received from the community.

Temboo ignited with a Spark

This is a re-post of an article I wrote as a guest blogger for Temboo's blog.

I am working with connected devices and was looking for a cloud service. While surveying the field Temboo caught my eye because of the large number of supported premium web sites and the promise to connect IoT devices to the Internet in a breeze.

spark_size The connected device I am using is a Spark Core. The Spark Core is a sleek small board that offers a powerful 32 bit ARM CPU paired with WiFi. The product grew out of a Kickstarter campaign and is rapidly gaining in popularity. The Spark is nicely priced and everything is open source. The team supporting the Spark Core is smart and supportive and made a great choice to port most of the main Arduino APIs to their platform.

As outline in a blog post here migrating Arduino Libraries to the Spark Core often turns out to be pretty easy. With Temboo providing an open source library for Arduino I was tempted to give it a try. However, I had no Temboo-Arduino setup so I was not sure how hard it would be to get it all up and running.

Well, I am happy to report that is was easier than expected. Temboo's code is well written. I only had to work around some AVR specific optimizations that Temboo did to save program memory. As the Spark Core is built around a STM32F103 chip resources are not as tight as with the AVR so I simply bypassed these optimizations.

Here are some brief instructions how to install the Temboo Arduino Library. The instructions use the Spark command line SDK setup:

  1. Download the modified Temboo Arduino Library source code from github:
  2. Get the Spark Core firmware:
  3. In older Spark firmware there is a small problem that the spark team already fixed. Open the file core-firmware/inc/spark_wiring_ipaddress.h and uncomment the line 54 with your favorite editor:
  4. Save your TembooAccount.h you generated with DeviceCoder to temboo-arduino-library-1.2\Temboo
  5. Now it is time to build the Spark application:
  6. Connect your Spark Core to your computer via a USB cable
  7. Push both buttons, release Reset button and continue holding the other button until RGB-LED lights up yellow
  8. Download application into Spark Core

Temboo Examples

Two simple Spark application examples are included:

  • core-firmware/src/application_gxls.cpp - Example demonstrates the Temboo library with Google Spreadsheet
  • core-firmware/src/application_gmail.cpp - Example demonstrates the Temboo library with Gmail

to change the example that is built, edit the first line in the core-firmware/src/build.mk file:

or

Building this code was tested under Windows 8.1 using cygwin and the MINGW version of the ARM GCC compiler tool chain. It should be easy to use this Temboo Library with the Spark Cloud based SDK. To configure the Library to support Spark all the is required is to define the following label:

or add a

to the source code. Temboo support for the Spark Core is a lot of fun. It is easy to setup your own Temboo account and compile the Temboo Arduino Library that now supports the Spark Core platform. To learn more about similar projects please visit my blog at http://bentuino.com.

Google polls Spark Core

appsIn a previous blog post I was describing an example of how a Spark Core can be used to read weather sensors. The setup was really no different from any simple Arduino Uno setup. It only demonstrated how easy it is to port Arduino Sketches to a Spark Core.

With the integrated WLAN I was interested to connect the Spark Core to the internet cloud. One of the simplest ways I found, was using Google's Spreadsheet service. I stumbled over this idea in this Spark forum post.
Here is how it works: a Google Script is periodically reading data from the Spark Core via the RESTful Spark API and then appends the data to a Spreadsheet. The code below is a minimalistic Spark code to test the such a setup:

It publishes a variable for cloud access and then increments it in regular intervals. Together with the following Google script I was able to quickly pull data from my core.

However when I setup a time trigger to run the script in regular intervals I found the setup to be very unreliable. This is discussed and documented by several Spark Users and as of this writing I have not seen a fix for this problem.

One thing to note is, that this approach is pulling data from the Spark Core rather than the core pushing them to the cloud. This has a significant flaw as we cannot put the core into standby between the measurement intervals. Therefore this solution is anyway not a good choice for low power applications.

So stay tuned, I am experimenting with a better solution that I will blog about in my next post.

Using Arduino code on Spark Core

In an earlier post Spark Core Weather Station I presented code that was reading weather sensors and sent the data over USB to a PC terminal. However the approach I took there to combine all code into a single file is not very practical. So now that the Spark IDE supports multiple project files we can just include Arduino library files.

Screenshot 2014-04-16 20.49.13

So here are some hints how to get Arduino library code and sketches to compile in the Spark framework. I use a TMP102 I2C temperature sensor to demonstrate the porting. Below is an example Sketch that reads the temperature values and sends them to the serial interface:

The main reason this code fails to compile is the Arduino Wire.h library include statement fails on the cloud based Spark Core IDE. The TMP102 sketch code use I2C. So it includes the Arduino Wire Library with the following line:

This triggers an error in the Spark IDE. To solve this problem I created a new library named Wire.h with the following content:

With this library all we have to do is to uncomment or delete the original include statement. By adding Wire.h file the IDE automatically adds a line like this to the source code:

As long as the Arduino source follows basic guidelines for portable code I did not face much difficulties to use existing Arduino sources on the Spark Core.

RedBear BLE Module

Bluetooth Low Energy (BLE) transceiver board. BLE is a new protocol introduced in the 4.0 revision of the Bluetooth standard. It is a wireless personal area network (PAN) technology aimed at novel applications in the healthcare, fitness, security, and home entertainment industries.  BLE is not backward-compatible with the previous Bluetooth protocol. However it uses the same 2.4 GHz frequency bands but a simpler modulation scheme. The picture below shows the RedBearLab BLE Mini Board.MKRBL2-2T

The module is a combo of a RBT01 BLE module featuring the Ti CC2540 Single chip Bluetooth (SoC) and a breakout board that offers a micro USB connector and a 3.3Volt UART connectors. The main reason this board caught my attention is the fact that it also has solder points for additional GPIOs on the back of the board. These GPIOs can be custom programed. So it should be possible to hook up  I2C or SPI sensors to the board and with a bit of software be able to monitor them.
Using such a BLE board  is not exactly an IoT solution as the connectivity to the cloud would have to be implemented with for example a Wireless enhanced Gallileo. Also the BLE's range is rather limited. However when it comes to power consumption BLE has a leg up as it was specifically designed for very low power.

Spark Core compiler toolchain under cygwin

There are good instructions how to install the local toolchain to compile the Spark Core firmware. I don't want to replicate them here. Go to the Spark Core Github and check the Readme.md or search for a tutorial.
The purpose of this page is to show the steps and pitfalls when installing this toolchain under cygwin on a Windows 7 64 bit computer.

I assume you have at least the base cygwin installation on your machine. For instructions please head over to cygwin.org:

Screenshot 2014-04-07 22.05.40

Make sure you install git and any other goodies you like under cygwin.

  1. GCC for ARM Cortex processors - ARM cross compiler tool chain for Windows
  2. Make - Windows version of gmake
  3. Device Firmware Upgrade (DFU) Utilities - Utility to download the firmware to the Spark Core
  4. Zatig - USB driver for firmware downloads

Now install gcc for ARM and gnu make for Windows. Yes, cygwin also includes gmake. However there is a problem with it around dependencies. The MINGW GCC compiler uses Windows path notation in the *.d dependencies files that gmake under cygwin chokes on. So if you, the second time you try to compile, get an error like this:

it is because we use MINGW GCC under cygwin instead of a native cygwin compiler. You can also checkout a related posts on the Spark Community Board. It is now time to install the firmware. Pull the following three repositories from Github:

These repositories contain all the Spark Core firmware. Once the source code is downloaded, go to the build directory in the firmware folder and start the compile:

If everything went smooth you should now have a core-firmware.bin file.  Run the Zatig program to install the USB driver. The moment has come to flash the firmware into the Spark Core. For this, push the left button to reset the core while holding down the right button. Release the reset button and wait until the RGB-LED is flashing yellow. You can now download the firmware with the following command:

Screenshot 2014-04-08 03.03.57

Note the DFU utility always indicates an error "Error during download get_status" This is normal and as long as you see "File downloaded successfully" everything is fine.