Difference between revisions of "CometMeWindLoggingProject"

From TAMI
Jump to navigationJump to search
 
(39 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Redoing everything, step by step =
+
= Synopsys =
* Start with an empty SLboat_Mod TP-LINK TL-WR703N
+
This project started as a means to copy data from a sd accessible device to the interwebs.
* remove ~/.ssh/known_hosts offending lines for ssh access
+
 
* ssh into it. default password is ? by default it isn't set. set it and add ssh pub key via: http://192.168.1.1/cgi-bin/luci/;stok=050e6ecb7ab80178f16ec1b9aa83b690/admin/system/admin (just an example, stok will be different)
+
The sd carrying device (SCD) logs data every set amount of seconds about wind direction, power, and battery voltage. The problem was getting that data from the SCD to a website.
* setup networking via host
+
 
** If host is connected via wireless, and in the hackerspace, and host has IP of 192.168.1.239:
+
<diagram missing here: SCD->Flashair->OpenWRT->3G->Website>
*** host: masquerade_via_wlp3s0 (see below)
+
 
*** host: sudo iptables -F (fixme)
+
= Installation =
*** wrt: route add default gw 192.168.1.239
+
== Installing the router ==
*** wrt: echo nameserver 10.81.2.1 > /etc/resolv.conf
+
'''Setup of a new Barrier Breaker (14.07) flashed router (tested with TL-WR703N and MR-3020)'''
* change hostname to melogger: admin/system/system
+
 
* set timezone to Beirut: admin/system/system
+
Setup notes:
* opkg update
+
# Use a usb hub to connect the dongle because of https://dev.openwrt.org/ticket/16733
* opkg install packages for gsm and lua:
+
 
** opkg remove usb-uhci
+
Steps:
** opkg install usb-modeswitch usb-modeswitch-data comgt kmod-usb-serial kmod-usb-serial-option kmod-usb-acm luci-proto-3g luasocket luaposix
+
# Start with an empty SLboat_Mod TP-LINK TL-WR703N / 14.07 installed MR-3020/MR-3040
** opkg install usb-ohci kmod-usb-serial-wwan
+
# connect ethernet to laptop, laptop uses dhcp
* reboot. perhaps not required, but I had problems with uhci/ohci or this. Need to redo instructions to test this point.
+
# remove ~/.ssh/known_hosts offending lines for ssh access
* setup flashair card wireless as client
+
# ssh into it. default password is unset, with telnet enabled. Login via telnet, set password. Add ssh pub key via: http://192.168.1.1/cgi-bin/luci/;stok=050e6ecb7ab80178f16ec1b9aa83b690/admin/system/admin (just an example, stok will be different)
** use wizard: scan, choose flashair, use password. called wwan (the default)
+
# setup networking via host
** go to interfaces, select wwan, advanced, disable both "Use default gateway" and "Use DNS servers advertised by peer".
+
## If host is connected via wireless, and in the hackerspace, and host has IP of 192.168.1.239:
* connect gsm dongle, insert sim
+
### host: masquerade_via_wlp3s0 (see below)
* setup gsm networking
+
### host: sudo iptables -F (fixme)
** create new interface called '''umts''', for cellcom APN is internetg, the rest is empty (no username, no password)
+
### wrt: route add default gw 192.168.1.239
* create an ssh user key:
+
### wrt: echo nameserver 10.81.2.1 > /etc/resolv.conf
** mkdir /root/.ssh
+
# change hostname to melogger: admin/system/system
** chmod 700 /root/.ssh
+
# set timezone to Beirut: admin/system/system
** dropbearkey -f /root/.ssh/id_rsa -t rsa
+
# opkg update
* copy over the public key to cometme server at /.ssh/authorized_keys
+
# opkg install packages for gsm and lua:
* test that you can ssh to cometme: ssh -l user -i /root/.ssh/id_rsa server
+
## opkg remove usb-uhci (not required for 14.07, just for previous SLboat_Mod)
* host: git clone git://gitorious.org/air-sd-logger/air-sd-logger.git
+
## opkg install usb-modeswitch usb-modeswitch-data comgt kmod-usb-serial kmod-usb-serial-option kmod-usb-acm luci-proto-3g luasocket luaposix
* host: cd air-sd-logger
+
## opkg install usb-ohci kmod-usb-serial-wwan
* host: make update (assumes ip of router is 192.168.1.1)
+
# reboot. perhaps not required, but I had problems with uhci/ohci or this. Need to redo instructions to test this point.
** copies over main executable to '''/usr/bin/syncsdtocomet''' and config file to '''/root/config.lua'''
+
# setup flashair card wireless as client
* test: run executable locally once.
+
## use wizard: scan, choose flashair, use password. called wwan (the default)
** syncsdtocomet /root/config.lua
+
## go to interfaces, select wwan, advanced, disable both "Use default gateway" and "Use DNS servers advertised by peer".
* setup cron job: FILLME
+
# connect gsm dongle, insert sim
* test cron job (check the output, setup time close to it, wait)
+
# setup gsm networking
 +
## create new interface called '''umts''' (14.07: type UMTS), for cellcom APN is internetg, the rest is empty (no username, no password). For MF-180 choose /dev/ttyUSB2, make sure you rebooted after the opkg step (if you don't there will be no ttyUSB{0,1,2} available)
 +
## (check) MF-180: led will blink green. solid red means problem in config.
 +
## move it to "wan" zone
 +
## open ssh on wan zone by adding the following in /etc/config/firewall before the final 'include' line:
 +
### config rule
 +
###        option name  'accept ssh on wan'
 +
###        option src              wan
 +
###        option dest_port        22
 +
###        option target          ACCEPT
 +
###        option proto            tcp
 +
## mkdir /root/.ssh
 +
## chmod 700 /root/.ssh
 +
## dropbearkey -f /root/.ssh/id_rsa -t rsa
 +
# copy over the public key to cometme server at /.ssh/authorized_keys
 +
# test that you can ssh to cometme: ssh -l user -i /root/.ssh/id_rsa server
 +
# host: git clone git://gitorious.org/air-sd-logger/air-sd-logger.git
 +
# host: cd air-sd-logger
 +
# host: make update (assumes ip of router is 192.168.1.1)
 +
## copies over main executable to '''/usr/bin/syncsdtocomet''' and config file to '''/root/config.lua'''
 +
# test: run executable locally once.
 +
## syncsdtocomet /root/config.lua
 +
# setup cron job:
 +
## echo '30 0 * * * /usr/bin/syncsdtocomet /root/config.lua > /root/syncsdtocomet.last.log' > /etc/crontabs/root
 +
## /etc/init.d/cron start
 +
## verify via logread that no parsing errors occured in /etc/crontabs/root
 +
## verify symlink in /etc/rc.d
 +
# test cron job (check the output, setup time close to it, wait)
 +
## date 00:29
 +
## logread -f
 +
## wait 60 seconds
 +
## ^C
 +
## you should see a line: Nov 12 00:30:01 melogger cron.info crond[2727]: crond: USER root pid 2762 cmd /usr/bin/syncsdtocomet /root/config.lua > /root/syncsdtocomet.last.log
 +
# verify everything is running after a reboot
 +
## verify crond is running, there is an internet connection, and an sd card connection.
 +
# verify everything is running without ethernet connection
 +
## remove ethernet cable.
 +
## reboot forcibly (remove power).
 +
## wait 60 seconds.
 +
## verify: modem is blinking green (ZTE 190), wrt led is solid blue.
 +
# setup dynamic dns
 +
## create an account at duckdns.org
 +
## follow wrt instructions
 +
### opkg install ddns-scripts luci-app-ddns
 +
### update configuration via luci: services->dynamic dns. The config file:
 +
#### config service 'duckdns'
 +
####        option enabled '1'
 +
####        option domain '<the domain>'
 +
####        option username 'NA'
 +
####        option password '<the token>'
 +
####        option force_interval '72'
 +
####        option force_unit 'hours'
 +
####        option check_interval '10'
 +
####        option check_unit 'minutes'
 +
####        option update_url 'http://www.duckdns.org/update?domains=[DOMAIN]&token=[PASSWORD]&ip=[IP]'
 +
####        option interface 'umts'
 +
####        option ip_source 'interface'
 +
####        option ip_interface '3g-umts'
 +
# setup constant ip for cellcom:
 +
## change APN to statreal in /etc/config/network or via luci
 +
 
 +
== Installing a USB Modem presenting as an Ethernet card ==
 +
Tested working modem: HUAWEI E3131 05c6:9025 Qualcomm, Inc. Qualcomm HSUSB Device with 14.07, 3.10.49 ([[User:AlonLevy|AlonLevy]] ([[User talk:AlonLevy|talk]]) 12:42, 19 June 2015 (UTC))
 +
 
 +
# opkg install kmod-usb-net-cdc-ether
 +
# create an interface called UMTSETHER using the new ethernet interface (eth1 sould be)
 +
 
 +
HUAWEI E3276 (LTE USB Rotator)
 +
 
 +
# need to upgrade to 15.05 (CHAOS CALMER)!!
 +
# opkg install comgt-ncm kmod-usb-net-cdc-ncm  kmod-usb-net-huawei-cdc-ncm
 +
 
 +
== Installing the flashair card ==
 +
=== What is the Flashair ===
 +
It is a SDIO card that is also an Access Point. We use the 16 GB class 10 card.
 +
* http://www.toshiba-components.com/FlashAir/
 +
* https://www.flashair-developers.com/en/about/overview/
 +
=== Configuration of the Flashair card ===
 +
The card contains a SD_WLAN/CONFIG file ([https://www.flashair-developers.com/en/documents/api/config/ guide]), it needs to be edited to contain the following:
 +
APPSSID=cometmelogger
 +
APPAUTOTIME=0
 +
APPNETWORKKEY=haleycometme
 +
LOCK=1
 +
 
 +
== Setup on the remote server ==
 +
 
 +
= Notes =
 +
== rsync for flashair ==
 +
openwrt on wrt701 connected to flashair, with 3g modem connected to internet
 +
 
 +
* sd & wireless card combo (wifisd, flashair)
 +
* http server on port 80, with whole filesystem contents.
 +
* Files are stored at: SD CARD/CSVFILES/LOG
 +
* 192.168.0.1/24
 +
* (development on ethernet, 192.168.1.1/24)
 +
* periodically connect to a remote server via ssh
 +
 
 +
== Data flow overview ==
 +
* communication:
 +
** flashair -> openwrt wr703n w/ 3g modem -> host-on-cloud
 +
* communication paths
 +
** logger [sd slot]-> flashair sd card [flashair wifi]<-> openwrt [usb socket]-> usb extension cable 10m (hub + 2x5m) [usb socket]-> 3g modem [gsm]->[internet]->[ssh]-> host on cloud
 +
* physical
 +
** logger
 +
*** flashair sd card
 +
** TL-WR703N
 +
*** power adapter
 +
*** usb extension 5 m
 +
**** hub
 +
***** usb extension 5 m
 +
****** 3g modem
 +
 
 
== Helper: Masquerade script ==
 
== Helper: Masquerade script ==
 
  #!/bin/bash
 
  #!/bin/bash
Line 50: Line 161:
 
  iptables -t nat -A POSTROUTING -o $TARGET -j MASQUERADE
 
  iptables -t nat -A POSTROUTING -o $TARGET -j MASQUERADE
  
= Architecture =
+
== Troubleshooting ==
 
+
* Modem should ''blink green'' when connected. It's two faulty modes are:
sd&wireless card combo (wifisd, flashair)
+
** (incorrect) ''Solid Red''. That means the sim didn't register. It is a sim problem, not an openwrt one. Try a different sim. Try sim in phone. Contact gsm provider if nothing works and you can reproduce in a phone.
http server on port 80, with whole filesystem contents.
+
** ''Solid Green''. Sim is registered but not in data mode. This is an openwrt configuration error. Try rebooting. Then use logread to look for ppd, or ps. Use '''ubus call network.interface.umts''' status for more details. Try removing usb-uhci and installing usb-uhci, and rebooting after.
 
 
Files are stored at: SD CARD/CSVFILES/LOG
 
 
 
192.168.0.1/24
 
 
 
openwrt on wrt701 connected to flashair, with 3g modem connected to internet
 
(development on ethernet, 192.168.1.1/24)
 
 
 
periodically connect to a remote server via ssh
 
  
 +
= Notes =
 +
== Dropbear ==
 
[http://matt.ucc.asn.au/dropbear/dropbear.html dropbear] (minimal disk space implementation):
 
[http://matt.ucc.asn.au/dropbear/dropbear.html dropbear] (minimal disk space implementation):
 
* dropbearkey
 
* dropbearkey
 
* https://yorkspace.wordpress.com/2009/04/08/using-public-keys-with-dropbear-ssh-client/
 
* https://yorkspace.wordpress.com/2009/04/08/using-public-keys-with-dropbear-ssh-client/
 +
* comet-me server is hosted weirdly, the root is at /, so you need to place the public keys in /.ssh/authorized_keys (permissions 700 for directory, 600 for file)
  
comet-me server is hosted weirdly, the root is at /, so you need to place the public keys in /.ssh/authorized_keys (permissions 700 for directory, 600 for file)
+
== Cronjob ==
  
= Notes =
 
 
Copying is done 1 per minute with a cronjob
 
Copying is done 1 per minute with a cronjob
  
 
* * * * * /root/pushlog
 
* * * * * /root/pushlog
  
 +
== Wireless connection as client on openwrt ==
 
The flashair client wireless connection was created with the "wizard" from the scanned access points and then manually set to
 
The flashair client wireless connection was created with the "wizard" from the scanned access points and then manually set to
 
# ignore routes
 
# ignore routes
Line 90: Line 195:
 
= GDM/3G Modem with Antenna =
 
= GDM/3G Modem with Antenna =
 
One of the requirements is to have the modem in a cave, and the antenna external. One solution would be a long usb cable, but it looks like it would be less robust to random errors, so we are going with a long antenna.
 
One of the requirements is to have the modem in a cave, and the antenna external. One solution would be a long usb cable, but it looks like it would be less robust to random errors, so we are going with a long antenna.
 +
 +
== ZTE 190MF ==
 +
That's what we currently use, supported well with openwrt.
  
 
== Huawei 3131 ==
 
== Huawei 3131 ==
Line 118: Line 226:
 
* https://en.wikipedia.org/wiki/FME_connector
 
* https://en.wikipedia.org/wiki/FME_connector
  
= SD Carc with Wifi =
+
== ROOter ==
 +
Using this firmware because setting up cdc_ncm to work with HUAWEI E3276 otherwise was painful.
 +
 
 +
Website: http://ofmodemsandmen.com/
 +
 
 +
=== Funnelweb ===
 +
How to install extra packages:
 +
* the default location is broken: http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages
 +
* update it to this: http://downloads.openwrt.org/snapshots/trunk/ar71xx/generic/packages/base
 +
 
 +
= SD Card with Wifi =
 
This is hopefully identical to what Noam has:
 
This is hopefully identical to what Noam has:
 
* 4GB SD (Secure Digital) Eye-Fi card - Includes Wifi module that allows wireless data transfer to any wifi-enabled device - Easy setup and photo sharing with included software - Automatic file upload frees memory card space - Device compatibility with SDHC format - Included USB adapter - Class 4 Speed rating
 
* 4GB SD (Secure Digital) Eye-Fi card - Includes Wifi module that allows wireless data transfer to any wifi-enabled device - Easy setup and photo sharing with included software - Automatic file upload frees memory card space - Device compatibility with SDHC format - Included USB adapter - Class 4 Speed rating
Line 258: Line 376:
 
== Sync SD to Comet ==
 
== Sync SD to Comet ==
 
https://gitorious.org/air-sd-logger/air-sd-logger/
 
https://gitorious.org/air-sd-logger/air-sd-logger/
 +
 +
= Visualization =
 +
D3: http://tonygarcia.me/slides/d3chartintro/
 +
 +
Or maybe python: https://github.com/ContinuumIO/Bokeh
 +
Also see this completely unrelated [http://www.talyarkoni.org/blog/2012/06/08/r-the-master-troll-of-statistical-languages/ post] on python becoming the defacto language for scientific data processing
 +
 +
= Buying List =
 +
TODO: where are the eyecards
 +
 +
= Development =
 +
Using [http://wiki.openwrt.org/doc/howto/docker_openwrt_image docker]
 +
== Status ==
 +
=== MR3020 ===
 +
4 MB router. Having problems with:
 +
- ROOter - lack of space to install luaposix + luasocket, and luabitop (because of missing support for posix.bit32, which posix seems to be looking for - broken)
 +
- 15.03rc3 - lack of space as well.
 +
 +
Options:
 +
- weird that I cannot have more space available.
 +
- how much do I need?
 +
 +
Removing packages
 +
- after the fact
 +
- create a new image without them
 +
 +
The "create new image" option is what I'm going after.
 +
 +
ROOter gives a good out of box modem experience, and is basically just OpenWRT stuck to a slightly older version, so should be workable as a starting point in learning openwrt image building.
 +
 +
Currently following it's instructions from: http://ofmodemsandmen.com/build.html
 +
 +
Using a git checkout I have (which is just a clone of the svn).
 +
 +
==== Open Problems ====
 +
* some MR3020 don't work with HAUWEI LTE modems - see https://dev.openwrt.org/ticket/16505 (-145 error, seems it means timeout)
 +
 +
=== Different Wireless SD cards ===
 +
Initially this was developed with the Toshiba AirSD (typo?) card. That card turns out to occasionally not work. Since I cannot find any explanation for those problems (only one card so far) I'm trying other brand