Tag Archives: openwrt

mosquitto in OpenWrt which versions are where

Posted by KarlP on 8 May, 2015 None comments

With more and more versions of OpenWrt and mosquitto being released with more dependencies and features, here’s a quick overview of what versions are available where.

This is the out of the box versions. An extra packages feed, https://github.com/remakeelectric/owrt_pub_feeds is available for running mosquitto and it’s dependencies on older OpenWrt releases, should you need that functionality. At the time of writing, this feed contains mosquitto version 1.4.2, and libwebsockets 1.3.x.

I should clarify, this is what you get if you build your own images out of the box. Not the version that was available at release day.

Openwrt Backfire (10.03)

Attitude Adjustment (12.09)

Barrier Breaker (14.07)

Chaos Calmer (15.05)

master/trunk

Mosquitto version unavailable 0.15 1.3.5 1.4.2 1.4.2
libwebsockets n/a n/a n/a 1.3.x 1.4.x

Rust on OpenWrt – baby steps

Posted by KarlP on 3 October, 2014 1 comments

Well, it works, at least hello world. I followed the steps here: http://www.cnblogs.com/yido9932/p/3980362.html but had to make a few changes. First, I’m using the old atheros target, so I used “mips-unknown-linux-gnu” instead of mipsel. I still had to make symlinks into the STAGING_DIR path, because rust required the compiler to be with a certain name, and the target triplets don’t line up with OpenWrt.

./configure --target=mips-unknown-linux-gnu --disable-docs --prefix=/home/karlp/dummylib

I also had to remove the -mno-compact-eh flags from [rustroot]/mk/platform.mk in a couple of places. This is an unknown option in gcc as far as I can tell, it appears to be only in CodeSourcery/Mentor’s hacked up version? Then to actually compile the code I needed a few more incantations…

LD_LIBRARY_PATH=/home/karlp/dummylib/lib ~/dummylib/bin/rustc --target=mips-unknown-linux-gnu -C linker=mips-linux-gnu-gcc -L /home/karlp/dummylib/lib/rustlib/mips-unknown-linux-gnu/lib hicore.rs

And, there’s been a rust language change since that blogpost was made. Using the sample code from that article, you’ll get an error on the “hicore.rs” example about error: language item required, but not found: `fail_fmt`, so you need to add an extra magic line into your sample code, 

#[lang = "fail_fmt"] fn fail_fmt() -> ! { loop {} }

This is outlined on: http://doc.rust-lang.org/guide-unsafe.html

All in all, somewhat neat, maybe, I guess. But the rust std library is huge, as noted in the google translation of the chinese article linked above. And writing it all in “unsafe” code sounds kinda pointless. So, maybe not so interesting for OpenWrt at this point, but it runs at least.

CPU Profiling applications on OpenWrt with perf or oprofile

Posted by KarlP on 30 November, 2012 1 comments

I have a problem were two applications running on an OpenWrt router are using too much CPU time. I can reproduce the problem fairly well, but I don’t have any easy obvious clues as to the cause yet. Worse, it’s behaves more like a memory leak. The task stays the same, but CPU usage slowly creeps up over an hour or so. I’ve tried setting up the same applications on a linux desktop, and I can make it happen, but it’s so very very slow. So while that was running, I looked into CPU profiling on linux, and particularly, which of them would run on OpenWrt. So immediately things like valgrind/cachegrind and dtrace and friends are ruled out. I scratched google cpu profiler as being too invasive, and gprof as having too many criticisms. Also, I’d never used any of these tools before, and I didn’t want to spend all day testing them individually. Searching for CPU profiling linux kept turning up two applications, oprofile and perf. First step then was, “which of these, if any, is already packaged for OpenWrt?”

Background requirements / assumptions

  • You’re building your own OpenWrt images here…
  • I started testing this on Backfire, but then moved to Attitude Adjustment when I started working with perf.
  • You are free to recompile the code under test, and aren’t trying to test a live system

Turns out the both are, but you have to jump some hoops. In “make menuconfig“, you need to got to “Gobal build settings” and enable “Compile the kernel with profiling enabled“. That’s all you need for oprofile, but for perf, go also to “Advanced configuration options (for developers)” and enable “Show broken platforms / packages“. I’m working on the atheros platform (ar2317), and apparently (I’m told on IRC) it’s a packaging problem more than a broken tool. It works on atheros at least, YMMV.

Continuing, you need to turn on the libelf1 library under “Libraries“, because the perf package doesn’t select it properly for you. Finally, under “Development” enable oprofile, perf or both. (When I started this, I didn’t have BROKEN packages enabled, so just used oprofile, and later used perf. If perf works, I’d only use perf, and skip oprofile altogether. I won’t talk about oprofile any further, but this was all you needed to get it to work)

So, that will make an image that can run the tools, but all binaries are (rightfully) stripped on OpenWrt by default in the build process, so you won’t get particularly useful results. In theory, you can record performance data on the router, and then generate the reports on your linux desktop with all the symbols, but I never worked out how to do this.

So, we want to run non-stripped binaries for anything we want to actually investigate. One way of doing this is the following magic incantation in your OpenWrt buildroot (Provided by jow_laptop on IRC)

make package/foo/{clean,compile} V=99 CONFIG_DEBUG=y

You can then copy the binary from [buildroot]/staging_dir/target-arch_uClibc-0.x.xx.x/root-platform/usr/bin/foo onto your device and run it. Actually, if you only want to analyse a single program, that may be enough for you. You can copy that non-stripped binary to the ramfs on /tmp, and just start profiling…

$ start_your_program
# capture 120 seconds worth of profiling data
$ perf record -p $(pidof your_program) sleep 120
# wait…
$ perf report

If your program actually runs to completion, rather than being a daemon of some sort, you can just use “perf record your_program”. This may actually give you enough information. Lucky you!

Here’s an example of recording a stripped binary, and recording a non-stripped binary. (This is a real program, but it’s not the point here.)

# Overhead  Command           Shared Object  Symbol
# ........  .......  ......................  ......
#
    59.34%  mlifter  [kernel]                [k] 0x800cd150
    25.04%  mlifter  libuClibc-0.9.33.2.so   [.] 0x37990 
     5.17%  mlifter  libgcc_s.so.1           [.] 0xaaa4  
     2.96%  mlifter  libpthread-0.9.33.2.so  [.] 0xfe54  
     1.68%  mlifter  libjson.so.0.0.1        [.] 0x4e00  
     1.25%  mlifter  [nf_conntrack]          [k] 0x22e4  
     0.87%  mlifter  libmosquitto.so.1       [.] 0x17d0  
     0.87%  mlifter  ld-uClibc-0.9.33.2.so   [.] 0x1478  
     0.64%  mlifter  mlifter                 [.] 0x6bac  
     0.61%  mlifter  [ip_tables]             [k] 0x5c8   
     0.26%  mlifter  libuglylogging.so       [.] 0x96c   
     0.26%  mlifter  libmodbus.so.5.0.0      [.] 0x2e60  
     0.23%  mlifter  [xt_conntrack]          [k] 0x2d0   
     0.20%  mlifter  [nf_conntrack_ipv4]     [k] 0x320   
     0.15%  mlifter  libcrypto.so.1.0.0      [.] 0x84e58 
     0.12%  mlifter  [iptable_raw]           [k] 0       
     0.12%  mlifter  [iptable_nat]           [k] 0x170   
     0.09%  mlifter  libuci.so               [.] 0x2cec  
     0.06%  mlifter  [nf_nat]                [k] 0x5e4   
     0.03%  mlifter  [nf_defrag_ipv4]        [k] 0x34    
     0.03%  mlifter  [iptable_mangle]        [k] 0xe0    
     0.03%  mlifter  [iptable_filter]        [k] 0x40

And here’s the same thing, non-stripped

# Overhead  Command           Shared Object                         Symbol
# ........  .......  ......................  .............................
#
    59.29%  mlifter  [kernel]                [k] 0x800780e0
    26.30%  mlifter  libuClibc-0.9.33.2.so   [.] 0x39ff0 
     5.66%  mlifter  libgcc_s.so.1           [.] 0xa154  
     2.83%  mlifter  libpthread-0.9.33.2.so  [.] 0xaa20  
     1.70%  mlifter  libjson.so.0.0.1        [.] 0x4b54  
     0.85%  mlifter  [nf_conntrack]          [k] 0x6020  
     0.60%  mlifter  [ip_tables]             [k] 0x630   
     0.58%  mlifter  libmosquitto.so.1       [.] 0x39cc  
     0.47%  mlifter  ld-uClibc-0.9.33.2.so   [.] 0x29f0  
     0.23%  mlifter  libuglylogging.so       [.] 0x7d0   
     0.19%  mlifter  libmodbus.so.5.0.0      [.] 0x55cc  
     0.19%  mlifter  [nf_conntrack_ipv4]     [k] 0x2fc   
     0.17%  mlifter  mlifter                 [.] 0x1880  
     0.17%  mlifter  libcrypto.so.1.0.0      [.] 0x35b7c 
     0.15%  mlifter  mlifter                 [.] msg_add_branch_readings_full
     0.15%  mlifter  [xt_conntrack]          [k] 0xb0    
     0.15%  mlifter  [iptable_nat]           [k] 0x360   
     0.06%  mlifter  [iptable_filter]        [k] 0x40    
     0.04%  mlifter  mlifter                 [.] run_tasks
     0.04%  mlifter  mlifter                 [.] sfw_read_smart_bar
     0.04%  mlifter  [nf_defrag_ipv4]        [k] 0x54    
     0.02%  mlifter  mlifter                 [.] sfw_provides_circuit_readings
     0.02%  mlifter  mlifter                 [.] publish_backoff
     0.02%  mlifter  mlifter                 [.] sfw_verify_modbus_connection
     0.02%  mlifter  mlifter                 [.] task_modbus_poll
     0.02%  mlifter  mlifter                 [.] task_check_state_complete
     0.02%  mlifter  [nf_nat]                [k] 0x610   
     0.02%  mlifter  [iptable_mangle]        [k] 0x50    
     0.02%  mlifter  [fuse]                  [k] 0x70c

Not much of a difference. We can see calls into our own code, but well, we don’t spend much time there :) Really, to get useful output, you need non-stripped libraries, and the kernel symbols. You’re never going to fit all of that on a ramdisk on your OpenWrt router. It’s time for some remote filesystems.

I used sshfs for this, which was very easy to setup, but it’s pretty cpu heavy on a small router. NFS probably would have been a better choice. However you do it, mount your OpenWrt buildroot somewhere on your router.

Then, even without any changes in how you call your application, you can add the kernel symbol decoding, just add the -k parameter. “perf report -k /tmp/remote/build_dir/linux-atheros/linux-3.3.8/vmlinux” Now we get the kernel chunks broken out, like in this snippet.

# Overhead  Command           Shared Object                             Symbol
# ........  .......  ......................  .................................
#
    26.30%  mlifter  libuClibc-0.9.33.2.so   [.] 0x39ff0 
     5.85%  mlifter  [kernel]                [k] finish_task_switch.constprop.58
     5.68%  mlifter  [kernel]                [k] __queue_work
     5.66%  mlifter  libgcc_s.so.1           [.] 0xa154  
     4.07%  mlifter  [kernel]                [k] n_tty_write
     2.83%  mlifter  libpthread-0.9.33.2.so  [.] 0xaa20  
     2.83%  mlifter  [kernel]                [k] __wake_up
     2.41%  mlifter  [kernel]                [k] tty_write
     1.85%  mlifter  [kernel]                [k] handle_sys
     1.70%  mlifter  libjson.so.0.0.1        [.] 0x4b54  
     1.58%  mlifter  [kernel]                [k] fsnotify
     1.57%  mlifter  [kernel]                [k] put_ldisc
     1.30%  mlifter  [kernel]                [k] vfs_write
     0.94%  mlifter  [kernel]                [k] tty_insert_flip_string_fixed_flag
     0.91%  mlifter  [kernel]                [k] __do_softirq
     0.91%  mlifter  [kernel]                [k] memcpy

We’re still missing symbols from uClibc, and any other user libraries we may call. (like libjson there) Fortunately, when we rebuilt our package with CONFIG_DEBUG=y earlier, it rebuilt the libraries it depends on in debug too. But now, we need to make sure we call our binary using the correct debug libraries. (This is the bit I think should be doable offline, but this is the way I got it to work at least!)

This is setting up the library path to use the non-stripped libraries. We then start perf using this process ID. (/tmp/remote is where I mounted my OpenWrt buildroot)

$ LD_LIBRARY_PATH=/tmp/remote/staging_dir/target-mips_uClibc-0.9.33.2/root-atheros/lib:/tmp/remote/staging_dir/target-mips_uClibc-0.9.33.2/root-atheros/usr/lib /tmp/remote/staging_dir/target-mips_uClibc-0.9.33.2/root-atheros/usr/bin/mlifter

And now we can see all the details….

# Overhead  Command           Shared Object                             Symbol
# ........  .......  ......................  .................................
#
     6.39%  mlifter  [kernel]                [k] finish_task_switch.constprop.58
     5.83%  mlifter  [kernel]                [k] __queue_work
     5.01%  mlifter  libuClibc-0.9.33.2.so   [.] _ppfs_parsespec
     3.98%  mlifter  [kernel]                [k] n_tty_write
     3.27%  mlifter  libuClibc-0.9.33.2.so   [.] _vfprintf_internal
     3.24%  mlifter  [kernel]                [k] __wake_up
     2.77%  mlifter  libuClibc-0.9.33.2.so   [.] __libc_write
     2.48%  mlifter  [kernel]                [k] tty_write
     1.60%  mlifter  [kernel]                [k] vfs_write
     1.60%  mlifter  [kernel]                [k] put_ldisc
     1.57%  mlifter  [kernel]                [k] fsnotify
     1.53%  mlifter  libuClibc-0.9.33.2.so   [.] _fpmaxtostr
     1.45%  mlifter  [kernel]                [k] handle_sys
     1.31%  mlifter  libgcc_s.so.1           [.] __muldf3
     1.11%  mlifter  libuClibc-0.9.33.2.so   [.] __stdio_WRITE
     1.08%  mlifter  [kernel]                [k] tty_insert_flip_string_fixed_flag
     1.06%  mlifter  libgcc_s.so.1           [.] __unpack_d
     1.04%  mlifter  libuClibc-0.9.33.2.so   [.] __stdio_fwrite
--- snip ---
     0.19%  mlifter  libjson.so.0.0.1        [.] json_object_put
--- snip --- 
     0.11%  mlifter  libjson.so.0.0.1        [.] json_object_double_to_json_string

We can now see into uClibc, and into user libraries like libjson. (Trust me, I just didn’t include _alllll_ the output.) This is not actually the same point in the program, so don’t read too much into the actual numbers here. If you want, you can comment that this program is spending far too much time printing something to the console. (which is true!)

That’s it. You can now get symbol level information on what’s happening and how much. Whether this is useful is a different story, but this was fairly complicated to setup and test out, so I thought this might be helpful to others at some point.

Automatic OpenWRT builds and conflicting package versions

Posted by KarlP on 1 October, 2012 None comments

I’m doing some automated builds of OpenWRT here, and was running into problems with the .config file, package dependencies and versions across multiple feeds. Specifically, I have a very very tiny .config file that lists a package and platform, and that pulls in just about everything else it needs. However, depending on the order of feed update and install, (And particularly, when using ./scripts/feeds/install -a) the wrong version of one of the dependencies was being pulled in.

Now, if all feeds were instantly and magically updated in the OpenWRT upstream repository, this wouldn’t be a problem of course :) Unfortunately, that problem isn’t going away, so what to do?

Well, basically, get simpler. Originally I had a giant .config file copied out of a manually configured buildroot, a feeds.conf with all sources, and the build script ran

  1. ./scripts/feeds/update -a && ./scripts/feeds/install -a
  2. make -j5 V=s

That way’s crap. The new way runs with a much much much smaller .config file, onyl about 10 lines (enough to specify the platform, a “master” package that depends on what we want, and one or two things we explicitly don’t want) Then, it’s a matter of calling feeds install -p feedname -a IN ORDER so that if there’s a package in one feed that depends on a package in another feed, that the dependency’s feed is installed first.

Specifically, this happened here when our “master” project depended on mosquitto, which is both in the stock feed for trunk, (at v0.15) as well as in the company maintained public feed (at v1.0.3) The order of the install -a call was pulling in the mosquitto dependency from trunk first.

So, the new cool way…

  1. cp tiny.config .config
  2. ./scripts/feeds update -a
  3. ./scripts/feeds install -a -p owrt_pub_feeds
  4. ./scripts/feeds install -a -p private_feeds
  5. make defconfig
  6. make -j5 V=s

Much better.

Local development for LuCI

Posted by KarlP on 18 May, 2012 None comments

So, you’ve got some LuCi modules packaged up, and you can build them via the OpenWRT build root, and you can install them with opkg, and you can edit the final files in /usr/lib/lua/luci/{controller,views,etc} and the changes take effect immediately, but you don’t like working on the target router. You don’t get to use the editors you want, it’s hard to integrate into your version control to see what you’ve been doing. You’ve seen this page about a “local development environment”, but it didn’t help as much as you had hoped?

Here’s a few things that worked for me, based on lots of advice and snippets from jow_laptop and some experimentation.

First, I’m assuming you’ve got luci trunk checked out, and I’m going to refer to that directory as [LUCI_DIR]. I’m assuming that your working luci package, which has the OpenWRT opkg Makefile, and a luasrc and htdocs directory are in [YOUR_PKG].

Now, the opkg Makefile that does a great job of building your package normally isn’t going to cut it for running inside luci. Make a new directory in [LUCI_DIR]/applications/your_name and add a Makefile with the following contents

include ../../build/config.mk
include ../../build/module.mk

That’s it. Now, you need the luasrc/htdocs etc directories from your project. This is a touch tedious…

  1. [LUCI_DIR]/applications/your_name$ ln -s [YOUR_PKG]/luasrc
  2. [LUCI_DIR]/applications/your_name$ ln -s [YOUR_PKG]/htdocs

At this point, [LUCI_DIR]$ make runuhttpd will churn away, and copy things all over the place, and you should have your package working locally.

I had problems where controllers were simply not detected, with the page’s “entry” links simply returning a 404 “No page is registered at blah/wop” As best I can tell, this was a problem in the fast indexer, and editing [LUCI_DIR]/libs/web/dispatcher.lua, in the createindex function, to use the plain indexing instead of the fast indexing meant that all my controllers showed up properly. The controllers were all working perfectly on the target mind you.

So, with this, I don’t have to fiddle around with sshfs, and I can easily use things like git diff to see what I’ve actually done recently. However, because of the way make runuhttpd works, by copying all the files first into a dist directory, then copying that into the [LUCI_DIR]/host/ tree, you have to continually stop and start the server after any changes. A project for another day….