Who pays for shipping? FedEx, TI and overseas agents

Update: I lost this in my drafts folder for over a year!

I live in Iceland. It’s a nice place to live, but it’s small, so I often have to buy things from overseas, particularly in for hobby electronics gear, but anything small and cheap has this problem.

Now, let’s get this straight, I’m not complaining about customs or VAT. (Not today anyway) VAT here in Iceland is 25.5%, and is applied at the end, on top of shipping charges. Customs rate isn’t always as easy to work out.

Now, in this example, I bought a device from Texas Instruments, for $US4.99, or about 630 ISK. This was advertised as shipping included, worldwide. Add 25.5% VAT, 790 ISK. Not bad. So it arrived yesterday.

Now, what does that even mean? They’ve charged me ~473 ISK for “customs handling” It doesn’t seem to be anything like the VAT rate, or the VAT rate plus the customs rate. (FIXME – finish getting customs rate)

Customs officially don’t collect anything like this, they simply say that agents are allowed to charge for filing paperwork.

So, this seems to be some arbitrary amount they just felt like charging. Next we have 750 ISK for being “an unregistered customer.” Well, that sounds awesome. I didn’t choose this company, I ordered a product from Texas Instruments with shipping included. Who knows if it’s even possible to become a registered customer.

Then there’s VAT on those charges. So, a whole 120% charge (of original purchase) on being the wrong sort of customer and another 75% (of original purchase) for “handling”. I can’t actually see where they charge the VAT/Customs on the product itself, but the end result is that I’ve paid 170% of the original purchase ON TOP of the basic vat rate + the product price. (1345 ISK + 790 ISK)

How is this remotely reasonable for “shipping included” ? And who’s responsible? Texas Instruments clearly believes they paid FedEx, (or presumably FedEx wouldn’t accept the shipment?) I can only assume that FedEx delivers the package to Iceland, believing that they’ve been paid, but I have zero insight into what the deal is between the local agent (Hraðflutningar ehf)

I believe the party responsible is IceTransport. As they said earlier, local customs agents, defined here, are allowed to charge basically whatever they like. So they do. I presume they treat customers shipping outbound through them quite a lot better, but what are my options at the end of the day? I refuse to pay their arbitrary charges, they hold the parcel, and they’ve already been paid by FedEx/Texas Instruments? So they bin the package, and are already in front? Or do I just suck it up, and pay through the nose? They’re both distasteful.

Customs official answer on handling charges:

http://www.tollur.is/displayer.asp?cat_id=2816&module_id=210&element_id=10177

Decoding Vendor Specific USB protocols with Wireshark lua plugins

Earlier this week I was doing some reverse engineering and confirmation of behaviour for a USB tool. I got wireshark to sniff the traffic, (Not going into that here, it’s relatively straightforward and documented enough on the web to get by) but as it’s a vendor specific protocol, it was just lots of bytes.

I was decoding them by hand, and then copying and pasting into a python script (I had pretty good sources of what all the bytes meant, I just had no good way of visualising the stream, which is where wireshark and this post comes in)

I have written a custom wireshark dissector before, but I wasn’t super happy with the mechanism. I have been doing some work with Lua in my dayjob, and had read that wireshark supported dissector plugins could be written in Lua now. Seemed like a better/easier/more flexible approach.

I got to work, following these (somewhat) helpful resources:

These were all generally very helpful, but there two things I wrestled with, that I didn’t feel were at all well described. From here on out, I’m going to assume that you know what’s going on, and just need help with things that are not covered in any of the earlier links.

Little Endian

You declare ProtoField’s as just uint8, uint16, uint32. This is fine and sane. But there’s a few ways of working with it when you add it to the tree.

f.f_myfield = ProtoField.uint32("myproto.myfield", "My Field", base.HEX)
-- snip --
mytree:add_le(f.f_myfield, buffer(offset, 4)

This way works very well, selecting “myfield” in the packet view correctly highlights the relevant bytes. But say you want to get the value, to add to the info column for instance, you might do this.. (if you read the api guide well enough)

local val = buffer(offset, 4):le_uint()
-- normal add, we've done the LE conversion
-- if you don't do :le_uint() above, and do a :add_le() below,
-- the info column will show the backwards endian value!
mytree:add(f.f_myfield, val)
pinfo.cols["info"]:append(string.format(" magicfield=%d", val))

At first glance, this works too. The Info Column shows the right value, the tree view shows the right value. BUT clicking on the tree view doesn’t highlight the bytes.

Here’s how to do it properly:

local val = buffer(offset, 4)
mytree:add_le()(f.f_myfield, val)
pinfo.cols["info"]:append(string.format(" magicfield=%d", val:le_uint()))

Ok, a little fiddly, but you would probably get there in the end.

Reading existing USB properties

The docs talk about doing something like:

local f_something = Field.new("tcp.port")

Except, I didn’t find anywhere that described what magic strings were available. I tried using the values available in the display filter box, like, “bEndpointAddress” but never got anywhere. One of the samples led me to this tidbit:

       local fields = { all_field_infos() }
       for ix, finfo in ipairs(fields) do
            print(string.format("ix=%d, finfo.name = %s, finfo.value=%s", ix, finfo.name, getstring(finfo)))
        end

When you click on a packet, this will dump lots to the console, and you can hopefully work out the magic values you need!

Synchronising packets

TCP streams are easy, you have sequence ids to correlate things. USB isn’t quite the same. You can see the “URB” has two frames, from host to device and device to host, which are in sync, but for the very common case of writing to an OUT endpoint, and getting a response on an IN endpoint, you don’t get any magical help.

I found a way of doing this, but it’s not ideal, and tends to mess up the display in wireshark if you click on packets in reverse order. This is because I just set a state variable in the dissector when I parsed the OUT packet, and check it when I parse the IN packet. It works, but it was less that ideal. Sometimes you need to click forwards through packets again. Sometimes the tree view would show the right values too, but the info column would be busted. Probably doing something wrong somewhere, but hard to know what.

Actually hooking it up

Finally, and the most frustrating, was how to actually hook it up! The demos, all being TCP related, just do:

the_table = DissectorTable.get("udp.port")
the_table:add(9999, my_protocol)

Ok, well and good, but how on earth do I register a USB dissector? You need to register it by the class which is sort of ok, but good luck having dissectors for multiple vendor specific classes. I didn’t see a way of adding a dissector based on the VID:PID, though I think that would be very useful.

usb_table = DissectorTable.get("usb.bulk")
-- this is the vendor specific class, which is how the usb.bulk table is arranged.
usb_table:add(0xff, my_proto)
-- this is the unknown class, which is what shows up with some usb tools?!
usb_table:add(0xffff, my_proto)

Update 2013-Jan-16One of the sigrok developers got the vid/pid matching working, in another example of a lua plugin.

This skims over a lot, but it should help.
Final working code:

I’ve totally glossed over exactly _what_ or _why_ I was decoding vendor specific usb classes. That’s a topic for another day :)

Here’s some pictures though

How it looks before you write a plugin.

How it looks before you write a plugin.

With the plugin, note that the info column isn't always showing the proper values.  No idea why.  wireshark's weird

With the plugin, note that the info column isn’t always showing the proper values. No idea why. wireshark’s weird

PS: How about that title! linkbait to the max! keywords to the rescue! (or at least, the ones I had tried searching for)

Installing Eagle 6.5 on Fedora 18

I’ve spoken about this before, but here’s how to install current Eagle 6.5.0 on Fedora 18 64bit.

As before, this may be incomplete for a fresh install, ie, I’ve already installed all the 32bit compat libs. But Eagle expects to find OpenSSL 1.0.0. No other version at all.

You need openssl-libs.i686 to be installed, but that will be 1.0.1e, and none of the existing symlinks are suitable.

$ sh ./eagle-lin-6.5.0.run
/tmp/eagle-setup.11607/eagle-6.5.0/bin/eagle: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

However, you can just symlink it a bit more…

/usr/lib $ sudo ln -s libssl.so.1.0.1e libssl.so.1.0.0
/usr/lib $ sudo ln -s libcrypto.so.1.0.1e libcrypto.so.1.0.0

STM32 Unique ID Register on L1 parts

Last week I mentioned I was seeing duplicate “Unique” ID registers on some STM32L151 parts [1], and included some memory dumps of the three unique ID registers as proof.

However, I had foolishly assumed that on the L1 the Unique ID register was three contiguous 32 bit registers, as it is on the F0, F1, F2, F3 and F4. (The base address changes, but that’s normal and expected)

On the L1, the registers are actually at offset 0, offset 4, and offset 0x14. Thanks for nothing ST! :(
(Oh, and L1 Medium+ and High Density devices use a different base address too, good job)

Here’s some more complete memory dumps for the same three parts I was looking at last week.

UID31:0 UID63:32 UID95:64
Part A 0x0e473233 0x30343433 0x00290031
Part B 0x0e473233 0x30343433 0x00320032
Part C 0x0e473233 0x30343433 0x00380030

Reading other reference manuals, and seeing that the UID registers often have 8 bits of unsigned “Wafer number”, 7 bytes of ASCII Lot number, and 4 bytes of X/Y wafer coordinates in BCD, I would interpret my part “A” above as

Wafer Number Lot Number X/Y coords
Hex 0x0e 0x47323330343433 0x00290031
Natural 0x0e G230443 X=0029, Y=0031

For reference, here are some full dumps of that section of memory. “0x7b747800” is what I had been looking at as UID bits 95:64, note that there are other bits in this section with fixed values, no idea what they mean :)

Part A

(gdb) x /20x 0x1FF80050
0x1ff80050: 0x0e473233  0x30343433  0x7b747800  0x50505091
0x1ff80060: 0x00000011  0x00290031  0x11000000  0x11000011
0x1ff80070: 0x00000000  0x00000000  0x029f067e  0x035a0000
0x1ff80080: 0x035a0000  0x035a0000  0x035a0000  0x035a0000
0x1ff80090: 0x035a0000  0x035a0000  0x035a0000  0x035a0000
(gdb) 

Part B

(gdb) x /20x 0x1FF80050
0x1ff80050: 0x0e473233  0x30343433  0x7b747800  0x50505091
0x1ff80060: 0x00000011  0x00320032  0x11000000  0x11000011
0x1ff80070: 0x00000000  0x00000000  0x02a50685  0x035e0000
0x1ff80080: 0x035e0000  0x035e0000  0x035e0000  0x035e0000
0x1ff80090: 0x035e0000  0x035e0000  0x035e0000  0x035e0000
(gdb)

Part C

(gdb) x /20x 0x1FF80050
0x1ff80050: 0x0e473233  0x30343433  0x7b747800  0x50505091
0x1ff80060: 0x00000011  0x00380030  0x11000000  0x11000011
0x1ff80070: 0x00000000  0x00000000  0x02a50689  0x035e0000
0x1ff80080: 0x035e0000  0x035e0000  0x035e0000  0x035e0000
0x1ff80090: 0x035e0000  0x035e0000  0x035e0000  0x035e0000
(gdb)

[1] Again, these are STM32L151C6T6 parts, revision V, package markings “GH254 VG” and “CHN309”

STM32 Unique ID register not so unique (Or, how to read docs properly)

UPDATE: This post is WRONG! See updated information here

The findings below were based on expecting the UID register to be contiguous as it is on all other STM32 parts. This is not true on the L1 family, and I hadn’t taken enough care with reading the reference manual.

Original post below


Following up from when I wrote about it earlier, it turns out that the “unique” id isn’t as unique as it is meant to be.

On my desk I have three different STM32L151C6T6 revision “V” parts, with exactly the same 96bit unique id. The parts all have package labels “GH254 VG” and CHN309

UID[32:0] (0x1FF80050) UID[63:32] (0x1FF80054) UID[96:64] (0x1FF80058)
Hex 0x0e473233 0x30343433 0x7b747800
Decimal 239546931 808727603 2071230464

According to reports on the irc channel ##stm32, this has also been seen (at least) on stm32f407vet6 parts.

Not fun :(

Installing Eagle 5.12 on Fedora 19 64bit

Earlier I wrote about installing Eagle 5.12 on Fedora 17 64 bit, but on a new computer, I have an updated OS, so here’s a revised list.

This should be the minimal steps to install. (You probably need glibc.i686 as well, but it _should_ get pulled in by these other libs)

  • libXrender.i686
  • libXrandr.i686
  • libXcursor.i686
  • freetype.i686
  • fontconfig.i686
  • libXi.i686
  • libpng12.i686 (Changed names since Fedora 17)
  • libjpeg-turbo.i686

libstdc++.i686 might have been necessary as well, but I had quite possibly already installed it.

Bash tab completion for python scripts

Bash completion is normally pretty awesome. You can even tab complete hostnames to ssh, if you’ve ssh’d to them before. It can complete arguments to all sorts of commands, and there’s even a python module that will let you provide completion for your custom command.

And this is all handled by plugins, so if you are executing something new/different, you get plain old regular completion of files and directories. This mostly all works, but I ran into a problem with a python script I’d written the other day, and I have no solution at all, only a workaround.

In all cases I expect to get full path plus file completion and prompting as my command is “unknown”

Command

Outcome

asdfasdf fileprefix[tab]

Works perfectly

python my-script.py fileprefix[tab]

Works perfectly

python my-script.py --my-argument fileprefix[tab]

only completes directories!

chmod +x myscript.py
./my-script-py --my-argument fileprefx[tab]

Works perfectly

I can only assume there’s a bug in the python completion module? No idea how to diagnose it further though. Having to make my own scripts executable isn’t the end of the world though, so it works for now. This was with bash 4.25.45(1) on Fedora 18, and bash-completion version 2.1, fwiw

Sniffing 802.15.4 with wireshark and MRF24J40 modules

Wireshark screenshot showing my test data

This is what I did the last two evenings or so. The data being sent is from my example app for my MRF24J40 module library, simrf.

The streaming into wireshark is all thanks to one of the Contiki devs, George Oikonomou’s sensniff project. The peripheral code is just another app using my library.

Neat! This should make some things a bit easier to understand as I move into getting the full networking stack working in Contiki. (You can ignore the warnings from wireshark about the packets being invalid 6LoWPAN frames, all the frames are just raw 802.15.4, with “abcd” being sent as the frame payload.

Further adventures with Contiki OS

I got the (very) basic things working with my port of Contiki OS to my stm32l discovery and stm32vl discovery boards. It was more painful than I expected, there were some unexpected things, but generally, nothing too complicated. Now, I’m at the point where I was hoping to be, starting to try and use the networking stacks in Contiki OS. I want to use the 6LoWPAN implementation, the RPL implementation, the TCP (v4 and v6) implementations.

And here things unravel very quickly. The docs had lead me to believe that I just needed to implement “rtimers” and it would all start working. Turned out the docs referred to an API 5 years out of date. rtimers only actually support a single timer, so you can’t really actually use them in your own apps, you’re really just providing them for the exclusive use of the radio duty cycling (RDC) code. Of which there are multiple styles.

So, yeah, you don’t need to implement them at all. Think of them as rdc_timer and it all makes a lot more sense. They were presumably only introduced at all as the existing timer code (etimer/ctimer) are based on co-operative multi tasking, and RDC algorithms need relatively hard timing to do the radio strobing.

So, what’s the status of Contiki then? The mailing list gives an interesting picture. You’ve got some very advanced topics being discussed about packet loss modelling and distance calculation methods and tweaking the latest draft of CoAP implementation. Then you’ve got posts about simple things that still aren’t fixed like examples simply not even compiling.

Oh yeah, that’s right, did I mention that? There’s examples that don’t compile. You can apply some “fixes” to make them compile, but they’re clearly not the way the makefiles and the project were intended to be, and no-one who might know is
answering.

Stepping out again, it starts to feel like Contiki OS isn’t really an OS project that you can use as is. It’s an OS that’s an incubator for other projects. TCP and UDP for both ipv4 and v6, with all of it configurable via an undocumented mix of makefile variables and C preprocessor defines. A windowing toolkit, because why wouldn’t I want to VNC to my battery powered wireless sensor node and click on things. (I’m serious, the current tree includes a graphical calculator app and a vnc server) Three different RDC algorithms. Two different CoAP implementations. A rather large and complex java modelling framework. Two different file systems. Two different webservers and a webbrowser. This is all in the same tree, along with support for ~20 platforms with ~6-7 different architectures.

It starts to make a bit more sense when you realise that it all started out as an operating system for a commodore 64, but that doesn’t really excuse it per se. Shit’s broken. Shit aint improving real fast. Back to the two different sorts of threads on the mailing list and it starts making more sense again. There’s the CS research types, using hardware the previous grad student used, with the magic incantations from the previous researcher, working on really interesting science, and Contiki OS is the base. Then there’s people who are looking around for some modern networking stacks to use on a device of their own. These other people currently have a pretty raw deal. It’s like the Contiki itself really, some really neat cutting edge science, with sharks and hot burning sun and no water nor a soul in sight to ask for help.

Fortunately, there does appear to be a few birds in the sky, hinting at land. Contiki OS’s now a github project, and getting actual pulls, rather than just being hidden inside a Swedish University. There’s a bunch of people who seem keen to try and clean up some of the accumulated cruft of ten years of grad projects being dumped into a repository somewhere. Now if only there was a little bit of direction and documentation from central leaders and we could really sail!

Unaligned memory access fault on Cortex-M3

AKA A surprising thing that happened to me while porting Contiki to the STM32F1.
AKA Some steps to take when diagnosing an unexpected hard fault on ARM Cortex M3

I already have a STM32L1 port working (for the basic uses of Contiki) and the major difference with this port is that it should support pretty much any target that libopencm3 supports. So I made a new platform and tweaked the GPIO settings for the STM32F1, and flashed it to my STM32VL Discovery board, and…. it started, but then it crashed.

Program received signal SIGINT, Interrupt.
blocking_handler () at ../../cm3/vector.c:86
86	{
(gdb) bt
#0  blocking_handler () at ../../cm3/vector.c:86
#1  
#2  update_time () at contiki/core/sys/etimer.c:72
#3  

Now, I don’t see unhandled exceptions much these days. I consulted the Configurable Fault Status Register (CFSR) at 0xE000ED28 and compared that to the definitions in ARM’s “Cortex M3 Devices Generic User Guide” (link will google search to the current location of that doc)

(gdb) x /wx 0xE000ED28
0xe000ed28:	0x01000000
(gdb) 

Ok, some bit in the top 16bits. That’s the Usage Fault Status Register(UFSR). Let’s look at it a little closer because I can’t count hex digits in my head as well as some people.

(gdb) x /hx 0xE000ED2a
0xe000ed2a:	0x0100
(gdb)

Ok. That bit means, Unaligned access UsageFault. Awesome. One of the big selling points of ARM Cortex-M is that it doesn’t care about alignment. It all “just works”. Well, except for this footnote: "Unaligned LDM, STM, LDRD, and STRD instructions always fault irrespective of the setting of UNALIGN_TRP" Ok, so let’s see what caused that. GDB “up” two times to get to the stack frame before the signal handler. x /i $pc is some magic to decode the memory at the address pointed to by $pc.

(gdb) up
#1  
(gdb) up
#2  update_time () at contiki/core/sys/etimer.c:72
72	      if(t->timer.start + t->timer.interval - now < tdist) {
(gdb) x /i $pc
=> 0x80005c6 :	ldmia.w	r3, {r1, r4}
(gdb) info reg
r0             0x7d2	2002
r1             0x393821d9	959979993
r2             0x39381a07	959977991
r3             0x29d0fb29	701561641
r4             0x20000dc4	536874436
r5             0x2000004c	536870988
r6             0x0	0
r7             0x14	20
r8             0x20001f74	536878964
r9             0x20000270	536871536
r10            0x800c004	134266884
r11            0xced318f5	-825026315
r12            0x0	0
sp             0x20001fb8	0x20001fb8
lr             0x80005b9	134219193
pc             0x80005c6	0x80005c6 
xpsr           0x21000000	553648128
(gdb) 

Check it out. There’s an ldm instruction. And r3 is clearly not aligned. (It doesn’t even look like a valid pointer to SRAM, but we’ll ignore that for now) Ok, so we got an unaligned access, and we know where. But what the hell?! Let’s look at the C code again. That t->timer is all struct stuff. Perhaps there’s some packed uint8_ts or something, maybe some “optimizations” for 8bit micros. Following the chain, struct etimer contains a struct process, which contains a struct pt which contains a lc_t. And only the lc_t. Which is an unsigned short. I guess there’s some delicious C rules here about promotion and types and packing. There’s always a rule.

Changing the type of lc_t to an unsigned int, instead of a short and rebuilding stops it from crashing. Excellent. Not. It does make the code a little bigger though.

karlp@tera:~/src/kcontiki (master *+)$ cat karl-size-short 
   text	   data	    bss	    dec	    hex	filename
  51196	   2836	   3952	  57984	   e280	foo.stm32vldiscovery
karlp@tera:~/src/kcontiki (master *+)$ cat karl-size-uint 
   text	   data	    bss	    dec	    hex	filename
  51196	   2916	   3952	  58064	   e2d0	foo.stm32vldiscovery
karlp@tera:~/src/kcontiki (master *+)$

I’m not the first to hit this, but it certainly doesn’t seem to be very common. Apparently you should be able to use -mnounaligned-access with gcc to force it to do everything bytewise, but that’s a pretty crap option, and it doesn’t seem to work for me anyway. Some people feel this is a gcc bug, some people feel it’s “undefined behaviour”. I say it’s “unexpected behaviour” :) In this particular case, there’s no casting of pointers, and use (or lack thereof) of any sort of “packed” attributes on any of the structs, so I’d lean towards saying this is a compiler problem, but, as they say, it’s almost never a compiler problem :)

Here are some links to other discussion about this. (complete with “MORON! COMPILERS ARE NEVER WRONG” type of helpful commentary :)

I’m still not entirely sure of the best way of proceeding from here. I’m currently using GCC version arm-none-eabi-gcc (GNU Tools for ARM Embedded Processors) 4.7.3 20121207 (release) [ARM/embedded-4_7-branch revision 194305], and I should probably try the 4.7-2013-q1-update release, but if this is deemed to be “user error” then it’s trying to work out other ways of modifying the code to stay small for everyone where possible, but still work for everyone.

Not entirely what I’d planned on doing this evening, but someone enlightening at least.