Category Archives: software

Pachube dashboard brewery, with pygame and stomp

I send a bunch of data from my sensor network _to_ pachube, but this post is about using pachube as a data _source_ They have this app called a dashboard, which basically gives you some knobs and switches, that are hooked up as live inputs to a pachube data stream.

So, that got me thinking, here was a process control GUI pretty much done for me. The permissions are a bit wonky, so I’ll only include a picture here, but basically I get a knob for a set temperature, and a button to turn an alarm on or off. (I can add lots more, but this is all I’ve got set up so far)
Pachube Dashboard control panel

Neat, but now what? Well, the pachube api is pretty easy, so I just hooked up yet another consumer to my local network, (sensor data is dumped into a stomp/activemq message bus here, so that I can have infinite consumers of the data, without ever having to touch the code on my sensor nodes) that pulls data not just from the local network, but also from this pachube dashboard’s feed.

Add a little bit of pygame hackery so I can play sounds, and now I have a heating/cooling warning system for the kettle in the kitchen.

Example output

2011-01-22 11:05:08,649 INFO main - Current temp on probe is 31
2011-01-22 11:05:08,650 DEBUG main - threshold is 74, mode: heating
2011-01-22 11:05:08,650 INFO main - Turning music off.... we're below the threshold
2011-01-22 11:05:13,364 INFO main - Current temp on probe is 32
2011-01-22 11:05:13,365 DEBUG main - threshold is 74, mode: heating
2011-01-22 11:05:13,365 INFO main - Turning music off.... we're below the threshold
2011-01-22 11:07:01,408 INFO main - Current temp on probe is 33
2011-01-22 11:07:01,409 DEBUG main - threshold is 74, mode: heating
2011-01-22 11:07:01,409 INFO main - Turning music off.... we're below the threshold
2011-01-22 11:07:01,409 INFO main - Fetching current dashboard values from pachube, incase they've changed

This is far from any sort of automatic brewery, and was more an experiment in what was possible, and how easily. And it’s still a lot better than having me walk over to the kitchen every 5 minutes to check the current temperature. Now I can just turn the stove on, and get back to serious time wasting on the internet!

Source is over at github, https://github.com/karlp/karlnet/blob/master/consumers/pachube/consumer.pachube.dashboard.py

Things I found:

  • Pachube dashboard only updates if you drag the knobs. Using direct entry doesn’t update the datastream
  • Playing sounds from python is easy, but only when you find the right library, and only when you guess at the documentation a lot. I’ve no idea if this works on windows or osx. I tried a lot of other ways first, all of which failed miserably.

oracle destroying sun

Oh oracle, how I dislike thee. You took a perfectly good url like: “http://forums.sun.com/thread.jspa?threadID=5147616” clearly referencing a single thread, and you THREW IT AWAY giving me
http://forums.oracle.com/forums/main.jspa;jsessionid= 8d92079030d6c653c4adf7d34c40af87cbfdff6c4e77.e38Qb3qMa3eRbO0LaNqQaNaQax0Qe0?categoryID=84

And that just drops me at the root of _ALL_ the oracle forums.

“Welcome to our online community. Please choose from one of the forums below or login to start using this service”

Good job. Good fucking job.

WebMethods Designer 8 – take 2

Well, it’s still pretty crap. Here’s some more things that are just wrong/stupid.

  • The navigation tree on the left helpfully scrolls to the current flow. Except that it actually scrolls so that the top of the folder containing the current flow is at the top of the tree view. So for a folder with (too) many flows, the current flow is actually off the bottom of the screen.
  • To see your currently locked elements, you need to scroll to the very top, right click the server and choose properties.
    Fine, except that you can’t double click on an entry to go to that flow, and the dialog is modal, so you have to keep opening and closing it to have a list of what’s locked. Or, take a damn screenshot.

Doesn’t matter whether I’m using the stock version, based on eclipse 3.4, or installed into a working eclipse 3.6.

Contributing back to Open Source, trials and tribulations

So, back in March 2010, I started using xbee.py. It turned out not to do much of what I needed really, but it was a good base. The project seemed pretty dead in the water, but I fixed a bug or two, added some logging, and filed a bug with the diffs attached anyway. It wasn’t much, and it wasn’t pretty, but hey, if the project was alive, something would happen right?

And besides, it was now all working just fine for my needs.

Fast forward a few months, and the project has been resurrected by a third party. The code has been almost completely rewritten, and seems to have much fuller support for all the esoteric options I hadn’t needed. Oh, and the reply sent to my bug report never reached me, so I didn’t see any of this until just recently.

Of course, the released version _still_ didn’t have some rather important features I’d added, (it’s in trunk) and because of the rewrite, my diffs were now completely useless. And of course, the API has changed.

So where do I go now? Do I toss my local fork, resync with the “master” and try and get back on the train with the current open source base? Ignore it, and stay with my functional, working, battle tested, but feature limited version? (To be clear, my work mostly requires working with just escaped API mode, tx/rx only. The current xbee python project also supports transparent uart, remote AT commands, and remote IO transfer and ADC readings)

Seems like a bit of a loss all around really :( Should I have tried harder to get in contact with the original developer to get my patches in earlier? How much time should you really spend trying to patch open source software and how much time should you spend using it?

I’m well aware of the massive cost of maintaining local forks, if you actually do want the new features, but sometimes it just doesn’t seem worth it :(

FWIW, my version is still at github, and still very much in use here at home.

jetty:run maven plugin file locking on windows, a better way

If you use something like jetty:run while developing webapps on windows, sooner or later you’ll run into a problem with file locking. Any editor worth using normally lets you edit jsp files, css or js files and have the changes immediately be visible, without having to start/stop the servlet container, or (heaven forbid) rebuild the .war.

Jsp files don’t seem to be affected, because they have to be recompiled, but editing js and css files, say in a resources directory, fail.
Jetty has an extensive reference on this and if you search the web you’ll find hundreds of people parroting this in one form or another.

In my opinion, it’s a terrible solution. I most certainly do _NOT_ want to have to copy a jetty webdefault.xml file into my project, and maintain any changes made in the plugin upstream. I do _NOT_ want to have to hardcode a path to a local jetty install. I DO use more than one servlet container, and have no desire to include jetty specific garbage in my main web.xml file. But all this talk of changing the value of useFileMappedBuffer can be avoided completely.

Step back a bit, the problem is caused by NIO on windows. If you’re running jetty for embedded development purposes via jetty:run, do you really give a shit about the performance difference with hundreds of threads vs traditional blocking IO? No. Not in the least. So instead of hacking and poking jetty files, just change the plugin config to not use NIO. Like so…..

<plugin>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>maven-jetty-plugin</artifactId>
    <version>6.1.18</version>
    <configuration>
        <contextPath>${project.parent.artifactId}</contextPath>
        <connectors>
            <!-- work around file locking on windows -->
            <connector implementation="org.mortbay.jetty.bio.SocketConnector">
                <port>8080</port><!-- this connector defaults to 1300 for some reason -->
            </connector>
        </connectors>
    </configuration>
</plugin>

Database size, column types, and retention policies

So, earlier I mentioned that I had a 700meg database of samples, and wasn’t even looking at them.

I started trying to look at them, and quickly had to add some indexes, ballooning the db out to about 950meg. This was clearly going in the wrong direction.

I looked at the table schema, and quickly realised that keeping the sensor sample time (a unix timestamp) as a string instead of a number was pretty wasteful, and likewise for the sensor type and node id. Although both of those can be given meaningful names, all firmware on the nodes have those as numbers as well, because it has to go over the wire(less)

So, simply changing the database schema from some string columns to some integer columns, and my calculator tells me I’ve gone from 90 bytes per sample, down to 32 bytes per sample.

Much better. And now that the date sample is a proper number, searching the db for date windows is much faster, without even needing the index.

And for added deliciousness, the python consumer, that writes into this database didn’t have to change at all, is it a string, is it an integer? It’s both!

That’s all well and good, but it’s still going to just keep on growing forever and ever, and with samples every 5 seconds, and (currently) 3 nodes with 2-3 sensors on each one, this isn’t going to slow down.

Do I really really care about that resolution of data much longer than a few days/weeks/month in the past? Shouldn’t I really just be averaging the older data out to get some general trends? Perhaps :) And that’s exactly what RRDtool is for, I just find it a little obtuse to use at times. (Mostly, I have to decide, permanently and forever, right now, exactly how long I do want to keep things. I didn’t want to have to decide that)

Footnote: For the curious, the schemas for sqlite3 were

CREATE TABLE karlnet_sensor (sampleTime text, node text, sensorType text, sensorRaw real, sensorValue real);
CREATE TABLE karlnet_sensor2 (sampleTime integer, node integer, sensorType integer, sensorRaw real, sensorValue real);

And 32bytes per sample, with 4.7 million samples for the last three months, is still 150meg (I tossed some old data, when the node firmware was all in flux anyway)

Back in action… power meters and sensor probes.

So, after a six month break or so, and a 700meg database of sensor data I wasn’t actually looking at, I’m back in action!

I finally bit the bullet, bought a commercial power meter, and finished off my power meter project. It’s based on: Open Energy Monitor v3 but mine is in regular C, instead of arduinoC, because that fits in better with the common code for the rest of my nodes. (Not all my nodes are arduino compatible hardware) Calibrating was a bit of a pain, but it’s working! Now to get it boxed up and attached somewhere a bit more useful. It’s currently tying up my only spare AVR chip, so it’s not live.

What else? I bought some sexy radio boards from Dresden Elektronik that look pretty neat. Basically combining an AVR with an xbee, in a smaller package, for the same price. Why did I ever actually get into xbees? I should have done more research on the available radio options :) If I wanted just radios, I should have looked at these MRF24740 radios from Microchip I’ve got some on their way, but haven’t tried them out yet.

One problem with both the xbees and the dresden boards is that they’re on 2mm and 1.27mm pitch pin headers. This is nice and compact, but not very friendly for breadboards :)

Which brings me to customer service! Dresden actually phoned me up about 2 weeks after I bought the boards to check how I was going, and if there was any engineering help they could give me! Awesome! Pity Icelandic customs had only finally released the modules to me the day before, and I’d only just realised just how fine pitch 1.27mm really was :) Again, I should have read a bit more first.

Still on customer service, I ordered some pin headers from Samtec Seems 23pin female sockets, 1.27mm pitch are not something that mouser/farnell/RS stock, but Samtec is actually one of those lovely manufacturers who ship small quantities direct, worldwide, for reasonable prices! Excellent! So excellent in fact, that they shipped out the 1.27mm sockets as soon as they were ready, and the 2mm ones (for more xbee based boards) two days later. So I get two _enormous_ toblerone packages (triangle tube, about 13-14cm/side, 90-100cm long?) Delivered separately, with 20 rather teensy tiny little sockets all up. I’m sure if I was in a hurry this would have been excellent, but it felt a little excessive really :)

But socket headers are not going to help on a breadboard, it’s still fine pitch. So I’m also back into PCB design again, something I haven’t done since uni.

Lots of excitement. Oh yeah, and software too of course. I mentioned at the start that I wasn’t really looking at any of the data. Well, now, I am and also in other ways making more use of the fact that I was uploading all this data to pachube for so long!

It’s good to be back, but it has meant I’ve been spending far too much time in front of the computer again :)

Multiple schemas in WSDL, jaxws, cxf, soapui and webmethods.

This is the _short_ version.

We write a few wsdls here. We’ve been exposing them with plain stock JAXWS built into java 6, along with JAX-WS Commons and regular spring. This has mostly been working fine. We even have some maven archetypes to generate the templates around this and it’s all well and good.

As long as your schemas and types are all in a single .wsdl file

You see, jax-ws commons is basically unmaintained. As you can see in the quickstart, you should just need this…

<wss:binding url="/sub">
    <wss:service>
      <ws:service bean="#myService" />
    </wss:service>
  </wss:binding>
 
  <!-- this bean implements web service methods -->
  <bean id="myService" class="foo.MyService" >
      <property name="something" value="somevalue"/>
  </bean>

And, that actually works. Except…. Even if you started from a wsdl, and then created a service that implemented the wsdl, the wsdl jax-ws commons will expose at runtime is generated dynamically. So none of that documentation and and xml annotations you slaved over for hours/days/weeks are actually visible anywhere.

Ok, No big deal! It’s still all standards based, we just lost our documentation. Welllll. Only if you added more annotations specifying the namespaces. If you just have the @WebService(serviceInterface = “the.generated.from.wsdl.Interface”) jax-ws commons will make you a webservice whose runtime wsdl will have a namespace created from your implementation package name! (Instead of the originally defined wsdl namespace)

So you can add more annotations, oh, one of them is wsdl location! sweet! oh. no. It can’t reference into a jar.

So you look at the wss: namespace based config, oh look, we can specify the wsdl there too! Oh. no. It can’t reference into a jar. Hmm, the schema’s broken. primaryWsdl is an attribute, but can’t be used as such…

You check the documentation again, and give it a go… Ok, primaryWsdl is a child element? Ok! this seems to work!

Then you look at your runtime wsdl…

<xsd:import schemaLocation="Core_v1.xsd" namespace="http://core.vf.is/dom/v1"/>

Hmm, how’s it going to see that file at runtime? Oops. you’ve just made an invalid wsdl. Back to the docs….. what docs? There are none. but suffice to say, there’s a magic key that does what you want… ws:metadata It’s defined as taking a list of schemas. When you get this right, it knows how to magically create runtime links to them. Of course, there’s a bug for this, but no-one’s working on it.

Except, as we mentioned, jax-ws commons is effectively unmaintained. There’s a bug in the schema that only allows one metadata element, instead of a list. So you can go and use the full expanded plain old spring config. But good lord that’s ugly. And how on earth would you have survived this far?

    <wss:binding id="wscustomer" url="/services/customer1">
        <wss:service>
            <bean class="org.jvnet.jax_ws_commons.spring.SpringService">
                <property name="bean" ref="servicesCustomer1"/>
                <property name="impl" value="is.vf.test.ws.customers.v1.CustomersV1Impl"/>
                <property name="primaryWsdl" value="customers_v1.wsdl"/>
                <property name="serviceName">
                    <bean class="javax.xml.namespace.QName">
                        <constructor-arg index="0" value="http://core.vf.is/customers/jaxws/"/>
                        <constructor-arg index="1" value="customers_v1"/>
                        <constructor-arg index="2" value="tns"/>
                    </bean>
                </property>
                <property name="portName">
                    <bean class="javax.xml.namespace.QName">
                        <constructor-arg index="0" value="http://core.vf.is/customers/jaxws/"/>
                        <constructor-arg index="1" value="customers_v1SOAP"/>
                        <constructor-arg index="2" value="tns"/>
                    </bean>
                </property>
                <property name="metadata">
                    <list>
                        <value>Core_v1.xsd</value>
                        <value>Customers_v1.xsd</value>
                        <value>ProductCatalog_v1.xsd</value>
                    </list>
                </property>
            </bean>
        </wss:service>
    </wss:binding>

There’s patches that aren’t applied, bugs not fixed, it’s a zoo. But yeah, you can make it work with the very manual and repetitive raw spring bean config.

But where did webmethods come into this?

Well, basically, it just fails abominably when it gets one of these wsdls that references unreachable schemas. And if you were using the autogenerated ones, and weren’t super careful, you ended up with types in webmethods that weren’t actually the same type.

Did I mention that when you import a wsdl into webmethods, it creates a doc type for every type, even if the same type has already been imported via another wsdl? It gives you a warning for each duplicate type, but fortunately, at run time, it seems acknowledge that the two types are actually the same thing. Thank god.

Enough of this shit.

Want to do the same thing with CXF?

     <jaxws:endpoint id="wsCustomer" implementor="#servicesCustomer1" wsdlLocation="customers_v1.wsdl" address="/customer1"
                    xmlns:kz="http://core.vf.is/customers/cxf/" serviceName="kz:customers_v1"
                    endpointName="kz:customers_v1SOAP"/>

You still have to put in the serviceName and enpointName, even though they just copied from the wsdl itself but hey, it works, it’s way more intuitive, and it’s less typing.

(Don’t even get me started on the quality of the generated types from CXF vs JAXWS-Commons.)

bleh.

problems with webmethods developer/designer 8 (eclipse based)

So I’ve upgraded from v7 of webmethods developer (with the custom three column sort of view) over to v8 of webmethods designer (eclipse based)

Here are the stupid, annoying, weird things that I haven’t managed to fix (yet?)

  • Entire flows are marked in grey until you check them out for editing. I don’t want to edit it, but I want to be able benefit from having a colour screen. I also want to be able to separate disabled steps. Sure, the icons are still there, but it’s all grey.
    Which steps are active?

    Which steps are active?

  • Comments on a flow are now a single line property. They used to be a whole panel of free form text. Given how awkward it is to comment the “code” any other way, and given that this is the only place to put any overall comments on the whole flow, and given how crap the VCS integration is (at least in our setup?) this box is pretty important for staying sane. yes, you can use the “…” link to get into a box editor, and i’ll probably get used to this one, but it used to be a bit easier
  • No keyboard shortcut to expand the flow. And no config setting to always start expanded. Although right and left arrows will open and close, they only open/close a single level. That’s not good enough.
  • Pressing delete on the keyboard doesn’t work for flows/folders in the Package Navigator. It does work for flow steps. Even though right click “delete” shows Delete as being the keyboard shortcut
  • Enough for now. bleh.

eclipse 3.6 and maven archetypes with IAM/q4e

Ok, IAM for maven integrations fails the big one. It still uses the v1 archetype plugin in maven, so it completely and utterly fails to generate archetypes that are remotely recent.

Secondly, it only uses an internal list of archetypes, even if there’s more installed in your local ~/.m2 repository, and there’s no way of browsing on a remote repository, you have to type in all of it. (Intellij 9 also does this, but at least it can then do the generation steps!)

So, how about running archetype:generate from the command line, and then importing it into eclipse?

well, that kinda works. I made a new empty project, and chose to import a maven project. I pointed it at the base dir of my generated project, and it detected the two child modules, and with “experimental parents” enabled, it generated me 3 projects. Apparently that’s how eclipse works. kinda lame. My parent project has all the source of the inner projects.

This was using eclipse 3.6.1, and Eclipse IAM version 0.11.0.201001181900