oracle destroying sun

Oh oracle, how I dislike thee. You took a perfectly good url like: “http://forums.sun.com/thread.jspa?threadID=5147616” clearly referencing a single thread, and you THREW IT AWAY giving me
http://forums.oracle.com/forums/main.jspa;jsessionid= 8d92079030d6c653c4adf7d34c40af87cbfdff6c4e77.e38Qb3qMa3eRbO0LaNqQaNaQax0Qe0?categoryID=84

And that just drops me at the root of _ALL_ the oracle forums.

“Welcome to our online community. Please choose from one of the forums below or login to start using this service”

Good job. Good fucking job.

WebMethods Designer 8 – take 2

Well, it’s still pretty crap. Here’s some more things that are just wrong/stupid.

  • The navigation tree on the left helpfully scrolls to the current flow. Except that it actually scrolls so that the top of the folder containing the current flow is at the top of the tree view. So for a folder with (too) many flows, the current flow is actually off the bottom of the screen.
  • To see your currently locked elements, you need to scroll to the very top, right click the server and choose properties.
    Fine, except that you can’t double click on an entry to go to that flow, and the dialog is modal, so you have to keep opening and closing it to have a list of what’s locked. Or, take a damn screenshot.

Doesn’t matter whether I’m using the stock version, based on eclipse 3.4, or installed into a working eclipse 3.6.

Contributing back to Open Source, trials and tribulations

So, back in March 2010, I started using xbee.py. It turned out not to do much of what I needed really, but it was a good base. The project seemed pretty dead in the water, but I fixed a bug or two, added some logging, and filed a bug with the diffs attached anyway. It wasn’t much, and it wasn’t pretty, but hey, if the project was alive, something would happen right?

And besides, it was now all working just fine for my needs.

Fast forward a few months, and the project has been resurrected by a third party. The code has been almost completely rewritten, and seems to have much fuller support for all the esoteric options I hadn’t needed. Oh, and the reply sent to my bug report never reached me, so I didn’t see any of this until just recently.

Of course, the released version _still_ didn’t have some rather important features I’d added, (it’s in trunk) and because of the rewrite, my diffs were now completely useless. And of course, the API has changed.

So where do I go now? Do I toss my local fork, resync with the “master” and try and get back on the train with the current open source base? Ignore it, and stay with my functional, working, battle tested, but feature limited version? (To be clear, my work mostly requires working with just escaped API mode, tx/rx only. The current xbee python project also supports transparent uart, remote AT commands, and remote IO transfer and ADC readings)

Seems like a bit of a loss all around really :( Should I have tried harder to get in contact with the original developer to get my patches in earlier? How much time should you really spend trying to patch open source software and how much time should you spend using it?

I’m well aware of the massive cost of maintaining local forks, if you actually do want the new features, but sometimes it just doesn’t seem worth it :(

FWIW, my version is still at github, and still very much in use here at home.

jetty:run maven plugin file locking on windows, a better way

If you use something like jetty:run while developing webapps on windows, sooner or later you’ll run into a problem with file locking. Any editor worth using normally lets you edit jsp files, css or js files and have the changes immediately be visible, without having to start/stop the servlet container, or (heaven forbid) rebuild the .war.

Jsp files don’t seem to be affected, because they have to be recompiled, but editing js and css files, say in a resources directory, fail.
Jetty has an extensive reference on this and if you search the web you’ll find hundreds of people parroting this in one form or another.

In my opinion, it’s a terrible solution. I most certainly do _NOT_ want to have to copy a jetty webdefault.xml file into my project, and maintain any changes made in the plugin upstream. I do _NOT_ want to have to hardcode a path to a local jetty install. I DO use more than one servlet container, and have no desire to include jetty specific garbage in my main web.xml file. But all this talk of changing the value of useFileMappedBuffer can be avoided completely.

Step back a bit, the problem is caused by NIO on windows. If you’re running jetty for embedded development purposes via jetty:run, do you really give a shit about the performance difference with hundreds of threads vs traditional blocking IO? No. Not in the least. So instead of hacking and poking jetty files, just change the plugin config to not use NIO. Like so…..

<plugin>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>maven-jetty-plugin</artifactId>
    <version>6.1.18</version>
    <configuration>
        <contextPath>${project.parent.artifactId}</contextPath>
        <connectors>
            <!-- work around file locking on windows -->
            <connector implementation="org.mortbay.jetty.bio.SocketConnector">
                <port>8080</port><!-- this connector defaults to 1300 for some reason -->
            </connector>
        </connectors>
    </configuration>
</plugin>

Packaging samples and small orders: Sensiron

Following on from yesterday’s post about Microchip’s samples packaging, today we have Sensiron. No handling charges, so no risk of any VAT charges from customs nazis here. And delightfully, the samples came just in a regular business envelope, inside a little sleeve of bubble wrap. This was just happily posted through the letterbox like any other mail, instead of being held by the aforementioned customs nazis for a few days while they searched for a receipt to bill VAT from. WIN!

Sensiron samples packaging - 1xDFN6

Sensiron samples packaging - 1xDFN6

Packaging samples and small orders: Microchip

I’ve gotten a few samples from various companies over the years, or in various cases, bought a couple of parts direct from the manufacturer. It often amuses me the huge differences in packaging levels between the various manufacturers. I’ve got a few things recently, while looking for some particular parts, so I thought I’d run a short series on how some companies stack up.

Microchip sent me four SOT23 devices in this nice small cardboard box below. This is fine, and I might even find a use for the box again, but because it came in a box, customs here in Iceland held it up for a few days searching for a receipt. (first from me, and then inside the parcel when I couldn’t provide one) On the bright side, the parts list inside the box didn’t list the “handling” charge they charge for all samples, so customs didn’t try and charge me VAT. Small miracles :)

A simple envelope with maybe a sleeve of bubbles would have been plenty, but this is fine really. Nothing to complain about :)

Microchip samples package - 4xSOT23

Microchip samples package - 4xSOT23

Database size, column types, and retention policies

So, earlier I mentioned that I had a 700meg database of samples, and wasn’t even looking at them.

I started trying to look at them, and quickly had to add some indexes, ballooning the db out to about 950meg. This was clearly going in the wrong direction.

I looked at the table schema, and quickly realised that keeping the sensor sample time (a unix timestamp) as a string instead of a number was pretty wasteful, and likewise for the sensor type and node id. Although both of those can be given meaningful names, all firmware on the nodes have those as numbers as well, because it has to go over the wire(less)

So, simply changing the database schema from some string columns to some integer columns, and my calculator tells me I’ve gone from 90 bytes per sample, down to 32 bytes per sample.

Much better. And now that the date sample is a proper number, searching the db for date windows is much faster, without even needing the index.

And for added deliciousness, the python consumer, that writes into this database didn’t have to change at all, is it a string, is it an integer? It’s both!

That’s all well and good, but it’s still going to just keep on growing forever and ever, and with samples every 5 seconds, and (currently) 3 nodes with 2-3 sensors on each one, this isn’t going to slow down.

Do I really really care about that resolution of data much longer than a few days/weeks/month in the past? Shouldn’t I really just be averaging the older data out to get some general trends? Perhaps :) And that’s exactly what RRDtool is for, I just find it a little obtuse to use at times. (Mostly, I have to decide, permanently and forever, right now, exactly how long I do want to keep things. I didn’t want to have to decide that)

Footnote: For the curious, the schemas for sqlite3 were

CREATE TABLE karlnet_sensor (sampleTime text, node text, sensorType text, sensorRaw real, sensorValue real);
CREATE TABLE karlnet_sensor2 (sampleTime integer, node integer, sensorType integer, sensorRaw real, sensorValue real);

And 32bytes per sample, with 4.7 million samples for the last three months, is still 150meg (I tossed some old data, when the node firmware was all in flux anyway)

Back in action… power meters and sensor probes.

So, after a six month break or so, and a 700meg database of sensor data I wasn’t actually looking at, I’m back in action!

I finally bit the bullet, bought a commercial power meter, and finished off my power meter project. It’s based on: Open Energy Monitor v3 but mine is in regular C, instead of arduinoC, because that fits in better with the common code for the rest of my nodes. (Not all my nodes are arduino compatible hardware) Calibrating was a bit of a pain, but it’s working! Now to get it boxed up and attached somewhere a bit more useful. It’s currently tying up my only spare AVR chip, so it’s not live.

What else? I bought some sexy radio boards from Dresden Elektronik that look pretty neat. Basically combining an AVR with an xbee, in a smaller package, for the same price. Why did I ever actually get into xbees? I should have done more research on the available radio options :) If I wanted just radios, I should have looked at these MRF24740 radios from Microchip I’ve got some on their way, but haven’t tried them out yet.

One problem with both the xbees and the dresden boards is that they’re on 2mm and 1.27mm pitch pin headers. This is nice and compact, but not very friendly for breadboards :)

Which brings me to customer service! Dresden actually phoned me up about 2 weeks after I bought the boards to check how I was going, and if there was any engineering help they could give me! Awesome! Pity Icelandic customs had only finally released the modules to me the day before, and I’d only just realised just how fine pitch 1.27mm really was :) Again, I should have read a bit more first.

Still on customer service, I ordered some pin headers from Samtec Seems 23pin female sockets, 1.27mm pitch are not something that mouser/farnell/RS stock, but Samtec is actually one of those lovely manufacturers who ship small quantities direct, worldwide, for reasonable prices! Excellent! So excellent in fact, that they shipped out the 1.27mm sockets as soon as they were ready, and the 2mm ones (for more xbee based boards) two days later. So I get two _enormous_ toblerone packages (triangle tube, about 13-14cm/side, 90-100cm long?) Delivered separately, with 20 rather teensy tiny little sockets all up. I’m sure if I was in a hurry this would have been excellent, but it felt a little excessive really :)

But socket headers are not going to help on a breadboard, it’s still fine pitch. So I’m also back into PCB design again, something I haven’t done since uni.

Lots of excitement. Oh yeah, and software too of course. I mentioned at the start that I wasn’t really looking at any of the data. Well, now, I am and also in other ways making more use of the fact that I was uploading all this data to pachube for so long!

It’s good to be back, but it has meant I’ve been spending far too much time in front of the computer again :)

common object models in webmethods and java webservices

What all the ranting and soap tedium of the previous post was leading to, was that we want to have some common object models shared across multiple java/.Net webservices, as well as webmethods business processes.

So, we made a few nice schemas, all in one namespace, and made a few test webservices and clients, and made sure that the objects could be happily passed from one to the other without needing to do any field by field level copying.

To repeat, the object model of customers and products, is multiple schema files, all in one namespace. This makes it easier to work on one part at a time, and means less people have to get their heads around 20 different namespaces.

Each service then has it’s own separate namespace, to allow later deployment of different versions side by side, but uses the same common object model.

Because this leads to having wsdl documents that reference multiple schemas, that may reference further schemas, you need to be very fucking sure that your webservices library is doing the right thing&tm; so that everyone actually gets the types they were expecting.

JAX-WS Commons doesn’t do a very good job of this. You can _make_ it do the right thing, but it takes a lot of work. And when you want to generate code from the wsdls, and include niceties like toString and equals and hashcode and the other tasty xjc plugins, you get a world of hurt with overlapping maven plugins and snapshot dependencies. Again, you can make it work, but it’s more difficult than it should be.

CXF seems to just work so far.

And that means that webmethods can actually import wsdls from CXF, and we can be confident that those are the same ones that we wrote, and that the objects will be portable.

WebMethods gives us a nice big warningwebmethods warning about duplicate types but it works. As best I can tell, as long as your “duplicate” types really are the same, you can safely ignore this warning.

Multiple schemas in WSDL, jaxws, cxf, soapui and webmethods.

This is the _short_ version.

We write a few wsdls here. We’ve been exposing them with plain stock JAXWS built into java 6, along with JAX-WS Commons and regular spring. This has mostly been working fine. We even have some maven archetypes to generate the templates around this and it’s all well and good.

As long as your schemas and types are all in a single .wsdl file

You see, jax-ws commons is basically unmaintained. As you can see in the quickstart, you should just need this…

<wss:binding url="/sub">
    <wss:service>
      <ws:service bean="#myService" />
    </wss:service>
  </wss:binding>
 
  <!-- this bean implements web service methods -->
  <bean id="myService" class="foo.MyService" >
      <property name="something" value="somevalue"/>
  </bean>

And, that actually works. Except…. Even if you started from a wsdl, and then created a service that implemented the wsdl, the wsdl jax-ws commons will expose at runtime is generated dynamically. So none of that documentation and and xml annotations you slaved over for hours/days/weeks are actually visible anywhere.

Ok, No big deal! It’s still all standards based, we just lost our documentation. Welllll. Only if you added more annotations specifying the namespaces. If you just have the @WebService(serviceInterface = “the.generated.from.wsdl.Interface”) jax-ws commons will make you a webservice whose runtime wsdl will have a namespace created from your implementation package name! (Instead of the originally defined wsdl namespace)

So you can add more annotations, oh, one of them is wsdl location! sweet! oh. no. It can’t reference into a jar.

So you look at the wss: namespace based config, oh look, we can specify the wsdl there too! Oh. no. It can’t reference into a jar. Hmm, the schema’s broken. primaryWsdl is an attribute, but can’t be used as such…

You check the documentation again, and give it a go… Ok, primaryWsdl is a child element? Ok! this seems to work!

Then you look at your runtime wsdl…

<xsd:import schemaLocation="Core_v1.xsd" namespace="http://core.vf.is/dom/v1"/>

Hmm, how’s it going to see that file at runtime? Oops. you’ve just made an invalid wsdl. Back to the docs….. what docs? There are none. but suffice to say, there’s a magic key that does what you want… ws:metadata It’s defined as taking a list of schemas. When you get this right, it knows how to magically create runtime links to them. Of course, there’s a bug for this, but no-one’s working on it.

Except, as we mentioned, jax-ws commons is effectively unmaintained. There’s a bug in the schema that only allows one metadata element, instead of a list. So you can go and use the full expanded plain old spring config. But good lord that’s ugly. And how on earth would you have survived this far?

    <wss:binding id="wscustomer" url="/services/customer1">
        <wss:service>
            <bean class="org.jvnet.jax_ws_commons.spring.SpringService">
                <property name="bean" ref="servicesCustomer1"/>
                <property name="impl" value="is.vf.test.ws.customers.v1.CustomersV1Impl"/>
                <property name="primaryWsdl" value="customers_v1.wsdl"/>
                <property name="serviceName">
                    <bean class="javax.xml.namespace.QName">
                        <constructor-arg index="0" value="http://core.vf.is/customers/jaxws/"/>
                        <constructor-arg index="1" value="customers_v1"/>
                        <constructor-arg index="2" value="tns"/>
                    </bean>
                </property>
                <property name="portName">
                    <bean class="javax.xml.namespace.QName">
                        <constructor-arg index="0" value="http://core.vf.is/customers/jaxws/"/>
                        <constructor-arg index="1" value="customers_v1SOAP"/>
                        <constructor-arg index="2" value="tns"/>
                    </bean>
                </property>
                <property name="metadata">
                    <list>
                        <value>Core_v1.xsd</value>
                        <value>Customers_v1.xsd</value>
                        <value>ProductCatalog_v1.xsd</value>
                    </list>
                </property>
            </bean>
        </wss:service>
    </wss:binding>

There’s patches that aren’t applied, bugs not fixed, it’s a zoo. But yeah, you can make it work with the very manual and repetitive raw spring bean config.

But where did webmethods come into this?

Well, basically, it just fails abominably when it gets one of these wsdls that references unreachable schemas. And if you were using the autogenerated ones, and weren’t super careful, you ended up with types in webmethods that weren’t actually the same type.

Did I mention that when you import a wsdl into webmethods, it creates a doc type for every type, even if the same type has already been imported via another wsdl? It gives you a warning for each duplicate type, but fortunately, at run time, it seems acknowledge that the two types are actually the same thing. Thank god.

Enough of this shit.

Want to do the same thing with CXF?

     <jaxws:endpoint id="wsCustomer" implementor="#servicesCustomer1" wsdlLocation="customers_v1.wsdl" address="/customer1"
                    xmlns:kz="http://core.vf.is/customers/cxf/" serviceName="kz:customers_v1"
                    endpointName="kz:customers_v1SOAP"/>

You still have to put in the serviceName and enpointName, even though they just copied from the wsdl itself but hey, it works, it’s way more intuitive, and it’s less typing.

(Don’t even get me started on the quality of the generated types from CXF vs JAXWS-Commons.)

bleh.