Category: Kev’s Blog

Posted on

Revisiting my spotviz.info webapp: visualizing WSJT-X FT8 spots over time – part 5: MongoDB Aggregate Queries

Now I have my http://www.spotviz.info app up and running with some collected spot data for testing, one of the first things I want to enhance is the date range selection for visualization, and specifically, adding more data to the heatmap to help you pick a good date range with available data.

The heatmap display I originally added gives you a visual representation of which days have data to view, but it doesn’t help you pick a time range within that day where there’s data. Here’s what the heatmap looks like right now:

The darker the color the more data there is for that day. The problem as it is right now though is that say for one day you ran WSJT-X for 1 hour between 1pm and 2pm and received spots from 2000 stations. Without knowing that there’s only data between 1pm and 2pm, you could select a range of 9am to 6pm for playback, and you’d get nothing displayed on the map until it animated through the 1pm to 2pm block and then a gap of nothing again.

My first enhancement here is to show the earliest and latest signal received times for a given day, to help you pick a good range.

The original MongoDB query (which I discussed here) retrieves a count of spots per hour for a given day:

db.Spot.aggregate( [   
   {$match: {spotter: "kk6dct"}},
   {"$group": {
     "_id": {
       "$subtract": [
         { "$subtract": [ "$spotReceivedTimestamp", new Date("1970-01-01") ] },
         { "$mod": [ { "$subtract": [ "$spotReceivedTimestamp", new Date("1970-01-01") ] }, 1000 * 60 * 60 * 24 ] }
       ]
     },
     count:{$sum: 1 }
   }
 }
])

There’s only one calculated value returned in the results in this query, count, but in order get the earliest and latest spots per day, this is a simple as adding two more calculated values following the return of 

count : { $sum : 1 }

so the last part of the query is now:

    count : { $sum : 1 },
    "firstSpot" : { "$min" : "$spotReceivedTimestamp" },
    "lastSpot" : { "$max" : "$spotReceivedTimestamp" },

Done!

Posted on

systemd quick reference

sudo systemctl start|stop|restart|status servicename

sudo systemctl enable servicename : configures service to start at boot

If changing service config files in /etc/systemd/system/servicename.conf, rerun systemctl daemon-reload after changes and restart changed services

Posted on

Revisiting my spotviz.info webapp: visualizing WSJT-X FT8 spots over time – part 2: jaxb with Java 9 and beyond

Continuing from my efforts to get my SpotViz.info site up and running again on a current version of WildFly before refreshing the tech used to run the app, I ran into the first major issue: jaxb and support on Java 9 and beyond.

This is the first time I’ve run into this in practice, but was aware of the move to remove jaxb from Java SE in releases after Java 9. Deploying and running the app locally on WildFly17 and Java 8 everything deploys and works unchanged, but now I’m deploying to a test Ubuntu18.04 server in a VM on my local rack server before moving to the cloud, I’m running into all sorts of exceptions at runtime with part of the app that uses jax-ws and jaxb to call a remote xml based webservice.

At runtime as the MDB is picking up messages to process from the queue, I’m getting this exception:

Caused by: java.lang.NullPointerException
     at org.jboss.resteasy.resteasy-cdi@3.7.0.Final//org.jboss.resteasy.cdi.CdiInjectorFactory.createConstructor(CdiInjectorFactory.java:91)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.spi.ResteasyProviderFactory.injectedInstance(ResteasyProviderFactory.java:2809)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$AbstractInterceptorFactory.createInterceptor(JaxrsInterceptorRegistry.java:170)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$OnDemandInterceptorFactory.initialize(JaxrsInterceptorRegistry.java:188)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$OnDemandInterceptorFactory.checkInitialize(JaxrsInterceptorRegistry.java:203)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$OnDemandInterceptorFactory.getInterceptor(JaxrsInterceptorRegistry.java:214)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$AbstractInterceptorFactory.postMatch(JaxrsInterceptorRegistry.java:158)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry.postMatch(JaxrsInterceptorRegistry.java:421)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientConfiguration.getRequestFilters(ClientConfiguration.java:112)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.getRequestFilters(ClientInvocation.java:408)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:582)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:440)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:464)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.get(ClientInvocationBuilder.java:189)
     at deployment.callsignviz-1.0.war//kh.hamqthclient.HamQTHClient.logon(HamQTHClient.java:70)
     at deployment.callsignviz-1.0.war//kh.callsign.spotcollector.service.CallsignProcessorService.(CallsignProcessorService.java:35)

The clue here is the NullPointerException is coming from the HamQTHClient.logon() call, which is my library using jax-ws to call the remote HamQTH service.

Running my JUnits locally, this is definitely a JDK version issue. On Java 8 this code runs fine, but on Java 10 and 11 I get this:

Caused by: java.lang.ClassNotFoundException: javax.xml.bind.PropertyException
     at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)

… and now I’m remembering the jax-ws removal timeline from JEP-320:

  • Java 9 : deprecated for future removal
  • Java 11: complete removal of javax.xml.bind (jaxb) and javax.xml.ws (jax-ws) (also see here)

I don’t currently have Java 9 installed locally, but I suspect the code would work, but my test local Ubuntu 18.04 has OpenJDK 11, so this is currently my target JRE.

Ok, so what are the migration steps for using jax-ws after Java 8? A quick search found this useful article on migration, and adding explicit dependencies on the jaxb-api and the Glassfish reference implementation was exactly what I needed to get the code to run unmodified on Java 11:

<dependency>
    <groupId>javax.xml.bind</groupId>
    <artifactId>jaxb-api</artifactId>
    <version>2.3.1</version>
</dependency>
		
<dependency>
    <groupId>org.glassfish.jaxb</groupId>
    <artifactId>jaxb-runtime</artifactId>
    <version>2.3.1</version>
</dependency>

 

Posted on

Essential best practices and principals of effective software development

The software development industry is full of best practices and recommendations, many that evolve and change as our technology changes. There are some core concepts that remain beneficial even though programming languages and runtime technology platforms have changed over time.

Here’s a summary of some core principals that can help us develop software systems more effectively:

KISS – Keep it Simple, Stupid

Unnecessary complexity in code is technical debt that will to cost you at some point in the future – it’s more likely to contain unexpected and hard to find bugs, and complicated code it harder to understand, and therefore harder to maintain. Keep it simple.

DRY – Don’t Repeat Yourself

Avoid duplication in code. Duplicated blocks of code are a maintenance risk. If there is bug in a block of code that exists in multiple places, fixing the issue in only one place means that the issue still remains in those other blocks of code. When other developers are maintaining the code in the future, it may not be obvious that the same code exists elsewhere.

YAGNI – You aren’t gonna need it

The cost of maintaining a system increases in relation to the more code that is part of your product or solution. Why? Because the more code you have, these’s a higher risk for things to break, go wrong, increased effort for testing, and so on. You won’t have bugs in code you don’t have (you may have missing features but that’s a different problem). It makes sense therefore that you shouldn’t build additional features that you or your customer doesn’t need or want. The YAGNI rule is that if you don’t need it, don’t build it. Don’t make your job harder by increasing the possibility for things to break, or increasing the overhead of maintenance by building unneeded features that need to be fixed, updated, tested in the future. Don’t write code you don’t need.

Design/plan/structure your code to enable easier unit testing

Not everyone agrees whether Test Driven Development (TDD) should be followed as an absolute rule or not, but thinking about your tests first has a benefit that can be taken and used to your advantage even if you don’t subscribe 100% to the TDD approach. If you think about structuring your code in a way that makes it easier for you to test, then it’s immediately easier and takes less effort to write your unit tests, compared to code where there was no effort or thought into how the code could be tested. Simpler code is (usually) easier to test, compared to overly complex code that is usually much harder. Simpler code is easier to understand and maintain.

Proudly powered by WordPress